Versions Compared

Old Version 8

changes.mady.by.user Nicole Mc Neilly

Saved on

compared with

New Version 9

changes.mady.by.user Nicole Mc Neilly

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.
info
Panel
bgColor
#DEEBFF

Pros: light-touch, easy to do and apply in a workshop setting (online or in-person), can be built into existing meeting formats, doesn’t take much time, gives results straight away. This could be seen as a ‘minimum’ approach you can take to evaluating your impact assessment process. Plus, you can easily apply it to other circumstances. 

Cons: applying this method at a general level (e.g. looking at your whole impact assessment process without focussing on specific elements or phases) might mean that your results are also very general.

...

Download the template!

...

panelIconText
Panel
panelIconId1f91d
panelIcon:handshake:
🤝bgColor#E6FCFF

Team exercise: ten steps for a team debrief / retro

Task: schedule a meeting with the core colleagues involved in your impact assessment activity

Time needed: up to 90 minutes (or more if the project has experienced some difficulties or has a more complicated context. We recommend that your meetings are no more than 90 minutes long, so consider additional meetings if you need to discuss more topics)

Checklist: before you start

  • Schedule a meeting of between 45 minutes to 90 minutes. Invite colleagues who have been involved in the impact assessment. Ensure that everyone knows why you are meeting and what you are evaluating.

  • Prepare your debrief/retro digital board or space. This should be accessible by everyone in the meeting. You can use the template in the appendix. Leave plenty of (digital) post-it notes close to the questions: 

    • What worked well? 

    • What didn’t work well? 

    • What can we improve next time? 

    • You can add additional prompts to each of these questions like those in the table below.

  • Review any initial plan or objectives you created for your original impact assessment. Set these out on the board or shared document or share these in advance with participants (e.g. by attaching them to the meeting invitation).

  • If people can’t attend, ask for their perspectives and/or share the digital board or document in advance. 

In the meeting

  1. Agree what you want to achieve in your meeting: emerging with a list of recommendations that you can embed in future or ongoing work. 

  2. Outline what part(s) of your impact assessment approach (activity) you will evaluate.

  3. Agree whether your participants will contribute to the board directly or whether you need one central note-taker. Appoint a note-taker if necessary. 

  4. If you are using a digital board, you might need to introduce your colleagues to how they can use it, e.g., add digital post-it notes, and how to navigate the digital board. 

  5. Review the objectives of your original brief and impact assessment objectives as a group. 

    • What did you want to achieve through your impact assessment? Have you accomplished this? 

    • What were your success metrics (Phase one)? 

    • If anything you discuss is positive or negative, add it to the respective sections of your board. 

  6. Discuss the first question:  what worked? (positive)

  7. Discuss the second question: what didn’t work? (negative)

  8. Ask your colleagues to spend two minutes individually reviewing the lists of what worked and what didn’t work. Then ask them to share what they would recommend as future improvements - answering Question 3. Capture this. 

  9. Review the recommendations that you and your colleagues have captured. Are they all actionable? Have you attributed who would have to action these, and what barriers might stand in the way? You might have a lot of recommendations. Do you want to prioritise these?

  10. Thank your colleagues and reflect on the value that what they have shared can add to your future work. You might want to reflect on the recommendations and edit these, so that they’ll be clear for you to use in the future. 

After the meeting

Save the materials (e.g. if you are using Miro, you can download a PDF) in a shared drive or other safe space. Share the link to the recommendations with everyone involved and anyone else whose work is involved. 

panelIconText
Panel
panelIconIdatlassian-light_bulb_on
panelIcon:light_bulb_on:
:light_bulb_on:bgColor#DEEBFF

Tip

Your colleagues might feel uncomfortable sharing criticism in front of others, particularly if it concerns others’ work. Encourage a respectful and critically constructive approach to sharing feedback by setting out how you would like your colleagues to participate. If you feel that the feedback might be quite sensitive to give in the meeting, you can share the questions with your colleagues via a Google form or other survey software and draw out the recommendations yourself.

...

The Europeana case study - Europeana XX’s Phase four workshop

Expand
titleClick here to find out what we learned by applying the debrief/retro model.
Panelinfo
bgColor#DEEBFF

About the Europeana XX Phase four workshop: we conducted a debrief after the final deliverables had been submitted but before the project review meeting (something that happens at the end of every European-funded project).

What we did: we adapted the template below to look at the three main stages of the Europeana Impact Playbook approach followed. We used Miro to do this.

Who was at the meeting: the lead on the impact approach and two other project partners involved in various aspects of the impact assessment (as well as the Europeana Impact Advisor who helped as the facilitator).

What worked well

  • Impact was built in from the proposal writing stage (using Phase one design). The kick-off meetings then also took the impact thinking one step further, which embedded an impact approach from the outset and validated the impact plan and helped better map relevant stakeholders.

  • The funding proposal received a good mark on the impact section, thanks to the impact approach taken.

  • Stakeholders were defined early, and activities were connected to the stakeholders for whom they would deliver benefit.

  • It was helpful that one member of the team had significant experience with the Europeana impact approach.

  • Improvements were made iteratively to the data collection plan because user engagement events happened throughout the project period and because feedback loops and evaluation were built in.

What didn’t work so well

  • Impact design

    • The original impact assessment plan had to be adjusted because of the COVID-19 pandemic and a change in the project activities.

    • Some outputs/activities happened or were delivered late in the project period, making it difficult to understand if they were delivering value for stakeholders.

  • Impact assessment could be a stronger part of other parts of the project, e.g. the qualitative impact of the editorial could have been captured to provide an impact narrative that goes beyond visitor statistics and thus steer the future editorial plan.

  • The team collecting data around editorial engagement could have shared this faster with the team who could then make improvements and adjustments.

  • It is challenging to track standardised data across very different activities (in particular, in physical events) coming from many sources (project partners) and some data can’t be compared.

  • It was difficult to coordinate data collection across so many activities (because of a lack of capacity).

  • It is still challenging to measure shifts in mindset stimulated by the project activity, also because of the impact assessment limited to the project’s duration.

Ideas for improvements - Europeana XX recommendations for future

  • Put impact on the agenda for review meetings and build in an impact ‘sprint’ after the project’s end but before the project review (relevant for funded projects only).

  • Ensure that impact assessment is part of all relevant project elements.

  • Future research could investigate how to articulate the impact of the enrichment of Europeana data and metadata (relevant also for other generic services).

  • Bring together more team members with different expertise and skills to evaluate the impact assessment, as this provides a deeper experience.

  • Provide indicators that could help the measurement of long-term outcomes in the final activity report or deliverable and consider what your forecast could be.

  • Consider how longitudinal evaluation could be built in after the end of the project.

  • Set out a clear approach to agile evaluation and improvements throughout the project processes.

  • Shape the review process using the impact narrative to make the final results as powerful as possible.

  • Collect baseline data on partner expectations at the beginning of the project and use this in the evaluation at the end of the project.

What changes or actions did we take afterwards?

This evaluation meeting inspired the project lead to structure the subsequent final project review meeting based on the findings of the impact assessment. Having a dedicated moment, or moments, to reflect on the impact assessment process, to identify its strengths and areas with a room for improvement is very relevant for ongoing work but can also help better structure and plan impact assessments in future projects.

It is important to be smart about choosing evaluation moments. On reflection, Phase four doesn’t necessarily have to come as the last phase. It can be used throughout the assessment, supporting agile adjustments in the project based on the gathered data.

...

Next step