Step 1. Agree your evaluation focus

Owned by Nicole Mc Neilly
Last updated: Aug 23, 2023
3 min read

Intended Learning outcomes (ILOs)

This page is designed to help you:

  • Determine the focus of your evaluation

  • Have a clear idea about what part of your impact assessment you want to evaluate.

  • Feel comfortable with what we mean by evaluating your impact assessment process.

How to focus your evaluation

Your impact assessment will have followed a number of stages, whether or not you followed the whole of the Europeana Impact Playbook methodology. You could evaluate your impact assessment in different ways - we outline three of them and their pros and cons below. 

Ways to evaluate your impact assessment

Pros

Cons

1. Each step in your impact approach (i.e. systematically going through each phase)

  • Ensures nothing is forgotten

  • Rigid process

  • Might take more time

2. The whole process (i.e. leave it open for your colleagues to share what they thought went well and didn't)

  • Most important discussion points likely to surface organically

  • Some aspects may have been forgotten about, particularly after some time has passed

3. Specific part(s) of your impact assessment (i.e. choose one or more parts of the impact assessment, such as impact measurement, narration, etc.)

  • Helps focus on improving specific areas of your impact assessment process

  • Useful if you don’t have much time

  • Useful if you are already very skilled in impact assessment and want to improve in specific areas

  • Your priority areas might be different from others in your team

  • May lose the opportunity to collect feedback and improve in other areas

Evaluating your impact assessment step by step 

Below we list a number of options you could keep in mind to help you start evaluating your impact assessment approach. You might not be able to evaluate some of these things because not much time will have passed. In other cases, you might have had a lot of time to reflect.

Phase one.

Think about…

  • Your stakeholders and prioritisation. Were they the right ones? Do you know more about your stakeholders now?  

  • Your change pathway. Was it fit for purpose? Were any significant changes left out? Should you have validated it further? Was it updated throughout the impact assessment process? Did you refer to the change pathway at all throughout the process? 

  • Your assumptions. What did you learn about your assumptions about the outcomes created by your activities? Where have you been proven wrong or right, and why might this have been the case? 

  • Your planning - inclusion. How were partners and different voices involved? Was anyone’s voice or perspective missing? Who should be involved in the future, and how could you plan for this? 

  • Your planning - timing. Did you plan at the right time? Should you have planned at the project proposal stage or did you plan too early, when no details were available? 

  • Your planning - delivery. Did you follow the plan you set out? What changed and why? Was it flexible enough? Did it have to change a lot? How was the plan referred to and updated throughout the process? 

Phase two.

Think about…

  • Your purpose. Were your objectives SMART (specific, measurable, actionable, realistic and timely)? 

  • Data collection - questions. Could you have improved the questions you asked? What would you change in future? Did the questions give you the data that you needed? What was missing?  

  • Data collection - platform(s). Could how you collected the data have been improved? Was the platform or service a barrier to your audience, e.g. in terms of accessibility? Did the platform(s) or approach chosen help in the data collection process? 

  • Communication with the subjects. Could you have improved how you communicated with the research participants, e.g. in the invitations to interview or in the detail shared in invitations to complete a questionnaire? Was adequate and necessary information shared, like your privacy statement and the closing date of the questionnaire? 

  • Validity. Was your sample sufficient? How could you improve the sample size and validity of your data? 

  • Interpretation. Are you happy with your interpretation of the data? Have others shared or disagreed with your interpretations? Was your interpretation an adequate representation of the findings in the data? How have you interpreted or acknowledged negative or less positive findings? 

Phase three.

Think about…

  • Perspective and objectivity. How did you respond to less positive results? How did your audiences respond to the narrative?

  • Constructing the narrative. Was there something missing?  What can be added or dismissed in the future? Could the narrative have been simpler, e.g. avoiding jargon, or more concise? 

  • Visualisation. Did a visual element(s) support your narrative? Was it effective? What would you change or improve in future?

  • Dissemination of the findings. How widely your findings were shared? Were there limitations in sharing the results? What could have been improved? 

  • Using the findings. How have your findings been used? What barriers have there been to change? 

  • Impact. Have the findings led to the desired impact?

Next step