Creating your indicators
In an earlier exercise we shared a case study and asked you to work out if certain outcomes were likely to happen in the short or longer-term. Now we’d like you to use the same hypothetical case study to think about what indicators might help you assess if the changes have taken place and measure this change.
The case study
A museum has developed an online learning resource for educators to use before a schools visit to the museum. It uses the museums digital collections, sourced from Europeana. The museum project lead has created a change pathway for the project. The goal is to understand if the project contributes to a bigger impact of ensuring digital heritage is more embedded in educational resources and used to enhance museum visits. If positive outcomes are observed, the impact assessment will help the museum lead advocate for more funding and training to support this goal. But first, we need to know what to measure.
We now want to extend the Change Pathway by adding in the indicators we want to measure to ascertain if our outcomes have occurred. We’ll start using the extension that we share later in the guide to creating your data collection plan.
outcome (we’ll take these directly from the Change Pathway) | indicator(s) of change |
the changes that have occurred in (or for) your stakeholders that can be directly attributed to your activities (see Phase one for a recap!) | what are the things that show us that the outcome has been achieved? |
(Short-term) Pupils are more engaged in the museum visit than a traditional museum visit |
Note that the results must be compared in some way to a traditional museum visit |
(Short-term) Pupils learn more depth about the topic than a traditional museum visit |
Note that the results must be compared in some way to a traditional museum visit |
(Long-term) Pupils have a stronger sense of connection with heritage* |
*This is a hard one! It is also dependent on a lot of factors - that there is space for change (that pupils did not already feel very connected to heritage); that educators use more (digital) heritage in the classroom, etc. |
(Long-term) Educators more likely to use digital heritage in the classroom in future |
This is easier to monitor but requires long-term engagement. Measuring this will require good connections to the educators. The results are likely to be both qualitative and quantitative. |
What do you think? What indicators would you measure for each of these outcomes?
Prioritise the indicators - materiality and accountability
We now need to prioritise the indicators above. We can do this according to three criteria.
Is it a material change?
Are we accountable for the change?
Is it a priority to measure, based on our overall impact assessment purpose?
Now, we can use the following checklist to strengthen the indicators:
Are your indicators SMART?
Are all of these indicators important for you to measure?
Do we have a mix of subjective (e.g. opinions and perceptions) and objective indicators (e.g. facts)?
Have you looked at indicators used in other projects to see what you can learn?
Do you have a baseline for any of these indicators? Are you already collecting these measurements in other projects that you could compare to?
If you collect this evidence, do you have enough or the right data to tell the impact story you want to tell?
Tip.
Whether you use standardised indicators, or create your own, good indicators are SMART. You can use this checklist to help you both before you start and after you have created your indicators.
Specific: it’s clear what change the indicator represents
Measurable: it can be counted or observed
Attributable: there is a link between your activities and the outcome
Realistic: the information can be collected with the resource you have
Timely: that you can collect it when you need it and at an appropriate time
Tip.
In creating your indicators it’s likely that you’ll also be thinking about how to measure them. Take notes or use the Change Pathway extension to document this.
More complex Change Pathways are likely to require extensive prioritisation. Prepare to ask yourself some hard questions!