Creating your indicators

Owned by Nicole Mc Neilly
Last updated: Aug 17, 2023
4 min read

In an earlier exercise we shared a case study and asked you to work out if certain outcomes were likely to happen in the short or longer-term. Now we’d like you to use the same hypothetical case study to think about what indicators might help you assess if the changes have taken place and measure this change.

The case study

A museum has developed an online learning resource for educators to use before a schools visit to the museum. It uses the museums digital collections, sourced from Europeana. The museum project lead has created a change pathway for the project. The goal is to understand if the project contributes to a bigger impact of ensuring digital heritage is more embedded in educational resources and used to enhance museum visits. If positive outcomes are observed, the impact assessment will help the museum lead advocate for more funding and training to support this goal. But first, we need to know what to measure.

Our hypothetical case study’s Change Pathway

We now want to extend the Change Pathway by adding in the indicators we want to measure to ascertain if our outcomes have occurred. We’ll start using the extension that we share later in the guide to creating your data collection plan.

outcome (we’ll take these directly from the Change Pathway)

indicator(s) of change

the changes that have occurred in (or for) your stakeholders that can be directly attributed to your activities (see Phase one for a recap!)

what are the things that show us that the outcome has been achieved?

(Short-term) Pupils are more engaged in the museum visit than a traditional museum visit

  • Pupils ask more questions

  • Pupils answer more questions

  • Pupils remain engaged in the tour for longer

Note that the results must be compared in some way to a traditional museum visit

(Short-term) Pupils learn more depth about the topic than a traditional museum visit

  • Pupils answer more questions correctly

  • Pupils ask more in-depth, topic specific or focussed questions

Note that the results must be compared in some way to a traditional museum visit

(Long-term) Pupils have a stronger sense of connection with heritage*

  • Pupils are more likely to use digital heritage assets in their school work

  • Pupils suggest or request museum visits, or to visit a digital collection, as part of their work

*This is a hard one! It is also dependent on a lot of factors - that there is space for change (that pupils did not already feel very connected to heritage); that educators use more (digital) heritage in the classroom, etc.

(Long-term) Educators more likely to use digital heritage in the classroom in future

  • Educators use the museum's digital collections, or other digital collections, in their teaching resources in future

  • Educators use pre-made digital heritage educational resources in the classroom

  • Educators who had never previously used digital heritage educational resources in the classroom are now using these

This is easier to monitor but requires long-term engagement. Measuring this will require good connections to the educators. The results are likely to be both qualitative and quantitative.

What do you think? What indicators would you measure for each of these outcomes?

Prioritise the indicators - materiality and accountability

We now need to prioritise the indicators above. We can do this according to three criteria.

  • Is it a material change?

  • Are we accountable for the change?

  • Is it a priority to measure, based on our overall impact assessment purpose?

 

Now, we can use the following checklist to strengthen the indicators:

  • Are your indicators SMART?

  • Are all of these indicators important for you to measure?

  • Do we have a mix of subjective (e.g. opinions and perceptions) and objective indicators (e.g. facts)?

  • Have you looked at indicators used in other projects to see what you can learn?

  • Do you have a baseline for any of these indicators? Are you already collecting these measurements in other projects that you could compare to?

  • If you collect this evidence, do you have enough or the right data to tell the impact story you want to tell?

Tip.

Whether you use standardised indicators, or create your own, good indicators are SMART. You can use this checklist to help you both before you start and after you have created your indicators.

  • Specific: it’s clear what change the indicator represents

  • Measurable: it can be counted or observed

  • Attributable: there is a link between your activities and the outcome

  • Realistic: the information can be collected with the resource you have

  • Timely: that you can collect it when you need it and at an appropriate time

Tip.

In creating your indicators it’s likely that you’ll also be thinking about how to measure them. Take notes or use the Change Pathway extension to document this.

More complex Change Pathways are likely to require extensive prioritisation. Prepare to ask yourself some hard questions!

Next step