Phase two - Impact measurement

Phase two Intended Learning outcomes (ILOs)

Phase two of the Impact Playbook is designed to help you:

  • Create indicators based on outcomes

  • Identify the skills available in your team and (if applicable) have ideas of where you might get other support

  • Identify methods for data collection that best support your impact assessment

  • Talk about the validity of your data

  • Move forward in terms of the data collection needed for your impact assessment, outsourcing this if required

What is impact measurement?

If you followed Phase one (impact design), you should have a clear understanding of the type of impact you want to create or you think you are creating for your stakeholders. In prioritising your stakeholders, you should know who you want to collect data from. Now you need to kick off the data collection process.

In Phase two, we expand the change pathway - where you have documented your outcomes - to identify your indicators, consider the sample of data you need to collect, and develop your data collection and management plan. You’ll then analyse and interpret the data.

You might come to Phase two already knowing what you want to measure from whom. You might need some tips on how to plan and implement your data collection.

Who is it for?

You are likely to be a heritage professional trying to better understand their audience and the impact of your work. We don’t expect you to be an expert in data collection methods, though there will be points at which we’ll ask you to evaluate if you think you have the right skills internally and if external help might be needed. You might also be an experienced researcher or evaluator looking for a new approach or inspiration.

When should I use it?

You are doing or you have done something and you want to know how you can measure its impact. You want to collect data, evidence, something concrete to help you tell the story of the impact you have (or you think you have) created.

Phase two is a more complex phase, so you should give yourself more time to work through it and to get to grips with the different components.

What skills and competences do I need?

We think that Phase two requires all the skills of Phase one (like workshop facilitation and project management), and these specific skills and competences:

  • Confidence in developing and measuring indicators (we can take you through this - the best way is to learn by doing!)

  • Awareness of and experience with different data collection methods (we can help you find the right method for you, but you might need extra help using it in practice)

  • Confidence in qualitative and quantitative data analysis (we can give you tips on this, but you can’t learn this just by using Phase two. You might need external expertise or to learn using other resources - we can give you some tips for this!)

  • A practical mindset and project management skills that helps you evaluate the availability of resources (e.g. your and your colleagues' time) and plan your data collection. This will effect what and how much data you can collect in this phase, as well as how effectively you collect the data.


Impact measurement - skills and competences checklist

What skills do you or your colleagues have, and what skills might you need to get help with, to conduct your impact assessment? 

  1. I can differentiate between outcomes and outputs.

    1. Yes

    2. No

Yes - fantastic! It looks like you have really internalised your knowledge from Phase one. Keep going!

No - don't worry! If you need to refresh yourself on the change pathway terminology like outputs, outcomes, etc, go back to our training and guidance.

 

  1. I am comfortable with the concept of indicators.

    1. Yes

    2. No

Yes - great - some of the exercises in Phase two should be plain sailing for you. Keep going!

No - don’t worry - we cover this in Phase two.

 

  1. I have experience in analysing quantitative data, e.g. using Excel or other software.

    1. Yes

    2. No

Yes - fantastic! You're all set.

No - don’t worry - it's likely that someone in your organisation can help, depending on the type of data you want to collect. You can also consider sourcing external expertise to help you in this stage of your impact journey.

We assume that most users of the Impact Playbook have competencies in basic analysis in Excel or Google sheets. In some cases, you might have to or want to use more advanced analysis software. You might want to ‘learn by doing’, e.g. consult other learning resources, to help you analyse the data. You might also outsource your analysis to an external partner.

 

  1. I have experience in analysing qualitative data.

    1. Yes

    2. No

Yes - fantastic! You can still check out our tips and good practice relating to qualitative data for inspiration.

No - don't worry if you don't have the experience. There are plenty of resources to help you on your journey to delve into your qualitative data and methods. If you don't feel comfortable, think about asking others in your organisation if they have experience or get some external expertise.

We assume that most users have at least some competencies in analysing textual data, otherwise known as ‘coding’. You might want to ‘learn by doing’, e.g. consult other learning resources, to help you analyse the data. You might also outsource your analysis to an external partner.

  1. Do you feel ready for impact assessment? 

    1. Yes - let’s go!

    2. Sort of - I’ll decide once I’ve looked through Phase two.

    3. No - I feel out of my depth.

Whether you feel ready to collect and analyse your data, or whether you're feeling out of your depth, don't worry. You can always try to find external expertise, read our case studies and tips, and build up your skills.

See more below about finding partnerships and getting the skills you need.


I don’t think we have the right skills in the team

If you work for a larger organisation, you may have the support of specialised teams who can help you to manage the data collection and analysis phase. In a smaller organisation, the entire impact assessment may be done by the same people or person. The skills and competencies you had for Phase one, for example, might not be enough to get you through Phase two. You might have to look outside for additional support.

This can be a great opportunity. Consider, for example:

  • Partnering with a local higher education institution or University. Maybe there will be a strategic partnership that you can build.

  • Hiring a student or intern with data analysis skills who might help you conduct your impact assessment and at the same time, benefit from the opportunity (e.g. for their own research or professional experience).

  • Reaching out to local consultants or experts who might be able to help. This might be a valuable way to build connections with the local community.

  • Reaching out to local artists or creatives working with data analysis and presentation. They might have create ways to collect and present your data that could bring unexpected value for all.


What will the end result look like?

You’ll have your numbers crunched and have ideas about how to write about and present your data. You’ll be ready for Phase three!

If you follow Phase two, you’ll have:

  • A data collection plan, including agreed methodologies to collect data

  • Collected data from one or difference sources

  • A secure back-up of your data



Next step