Step 1. The data collection plan
- Former user (Deleted)
Intended Learning outcomes
This page is designed to help you:
Understand what the components of a data collection plan are and how it will help you collect the data you need
In Phase one you will have emerged with a completed change pathway.
You might have used Word or Excel (or something else) to map out each change pathway for your stakeholder, like this.
Or this example, which uses a slide deck.
It’s now time to extend your change pathway to document the indicators that you will use to measure change, how you will measure them, and when and from whom you’ll collect the data. This will give you a data collection plan.
Create a data collection plan
The steps below relate to the development of your data collection plan and not phase two more generally.
You’ll complete different parts of your data collection plan over the course of Phase two!
1. Agree the scale of your impact assessment
“If your budget is small, or you’re still at the very beginning of the project, it’s best to start with small, straightforward measures that you can expand little by little.”
It’s good to be ambitious (you are - you’re already here!) but ensure that what you invest in the data collection is balanced with the quality of data you’ll get in return and the scale of the project. You can always build up to a more in-depth approach at a later point. How much money and time you have is also important. How much of both are you able to invest? How much time do your team or colleagues have to support you? This will change and influence your approach. You might also want to do a quick pilot or test before launching anything large scale.
2. Create your indicators
See the dedicated page on how to work with indicators.
3. Set your ideal sample size and response rate
The data that you collect should be representative and robust. You need to know how many responses you need to draw valid conclusions about those whom you are surveying.
Europeana Network Association impact assessments - sample size
In 2020 and 2021 we shared impact assessment and satisfaction questionnaires with the Europeana Network Association membership. The questionnaire was sent to the whole group (over 3,000). We wanted a minimum of a 10% response rate to give us a valid insight into Network membership, and we achieved this in both 2020 and 2021. We know, however, that the bigger the sample size, the stronger the findings.
We assessed the responding sample against other data (e.g. what we know about length of membership, experience in the sector, location) to understand to what extent this was reflective of the full group of members. Read the impact assessments on Europeana Pro.
The sample size you need will influence how you collect data. You might not expect a high response rate for a survey, so you could instead interview a smaller sample of stakeholders. This might also give you richer insights, but it might not fulfil the desires of your Board, for example, if they want easily shareable infographics about their impact.
4. Think about your baseline
What is a baseline?
A baseline is a measurement of the starting point of something, for example, the situation at the beginning of your project. For example, if your objective was to evaluate the effectiveness of an activity in terms of how it improved the reactions of visitors to your website, you could measure their satisfaction with their experience.
If you know how you will collect your data you now need to think about the design of your data collection.
Can you collect and then compare a pre- (baseline) and post-situation (endline)?
Do you want to continually monitor the effects of your work (and thus collect data on a continuous basis)?
Are you doing retrospective research (looking back on what happened) focused on a certain time period?
Will you collect data at multiple points after the activity has finished?
Q&A
Question: I can’t collect baseline data. Does this mean my impact assessment isn’t valid?
Answer: An assessment of change is strongest and most valid when the initial situation (baseline) and the post-intervention situation can be compared. Collecting the baseline data, however, is not always possible. That doesn’t mean you shouldn’t do an impact assessment; nor does it mean that you should stop trying to collect baseline data.
In your post-intervention survey, you could ask the participant to reflect on processes of change or to rate the situation at the beginning quantitatively (e.g. on a scale) compared to the current situation. There is nonetheless a risk of recall bias that you should bear in mind.
If you are aiming to conduct impact assessments in the future, the current post-intervention data can be used as baseline then.
5. Think about short-term vs long-term
Tip.
The data you want to collect is what you will use to tell your story of change. That change can take place over the long-term or short-term. Think about how you can best collect data to reflect this.
Asking people questions about an event that took place a year ago may not give you very reliable information if you don’t have data from closer to the time to compare it to. At the same time, asking people questions too soon after an intervention may lead to you missing out on relevant changes that are still to occur. Therefore, you could collect data from stakeholders in the short-term and long-term, if resources allow.
Long-term outcomes can take a long time to appear. You might not have the resources to continue an impact assessment to the point where you can begin to evidence these outcomes. The goal of the Impact Playbook is to empower you to show that you’ve helped create shorter term outcomes that can help lead to this longer-term change and impact, and to give you something that you can learn and improve from.
Tip.
Think about how you are communicating to your stakeholders about your data collection. You might be able to ensure their participation and improve your response rates by telling them why collecting this data is important for you. When you are planning a follow-up questionnaire some time after the activity happened, tell the respondent that they should expect this questionnaire and why it’s important to you that they fill it in. Clear communication never hurts!
When we sent longitudinal questionnaires to attendees of our national workshops and Europeana 2019, there was a very low response rate. This might be because the participant thinks that the activity under question is no longer relevant, or that they have other priorities to reflect on than an activity that was so long ago. The risk of low response rates should be borne in mind.
6. Agree what methods you will use
Before going into research for your methods, be aware that it can a time-consuming procedure if you have no experience. Keep in mind that external advice might be good to get at this point.
Take a look at:
Our advice on qualitative, quantitative and mixed methods approaches.
Our overview of different types of commonly-used methods.
7. Find or create your ‘data points’
The best data collection method for your research will depend on when and where you can collect data. How can you reach out to your stakeholders so that you can collect data from them? When and where can you collect this data? Consider:
Where your stakeholders are located
How you can access or contact them
Whether a group or subgroup can be representative of your stakeholder, and what size this group should be
How can you collect data in an unobtrusive way
Tip.
Sometimes your stakeholder might not be the one you are able to collect data from directly. For example, asking young children to reflect on how they experienced an activity may be difficult, instead you could interview their parents or teachers.
8. Set realistic timelines
By now you should have a feeling of what is a realistic timescale to set for your data collection and analysis. Some methods are more time-consuming than others, depending on the scope and rigour you have decided on. For example, you may want to focus in more depth on a smaller sample of stakeholders, but that will require resources over a long time period. Thinking through what is a realistic timescale for you and your team will help you set a realistic timeline. It’s also important to be flexible in the longer-run, as things might change or not run according to plan (e.g. a project might be extended or reduced in size).
Tip.
Take practicalities into consideration, e.g. travelling for face-to-face interviews, sending reminders for online surveys or getting enough people together for a focus group.
9. Ensure that you are confident about ethics, data protection and privacy approach and GDPR compliance
Before going any further, ensure that you are up to date with what data protection laws govern your data collection and how you should ethically approach data collection. Read our short guidance by clicking the link below.
You can also read in more detail the section on data protection in Europeana’s Standardised Question Bank.
You’re now ready to create your data collection plan!
We’ve developed a simple ‘extension’ to the change pathway of Phase one to help you do this. For each outcome (short, medium or long-term), you will:
Map the indicator(s) that will help you measure change
Identify the source of where the data comes from
Identify what sample you need to make it valid
Outline how you’ll collect the data (when/where/how)
outcome (you’ll already have these) | indicator(s) of change | source and sample | data collection method and plan |
the changes that have occurred in (or for) your stakeholders that can be directly attributed to your activities (see Phase I for a recap!) | what are the things that show us that the outcome has been achieved? | where can we get the data from (existing data sources or new data needs to be collected)
| surveys, interviews, analysing existing data sources...consider the pros and cons
when and how can we collect this data? |
e.g. pupils engage more in heritage as a result of going to Transcribathon event | e.g. number of pupils who use more digital heritage in homework assignments in the four months after the Transcribathon event | e.g. at least 10% of educators need to participate | e.g. follow-up interviews with educators after Transcribathon event. Challenge: small data sample, need a native speaker trained in qualitative methods |
Download the extension Excel template
Download the extension Word template
Final checklist
Take a final moment with your team to check:
Do your indicators still make sense in relation to the outcomes?
Has your perspective on your priority outcomes changed?
When is the best time to collect the data? Are you missing any opportunities?
Will your sample size give valid results?
Is your timeline manageable?