| Panel |
|---|
| panelIconId | 1f3eb |
|---|
| panelIcon | :school: |
|---|
| panelIconText | 🏫 |
|---|
| bgColor | #C0B6F2 |
|---|
|
Intended Learning outcomes This page is designed to help you: Understand in detail when to apply certain methods Understand the pros and cons of commonly-used methods Explore new ways of collecting data
|
You or someone in your team might be familiar with the data you want to collect. Or know which methods work well for your audiences. But before you decide on your data collection methods, remind yourself of the options available to you.
Common data collection methods
Below we share a number of commonly-used data collection methods. At the bottom of the page, we also link to separate pages that show methods used to measuring environmental and economic impact.
| Panel |
|---|
| panelIconId | 2139 |
|---|
| panelIcon | :information_source: |
|---|
| panelIconText | ℹ️ |
|---|
| bgColor | #DEEBFF |
|---|
|
Primary and secondary data Is the data collected by your organisation for this purpose only? Then it’s primary data. Secondary data describes data that has been collected by another organisation or for a different purpose. Don’t rule out looking at secondary data. It can be a great starting point for your research as the hard work has already been done by others. These data can work as a guide for you in many ways. For example, you can use the data but also learn from their approach. What does their baseline say? Can these data act as a baseline? Can you collect data in the same way, and then compare? |
The methods described below are qualitative.
1. Interviews
| Expand |
|---|
| title | Expand for more information |
|---|
|
What: Face-to-face, phone or online conversations to gather deep insights, first-hand experiences or expert opinions When: If you are interested in gaining more context and understanding of your stakeholders (e.g. to better interpret quantitative data) Advantages In-depth and often rich answers Also open to people with low literacy skills Opportunity to explore unknown or sensitive issues in more depth Generally people are willing to participate Flexibility in the questions you pose
| Disadvantages Time consuming to schedule, conduct and transcribe Need for trained and confident interviewers Possible interviewer bias (learn about interview bias and how to avoid this) Sensitive issues can be challenging for the interviewer and interviewee Difficult to compare between interviewees Small samples Challenge of interviewing those who speak different languages
|
See our tips on interview etiquette. Interview etiquette |
2. Focus groups
| Expand |
|---|
| title | Expand for more information |
|---|
|
What: Group discussion to stimulate conversation about a (range of) topic(s) When: If you are looking for a better understanding of an issue that is shared by multiple people and you assume the group dynamic will give you additional relevant information Advantages No literacy requirements Opportunity to explore complex issues Different stakeholder perspectives Conversation between participants can give you more insights and nuance Readily available tips and guidance for effective methods to use
| Disadvantages No privacy or anonymity Difficult to bring together a balanced and representative group of participants The strongest voices are likely to dominate the conversation Need a skilled and sensitive group leader
|
|
3. Observation
| Expand |
|---|
| title | Expand for more information |
|---|
|
What: Extensive description of what is happening by observing people in their natural setting When: If you do not just want to ask people about how they do or experience something but rather see it with your own eyes to get a better idea of their experience Advantages | Disadvantages Only gives you a snap-shot of the situation at a certain point, unless repeated There are ethical considerations to bear in mind Observer judgement (learn about observer judgement and how to avoid this) People may change their behaviour when observed
|
|
4. Creative and arts-based methods
| Expand |
|---|
| title | Expand for more information |
|---|
|
What: arts-based research methods are generally seen as part of the field of creative methods, where artistic practices are used to help answer research or evaluation questions, or where more commonly-used methods are combined with artistic practices. The artist is likely to be involved at both the research delivery and the analysis stage, and the participants too, including: Story-stems - giving participants the beginning of a story and asking them to complete the story (using words, drawings, creating characters using craft materials, puppets, etc). Writing - documenting through writing, which could include poetry, graphic novels, cartoons, short stories and more. Visualisation - visualising an experience or result in some way including through, for example, drawing, collage, sculpture, collaborative film-making and photography. Performance - making a performance (e.g. through writing or improvising music or a play). Photovoice- an established technique combining photography and accompanying words or text. Creative mapping - asking participants to draw their journey in the form of a map, which can be helpful to understand a user journey and to think about how an experience or emotion has changed over time. This could also be combined with emotion mapping. Crafting - making something practically, e.g. with textiles, knitting, or other types of physical materials. Digital story-telling - a participant (or group of participants working together) shapes a narrative about their experience, often combining words (e.g. through sub-titles and structures), visuals (e.g. through videos or photography) and sound (e.g. spoken description). Creative drawing or writing in response to a prompt - creating an art-work in response to a prompt, and then analysing that art work to understand the meaning(s) behind it, either with or separately from the participant. Scrapbooking or journalling - asking participants to keep a visual or written diary of their experience, which they can later share or report on. Video-journaling - creating vlogs or short videos describing an experience.
When: many situations, for example, when you want to understand people’s physical and emotional interaction in a physical or digital space; when the project results in a creative output Advantages Suits visual thinkers and communicators A creative and fun method that engages people’s memories and sense of experience Valuable with younger participants, and fun for others A fun data collection method for older participants Useful when used in triangulation with other data sources Visual way to tell an impact story Can be more intuitively embedded into (digital) exhibition mediation or guides, educational materials, etc
| Disadvantages Software may be costly or not available Few set or agreed methodological approaches May not be an accepted methodological approach in some scientific or research circles Communities of practice are only just emerging, e.g. the Arts & Evaluation Community of Practice in Canada
|
See our case study from St Fagans Museum of National History, Wales, UK, for more insights into how creative mapping methods were used and combined with more conventional methods like interviews. Phase two case study: storytelling and mapping - creative methods |
5. Outcome Harvesting
| Expand |
|---|
| title | Expand for more information |
|---|
|
Read our case study on Outcome Harvesting and how we used it in an impact assessment of EuropeanaTech and the implementation of IIIF. Phase two case study: Outcome Harvesting, EuropeanaTech and IIIF |
The methods described below use mixed methods.
1. Questionnaires
| Expand |
|---|
| title | Expand for more information |
|---|
|
What: An online or paper form that allows you to ask the same questions to a large number of people When: If you want to ask a lot of people the same easy-to-answer questions to effectively compare experiences and get a good high-level overview of what change occurred for them Advantages Low cost or free Large samples are possible Wide reach, can be easily shared Can be anonymous Easy to analyse Can be multilingual Can be used to identify and recruit interview participants No transcription needed Can be conducted on different platforms and at different scales, e.g. during registration for an event, social media, pop-ups on your website or digital exhibitions, at events (e.g. in online polls or interactive slides), online, etc
| Disadvantages Bias due to self-reporting Low response rate Inflexible Response can lack context Responses may not be complete or valid Can be very long Time consuming to analyse large amounts of open text question responses
|
See our tips on writing a good questionnaire. Writing a good questionnaire |
| Expand |
|---|
| title | Expand for more information |
|---|
|
What: analysing engagement with your social media over time to understand responses to your digital offer (promoted or displayed on social media) or how our online communities are engaging with each other. This could include using the comments tool on Facebook or analysing Twitter or Instagram replies. Some tools help you scrape these comments and the text can be analysed manually or automatically (e.g. using sentiment analysis) When: If you have a digital or other campaign with the aim to increase awareness of your organisation, a certain topic or event, or if you want to understand your audience’s reaction with an online or physical project or activity Advantages Easy to assess if a baseline is collected Useful when used in triangulation with other data sources Opportunity to analyse qualitative data ‘Natural’ setting where participants do not feel like they are being observed for a specific purpose Tools like sentiment analysis can give rudimentary insight into qualitative responses Tools exist to help scrape social media comments from websites into more workable formats e.g. Excel spreadsheets
| Disadvantages Likely to contribute outputs rather than outcomes if numbers alone are used Need to be clear about ethics and data protection, including anonymising reporting what you have learned Qualitative analysis of comments, etc, takes time
|
|
The methods described below are quantitative.
1. Digital user statistics
| Expand |
|---|
| title | Expand for more information |
|---|
|
What: many options, including monitoring your API usage, tracking engagement with your website or online exhibitions or collections, and YouTube or other audio/audiovisual platform views When: If you are interested in what parts of an online exhibition people are engaging with, or what is most interesting; when you have published materials on your website or other platforms for your users to look at; when are able to track user interaction using, for example, a heat map Advantages Easy to set a baseline Software may already be in place on your website Data collection tools, e.g. Google analytics, may be free Useful when used in triangulation with other data sources Pop-up web tools may give an opportunity to collection qualitative data and to find out more about users’ preferences or experiences There are open source alternatives now readily available
| Disadvantages Likely to contribute outputs rather than outcomes if statistics alone are used Lacks the richness of qualitative data if open text responses are not possible Some tools are not privacy-friendly Some tools may require a lot of adaption to be implemented and deliver the desired results
|
|
2. Social network analysis
| Expand |
|---|
| title | Expand for more information |
|---|
|
What: study of the social connections between people, animals, etc (whereas network analysis can be used for non-living networks, e.g. electrical networks) When: you have data about connections between participants or professionals involved in the activity, and when their network outcomes are important for you to measure Advantages This is an increasingly used interdisciplinary field of research that explores the importance of connections between people in different formats Can work well with small samples Free tools are available, like NodeXL (with upgrade options available) Presents interesting data visualisations in network graph format
| Disadvantages Requires you to invest time to understand the principles and tools Not widely used in the cultural sector Full potential still to be explored in the cultural sector
|
See our impact assessment of Europeana’s annual conference in 2019 for an insight into how we applied a simple networking analysis to the data we collected from participants. https://pro.europeana.eu/post/impact-assessment-report-europeana-2019  |
3. Economic impact analysis methods
| Expand |
|---|
| title | Expand for more information |
|---|
|
Go to our guidance on measuring economic impact. Measuring economic impact |
4. Approaches to measuring your environmental impact
| Expand |
|---|
| title | Expand for more information |
|---|
|
We have a dedicated page to ways of assessing your environmental impact. Assessing environmental impact |
| Panel |
|---|
| panelIconId | 2714 |
|---|
| panelIcon | :heavy_check_mark: |
|---|
| panelIconText | ✔️ |
|---|
| bgColor | #B3F5FF |
|---|
|
How do you choose the right method? Some resources. Check out Social Impact Navigator’s guide to data collection methods, with a table showing which method works best in different circumstances. You can also use BetterEvaluation’s rainbow checklist. This takes you through a number of steps to identify appropriate methods for your case.
|
...
Next steps
Learn from Europeana’s impact assessments - Europeana has published a number of our own impact assessments. In each, we set out our methodology and we’re open about what we learned and the limitations. Take a look and see how we have applied different methods in our research.
Methods quiz