Get the data (right): Impact Playbook Phase 2

About 

Phase Two of the Impact Playbook talks through six stages from planning your data collection to analysis to developing your recommendations, illustrating these with real life examples in the same intuitive and helpful style as Phase one.

This practical webinar talks through the Playbook process and the challenges and opportunities Europeana colleagues and sector professionals have encountered as they’ve gone through impact assessment and data collection processes themselves. The first 45 minutes follows a standard webinar format, and for the last 45 minutes all attendees join a bigger discussion about the questions and issues raised in the presentations. 

This webinar was organised in collaboration with Common Culture. It took place on 25 November 2020 from 15:00 - 16:30 CET. 

Speakers

• Nicole McNeilly (Europeana Foundation) introduces Phase two of the Europeana Impact Playbook and share how this has been applied in different cases in the Europeana initiative across 2019/2020.
• Jenny Kidd (Cardiff University) presents a study on the experience of users of an immersive mobile digital story-telling experience at St Fagans Museum of National History in Wales.
• Rob Davies (Cyprus University of Technology) presents a case study based on the impact assessment carried out in the ‘A Million Stories: refugee lives’ digital storytelling project, carried out by public libraries in Denmark, Germany, Greece and Sweden during 2017.
• Shalen Fu (King’s University London) presents a list of impact indicators developed to demonstrate the economic, social, innovation and operational impacts of digital museum resources and the preliminary results from her survey research with museums and the general public in China.

Resources

• Join the Europeana Impact Community to find out more about impact and cultural heritage, network with peers and hear about relevant events, resources and opportunities.
• Find out more about this webinar series. 
• Download the Europeana Impact Playbook. 
• Access the webinar slide deck.
• Read a summary of the webinar below: 

Nicole McNeilly, Impact Advisor at Europeana Foundation, was first to take the floor. (See 07:33 in the recording) She set out the six steps of Phase two of the Impact Playbook (download this now from Europeana Pro), which include practical tips for how to develop your indicators to measure your impact and an overview of the pros and cons of the different methods and approaches you can take, as well as building in the steps you need to take to get your team and colleagues on board. She finished with a quick overview of the results of a forthcoming impact assessment, which measured the impact of Europeana 2019 (and also had a look back at the impact of past annual conferences and AGMs). The approach taken in considered the impact for participants (with network development emerging as the strongest impact), as well as the economic impact for the host city and the environmental impact of air travel. These impact assessments will be published in 2021, so keep an eye out!

Next to speak was Dr Jenny Kidd, Reader and Director of Postgraduate Research in the  School of Journalism, Media and Culture at Cardiff University. (See 20:22 in the recording) Jenny presented a study on the experience of users of an immersive mobile digital story-telling experience at St Fagans Museum of National History in Wales. Taking a very qualitative approach, they used creative mapping methods to better understand how users interacted with the digital experience. The users then talked through the map they created with the researchers, ‘as a start point for discussions about experience in place’. This drew out themes of usability and accessibility, emotional response, for example, to gain insights ‘into what had been significant’ for them. This was followed by a more structured interview approach, and the data was then analysed and summarised into a report. Jenny reflected on how a longitudinal approach could help to answer other research questions, as well as how a more quantitative approach by exploring the broader data from the mobile experience might have revealed more about people’s physical interaction with the experience, and help triangulate and strengthen the qualitative findings. The qualitative approach taken helped them learn about experiences ‘in which the senses, story and place can come together to create stirring and emotional, immersive encounters’. The study helped the museum and the researchers talk more confidently about ‘the value and relevance of mobile digital story-telling’. We hope to publish this case study on Europeana Pro as soon as possible. 

‘...we understand better going forward what works in order to facilitate rich immersion and experience and social interaction in immersive experiences, and then of course what some of the challenges and barriers are to these sort of experiences, as well. To use the language of the Playbook, we can talk more robustly about the Traces projects in terms of the social impact, innovation impact and operational impact as well.’ 

Shalen Fu (King’s College London) then presented a list of impact indicators developed to demonstrate the economic, social, innovation and operational impacts of digital museum resources and the preliminary results from her survey research with museums and the general public in China. (See 35:45 in the recording) Shalen was able to share her methodology, which follows the Balanced Value Impact model (also at the heart of the Europeana Impact Playbook), and involves surveys and follow-up interviews. A number of indicators that might be used to assess the impact of digital heritage activities were shared with museum professionals using a scale on which they could assess their strength. Across the indicators, under economic impact, we learned that heritage professionals and the Chinese public most strongly agree with the idea that museums should support the participation of local people in heritage as well as contributing to growth in visitor spending. Under social indicators, accessibility was important, but creating social ties (i.e. community networks) was rated lowest. Shalen noted how the results might differ in different cultural contexts. Other insights emerged, for example, that ‘nowadays, young people do not only value the outward impact, but also the inward impact digital resources have on museum management, and they do care how well a museum operates’. Shalen shared a draft version of the indicator impact framework and will be sharing more of her work with us in 2021!

‘To get the data right, we have to get the indicators right’

Rob Davies (Cyprus University of Technology) presented a case study based on the impact assessment carried out in the ‘A Million Stories: refugee lives’ digital storytelling project, carried out by public libraries in Denmark, Germany, Greece and Sweden during 2017. (See recording 44:00) After showing the project website and introducing the project, Rob set out the aims and objectives of the research and the mixed methods methodology that was followed. This included both a process evaluation (looking at how the project was going, including, for example, convening internal focus groups) and an impact assessment of the experiences of those who shared their stories of their recent journey to the EU. The latter focussed on the experiences of the refugee participants as well as the host population, people from the areas to which they had moved. Rob was responsible for analysing all of the data:t most of the partner libraries, had never conducted an impact assessment before. We got an insight into the analytical process he took, into the outcomes for both the migrant and host population, and into the challenges encountered relating to the impact assessment process. 

In the discussion afterwards, we had a short but in depth discussion. This was not recorded in order to encourage a frank and open discussion. Two questions brought up the need to know who is benefitting from your activities and how to reach them in order to understand the impact they experience. One question related to understanding the impact of open resources, such as those who use APIs to access digital cultural heritage but who are not able to be tracked. There is no solution to this, but it’s something to think creatively around. Europeana has some ways to survey this audience such as, for example, Hotjar pop-up surveys. Voluntary or obligatory registration could be introduced for API users, or an introductory webpage could ask users to complete a survey, though the sample might be small. We’ll take a look for any case studies that address this. 

‘It’s a challenge, and I think it’s a challenge that we can figure out in the next years, as it’s going to be a question and a challenge which is getting larger and larger, because of so much content going online and so many people reaching out to us who are anonymous’ Maaike Verberk, DEN

Jenny was asked who had been involved in the design process of the impact assessment she conducted, and let us know that this was a really broad range of stakeholders, including University PhD researchers, those who designed the  and . In response to a later question about methods that could be used by smaller organisations, she noted that observational research could have been used but wasn’t in this instance. The approach has to reflect the questions that you’re interested in - this should guide the impact design. 

Rob was asked about the interaction of the refugee participants with the impact assessment process, especially as he was working with those who had very recently arrived to their new home country. A perspective of those that were in the country for longer would have added value, but this was not as relevant due to the objectives of the research. Each approach has its own challenges (see Rob’s slide for more insight).