Skip to the report's conclusion

Research Questions and Objectives

This research project addresses the use, quality and possibilities of technology-linked evaluation methodologies and strategies that could be used to help evaluate the development of cultural value in different settings.

This research addressed the following questions: (1) How can technology be harnessed to aid investigation into a full range of impacts from arts and culture experiences? (2) Can such technologies be used to gather valid impact evaluation evidence from social networking website discussions and other naturally occurring digital forms of data? (3) What range of technologies have potential for supporting evaluation in new domains of cultural impact (e.g. online and social media)? (4) To what extent is it possible to use technology to automate the evidence-gathering process relating to cultural value? (5) What are the strengths and weaknesses of different research-related technologies for evaluating arts and cultural impacts?

The project report does the following:

(1) Develop a catalogue of current uses of technology-enhanced evaluation methods and approaches to measure cultural value and related impacts. (2) Critically review the strengths and weaknesses of different technologies for evaluating different components and dimensions of cultural value in a range of contexts. (3) Develop report describing different technology-enhanced methods of evaluating cultural value, including a master table of methods, technologies and approaches along with a critical assessment of their strengths/weaknesses.

Contribution to the Cultural Value Project

Methods of evaluating cultural impact still need substantial development to support long-term empirical research on cultural value. Meanwhile, technological innovations have raised new possibilities for evaluating cultural value that promise to increase efficiency, reach and validity beyond what is possible with conventional methods.

This project considers several technology-enhanced methods for evaluating cultural value, providing descriptions and critical assessments to elucidate their strengths and weakness. It focuses on evaluation methodology with the aim of supporting future empirical evaluation and research on cultural value as well as providing an appropriate framework for making best use of digital technologies. The project builds on existing methodological literature and reviews current literature on the use of technologies to evaluate the development of cultural value through a variety of events, settings, institutions and digital platforms such as Facebook, Twitter, individual webpages, etc. The results of this critical literature review will be able to broadly be applied to a framework that can advance the way in which the value of cultural engagement is evaluated and employed.

Research Methods

The methodology employed for this project integrates the existing knowledge of arts/culture engagement impact evaluation, software design, production and implementation, mixed methods data analysis, visualisation of digital programmes and the requirements for integrating these new modes with more traditional cultural impact analysis. At its core, the project is focused on how we can capitalise on digital technologies to build a robust, evidence-based evaluation framework, while at the same time maintaining a critical perspective with respect to the limits of the technologies’ contribution to evaluating cultural value.

The first phase involved an initial scoping review seeking evidence from published academic papers and books; grey literature including reports, evaluations, other unpublished reviews; and, where possible, on-going research of appropriate quality.

This review revealed that while there is literature on the role of specific new technologies in delivering public engagement with arts and culture, there is remarkably little material on the use of technology in evaluation and visitor research.

Critical Review's Conclusions (updated July 2014)

It is clear that the UK’s major arts funding organisations have grasped the fact that there are still significant problems associated with measurement (and especially the measurement of cultural value) in the context of the arts and cultural sector, and are beginning to invest serious money into addressing such challenges. However, this review suggests that, in the absence of any common measurement frameworks or toolkits, most of the organisations which are delivering artistic programmes on a day-to-day basis still have a number of high mountains to climb before they feel confident in being able to measure and make sense of cultural value. There is similarly still a long way to go for most of them to embrace new digital technologies in support of such measurement.

It is important to point out that, in attempting to measure cultural value, we appear to have missed out the stage in the process which enables us to understand what ‘cultural value’ is and, especially, that it is more than simply ‘marketing and communications’ or ‘audience development’. Instead, other things that may or may not be evidence of cultural value have often been measured – usually using inappropriate methods and tools. The emergence of a wide range of new digital technologies, a few of which have been described in this report, offers the arts and cultural sector a number of different ways of understanding what cultural value actually is. In particular, through pattern recognition, it offers techniques that can feed on diverse qualitative and quantitative data which are harvested in real-time through participative processes involving audiences themselves.

The vast majority of practitioners and consultants working on the task of measuring cultural value in the arts and culture sector have not had the social scientific research methods training required to produce valid impact evaluations or be a critical consumer of visitor research and impact evaluations conducted by others. This no doubt helps to explain the ubiquity of very poor quality visitor research and evaluation in this field, including in the published empirical museum studies literature. The most basic principles of survey and research design are routinely violated in evaluations by (and commissioned by) institutions large and small. Training is an option for addressing this shortcoming. However, it takes time to learn how to design and then to actually conduct valid data collection and analysis, and most practitioners are just too pressed with other priorities to gain these skills, even if they sincerely want to. This situation makes technology an appealing solution.

This possible solution was explored through the Qualia project (funded by the Digital R & D Fund for the Arts), which has just finished: qualia.org.uk. The goal was to use the latest technologies to enable the design of evaluation systems that could be automated so that, once they are set up in consultation with a social scientist, they could be used by institutions/practitioners without any skills in social scientific analysis. The goal was to build a high quality open source system that could be used by arts and culture institutions across the UK with a bare minimum of customisation required to deliver automated evaluation results. While this project proved the potential of such a system, it also revealed more barriers. It must also be recognised that the barriers to achieving an infrastructure which supports effective cultural value measurement are considerable.

These barriers are less technological, and much more to do with the enablement of open and participatory engagement of all ‘players’ across the entire value-adding artistic and cultural supply chain – including (among many others) educators, artists, funders, politicians, audiences, and systems developers. Moreover, the lack of genuine expertise in valid measurement practice amongst those employed or commissioned to assess culture value means that viable partners for ensuring effective implementation of technology-enhanced evaluation systems are hard to come by. Clearly, the tide of history is carrying the cultural sector towards greater adoption of technologies for measuring public impacts. Yet, without serious work to develop the theory and methodology of cultural impact evaluation, no amount of technology will achieve the challenging task of robustly measuring cultural value.