Audience Research and Evaluation
Eric Jensen has conducted research and impact evaluation at a wide variety of public engagement and informal learning settings. Here he offers some thoughts on different evaluation methods.
Studies involving audience analysis
Preaching to the scientifically converted: Evaluating inclusivity in science festival audiences. International Journal of Science Education Part B: Communication & Engagement.
Why people attend science festivals: Interests, motivations and self-reported benefits of public engagement with research. Public Understanding of Science, 23(5): 557-573.
Re-considering ‘The Love of Art’: Evaluating the potential of art museum outreach Visitor Studies, 16(2): 144-159.
Towards a ‘contextual turn’ in visitor research: Evaluating visitor segmentation & identity-related motivations. Visitor Studies, 14(2): 127-140.
Blog entries (skip to content):
Using QR codes for museum interpretation - good idea? (Q and A from professional email list)
Open access articles available here:
The problems with science communication evaluation
Blog entries
The siren call of the ‘easy option’ in public engagement and informal learning evaluation: Post-it (sticky) notes, comment cards, drop a token and tap a button options
I regularly encounter people eager for an easy option with audience feedback or evaluation who have been seduced by one of the following methods:
• Post-it note feedback
• Anonymised comment cards
• Drop a ball or token into a container divided into sections to indicate your response
• Tap a button on the way out of an event to indicate your views
The main limitations of these approaches are the same as any other self-selection method (e.g. having a stack of comment cards sitting at the exit or information desk). There is nocontrol over the parameters of the sample, so it is impossible to know how representative it is. This limitation makes it particularly invalid to make claims about the impact of an exhibit (or similar) on a visitor population on the basis of a self-selection method. The post-it note approach adds an extra layer of potential bias though by providing the respondents access to what other respondents have already said. Thus, the basic advantage of the questionnaire method (standardisation) is undermined as the stimulus for people’s responses will be constantly shifting over the data collection period. As a source of purely qualitative data about a range of possible responses to an exhibition, it may be okay. However, the lack of any contextual information (age, gender, ethnicity, etc.) is a problem for interpreting such qualitative data.
The lack of control over sampling makes the other options in this category equally unhelpful, except with a button or ball or freestanding kiosk you have even less information about whether the content of the data is valid, that is, whether they truly represent people’s views. For example, it is common for children to ‘participate’ in these types of data collection exercises by tapping buttons or dropping in tokens without regard for the labels on the categories. There is only one good use of post-it note evaluation I have encountered to date. A question/prompt for the post-it note responses along the lines of ‘What are we missing?’ could conceivably generate valuable responses for organisers / practitioners to learn from.
Managing missing demographic data to minimize sampling bias risk: Q and A from professional email list
QUESTION
I have a statistical methodology question - what could be more exciting for a damp warm Friday? I realise it's not entirely in the comfort zone of this group, but I thought I'd try before exploring it with statistician contacts and broader research online.
It's about managing missing data in survey response, where the missing data is Missing Not At Random (MNAR) aka nonignorable nonresponse. I'm interested in any established models to correct for bias. Maybe those of you who have conducted surveys have come across this and found a good, understandable solution?
I'll explain the problem. Imagine you're conducting a survey where some of the questions are within the 'sensitive data' realm: race, gender, sexuality, disability. Imagine you're getting high 'prefer not to answer' levels , eg 50%. One flawed approach is listwise deletion, meaning that the 50% PNTA is simply excluded from analysis. This introduces a bias risk, because it's unlikely that the nonresponse is random, it's more likely to be meaningful - eg you might argue that over-represented cases - white, heterosexual males without disability - are slightly more likely to PNTA than under-represented cases. So deleting the PNTA is likely to introduce bias in your analysis, even if that nonrandomness is low level. A concrete example: removing 50% PNTA from the gender question might bias your analysis towards misleadingly high % female.
There are complex statistical methodologies for approaching the management of this problem - multiple imputation, maximum likelihood estimation, etc - but the complexity is daunting to a non-statistician without a software package like Stata. So I wonder if any of you have done this and either found a simple solution or developed a complex solution which is transferable - in other words, does anyone have some Python they can give me / direct me to??
ANSWER (ERIC JENSEN)
No amount of fancy statistical tests are going to allow you to magically reverse engineer your data to identify what systematic biases might have been introduced in the non-response to demographic survey items.
For face-to-face surveys, best practice dictates the use of a ‘refusal log’, where you track any visible characteristics of the respondent (e.g. ‘white’ or ‘non-white’, ‘apparent gender’) to identify systematic biases that may have affected the data. If your survey is purely online and does not afford these options, you may be stuck just acknowledging this as a limitation of your data.
I am not aware of any robust evidence in the UK showing that there is a persistent pattern of non-response to demographic questions affecting one type of respondent more than another.
I would strongly advise against one of the possible solutions suggested by on the list previously :
'Could you re-run the survey without PNTA as option'
This would be poor practice and could result in people exiting your survey altogether at this point, or putting down false information if they would in fact prefer not to answer.
The second suggestion on this list of indicating how important this data is (and I would also stress what you are going to do with it) does sound like a promising approach:
'preface it with a statement about how ticking the PNTA box might lead to skewed results'
If you are getting high levels of non-response to demographic questions, it is worth reviewing the quality of the question and response options to ensure they are a good fit with your respondents and easy for them to answer (some pilot testing may be in order).
Your concern for the quality of your data is very admirable, by the way!
Using QR codes for museum interpretation - good idea? (Q and A from professional email list)
QUESTION
I’m guessing that this one has been discussed previously, but I’m wondering if anyone uses QR codes at their site to engage visitors? If so, how do you utilise them and how successful have they been? Have they successfully engaged a particular audience for example?
Equally, if you’ve considered using QR codes for interpretation but have actually decided on another solution, I’d like to hear from you.
ANSWER (ERIC JENSEN)
All the research I have seen published on this topic (and my own observations while conducting audience research studies) points to the idea that QR codes are a poor option for reaching people with interpretation.
The proportion of people who would actually use them is usually very low, even in the best of circumstances (these two articles [first] [second] are about university students’ usage of them in museum like spaces).
The most optimistic statistical accounts about QR code usage levels tend to focus on the best case scenarios: People using them for coupons to save money, questions about whether people have ever used one, etc.
In my experience, an easy to type / remember link is more likely to be used than a QR code (although still low usage level compared to something brief and easy to access available on site).
About Dr Eric Jensen (@JensenWarwick)
Dr. Eric Jensen (Associate Professor, Dept. of Sociology, University of Warwick) is a leading social scientist specializing in innovative methods of conducting impact evaluation research in informal learning and public engagement contexts. Jensen is author of Doing Real Research: A Practical Guide to Social Research (SAGE). He has extensive experience designing and conducting quantitative and mixed methods evaluations at institutions including the National Gallery, London Zoo, Natural History Museum, Cheltenham Literature Festival, the British Museum, San Diego Zoo and Bronx Zoo. Dr. Jensen holds a PhD in Sociology from the University of Cambridge (UK). He teaches social research methods at the University of Warwick.
