Part 2
In this section
- Step 5: Resources
- Step 6: Evaluation Methodology & Tools
- Step 7: Values and Data Protection
- Step 8: Conduct your evaluation
Step 5: Resources
Unfortunately, there is not a cupboard full of evaluation pixies ready to leap out and do the evaluation; I have looked for one on many occasions and can confirm those pesky pixies always disappear as soon as you open the cupboard door.
So have a think as to what resources are realistically available to the project to create the evaluation tools; conduct the evaluation; gather and collate the data, evidence and indicators and synthesise these into the final report, or otherwise (more on reporting below). And write down the name of the individual(s) who will be responsible for the tasks in your evaluation plan: is this your good-self; a team member or colleague in a partner organisation? Or do you have funding for evaluation to enable you to contract an external evaluator?
Being clear on what resource is available will enable you to develop and deliver an effective and feasible evaluation. Effective evaluations do not have to be resource intensive – the key is to prioritise what is most important to find out and to focus your resources accordingly – do not try to evaluate everything! Furthermore, make sure your evaluation is commensurate to the scale of the project.
Furthermore, by planning your evaluation at the start of the engagement project makes it so much more efficient as it can often be embedded into the project and activities.
Step 6: Evaluation Methodology & Tools
The next step is to explore and select the evaluation approach, i.e. the methodology and tools to be used to gather the indicators, evidence and data to explore if the key outcomes have been achieved and to answer your evaluation questions. In addition, decide on whether gathering quantitative or qualitative data will be most appropriate/ valuable, although in reality many evaluations adopt a mixed-methods approach – i.e., gather both.
Selecting the appropriate evaluation method and tools depends upon the nature of your engagement activity and its objectives; the purpose of your evaluation; your respondents and how you can reach them; and the resources available for the evaluation.
Example Evaluation Tools
Your evaluation tool does not have to be a questionnaire – there are many different and more creative tools that can be used. Download this Evaluation Tools Guide that summarises a range of example methods and tools. You can use one or several evaluation tools to enrich your data and evidence base; this also enables findings, that have been gathered using different approaches, to be cross-checked and compared [a.k.a. ‘Triangulation’].
Sampling
There is no need to ask all the folk involved in the project or activity (in evaluation terms – this is called the ‘population’) for their views, experiences and outcomes unless this is appropriate and feasible.
You can instead select a subsection of the population which will become your sample. For qualitative work, you will likely have a much smaller sample than for quantitative work, as it is more in-depth. Examples of different sampling methodologies are provided below:
Census: all members of the population
- Random: randomly selected sample
- Systematic: select – for example, every ‘nth’ person (e.g., 10th on a list or when the 5th person leaves a venue etc)
- Quota: sampling people based on a particular characteristic or role in the project
- Self-selecting: is where it is up to the members of a population to choose to respond or not. e.g., questionnaires left at the site of the engagement. Try to avoid this where possible; but if it is the only option open, be aware that it will likely introduce bias as they tend to be completed by those that have had a very positive or negative experience; and are unlikely to be representative of the population; and be sure to introduce this as a caveat when reporting your findings.
Step 7: Values and Data Protection
Honesty
Honesty is so important in evaluation including the need for you, the team and partners to be honest with yourselves. For example, be as keen to find out any negative outcomes and challenges as well as the positives and successes. Be your own critical friend when thinking about and sharing reflections and what you would do differently next time.
Another key focus is to enable evaluation respondents and participants to share their true reflections and ‘feel’ they can be honest in their feedback. As such, avoid at-all-costs asking leading or biased questions and frame questions so people feel their feedback – whether negative or positive – is constructive. For example, ask – “In what ways could the project be improved and why?” rather than “What was wrong with the activity?”.
Trust, Respect and Protection
Ensure you maintain respect and are worthy of people’s trust when carrying out data collection and synthesising and sharing the results. This includes developing an inclusive evaluation that is appropriate and pitched at the right-level to the target groups and enables all participants and respondents to take part fully. Take the time to create good quality evaluation resources as ambiguous questions and poorly produced tools are unlikely to make the participants feel that the evaluation matters to you.
Where you are collecting personal data (i.e., where an individual can be identified) you must ensure that the work complies by GDPR including seeking informed consent as may be required, communicating the purpose of the evaluation; how the data will be used and stored; and the right to withdraw.
Ethics
Do you need ethical approval to do an evaluation? In short – it depends! Where ethical approval might be needed includes:
- If the findings are to be used in any published research outputs; as the evaluation then also becomes research and needs to go through the usual ethical processes as any piece of research would do.
- Will the evaluation activities have the potential to cause distress in any way? For example, exploring engagement with sensitive and challenging topics?
- In many cases, evaluation will not necessarily need to go through the University’s ethics processes; but there is still a need to explore your evaluation purpose, methods and use of the findings with an ethical lens – and to do what is right, considerate and appropriate for all those participating in the evaluation – including researchers; respondents; participants and the evaluators.
[Note - For more advice please visit the R&IS pages on ethical approval]
Step 8: Conduct your evaluation
Are you now thinking it takes until Step 8 to actually start doing some evaluation? Then the answer is absolutely! Too many poorly planned evaluations jump straight to the selection of tools and collecting lots and lots and lots of data; much of which either won’t get used or was not that relevant to what you really wanted to find out.