Simulations

Download video transcriptLink opens in a new window

What can simulations assess?

Simulations offer performance-based assessment which can be used to evaluate across all learning dimensions: cognitive, behavioural and affective depending upon the intended learning outcomes to be measured and the nature of the simulation developed. They lend themselves especially well to assessing professional competencies, communication and interpersonal skills, application of knowledge, and decision-making.

What alternative assessments could I choose?

< Back to assessment methods selector

Design

Simulations can assess individuals or groups. When designing a simulation assessment, it is crucial to clearly delineate the intended learning outcomes, the purpose of the task and to ensure that these are strongly aligned. It is also important that students understand the process and the benefits of the task.

When students are assessed through simulation the primary consideration is that the simulation itself works. There is a wealth of literature on designing simulations in medical education, political science and international relations, and in business and management upon which to draw.

The videos below provide some advice for designing simulations across Social Science and Healthcare.

Download video transcriptLink opens in a new window

Game-based simulations

Game-based simulations have a longer history as a teaching method rather than an assessment method. They can be a powerful mechanism to change perspectives, understand complexity, and offer improved understanding of others’ mind-sets. Debriefing sessions and feedback are crucial if these functions are to be fulfilled, and therefore these may work most effectively as formative assessment methods. Gamed simulations can draw upon factual and / or fictional scenarios and cases.

In sum those who would assess the learning taking place in any given gamed simulation have two questions to answer:

are participants learning the model of the real world that the game is simulating?
does the model of the real world have verisimilitude? (Chinet al2009; 559)

Assessing simulations

What you are going to assess will be determined by your intended learning outcomes, the constraints within which you are operating the simulation, and which aspects of the simulation you plan to assess.

Preparation: assessing preparatory materials produced for the simulation [e.g. position paper; briefing memo] will enable you to assess aspects of student research, such as data collection and analysis, synthesis, critical thinking, evaluation. Asking students to submit a portfolio of research resources will enable you to evaluate research processes.

Interaction: the agreed and communicated criteria against which you assess interaction with or within the simulation will depend upon the nature of the simulation and intended learning outcomes. It might include: mastery of technical skills, adherence to rules of procedure, participation in formal debate, communication, effective representation, negotiation and compromise, decision-making, leadership, teamwork and/or inter-professional communication.

Some simulation mechanisms may offer outcomes / feedback - technologically or in person [e.g. a standardised patient] which can be used to evaluate performance. Even with clear assessment criteria evaluating participation may admit a degree of subjectivity and you will need to be aware of the potential for assessor bias.

Debrief: post-simulation you might assess students’ critical reflections on the experience of the simulation. Petranek, Corey and Black posited a reflective model for simulation based on consideration of the four ‘e’s: events, emotions, empathy, and explanation (1992, 174). Assessing student reflection may be particularly powerful in formatively assessed work.

Diversity & Inclusion Academic Integrity

Student and staff experience

Benefits

Research shows that students respond positively to simulations, finding them enjoyable and stimulating.

The use of simulations in clinical education is well-researched. The ability of simulations to train and evaluate technical, clinical and professional competencies and have a positive impact on patient outcomes is well-established.

Game-based simulations offer student-centred and active learning, and push students into higher order thinking requiring them to apply knowledge, synthesise ideas, and think critically and analytically through a series of decision-making processes.

As they simulate the ‘real world’ - they can also promote authentic learning and assessment.

Challenges

Simulations can also be a source of frustration and anxiety, especially if there are administrative difficulties, which will be heightened if assessment is summative and high stakes. As relatively unusual learning and assessment environments students can find simulations, especially in their initial phases, confusing if the purpose of the simulation is difficult to grasp.

If the simulation assesses group performance you will need to be particularly mindful of all aspects of group assessment, especially if your simulated scenario includes competition. A mixture of individual and group marks (see the Group WorkLink opens in a new window section), perhaps at different stages in the preparation, interaction and debrief, might be helpful to mitigate against inequity in distribution of tasks or perceptions of unfairness.

Assessment of student learning within simulations can be more complex than other methods of assessment. The constraints of space (room availability) and time (timetabling, preparation, multiple iterations for larger cohorts) may be an issue. Access to resources may also present a barrier.

Some simulations may require an appropriate level of conflict in order to be effective and this will need to be carefully managed by the person running simulation.

Student absence will be problematic. Absence of any students may compromise the experience of the other students within the simulation. Many simulations will not be replicable and therefore you will need to consider how to re-assess should student be absent or need to remedy failure.

Workload

There is evidence that the demands of the simulation may consume students’ time and energy outside of class in ways that are not productive for learning - providing students with outlines or instructional supplements can mitigate against this (Raymond and Underwood, 2013, 163).

The manageability of assessment using simulations will depend upon what element (preparation, interaction, debrief) you are going to assess. Summatively assessing participation during the simulation will require the session to be captured in some way for audit purposes and for the external examiner. The size of the simulation, and the number of participants involved may also present logistical issues; each assessor can only watch 5 people sufficiently closely and so you may need a number of assessors in the room, or run the simulation multiple times.

Preparation time to create a convincing 'world' or to generate scenarios could be massively decreased through the use of AI - both text-based, and visual.

References

Petranek, C. F., Corey, S., & Black, R. (1992). Three levels of learning in simulations: Participating, debriefing, and journal writing. Simulation & gaming, 23 (2), 174-185.

Raymond, C. and Underwood, S. (2013). Assessment in simulations. Journal of Political Science Education, 9 (2), 157-167.

Useful resources

Vermunt, J. D. (2023). Understanding, measuring and improving simulation-based learning in higher education: Student and teacher learning perspectives. Learning and Instruction,86, 101773.Link opens in a new window