Objective tests (Short-answer and Multiple choice questions)
Using objective tests to assess learning
Introduction
Objective tests are questions whose answers are either correct or incorrect. They tend to be better at testing 'low order' thinking skills, such as memory, basic comprehension and perhaps application (of numerical procedures for example) and are often (though not necessarily always) best used for diagnostic assessment. However, this still affords a great variety of both textual and numerical question types including, but not limited to: calculations and mathematical derivations, mcqs, fill-in-the-blanks questions and short essay (short answer) questions.
LSE (2019).
In brief, objectives tests are written tests that require the learner to select the correct answer from among one or more of options or complete statements or perform relatively simple calculations.
What can objective tests assess?
Objective tests are useful to check that learners are coming to terms with the basics of the subject in order that they have a firm foundation and knowledge. They are useful because:
- can test a wide sample of the curriculum in a short time
- can be marked easily; technology can assist with this
- less reliance on language skills of the students
- useful for diagnostic purposes: gaps and muddled ideas can be resolved.
The drawbacks are:
- students can guess rather than know
- the random nature of the questions does not help build mental maps and networks
- writing good questions is not easy
- they tend to focus on lower-order processes: recall rather than judge, explain rather than differentiate.
Design
Short-answer
Short answer questions (SAQs) tend to be open-ended questions (in contrast to MCQ) and are designed to elicit a direct response from students. SAQs can be used to check knowledge and understanding, support engagement with academic literature or a particular case study and to encourage a progressive form of learning. They can be used in both formative and summative assessment. SAQs may take a range of different forms such as short descriptive or qualitative single sentence answers, diagrams or graphs with explanations, filling in missing words in a sentence, list of answers. As the name suggests, the answer is usually short.
Gordon (2015, p.39)
Depending on the type of question, marking may simply involve checking against a list of correct answers. Alternatively a set of criteria may be used based:
- factual knowledge about a topic: have the questions been answered correctly?
- numerical answers: will marks be given on the process as well as the product answer?
- writing style: importance of language, structure, accuracy of grammar and spelling?
How to design good questions:
- express the questions in clear language
- ensure there is only one correct answer per question
- state how the question should be answered
- direct questions are better than the sentence completion
- for numerical questions be clear about marks for process as well as product and whether units are part of the answer
- be prepared to accept other answers; some of which you may not have predicted.
Multiple choice questions (MCQ)
The Centre for Teaching Excellence (no date) provides useful advice for designing questions including illustrative examples. Those guidelines are paraphrased and enhanced here for convenience.
Definition: A multiple-choice question is composed of three parts: a stem [that identifies the question or problem] and a set of possible answers that contains a key [that is the best answer to the question] and a number of distractors [that are plausible but incorrect answers to the question].
Students may perceive MCQs as requiring memorisation rather than more analytical engagement with material. If the aim is to encourage a more nuanced understanding of the course content, questions should be designed that require analysis. For example, students could be presented with a case study followed by MCQs which ask them to make judgements about aspects of the brief or to consider the application of certain techniques or theories to a scenario.
The selection of the best answer can be focused on higher-order thinking and require application of course principles, analysis of a problem, or evaluation of alternatives, thus testing students’ ability to do such thinking. Designing alternatives that require a high level of discrimination can also contribute to multiple choice items that test higher-order thinking.
When planning to write questions:
General strategies
- multiple-choice question tests are challenging and time-consuming to create; write a few questions, after a lecture when the course material is still fresh in your mind
- instruct students to select the best answer rather than the correct answer; by doing this, you acknowledge the fact that the distractors may have an element of truth to them
- use familiar language; students are likely to dismiss distractors with unfamiliar terms as incorrect
- avoid giving verbal association clues from the stem in the key. If the key uses words that are very similar to words found in the stem, students are more likely to pick it as the correct answer
- avoid trick questions. Questions should be designed so that students who know the material can find the correct answer
- avoid negative wording.
Designing stems
- ask yourself if the students would be able to answer the question without looking at the options. If so, it is a good stem
- put all relevant material in the stem
- eliminate excessive wording and irrelevant information from the stem
Designing answers
- limit the number of answers; between three and five is good
- make sure there is only one best answer
- make the distractors appealing and plausible
- make the choices grammatically consistent with the stem
- randomly distribute the correct response.
There are a number of packages that can analyse the results from MCQ tests for reliability and validity. Using the questions for formative purposes can generate the data needed and so pilot questions prior to their use for summative tests. In addition to asking student to give an answer we can also ask for their confidence rating - how sure they are about the answer they are giving. This not only reduces guessing, but also provides feedback to the learner about the extent of their comprehension / understanding.
Using online packages to administer the test allows instant feedback. Once a student has selected an answer they can be told if they are correct or not and be given an explanation of their mistake. Some of these packages select questions on the basis of previous results rather than randomly, which allow a check on whether the learner is gaining from the feedback provided [adaptive testing].
Diversity & inclusion
There is some evidence that males perform better than females in MCQ examinations as they are more willing to guess. Using MCQs for formative rather than summative purposes resolves this. Using short answer questions reduces reliance on language and so is more inclusive for those working in a second language.
Academic integrity
If used for summative purposes one needs to maintain the integrity of the question banks by not allowing copies out of examination room.
When used online it is important to have a large question bank to enable random generation of tests. (Click here for further guidance on academic integrity.)
When used outside of in-person exam conditions assessment may become less secure, as online working could facilitate collusion, or contract cheating, or the use of AI. Randomly generated questions (with different questions or questions in a different order) might mitigate against collusion.
Student and staff experience
Benefits
Short answer
Students: are often more familiar with the practice and feel less anxious than many other assessment methods.
Staff: short answer questions are relatively fast to mark and can be marked by different assessors, as long as the questions are set in such a way that all answers can be considered by the assessors. AI can support feedback generation.
They are also relatively easy to set.
Multiple choice questions
Students: good to enable self-assessment, particularly online e when the feedback is instant
Staff: are quick to mark, and be grouped into re-usable questions banks and efficient approach to testing large numbers of students.
Challenges
Short-answer
Tests lower levels of learning and may encourage surface approaches to learning. Rather like mcqs, to make this approach test higher levels it is the structure of the questions that becomes more complex rather than the content of the question itself.
If short answer questions are to be used in summative assessment they tend to be used alongside longer essays and other longer forms of assessment and thus time management is crucial.
It is very important to be very clear about the type of answers that you expect because these are open-ended and students are free to answer any way they choose; short-answer questions can lead to long answers if you are not careful.
Multiple choice questions
It is challenging to write questions that test higher order learning; the question structure tends to become more complex rather more than the content being tested (see Question Pro in Useful resources below). Students need practice before taking a summative mcq examination so that they are being tested on their knowledge of the material and not on their understanding of the question type.
Workload
Students
Taking full advantage of the feedback may be more time consuming for students than actually answering questions; but this is one of their strengths.
For staff
Multiple choice question writing is expensive in terms of time, but once a good item bank has been established then the use of the questions, and their marking, is of low demand in terms of time.
Short answer questions are relatively fast to mark and can be marked by different assessors, as long as the questions are set in such a way that all alternative answers can be considered by the assessors.
Useful resources
Multiple Choice
Question Pro: Multiple choice questions.
https://www.questionpro.com/article/multiple-choice-questions.html
Moodle Docs
https://docs.moodle.org/37/en/Multiple_Choice_question_type
Vanderbilt University, Center for Teaching. Writing Good Multiple Choice Test Questions
https://cft.vanderbilt.edu/guides-sub-pages/writing-good-multiple-choice-test-questions/Ce
Short Answer
Open University: Types of assignment: Short answer questions
https://help.open.ac.uk/short-answer-questions
Moodle docs: short-answer question types
https://docs.moodle.org/37/en/Short-Answer_question_type
Essay variants: essays only with more focus
Laboratory notebooks and reports
Objective tests
- short-answer
- multiple choice questions
Creative / artistic performance