Dr Trevor Hawkes, Mathematics Institute, University of Warwick
Special Requirements, Special Rewards
Some features of computer-aided assessment peculiar to mathematical subjects
Assessing mathematics online presents challenges, both to software developers and to authors (teachers writing questions and creating assessments). Happily, there is a flip side. As if to compensate for these hurdles, there is scope for real pedagogical gain when it comes to writing questions with mathematical content. This article presents work from an Education Innovation Fund project, CAA in the Sciences. Some examples of question formats are offered that allow tutors to stretch the testing capability of computer aided assessment software into deeper cognitive areas and to avoid the common traps of minor errors in answers and provision of meaningful feedback. Case studies in science departments at Warwick are outlined to illustrate the achievements to date, including the use of a team of IT-savvy postgraduates to contribute to question authoring and support to students.
Assessing mathematics online presents challenges, both to software developers and to authors (teachers writing questions and creating assessments). For instance:
Web browsers have a hard time displaying mathematical expressions
Students have a hard time entering mathematics into answer boxes
Computers have a hard time understanding what students have entered
Answers can have many equivalent forms (e.g. the expressions –(–1), 65/65, 0.99 recurring, cos2x + sin2x, –eiπ are all equal to 1)
Happily, there is a flip side. As if to compensate for these hurdles, there is scope for real pedagogical gain when it comes to writing questions with mathematical content.
One bonus is the ability to create clever question templates containing parameters and place-holders that are allowed to vary within given sensible ranges. When a question is generated from a template, the assessment software randomly inserts specific values for the parameters, and if the number and ranges of the parameters is sufficiently large, the computer can deliver literally millions of different questions, all pedagogically equivalent, from a single template. If every question in a test contains such parameters, the chances of two students getting the same test is vanishingly small. Such a question template (sometimes called a multi-question, but this is easily confused with a multi-part question) is an example re-usable assessment object (see McCabe ), and can be exploited
to give students virtually unlimited practice at problem solving without repetition
to minimize the risk of cheating (every student gets a different test)
to provide personalized feedback based on the individual parameters
even to change the subject context, by modifying units, scales and terminology accordingly
Find the two roots of the equation x2 – 4x– 21 = 0.
It has been generated from the template equation (x – a)(x – b) = 0 with the parameter values a = 7 and b = –3, which are therefore the roots of the given equation. If the software is instructed to choose the values of the whole-number parameters a and b randomly in the range from –9 to 9, but excluding zero, then it can create 324 different quadratic-equation problems, all testing the same set of skills.
Another plus is tolerance of arithmetical slips and minor mistakes. Very often a mathematical question requires a number of steps and calculations to reach the final answer. If the submitted answer is wrong, the examiner often can’t know the reason for the mistake: was it due to a simple calculation error? or to a serious conceptual failure? To reveal a student’s thought processes, a question can be broken down a sequence of smaller questions representing steps in the solution. In such multi-part questions, the software can then be instructed to store a student’s wrong numerical answers to earlier parts and to include them as correct in the calculations of subsequent parts, which can then be marked right or wrong independently of the earlier mistakes. This ensures that students are only penalised once for a slip; it also reveals more to the examiner about a student’s knowledge and understanding of the subject matter in the question.
Feedback, Feedback, Feedback. Here’s another area where mathematics offers special rewards to the canny assessor. The routine marking of 100s of scripts can be a laborious and soul-destroying activity; giving each student personalised and intelligent feedback is even more demanding and time-consuming. This is exactly what well-programmed computers, having no boredom threshold, can do quickly and accurately for us. Here are some special things you can do with feedback for mathematics questions:
You can reveal different levels of feedback: (i) whether the answer is right or wrong (ii) the correct answer; (iii) a full worked solution, showing all steps of an argument that leads to the answer
You can provide graduated hints for further attempts to help a student who gets the initial answer wrong
When using the question templates described above, you can arrange to base the feedback on the values of the parameters used in generating the question
- Having designed a question that reveals students’ calculations and thought processes on their way to the final answer, you can guess at their mistakes and misconceptions (called ‘malrules’ by Martin Greenhow, the brain-father of Mathletics) and then draw attention to them in the feedback (see Greenhow et al.  and )
- Even more searchingly, you can respond to a wrong answer with lower-level questions designed to ferret out exactly where a student is going wrong, much as you would during a face-to-face tutorial
Some preliminary, and largely unsurprising, findings, based on my experience leading this project:
Authoring questions in
CAA software with serious mathematical functionality is complex and requires considerable technical and pedagogical competence to learn
- Busy academics are usually don’t have time to master it and to stay proficient; the department management is reluctant to offer inducements, such as a reduced load.
- The currently available software meets the challenges listed at the outset with varying success; none scores 10/10 on all fronts
- Commercial and open-source approaches both have something valuable to offer; the maturity of the software and the investment put into it counts for a lot
- Exploiting the pedagogical pluses of RAOs cited in this essay is a time-consuming exercise, but correspondingly rewarding
- Students seem quite tolerant of the shortcomings of assessment software.
I aim to justify these observations in my final report.
One way to solve the steep-learning-curve/busy-academic problem? I have hired a small team of IT-savvy science postgraduates, who have collectively taught themselves to use three of the four
To find out more about the project and its findings, visit its website: http://go.warwick.ac.uk/caa4s
or peruse my musings on
Martin Greenhow, Daniel Nichols and Mundeep Gill: “Pedagogic issues in setting online questions” MSOR Connections Volume 3, Number 4: Nov 2003.
Michael McCabe: “Reusable Assessment Objects in Mathematics Teaching”, accepted for publication in the Proceedings of the 8th International Conference on Technology in Maths Teaching (ICTMT-8) Hradec Kralove, Czech Republic, July 2007 (meanwhile available on request from Michael.McCabe@port.ac.uk).
Daniel Nichols and Martin Greenhow: “Using Question Mark Perception v3 for testing mathematics” MSOR Connections Volume 2, Number 3: Aug 2002.
Citation for this Article
Hawkes, T. (2007) Approaches to Computer Aided Assessment in the Science Faculty. Warwick Interactions Journal 29. Accessed online at: http://www2.warwick.ac.uk/services/cap/resources/pubs/interactions/current/abhawkes/hawkes