Dr Trevor Hawkes, Mathematics Institute, University of Warwick

Special Requirements, Special Rewards
Some features of computer-aided assessment peculiar to mathematical subjects

Assessing mathematics online presents challenges, both to software developers and to authors (teachers writing questions and creating assessments). Happily, there is a flip side. As if to compensate for these hurdles, there is scope for real pedagogical gain when it comes to writing questions with mathematical content. This article presents work from an Education Innovation Fund project, CAA in the Sciences. Some examples of question formats are offered that allow tutors to stretch the testing capability of computer aided assessment software into deeper cognitive areas and to avoid the common traps of minor errors in answers and provision of meaningful feedback. Case studies in science departments at Warwick are outlined to illustrate the achievements to date, including the use of a team of IT-savvy postgraduates to contribute to question authoring and support to students.

Assessing mathematics online presents challenges, both to software developers and to authors (teachers writing questions and creating assessments). For instance:

• Web browsers have a hard time displaying mathematical expressions 
• Students have a hard time entering mathematics into answer boxes 
• Computers have a hard time understanding what students have entered 
• Answers can have many equivalent forms (e.g. the expressions –(–1), 65/65, 0.99 recurring, cos²x + sin²x, –e^iπ are all equal to 1)

Happily, there is a flip side. As if to compensate for these hurdles, there is scope for real pedagogical gain when it comes to writing questions with mathematical content.

One bonus is the ability to create clever question templates containing parameters and place-holders that are allowed to vary within given sensible ranges. When a question is generated from a template, the assessment software randomly inserts specific values for the parameters, and if the number and ranges of the parameters is sufficiently large, the computer can deliver literally millions of different questions, all pedagogically equivalent, from a single template. If every question in a test contains such parameters, the chances of two students getting the same test is vanishingly small. Such a question template (sometimes called a multi-question, but this is easily confused with a multi-part question) is an example re-usable assessment object (see McCabe [2]), and can be exploited

• to give students virtually unlimited practice at problem solving without repetition
• to minimize  the risk of cheating (every student gets a different test)
• to provide personalized feedback based on the individual parameters
• even to change the subject context, by modifying units, scales and terminology accordingly

As simple example of how a question template works, consider this exercise:

            Find the two roots of the equation x² – 4x– 21 = 0.

It has been generated from the template equation (x – a)(x – b) = 0 with the parameter values a = 7 and b = –3, which are therefore the roots of the given equation. If the software is instructed to choose the values of the whole-number parameters a and b randomly in the range from –9 to 9, but excluding zero, then it can create 324 different quadratic-equation problems, all testing the same set of skills.

Another plus is tolerance of arithmetical slips and minor mistakes. Very often a mathematical question requires a number of steps and calculations to reach the final answer. If the submitted answer is wrong, the examiner often can’t know the reason for the mistake: was it due to a simple calculation error? or to a serious conceptual failure? To reveal a student’s thought processes, a question can be broken down a sequence of smaller questions representing steps in the solution. In such multi-part questions, the software can then be instructed to store a student’s wrong numerical answers to earlier parts and to include them as correct in the calculations of subsequent parts, which can then be marked right or wrong independently of the earlier mistakes. This ensures that students are only penalised once for a slip; it also reveals more to the examiner about a student’s knowledge and understanding of the subject matter in the question.

Feedback, Feedback, Feedback. Here’s another area where mathematics offers special rewards to the canny assessor. The routine marking of 100s of scripts can be a laborious and soul-destroying activity; giving each student personalised and intelligent feedback is even more demanding and time-consuming. This is exactly what well-programmed computers, having no boredom threshold, can do quickly and accurately for us. Here are some special things you can do with feedback for mathematics questions:

• You can reveal different levels of feedback: (i) whether the answer is right or wrong (ii) the correct answer; (iii) a full worked solution, showing all steps of an argument that leads to the answer
• You can provide graduated hints for further attempts to help a student who gets the initial answer wrong
• When using the question templates described above, you can arrange to base the feedback on the values of the parameters used in generating the question
• Having designed a question that reveals students’ calculations and thought processes on their way to the final answer, you can guess at their mistakes and misconceptions (called ‘malrules’ by Martin Greenhow, the brain-father of Mathletics) and then draw attention to them in the feedback (see Greenhow et al. [1] and [3])
• Even more searchingly, you can respond to a wrong answer with lower-level questions designed to ferret out exactly where a student is going wrong, much as you would during a face-to-face tutorial

For a fuller discussion of feedback, see the entries on my CAA blog reached by adding the following phrases on the end of the URL http://blogs.warwick.ac.uk/caa4s/entry/  

• a_dish_best/, 
• deef_suggest_improvements/

The background to these observations is a project I have been leading called Computer-Aided Assessment for Science ( CAA4S).  It is paid for by Warwick’s Teaching Quality Enhancement Fund and aims to promote the uptake of CAA in the 8 departments in the Warwick Science Faculty. In Phase 1, which began in November 2005 and is now complete, I sounded out department CAA needs and identified software that was most suited to these needs while being adept at handling mathematics and symbolic expressions. In the current Phase 2, we are running five pilots to try out the software on real students in live learning situations.  

Some preliminary, and largely unsurprising, findings, based on my experience leading this project:

• Authoring questions in CAA software with serious mathematical functionality is complex and requires considerable technical and pedagogical competence to learn
• Busy academics are usually don’t have time to master it and to stay proficient; the department management is reluctant to offer inducements, such as a reduced load.
• The currently available software meets the challenges listed at the outset with varying success; none scores 10/10 on all fronts
• Commercial and open-source approaches both have something valuable to offer; the maturity of the software and the investment put into it counts for a lot
• Exploiting the pedagogical pluses of RAOs cited in this essay is a time-consuming exercise, but correspondingly rewarding
• Students seem quite tolerant of the shortcomings of assessment software.

I aim to justify these observations in my final report.

One way to solve the steep-learning-curve/busy-academic problem? I have hired a small team of IT-savvy science postgraduates, who have collectively taught themselves to use three of the four CAA packages that I am evaluating. Their main role has been to create questions and assessments at the behest of module organisers in science departments. They work on a part-time hourly basis and are paid out of a small surplus in the Project budget supplemented by some generous departmental contributions. As they near the end of their research degrees, they will hand down their skills to new recruits appointed to shadow them. Thus the team’s expertise will be preserved and project momentum maintained. As CAA practice gets firmly embedded in departmental practice, one hopes that departments will increasingly appreciate its worth and continue to fund its development.

To find out more about the project and its findings, visit its website: http://go.warwick.ac.uk/caa4s

or peruse my musings on CAA on the Project blog: http://blogs.warwick.ac.uk/caa4s

References

1. Martin Greenhow, Daniel Nichols and Mundeep Gill: “Pedagogic issues in setting online questions”  MSOR Connections Volume 3, Number 4: Nov 2003.
2. Michael McCabe: “Reusable Assessment Objects in Mathematics Teaching”, accepted for  publication in the Proceedings of the 8th International Conference on Technology in Maths Teaching (ICTMT-8) Hradec Kralove, Czech Republic, July 2007 (meanwhile available on request from Michael.McCabe@port.ac.uk).
3. Daniel Nichols and Martin Greenhow: “Using Question Mark Perception v3 for testing mathematics” MSOR Connections Volume 2, Number 3: Aug 2002.

 

Citation for this Article
Hawkes, T. (2007) Approaches to Computer Aided Assessment in the Science Faculty. Warwick Interactions Journal 29. Accessed online at: http://www2.warwick.ac.uk/services/cap/resources/pubs/interactions/current/abhawkes/hawkes
(Accessed )

Print Article

Cite Article