The Loughborough CAA Centre website provides very useful advice on writing good questions.
Don’t write more questions … write better questions
Once a class of students has taken a Perception test, you can see how well they performed. What too few tutors do is to check how well the test itself performed:
• Did it test what you thought it was testing?
• Was it overkill on some topics and a bit thin on others?
• Was the time limit realistic?
• Were the question values (how many marks they were worth) a consistent reflection of how difficult they were for the class?
• Did it produce the spread of marks you expected for this class?
If you have 100 or more results from a test, Perception is capable of analysing the effectiveness of the questions, allowing the test to be fine tuned so that it delivers a valid score with the minimum number of questions.
Two useful statistics which can be used to fine-tune Perception assessments are Difficulty and Correlation. Once your questions have been used on a large sample of students (possibly over several years), the values for Difficulty and Correlation can be added to the individual questions as tags, and tests can then be built up by selecting questions with particular tag values.
Difficulty or Facility
This is calculated as the average score for the question divided by the maximum achievable score for the question.
• A difficulty of 0.0 means that the candidates found the question to be very hard (no-one got it right)
• A difficulty of 1.0 means that it is very easy (no-one got it wrong).
• 0.5 is ideal (half the candidates got it right), but
• You may want to deliberately set a few questions with difficulties in the range 0.0 – 0.1 to identify the top 10% (to sort out the 1sts from the 2:1s) and so on.
Correlation or Discrimination
The Discrimination is the statistical correlation of the candidate’s scores for this question and their scores for the whole test, and it ranges from -1.0 to +1.0.
• A high correlation (i.e. close to +1.0) means that the question is measuring the same thing as the test.
• A low correlation means that there is little correlation between test takers getting the question right and getting a good score in the test as a whole.
• A negative correlation indicates that test takers getting this question right generally did badly in the test, i.e. the question is a trap for the unwary.