Definitions of terms within this document.
Assessment components: An assessment listed as part of a module (e.g. an examination or a presentation) where the assessment has been recorded within the module MA1.
Component mark: A mark for the completed assessment component (as defined above) and NOT an individual question within an assessment component.
Departments may have different titles for teaching and learning responsibilities. For the purposes of this document the following titles have been used but acknowledging that actual titles in Departments may differ
Module leader: Defined as the person identified as such on the module MA1 form. The module leader may not necessarily teach on the module, but will recommend the type of moderation to the Departmental Education Committee or its equivalent.
Departmental Education Committee: This is normally the body responsible for the management of education/teaching within the Department or equivalent academic unit.
Director of Education: This is normally the person responsible for the management of education/teaching within the Department or equivalent academic unit. They would normally Chair the Departmental Education Committee (or its equivalent structure, noting it may be known by a different name locally).
Director of Studies, Course Director, Year Director: This is normally the person responsible for the management of education/teaching within a specific Course(s) or a level of a Course(s).
|Why moderate?||What is moderation?|
- To ensure fairness, accuracy and consistency in both marking and the provision of feedback.
- To certify that marks accurately reflect achievement against the learning outcomes set.
- To assure that the quality and integrity of the University’s certification of student achievement is clear and robust for students and staff within the institution, as well as other stakeholders and external organisations (e.g. employers, accrediting bodies).
- To contribute to the continuous critical evaluation and enhancement of assessment practices in order to improve the quality of student learning opportunities.
It is a two-stage process by which the University ensures:
Stage 1 - at the module level:
- the consistency of marking within assessment components;
- the consistency of marking across assessment components within modules;
- the consistency of marking across markers within an assessment component;
- that any differences in academic judgement or procedural irregularity in marking are acknowledged, recorded and addressed;
- that marking within each assessment component is appropriate and that University-wide and Department-specific marking scales and criteria have been used consistently;
- that suitable feedback has been provided at the assessment component and module level (if the process of feedback review is not undertaken elsewhere within the Department’s processes).
Stage 2 - at the course and cohort level:
- that the summative assessment is in line with course learning outcomes;
- the quality of marking and feedback across modules on a course, and where appropriate across courses, cohorts and over time;
- an accepted form of scaling if appropriate is used for any assessment component within a module.
Moderation is required for all components of summative assessment within a module (listed as ‘components’ in the module approval form – including all forms of assessment, e.g. essays, exams, tests, labs, peformances, etc.), except where: (i) an exemption has been granted by the AQSC; or (ii) where the assessment is weighted 3 credits or less (i.e. 20% or less of the total credit for a 15 credit module or pro rata for other module credit weightings). Where exemption (ii) could be invoked, moderation is still required where peer marking, multiple markers or inexperienced markers are used.
Moderation is not required for assessment components that are automated (i.e. the answers are machine or optically read) or formative (i.e. the result does not contribute to the result for the module or course). However, it is good practice to follow procedures which ensure consistency of marking and feedback in all areas of assessment.
Departments are advised to choose the most appropriate form of moderation which provides the most reasonable level of confidence in the marking process constrained within the departmental available resource.
The Departmental Education Committee is responsible for approval of the form of moderation for each component of assessment and for ensuring the appropriate operation of moderation processes. The forms of moderation must be recorded in the minutes.
On an annual basis, the module leader in collaboration with the appropriate Director of Studies / Course Director, recommends the form of moderation for the assessment components within their module to the Departmental Education Committee. The Departmental Education Committee must review and approve all moderation choices and make a record of them in order that the External Examiner(s) be informed of the Departmental practice.
Moderation methods should be communicated to staff and students in an appropriate forum.
The Departmental Education Committee must ensure that all markers, moderators and External Examiners have a clear understanding of their roles and responsibilities.
The Departmental Education Committee will appoint moderators with appropriate expertise to moderation duties taking into account any local Departmental work-load model or other relevant information. This can be through the approval of a moderation policy through which moderators are chosen. Moderators should be members of teaching staff although they do not need to have taught on the module they are moderating.
The module leader is responsible for ensuring moderation takes place as agreed and is evidenced in an appropriate manner as defined by Departmental policy and for ensuring that moderation follows the rules of anonymity for individual students/groups of students if relevant.
When an assessment component is being marked by a team of markers it is good practice for the module leader, prior to marking, to call a meeting of markers to review marking standards and processes in the context of the relevant marking criteria. This could entail a sample of assessments being marked by the module leader and shared with the marking team to establish a benchmark or for each marker to mark a sample of papers and then share the marking and feedback with the module leader and / or other markers for calibration purposes.
Moderation methods that are sufficient to give reasonable assurance of standards as outlined below should be chosen. Moderation arrangements should be aligned with the credit weighting and form of assessment component according to resource.
Double marking is considered a form of moderation.
Process: Each marker marks the work independently, formulating their own judgement, and neither is aware of the other’s assessment decision when determining their own mark. A final mark is then agreed and recorded (which may be an average).
Normally, if the discrepancy is 10 percentage points or more, (e.g. marker 1 = Low 2.2 (52%) and marker 2 = Low 2.1 (62%)), and / or no agreement can be reached, an experienced colleague should be appointed as a third marker/adjudicator. They are not required to mark the work 'blind' although they might choose to do so (i.e. prior to looking at the marks and feedback from the first two markers). Having reviewed all available evidence including the marks and feedback , the third marker/adjudicator determines the final mark to be awarded.
Feedback should be recorded with the final agreed mark with all sets of markers' comments (or an agreed composite of comments) and the brief note about how agreement about the final mark was reached.
- Research project/dissertation/portfolios;
- Unique pieces weighted at 24 credits or more;
- Performance, practical and presentations unless clear reasons for not marking double blind have been outlined and explicitly approved by the Departmental Education Committee. N.B. Where it is not possible for two markers to be present, other examples of best practice include: moderation of speaker notes and/or recordings, moderation of marker notes, moderation of audio-visual recordings.
Double-Informed Marking (sometimes called Double-Seen marking)
Process: The first marker(s) marks and provides feedback, the second marker(s) second marks the full cohort with sight of the first marker’s marks and feedback. The mark is either confirmed or amended after discussion between markers
- Reviewing the work of new or inexperienced markers;
- Where sampled moderation highlights significant issues in the marking of an assessment and it is considered the best course of action by the module convenor;
- Where students are writing individual dissertations using identical data sets and supervised in groups.
Process: A sample is selected for moderation according to the defined requirements in the section ‘What sample should be moderated?’ below.
The first marker or a team of first markers marks and provides feedback. A moderator reviews the sample to confirm that the final module class is consistent with the marking criteria and that feedback is appropriate (a moderator does not generate a mark or feedback for individual students).
If the moderation process identifies concerns about the marking standards of the sample or has identified a systematic error in marking or marks processing, this should be communicated to the module leader (assuming they are not the moderator). The module leader will then review the work, consider the concerns raised, discuss the issue with the marker(s), and respond to the moderator(s) to indicate what action they intend to take, if any (e.g. further moderation; mark adjustment, complete remark of a group of assignments). Where the proposed action may adjust marks this must occur in a systematic and considered way so that all affected work is treated equally and not just the moderated sample. Where the moderator is the module leader, any further support in this process required should be sought from the Departmental Education Committee. The moderation report and module leader’s response should be documented and communicated to the Chair of the Board of Examiners so that they may decide whether to accept the response or require further action.
- All assessments of 3 credits or above unless covered by double blind or double-informed marking above;
- Where a single marker of an assessment component of less than 3 credits requires or requests further review;
- Where a single marker of an assessment component of less than 3 credits is a new or inexperienced marker;
- Where all elements of a module are assessments components of 3 credits or less, and as a result would lead to single marking of all assessment components. In this case at least two assessment components should be moderated using sampling;
Required for any assessment that is team marked (including those under 3 credits).
Performances, Presentations or Practical examinations
Where an assessment requiring moderation involves performance, presentation or a practical it is good practice for it to be observed by two markers and should ideally be recorded (with student permission) to permit moderation and later scrutiny by the External Examiner(s). Where it is not possible for two markers to be present, other examples of best practice include: moderation of student notes, recordings or slides, moderation of marker notes, moderation of audio-visual recordings.
Where marks are attributed to contributions to a group exercise, the material on which this assessment is based must be retained in a durable form in line with the University Retention Policy (e.g. written reports, video recordings, etc.) in order to permit moderation.
The sample must contain a meaningful proportion of the total candidates, which enables the purposes of moderation to be achieved.
The sample must enable the moderator (i) to establish that marking is consistent in line with the marking criteria and learning outcomes; and (ii) to confirm that marking at the boundaries of each classification band is accurate.
Sampling should normally include:
- examples from each represented class boundary;
- examples of all failed candidates;
- examples of the high first class range and above (88% and above);
- any individual candidate(s) the marker(s) finds significantly difficult to mark;
- for team marking, the sample should include examples from each team member
Sampling sizes: a minimum of 10% or 5 examples, whichever is larger should be chosen (up to a maximum sample of 30 assignments is normally recommended).
Notwithstanding the sampling sizes highlighted above it is necessary to moderate ALL failed candidates (even if this pushes the sampling size above 30 candidates). The module convenor will determine the proportion and minimum number of assignments to include in the sample in line with guidance above and must confirm that ALL failed candidates have not met the module learning outcomes.
Moderation is not required for assessment components that are (a) automated, (b) formative, (c) have an AQSC exemption, or (d) for assessment components of 3 credit weightings or less (unless all the assessment components of a module are weighted less than 3 credits).Where exemption (d) could be invoked, moderation is still required where peer marking, multiple markers or inexperienced markers are used.
For automatically marked assessments the module leader must check the marking criteria have been set correctly.
Clerical checking cannot be used in the place of moderation. Clerical checking should be appropriate to the form of assessment, some benefits of clerical checking are:
- to ensure all questions in an examination paper have been answered;
- to confirm that all pages of an assessment have been marked and annotated;
- to ensure that the correct total mark has been recorded;
- to ensure marks have been correctly uploaded.
In designing placement learning assessment or working with a partner institution, departments are required to consider moderation in line with the guidance above. Please see further examples: http://www2.warwick.ac.uk/services/academicoffice/quality/categories/placementlearning/assessment
Evidence of the moderation process and outcomes must be recorded for each assessment and made available to the Examination Board and the External Examiner. Records of moderation should be kept for the same time period as assessment components.
Departments can use the Tabula Moderation workflow to manage marking and moderation and this allows administrators to choose a moderation sample or set up double-blind or double-informed marking.
The workflow also will identify submissions that have been moderated, including whether a moderator has made any adjustments. There is also the ability to identify when a submission was not subject to moderation. Notwithstanding whether the Tabula moderation workflow is utilised or not a written record must be available for scrutiny by exam boards and external examiners which demonstrate a) the type of moderation used b) the size of the sample (and how this was selected and by whom) and c) comments/assurances from the moderator(s) on the standard of marking and feedback. An example of a template can be found in Appendix A and can be modified by Departments to suit their own pedagogy. These records should be made available to the Examination Board and the External Examiner if requested.
Elements of assessments which are marked according to objective criteria, such as online/offline examinations or tests, and which do not allow for the exercise of discretion in marking can be considered for scaling if appropriate. Scaling should not generally be used for subjectively marked assessments (either on-line or off-line) where the judgements intrinsic in the marking process allow for proper calibration via moderation. It should not be used for assessments which are individual in nature, such as dissertations, projects etc.
Scaling is the process by which a set of marks is raised or lowered (normally to an agreed average or by varying grade boundaries), in order to properly calibrate the performance of the cohort in terms of the achievement of learning outcomes and grade descriptors. The proper objective of scaling is to achieve a set of marks for a cohort of students which reflects the extent to which each student has achieved the relevant learning outcomes and has met the criteria set out in the grade descriptors.
Examiners may be prompted to consider scaling by a number of factors including:
- a module cohort having an unusually high or low average mark (e.g. marks with greater than 50 students might be expected to have an average ranging between 55-65%),
- the marks being inconsistent with marks achieved by the same students in other modules,
- the marks being inconsistent with marks achieved in previous years,
- other anomalous mark distributions.
However, such factors should not automatically lead to scaling. Scaling should only be undertaken if academic judgment identifies that raw marks do not appropriately reflect the standards achieved by the cohort.
If an assessment requires scaling this can take place via the scaling tool in Tabula using the University approved algorithm (a piecewise linear transformation with bounds of 0, pass, first and 100). Departments may undertake scaling of student marks outside of Tabula within their own systems, but if different algorithms are used then these must be approved by AQSC. If scaling is approved it must be reported to the External Examiner along with the academic justification for its use and the algorithm used.
Normalisation (also known as standardisation) is a statistical method using an algorithm to adjust the range of marks so that marks fall within a normal distribution curve. Normalisation is not accepted practice. Departments with specific reasons to request normalisation must seek AQSC approval on an annual basis. In the unlikely event that normalisation is permitted, all members of marking teams, external examiners and examination boards must be informed of the method being used. The method used must also be communicated clearly to staff and students.
Where a component mark is determined by application of the 20-point marking scale, any adjustment of marks in the process of moderation should be within the 20-point marking scale where appropriate (see LINK to marking scales https://warwick.ac.uk/services/aro/dar/quality/categories/examinations/marking).
The adjustment of marks for a category of assessments should only be used where marker and moderator are satisfied that the mark outcomes will be appropriate for all candidates. If the issue identified by the moderator affects a specific baseline mark (i.e. 48) then it is necessary to re-consider papers within this category on an individual basis.
Feedback (for non-examination based assessments) should be provided within the University’s 20-working days turnaround time deadline. When feedback is given before an examination board, it should be clearly communicated to students that the marks and feedback they are receiving are provisional marks. Some departments address this issue by giving provisional feedback before the exam board, but not marks. As a general principle, definitive feedback is only possible after the moderation and examination board have taken place, but the timing of an examination board should not delay students receiving provisional feedback.