From authoring to reporting - the eAssessment lifecycle

Understanding the (e)Assessment lifecycle

The (e)Assessment lifecycle outlines the essential steps involved in designing, delivering, and reviewing effective assessments online, ensuring they align with learning outcomes and provide meaningful feedback.

While this framework is not exclusive to Questionmark Perception (QMP) and can be adapted to other tools, we have included specific QMP functionalities where applicable to guide users in using its features.

By following this structured approach—from planning and creating question banks to delivering assessments and analyzing results—educators can enhance the reliability, validity, and impact of their assessments, fostering better learning outcomes and an improved student experience.

Step 1 - Plan your assessment

Define the purpose of the assessment

- Determine whether it will be a diagnostic test, formative weekly assessment, or summative end-of-term exam.
- Align the purpose with learning outcomes and course objectives.

Select appropriate question types
- Consider multiple-choice, essay, true/false, matching, or hotspot questions based on learning outcomes.
- Ensure a balance of question types to assess different cognitive levels (e.g., recall, application, critical thinking).

Design feedback and links to learning
- Decide the type of feedback: immediate vs. delayed, generic vs. personalized.
- Link feedback to additional resources for further study, such as reading materials, videos, or related modules.

Determine time limits and conditions
- Set appropriate time constraints based on the complexity and length of the assessment.
- Plan for accessibility accommodations and any conditional assessment pathways (e.g., unlocking additional sections based on performance)

Step 2 - Create a question bank

Organize topics and subtopics
- Structure your question bank by creating topics and subtopics.
- Align topics with the syllabus to ensure comprehensive coverage of learning outcomes.

Diversify question types
- Include a mix of question types (e.g., multiple-choice, multiple response, drag-and-drop, short answer, and essay) to test a range of skills and knowledge.
- Avoid over-reliance on a particular question type like multiple-choice to ensure a well-rounded assessment.

Enhance integrity through randomisation
- Randomise the order of answer options for multiple-choice and multiple-response questions to reduce the potential for pattern guessing, and over-shoulder copying.
- Where applicable, create question variants to randomise questions presented to different students.

Step 3 - Design the assessment

Select questions strategically
- Choose questions from various topics to ensure balanced coverage of the learning objectives.
- Align question selection with the intended difficulty level and cognitive skills being assessed.
Set time limits appropriately
- Determine the overall time allocation based on the number and complexity of questions.
- Include buffer time for students to review their answers, if applicable.
Incorporate feedback for formative assessments
- Provide targeted feedback to help students understand their strengths and areas for improvement.
- Use automated feedback options to ensure consistency and reduce manual grading workload.
Explore conditional assessment options
- Design pathways where students can unlock additional questions or resources based on their performance.
- Implement branching scenarios for adaptive assessments tailored to student proficiency.

Step 4 - Verification/peer review

Conduct a test run of the assessment
- Create a test version of the assessment, including all questions, marking schemes, and feedback.
- Schedule this test assessment to yourself and colleagues for review.
- Attempt all questions to check:
  - Accuracy of the marking schema.
  - Correctness and clarity of the feedback provided.
Simulate the student experience
- Schedule the actual assessment for yourself and colleagues, mirroring the conditions students will encounter.
- Evaluate the user interface for clarity, layout, and design.
Verify assessment functionality
- Check the following critical elements:
  - Is the layout and design intuitive and accessible?
  - Are accessibility features (e.g., screen reader compatibility) enabled and functioning?
  - Is a special character palette available if required (e.g., for math, science, or foreign language questions)?
  - Are the scoring and feedback accurate?
  - Have all possible alternative correct answers been accounted for?
  - Is the time setting appropriate?
Approval for student release
- Once verified, mark the assessment as "Approved" in the system to make it ready for students.

Step 5 - Scheduling

Set assessment timing
- Define the start and end time for the assessment.
- Confirm time zones if students are in different locations.
Enable secure exam delivery (if required)
- Determine whether a secure browser solution is needed to prevent academic misconduct.
- Configure the assessment settings to ensure secure delivery (e.g., lock screen, disable copy-paste).
Manage participant lists
- Confirm the list of students taking the assessment.
- Add students to the assessment via upload, for example.
- Identify and accommodate students requiring extra time or other adjustments (e.g., accessibility needs).
Set time limits and attempts
- Specify the total time limit for the assessment.
- Define the number of permitted attempts (e.g., single attempt or multiple retries for practice).
Plan physical or equipment requirements (if applicable)
- Book a room if the assessment requires in-person delivery.
- Ensure required equipment such as headphones, laptops, or calculators are available and functional.

Step 6 - Delivery

Communicate assessment details to students
- Ensure students receive an automated email confirmation with key details, including:
  - Assessment start and end times.
  - Access link or platform login instructions.
  - Equipment or software requirements, if applicable.
Provide access instructions and support
- Include step-by-step instructions for accessing the assessment in the email or learning platform.
- Share contact information for technical support in case students encounter issues during the assessment.
Enable timed release and completion tracking
- Ensure the assessment becomes available to students only at the designated start time.
- Track student submissions to confirm successful completion within the time limit.

Step 7 - Reports

Access reporting tools
- Users with administrative rights in Questionmark Perception can generate reports as soon as all students have completed the assessment.

Leverage reporting functionalities
- Utilise the following tools to gain insights:
  - Demographics filter: Analyze performance across specific student demographics to identify trends.
  - Question item performance analysis: Review how each question performed, identifying high or low discrimination items.
  - User grouping: Compare results among different student groups (e.g., by class section or demographic cohort).
  - Results breakdown: Generate detailed breakdowns by assessment, topic, or specific item to evaluate learning objectives.

Export results for further analysis
- Download results in multiple formats, including:
  - HTML: For immediate viewing in a web browser.
  - PDF: For sharing static reports.
  - CSV: For further data manipulation in spreadsheets or statistical software.

Share insights with stakeholders
- Provide tailored reports to academic staff, highlighting key findings and actionable recommendations.
- Use data to inform future assessment design and instructional strategies.

Step 8 - Review the assessment

Use available review tools
- Leverage Questionmark Perception’s assessment review and question performance analysis tools to evaluate the effectiveness of the assessment.

Analyse question performance
- Review items with particularly high or low performance indices:
  - Were the questions too hard or too easy?
  - Could there be alternative correct answers not accounted for?
  - Was there potential for misunderstanding or ambiguity in question wording?

Address student feedback
- Consider any complaints or concerns raised by students:
  - Is the feedback justified?
  - If not, clarify expectations in future assessments or lectures.
  - If changes are warranted, adjust the question or assessment design as needed.

Align questions with course content
- Review questions against the content of lectures and materials:
  - If you’ve changed the lecture content, ensure questions reflect the updated material.
  - Update references in feedback if the course literature has been revised.

Think creatively
- Explore and implement new functionalities in Questionmark Perception that could enhance the assessment (e.g., adaptive question types or improved reporting tools).
- Explore different tools - can the same or similar assessment be delivered using another tool (WAS, Moodle Quiz, Moodle Lesson, Moodle Assignment) to meet the learning outcomes, where this may bring other benefits.

Plan improvements for future assessments
- Document insights and changes made during the review to inform the design of future assessments.
- Share findings with colleagues to enhance collective assessment practices.

Online Exams

Some tips for applying the lifecycle to delivering an online exam.

Planning an online exam

Define objectives clearly: ensure the purpose of the exam is specific—diagnostic, formative, or summative—and aligns with course learning outcomes.

Plan for the online format: consider how the online medium might affect question design. For example, avoid overly complex diagrams that may not display well on small screens.

Anticipate technical needs: identify technical requirements early, such as secure exam delivery tools, stable internet access, and device compatibility.

Creating the question bank

Categorise questions by difficulty: tag questions as basic, intermediate, or advanced to ensure a balanced exam.

Prepare for randomisation: write multiple variations of key questions to enable randomisation while maintaining consistency in difficulty.

Designing the assessment

Incorporate real-world scenarios: application-based questions incorporating contextual prompts; require justification; build in decision-making.

Optimise time allocation: allocate time per question based on its complexity, using a rule of thumb (e.g. 1 - 5 minutes for MCQs, 10 - 30 minutes for 'essays' etc).

Verification and peer review

Run usability checks: ensure all multimedia (e.g., images, videos) load quickly and correctly on various devices.

Test navigation: simulate the student experience to check if the assessment flows logically and instructions are clear.

Scheduling exams

Stagger start times: if students are in different time zones, or you are concerned about system stability, provide multiple start times to ensure fairness.

Set a buffer window: allow a grace period at the end of the exam to account for minor technical disruptions.

Delivering exams

Offer a practice exam: provide a short, timed mock assessment to help students understand the platform and question formats.

Communicate expectations clearly: include rules about permitted materials, behavior, and how to handle technical issues during the exam.

Analyzing reports

Identify question-level insights: use item analysis reports to pinpoint questions with low discrimination or ambiguous wording.

Evaluate timing: review time-on-question data to determine if the time limit was appropriate.

Reviewing and improving

Capture student feedback: ask students for feedback on the exam experience, focusing on clarity, fairness, and technical functionality.

Document lessons learned: record insights and adjustments made to inform future online exams.