In these and other papers there are examples of different evaluation methods – see also LDC’s evaluation website.
• questionnaire survey
• interviews (including stimulated recall)
• logs of messages or web 'hits'
• pre test post test
• focus groups
• online discussion
• user diaries
However evaluation does not always make a good link between methods and pedagogy. This means that we do not always get a good idea of the learning objectives or how learning took place e.g. Hiltz et al (2002). In contrast a good example of linking evaluation to an explicit model of pedagogy is Fowell and Levy (1995).
If pedagogy is important then it might be possible to highlight different dimensions to online activity. These are outlined in table 2 below.
|Dimension||Characteristics||Some issues for evaluation||Examples|
Structure(low to high)
|Timetables, ground rules, tutor assessment, tutor direction, requirement to log on, assigned roles lead to high structure||
Coherence of curriculum design?
Match between design and learners preferred styles?
Gap between intention and practice
|Clarke (2002) an example of very low structure, Aviv, R. Erlich, Z., Ravid, G., Geva, A. (2003) provide two examples in a problematic paper|
and / or content
Content heavy - extensive predetermined course material,
Communication heavy - high exchange of ideas and information
Communication reich leads to focus on messages, message analysis and relationship of learners to messages;
Content heavy leads to study of imformation processing
|McConnell, D. (2000) a good example of high communication low content|
|Independance / collaborative learning||Collaborative learning often involves teams discussing cases or scenarios and constructing a group response, cooperative collaborative contested terms here.||Collaborative leads to focus on group processes||
Lockhorst, D., Admiraal, W., Pilot, A. and Veen, W. (2002) discuss case based approaches, Tsui, and Ki, (2002) discuss an open forum with independent contributionsDempster and Blackmore (2002) around student web publishing and critical analysis (see TELRI model below)
|Online / off line||On line learning suggests that most learning literally takes place at the machine||much neglected - how do we capture what happens away from the machine||
Henri and Pudelko (2003) discuss different types of online community
Chen and Hung raise questions over personal and group knowledge
Table 2 dimensions to online activity
The model devised by the TELRI project (Roach, Blackmore & Dempster, 2000) arose from consultation in departments across five research-led institutions. It is based around distinguishing between two types of learning: knowledge and practice-based learning (adoptive) and creative and higher order learning (adaptive).
ADOPTIVE LEARNING outcomes
|ADAPTIVE LEARNING outcomes|
|Knowledge and Practice of....||Formation and Generation of....|
|Facts, Assertions, Rules and Laws||Personal Interpretation and Meaning|
|Terminology, Language and Protocols||Evaluation and Decisions|
|Techniques and Procedures||Arguments, Reasoning and Justification|
|Organisation and Structure||Synthesis and Conceptualisation|
|Established Principles and Relationships||Originality, Creativity and Innovation|
By nature, assignments to develop higher order thinking tend to be more open, challenging and motivating than those that require the demonstration of established knowledge and procedures, characterised by adoptive learning outcomes. For courses that are particularly keen to support the higher level, adaptive type learning, tutors should aim to provide students with opportunities to obtain feedback on their ideas and their approaches through debate and discussion and sharing of work, as well as providing an environment for reflection. Extending this to peer assessment, students can learn valuable lessons when given access to the work of their peers by engaging in the evaluation of what is and what is not good quality work in their subject area, using the same open-ended approaches the tutor might use. In this way, students themselves can contribute to the course, bringing in resources that support the formation and expression of their ideas and justifying these.
The model supports the lecturer in locating the different learning resources, learning support and assessment methods used against the adoptive or adaptive learning criteria. It has the advantage of supporting the integration, or blending, of both online and classroom activities and resources. In so doing, the TELRI model offers a broad framework for course design and profiling that can be tailored to different disciplines.
Course review and e-learning mapping tools based on TELRI are provided in Annex A and available on the web at: www.telri.ac.uk/guidelines.pdf.
Laurillard (1993, 2002) offers examples of how learning and teaching methods, media/tools contribute to a didactic or dialogic relationship between the lecturer and the students. For example, she illustrates how the different e-learning tools relate to the different media forms representing these ‘conversations’
|Narrative||Print, TV, videocassette|
|Interactive||CD, DVD or web-based resources|
|Communicative||Web conferencing, asynchronous or synchronous|
|Productive||Tools for students to create models or descriptions|
Using this model produces a course design profile by mapping the tools used to resulting learning outcomes based on the different kinds and directions of dialogue supported.
An online toolkit based on this model is Media Adviser (Conole & Oliver, 2002): http://www.londonmet.ac.uk/ltri/demos/media_adviser_files/media_adviser.htm
Once the pedagogical aspects of the course objectives are clarified, the process of setting up the learning activity can be based on informed decision-making. The simple model represented below (Dempster, 2004) can be used to map learning activities against a hierarchy of increasing dialogic richness. Tools and methods can be selected that support increasing integration of communication and collaboration across the learning activities. The resulting course profile thus indicates the extent to which learning and teaching activities are focused on: presentation or practice of knowledge or techniques (e.g. content rich: teacher-student dialogue: adoptive) or on communication and collaboration (e.g. content light: multiple dialogues: group and task based dialogues: adaptive).