Evaluating e-learning is often concerned with whether learning has been enabled, enriched or enhanced through the use of technology. Recently, there has been a focus on the extent to which e-learning is inclusive and accessible (TechDis, 2002). Evaluation should aim to take account of the ways in which students are able to interact with technology-based teaching and learning and are supported in doing so. This has as much to do with management and implementation (processes), as with the design and delivery of tools or materials (outputs).
In designing an e-learning evaluation, it is helpful to clarify an initial “object” of reference. However, take a word of warning. The introduction of an e-learning mode of teaching and learning as a direct substitute for an existing non-e-approach is rare. Evaluating e-learning is therefore problematic for two main reasons.
Firstly, there are issues concerning “e-pedagogy”. E-learning facilitates new forms of resources, communication and collaboration and new patterns of study and group behaviours. These do not necessarily have comparable, existing counterparts. Evaluation is unlikely, therefore, to be a simple case of comparing the “e” group with a previous “control” group, though many attempt to do so. More often, e-learning is an adjunct – integrated or blended to support, extend as well as replace existing approaches. If e-learning offers new modes of learning, particularly in the ways in which students interact online and offline with each other and with learning resources, it changes considerably how they perceive and experience the learning environment as a whole. E-learning may demand new attitudes and skills. For both curriculum design and evaluation, traditional concepts and frameworks of learning and teaching may need remodelling.
Secondly, it is difficult to separate a specific e-learning intervention from the complex interplay of cultural and social influences on the learning experience. A holistic approach to evaluation is thus required, which takes into account these different domains. For this reason, there is increasing use of mixed techniques, including ethnographic methods typically used in the social science research (see Anderson, 1990). While there is no one model, it is sensible to focus not only on the pedagogical aims and learning and teaching processes, but also on particular aspects of the learning environment. This includes the technology itself, but importantly, the support surrounding the use of the technology. The effectiveness of e-learning is likely to be highly context-specific.
An effective evaluation might incorporate one or more stages of the design and development process, loosely defined above in terms of “types” of evaluation. Here, we add the focus of the evaluation (the objects of reference):
- diagnostic: identifying and analysing preconceptions and assumptions prior to use;
- formative: appraising implementation processes and usability at the point of use;
- summative: outcomes of the innovation after use, perhaps in terms of changes in learning and teaching behaviours and achievements.