Skip to main content

Evaluation Strategy

Version: TLTP3 /92 /03 /30/05/99
Evaluation scope

  • Research capabilities: curriculum development & technological approaches;
  • User conditions & requirements: stakeholders, IT infrastructure, staff development;
  • Cost-effectiveness: educational values & impact/resource implications;
  • Feasibility & sustainability of implementation;
  • Degree of transferability/applicability across disciplines and institutions;
  • Project management and team approaches;
  • Effective means of stakeholder engagement & dissemination;
  • Continuation.

  • Establish educational model for research capability development in undergraduate curricula;
  • Investigate appropriate technology-based approaches to suit research-orientated methods of learning
  • Map applicability, objectives, attitudes & obstacles across a range of disciplines & institutions;
  • Identify requirements for academic staff development & student training to meet TELRI approaches;
  • Implement TELRI methodologies in educationally and cost effective manner;
  • Encourage collaborative developments between individuals, departments and institutions;
  • Consider effects of new methods on student learning and motivation & staff development;
  • Consider how institutional context can influence and support the approaches;
  • Establish effective means of stakeholder engagement & dissemination;
  • Develop appropriate staff development and dissemination materials.
Stakeholder analysis

Primary S1 Secondary S2 Teriary S3
A Academic Staff A Heads of Departments A Employers
B Staff/educational developers B Senior managers B Careers Advisory
C IT/library support staff C Other HEFCE projects  
D Students D TLTP  

Project evaluation plan

  Issues Stakeholders 1a 1b 2 3
What are the characteristics of a research-led institution and a research-orientated curriculum?

Project profile
Stakeholder engagement in change management

S1, S2, S3  Y  Y  Y 0
To what extent can research capabilities be defined and to what extent does this vary between the disciplines? Developing learning outcomes and assessment criteria

What educational models are effective in implementation of research-orientated approaches to learning & teaching? Developing educational models
Mapping technological approaches
What are appropriate ways to evaluate whether research capabilities have been enhanced in the implementations? Measurable educational impact

What are the current conditions of teaching and learning in the participating departments? Current status

S1, S2

Y Y Y 0
What is the general level of understanding by academic staff & students of research capabilities, learning outcomes, and assessment criteria? Staff development & motivation

Student motivation

S1: A, B, D

Y Y Y 0
How does the development of students' research capability differ (if at all) between institutions and between the disciplines? Discipline specific curriculum approaches

Subject benchmarking

S1: A, B, D Y Y Y 0
What departmental or individual prerequisites assist effective course design discussions and implementation work? Implementation strategies, staff attitudes and competencies S1: A, B, C


Y Y Y 0
What are the current levels of learning technology usage in the five institutions?   All Y Y Y 0
What is the need for basic and high level IT competence in staff and students, how does this vary between institutions and disciplines, and how is this acquired in each institution/department? Staff and student IT training needs S1: A, B, C, D


S3: A

Y Y Y  
What technological approaches might enhance such research capabilities and how do these reflect any differences in discipline approaches? Technological approaches

All Y Y 0  
What courseware materials might be of value in each subject discipline and what are their characteristics? Identify applicability of courseware


S2: C, D

Y Y 0  
What are the criteria for evaluating TLTP and other existing materials and technological tools?   S1:

S2: C, D

Y Y 0  
What institutional provision, support and organisation structures assist effective implementation? infrastructure & support issues All Y Y Y C
How comparable are the 5 sites in terms of IT infrastructure, support and teaching & learning strategies?   S1: C
S2: C,D
Y Y Y  
What is the applicability of the project outcomes and phase 2 institutions, i.e. what criteria/prerequisites will enable HEIs to map onto the TELRI framework Applicability across the sector

Effective dissemination

All Y Y Y 0
What project approaches have worked most effectively and to what extent does this vary between pahse 1 and 2 institutions and departments Project team approaches, communication, dissemination Projec team
S2: C,D
In what form are the project outcomes most effectively disseminated Dissemination engagement          


1a Initial depts at Warwick & Oxford
1b Second round depts at Warwick & Oxford
2 Depts at Birmingham, Durham & Southampton
3 Consultancy & workshops, feedback forms

Evaluation methods

Specific evaluation of the implementation of new methods in the courses selected will be conducted using a variety of standard evaluation methods. The approaches to implementation, and evaluation thereof, will be strongly informed by continuous communication within the project management structure.

Project approaches
Those associated with the project will be evaluating their activities continually, both formally and informally, integrating their findings into practice to ensure continual improvement. To this end, each project team meeting will include evaluation of activities, reported on formally where appropriate. Quarterly reports presented by the project manager to the steering group, along with the bimonthly reports and the annual reports submitted to TLTP (and Tavistock/OU IET), will also have evaluative components. Project team members will, in their interactions with departments and individuals, be sensitive to feedback, both formal and otherwise, concerning the appropriateness and effectiveness of their work. Each intervention will be evaluated, in a process that will be devised by the project team, and that will take full account of the views of staff and students involved.

Evaluation will be made possible through information collected from:

Research Phase

Evaluation Area Evaluation method(s)
Current stages of departmental IT-based teaching innovation; preparation for QAA Subject Review; comparisons between departments at Warwick and Oxford; Semi-structured interviews
Research methodologies and capability components in curricula; Literature reviews
Discussion of TELRI model
Overall objectives of departments in TELRI project; current teaching methods; discipline comparison (subject benchmarking analysis); attitudes and priorities for areas to develop of academic staff in departments; student motivation, achievement and problem areas; involvement in other collaborative projects; number of staff to be involved; Semi-structured interviews

IT infrastructure capacity; current limitations; planned developments; support structures; Questionnaires
Existing surveys
Preliminary cost-benefits of current methods; existing evaluation data/studies; Researching appropriate cost-effectiveness criteria
Staff development needs; staff & student IT training implications; motivation and interest, areas of difficulty, understanding of concepts, integration into whole course programme (coherency and relevant importance), assessment and feedback. Semi-structured interviews & focus groups

Establishment phase

Focus groups, in-depth interviews & observation

  • Individual staff from similar disciplines at Warwick and Oxford; common issues identified and discussed, conclusions drawn; mapping educational model for research capability development; feasibility of technological approaches.
  • Student groups to look at motivation, understanding of course objectives and assessment criteria;
  • Ascertain interests & questions that staff development units & senior management may wish the project to answer.

Project team meetings and progress reporting
Reporting from project team officers as research & implementation work proceeds; summaries of important issues; outcomes and conclusions documented; effectiveness of communication and reporting mechanisms of the team; issues to be discussed by steering group.

Steering group meetings
Overall objectives and outcomes of TELRI project; monitoring progress of the project plan; institutional issues arising; national context; engaging stakeholders.

Cost-benefit analyses
Development effort/time, gains in teaching efficiency, educational effectiveness of new approaches, research profile improvement, developing new areas of the curriculum.

Dissemination strategy feedback
Feedback from project dissemination activities, workshop discussions, consultation with discipline-related bodies, web site feedback forms, email questionnaires, as appropriate.

Consultation with other partners
Consultation on proposed project plan, evaluation and dissemination strategies and criteria with representatives of consortium partners (Oxford, Durham, Southampton, and Birmingham) in addition to others - informally - including non-research-led HEIs.

Dissemination phase

Academic Papers Peer Review
Consultancy Questionnarie & focus groups
Materials Questionnarie
Continuation Structured interviews with commercial sponsers to develop technological tools for TELRI educational model
Production of TELRI software Trials & interviews

Cost effectiveness

The project will make it a priority to include a qualitative cost-benefit analysis in a number of the interventions. An approach is indicated below.

Educational effectiveness must first be measured, comparing the effectiveness of the new teaching and learning strategies with the previous approaches, including the role played by the technological intervention. Immediate and long-term benefits may occur in:

  • explicity in curricula about learning outcomes & assessment criteria;
  • enhanced student achievement in the skills that contribute to research capability;
  • improved access to learning opportunities;
  • cohesion of teaching and learning processes into a coherent strategy;
  • increased ratio of motivators to barriers (leading to increased uptake and success);
  • maintenance of quality in courses involving larger numbers of students;
  • reducing teaching staff work loads in preparation, delivery, support and marking;
  • increasing the quality time that staff can spend with students;
  • reducing the overall cost of methods and materials in the long term;
  • more effective use of IT infrastructure;
  • potential for distance learning development.

In terms of the benefits in improving student motivation and learning, one would ideally measure learning outcomes of groups making use of IT against those who do not. This might be achieved by evaluating outcomes with the previous years' equivalent groups. It is unlikely, given the relatively small scale of the project, that one would be able to identify major learning shifts. However, it should be possible for each intervention to derive qualitative data on changes in student capability, and this will be obtained through interviews with staff and students.

The cost of each intervention might also be measured, in terms of: improved or increased use of existing equipment; provision of new equipment; staff training and learning time; and production of new tools and materials. The intervention cost must be compared with the cost of current or where possible alternative methods. Cost in pilot courses will be projected to longer-term benefits of the intervention in terms of maintaining quality and values, staff development, student motivation and achievement, and other factors affecting sustainability (validity and viability).

Collaboration with the experts in the area of cost-effectiveness will inform our approaches and assist with the practical methodology, based on their experience at institutional levels. Gordon Doughty at University of Glasgow (previously TLTSN Centre) has already been approached for external consultancy.

Statements about cost-effectiveness may be derived by considering the overall benefits against the net cost of implementation. This can be guided by change management considerations that relate the likelihood of sustainable change in terms of the balance between desirability of change, net costs and extent of disruption required.

External evaluation

An external evaluation has been commissioned from the Centre for Educational Development, Appraisal and Research (CEDAR) at the University of Warwick. Dr Andrew Parker will carry out the external evaluation and consultation is taking place with the Project Director and Project Manager.

An interim evaluation will review the first eighteen months of the project, and will focus on the extent to which the project team has met its planned outcomes, and its preparedness for meeting the remaining outcomes, including the effectiveness of communication among all levels of the project and all participants.

A second external evaluation will review the project at three years, with a focus on educational and cost-effectiveness, both of the project as a whole and of the individual interventions that have been made. Extensive use will be made of evidence drawn from staff and students who have come into contact with the work of the project. On each occasion a project report will be provided to the steering group.