Skip to main content

Evaluating Potential Investments in Learning Technology

Gordon Doughty, University of Glasgow


Glasgow University's institutional TLTP project has studied "Teaching with Independent Learning Technologies" for up to 5 years. The project's aims included gaining benefits and reducing costs. This paper describes some successful outcomes. It focuses on the use of the concepts and language of costs, benefits and value-for-money to help teachers evaluate and negotiate their cases for investment in changing teaching methods. 

The investment in the reorganisation as a whole brings the benefits, so all aspects of a change should be evaluated approximately before deciding which count in choosing between alternatives. Managers must face the complexities that are obscured by considering all costs and benefits in financial terms alone, but gathering and interpreting data can be simple for individual courses, as illustrated here by case studies in teaching Electronics and Mathematics: 

  1. For efficiency and cost-effectiveness - the same learning at lower cost - alternatives were regarded as of equal benefit, so evaluation was of costs alone, in terms of time. 
  2. Costs need not be evaluated if alternatives have essentially the same cost. Benefits can be evaluated in terms of the quality of learning, such as assessment results and pass rates. 
  3. Learning Technology (LT) helped to restore learning quality for a decreased staff:student ratio. Both benefits and costs of alternatives were evaluated. 
Evaluating potential investments in LT

University teachers often say they would like to adopt more LT resources, but lack time, support staff, information and suitable materials. Teachers seldom know how to make a case for the necessary investment: educational justifications, cost-effectiveness, benefit:cost ratios and productivity. Investment decisions depend strongly on local conditions, and on which funding and resource allocation models are used. Decisions are usually more strategic than economic, but where economic analysis is possible, stakeholders and decision makers can be more confident. 

The approach to investment depends on whether big issues or local ones are concerned. To some the evaluation of costs and benefits of education must include all costs and all benefits to the whole of society - education is an investment in the future of a nation (Carnoy ). These considerations would be applied, for example, to whether or not the government should subsidise a loan scheme for students to buy computers. Accounting and estimating must find a monetary value for all factors, with discounting appropriate to a whole economy. But economists have to make and justify many assumptions, which tend to reflect the values of whoever has power. 

At each level in an institution, decision makers have their own objectives, and deal with problems, opportunities and threats at their level. Their ability to invest depends on their power to influence levels above and below through agreement, negotiation or imposition of their own values. Costs and benefits have to be expressed in one or more "currency" which adequately represent these values. 

Teachers, course teams and departments usually choose between potential changes to meet outside pressures. They tend to be only concerned with those costs and benefits over which they have control and need changes. It is often more appropriate to evaluate and compare the costs and benefits in terms closer to their experience - student and staff time, student numbers, assessment results - rather than try to assign money values. 

Most teachers in the TILT project wanted to move a course in at least one of three directions: better quality, cheaper provision (lower cost), or quicker delivery (less time). Generally it is not possible to improve in one of these directions without one or both of the other two deteriorating, unless more resources can be employed, or greater efficiency found. The teachers believed that LT permits better human activities, making teaching and learning more effective or efficient, but they needed investment in software, its physical and organisational infrastructure, in how to use it, and in integrating LT into the students' learning. Reflecting on the whole TILT project we can summarise how teachers could have identified good LT investments: 

  • identify driving problems and opportunities 
  • identify objectives of students, teachers, the institution, the nation 
  • identify and develop the possible alternatives 
  • choose criteria for evaluating inputs & outputs, costs & benefits 
  • consider all types of costs & benefits - identify those which are not relevant, those which are under your  influence, and those which may need action beyond your control 
  • estimate relevant costs in time and/or money 
  • estimate relevant benefits as satisfactory / unsatisfactory, on a points scale, or in money terms
  • estimate levels of confidence, risks if wrong, sensitivity to errors 
  • compare costs & benefits of each alternative 
  • consider wider implications of the direct change and consequential changes - go beyond economics and  accountancy methods and evaluate the softer benefits, the impacts in human and organisational terms. 

In order of potentially good value, LT investments seemed to be: infrastructure, organisation and training; communication; automating processes, eg assessment; low-cost local packages; importing learning packages; producing own high cost packages for better learning. 

1. Efficiency and cost-effectiveness

In seeking efficiency and cost-effectiveness, we consider alternatives which would provide benefits (eg assessment results) that are regarded as"good enough". Because alternatives are regarded as of equal benefit, choice between them can be made on costs alone. 

Many standard software packages can deliver far more than their specialised functions. Spreadsheets, (eg Tidball ) can be overlaid with text, diagrams, still and moving pictures and animations, cells can be programmed as interactive buttons, graphs can be linked to modifiable data. Although lacking the presentation and interaction possible with dedicated authoring packages, it may frequently be sufficient for its purpose, and is more likely to be an affordable, worthwhile investment of the teacher's and students' time. A lecturer invested in transforming lecture notes into low-cost interactive learning packages using spreadsheet files. The packages presented theory and set interactive tasks such as drawing graphs to simulate electric circuit behaviour, with pop-up hints and answers to numerical questions (early version). 

Evaluating Benefits 
Evaluators investigated whether this change provided benefits, mainly in the learning achieved, eg final examination questions based on this part of the course. Performance results were regarded as remaining "good enough". 

Evaluating Costs 
Most aspects of evaluating the costs of alternative actions are not unique to Higher Education (Management Information for Decision Making: Costing Guidelines for Higher Education Institutions). Generally, "opportunity costs" should be identified, the value of the next best alternative action to the one which is chosen. An opportunity cost to a teacher could be an afternoon of research given up to learn how to modify a learning package. 

There may be "fixed costs" such as for equipment, software licences, copyright, but for this course these had already been bought for general purposes and spare capacity was available. There may be "variable costs" - teaching assistants, extra licences, content update, consumables. Institutions may devolve costs of staff training, administration, LT support & maintenance. A department may need to consider "marginal costs", eg the costs of adding more students to an existing course. In many institutions the time of staff with no fixed hours of work is regarded as always available for marginal extra work, in others every hour is jealously costed. 

The "cost per student" varies with volume of activity (eg between little LT use to all LT use), the number of students involved and the phase in the life cycle of a course. Estimates are sensitive to time horizons and the rate of change of technology relative to rate of change of delivering learning. Computers, networks, buildings and books have different useful lifetimes. Because of rapid technological change there is little time to accumulate reliable predictive data. It is worth considering who pays for changes in teaching resources- students (who always pay for books, often photocopying, but seldom computers), teaching staff (time, stress, promotion prospects), budget holding units, and central administration's top slicing and pump priming. 

In this Electronics case, the important opportunity cost that was identified was measured in terms of the lecturer's time, and was little more than the time taken in each previous year to update printed course handouts and overhead transparencies - an hour or so for each topic. Before this courseware there were 24 contact hours of lectures. By 1996/7 the teacher spent about 8 hours lecturing to the whole class, 10 hours negotiating individual learning contracts and helping groups or individuals, and 6 hours timetabled on call, but seldom needed. Lecturing time was released for more individual attention and students became more independent learners. The main change in costs was almost 6 hours time saved each year (25%) and redirected to research. 

2. Benefits for no extra cost

There are many situations where all allowable alternatives have the same cost, eg better learning for the same cost, working within a fixed budget. If evaluating a benefit for no extra cost, costs need not be counted. Benefits can be evaluated in terms of the quality of learning, such as assessment results and pass rates. 

For no extra staff effort, computer provision or software cost, a package was obtained which let students discover their misconceptions in Electricity, informing the dialogue between teacher and students and between students on the student's conceptions or misconceptions (Kavogli). 

Evaluating Benefits 
Benefits of applying LT need to be evaluated. Teachers and managers tend to ask for evidence such as peer acceptance, fit to their theories of learning, evidence that successful learning took place and enthusiastic response from students. Benefits may be distributed between students, staff and administration.There could be unplanned or unexpected benefits of introducing a LT package - change towards more flexible, open learning; ability to provide distance learning; change to learning objectives. 

Alternatives may be compared by expressing benefits in money terms. Fortunately a local decision maker seldom needs to evaluate all of the benefits in money terms. It may be enough to estimate benefits on, eg, a 1 - 5 point scale. The total score may involve mathematical weighting ("Utility" to economists) of the value of the benefits. 

But crude analysis tends to obscure many complex quality issues, and the measure may depend on a hierarchy of interrelated benefits. In this Electricity case there were direct benefits in student achievement, and indirect benefits in retaining more students for the next session and in the quality of their subsequent work. Evaluation investigated whether this change provided learning benefits, through questionnaires, observation, interviews and assessment questions. All students benefited from the package, but the lower the previous qualification in Electricity, the greater the gain in learning. 

3. Improved benefit:cost ratio

LT helped to restore learning quality for an increased student:staff ratio. With staff numbers decreasing, small class Maths teaching was cut and a class of students had been placed in a lecture-based Maths course with 200 other engineers. Its high failure rate limited progress to the next year. The CALMAT Maths package was found (Cook, Tabor ), and it replaced almost the whole lecture course. 

The most noticeable costs of the change were £300 per year for the site licence and £2.50 per student per year for photocopying supporting material. Students could purchase copies for use on home computers. But using CALMAT saved more than 75 teachers' hours per year compared with the alternative of delivering lectures and providing class tutorials. 

Evaluators found that learning was improved and that time was saved for which there were plenty of alternative useful activities. Students' Maths skills improved dramatically, benefiting learning in other courses. Low failure rates stopped the loss of students to the university and preserved student fee income. £230,000 of fees have been retained so far. 

Evaluating benefit:cost ratios 
In the Electronics case the benefits were regarded as equal but costs were reduced. In the Electricity case the costs of alternatives were initially regarded as equal but learning benefited. Usually we cannot improve both benefits and costs. Alternatives need to be compared with a benefit:cost ratio. In this Maths case both the benefits and costs of the alternatives were different, so needed to be evaluated. Both were found to have improved. Students' skills improved, with better learning in other courses. 75 teachers' hours per year were saved. Low failure rates preserved income - a benefit of £230,000 so far for a cost increase below £1000.

Dr Gordon Doughty
Robert Clark Centre for Technological Education
University of Glasgow
66 Oakfield Avenue
Glasgow G12 8LS

Interactions Logo
bullet  Editorial
bullet  Articles 
bullet  News   
bullet  Innovations   
bullet  Resources   

 CAP E-Learning