A recent study by Harvard psychologists claimed that student reactions to randomly selected 30-second clips of soundless video tapes of actual class instruction were extremely accurate predictors of end-of-course student evaluations, even though the sound had been turned off. Moreover, the correlation decreased only slightly when the length of the film clips was reduced to 10 or 5 seconds. Because my source of information for this apparent indictment of student evaluations was the Boston Globe Sunday magazine, I tracked down the original article in the Journal of Personality and Social Psychology.

The actual purpose of the study was not to assess the validity of student evaluations, but to examine the accuracy of first impressions. End-of-course ratings were chosen as the comparison measure because ``student evaluations seem to be a valid means of evaluating teacher effectiveness.'' The methodology of the study, which used samples of only thirteen instructors and a quite different set of questions from those on the evaluation forms, left me skeptical about drawing any conclusions from it. Nevertheless, the findings raise serious concerns that merit further study.

Meanwhile, a variety of sources, including many of the legislators responsible for funding state-supported universities, are calling for increased use of student evaluations. Administrators, faced with a glut of data, often find creative ways to reduce it to meaningless numbers. I encountered one who insisted that it sufficed to consider only the question on overall effectiveness, because he had once seen a report that, on average, the average on this question equalled the average of all other questions. He persisted in this policy even in cases for which it was patently false.

Advocates often cite a few superficial studies in support of the reliability of student evaluations. However, other studies give a more complex picture. Moreover, survey results can be extremely sensitive to the wording of questions and the circumstances under which the survey is given. An evaluation which asks both ``Did the homework assignments increase your understanding and facility with the subject?'' and ``Were the homework assignments reasonable in length and difficulty?'' is more likely to provide meaningful information than ``Was the homework appropriate to the course.'' And where did the question about instructor's ``sensitivity to student difficulty with course material'' come from? Questions about an instructor's willingness to answer questions and/or the clarity of an instructor's answers may seem straightforward until one sees low ratings followed by comments such as ``often answers indirectly by working a similar problem or asking us questions.''

But simply bashing student evaluations serves little purpose. Some types of student feedback are useful, and many advocates of student evaluations are motivated by a genuine concern about the quality of education. However, many experienced faculty question the reliability of student evaluations as a measure of teaching effectiveness and worry that they may have counter-productive effects, such as contributing to grade inflation, discouraging innovation, and deterring instructors from challenging students.

Student evaluations need to be much more carefully investigated. Their deficiencies, their limitations, and the circumstances under which they can be useful all need to be thoroughly documented. Other mechanisms for evaluating and improving teaching effectiveness need to be explored and given greater emphasis.

While these issues concern faculty in all disciplines, mathematical scientists, because of their close link to statistics, bear a special role and responsibility. At the very least, mathematics faculty should insist that any numerical component to the evaluation process used at their institution meet minimum standards of statistical validity. In addition, the AMS and other organizations representing mathematicians should promote and publicize high-caliber studies which address these issues. Our commitment to excellence in mathematics instruction must extend beyond our own classrooms.

--Mary Beth Ruskai