Research by Warwick Business School at the University of Warwick warns that a key Government Strategy for Social Mobility is placing considerable reliance on a table which is simply replicating a well known statistical trap or artifact that may not be the true picture.
The policy proposal put forward by Nick Clegg in a strategy document published by the Government at the start of this month entitled: “Opening Doors, Breaking Barriers: A Strategy for Social Mobility” puts a great deal of faith in a graph that appears to show that children from poor families of high ability massively underachieve relative to children from wealthy families of similar ability, and that conversely children from wealthy families of low ability massively overachieve relative to children from poor families matched on ability.
This Feinstein graph follows below with Feinstein’s own labelling:
Warwick Business School Professor Daniel Read researcher says:
“While the proposal in the strategy may be good ones I am very worried that this graph is being used to shape policy when in fact many statisticians will instantly see that it simply replicates a statistical trap or artefact called “regression toward the mean”. The apparently shocking pattern of results in the graph is simply what statisticians would expect when you measure extremes of performance in two populations of differing ability”
Regression toward the mean (RTTM) occurs whenever measurements are selected for their extremeness (e.g., the high or low scores are chosen for further analysis), and then compared to other measurements of the same quantity.
For instance, imagine that you look at all the sprints made by Usain Bolt, and find his personal best. You then look at how fast he ran on his next sprint. It will probably be slightly slower, and his previous sprint will be slower as well. That is because we deliberately selected the fastest sprint and then looked at what happens on other sprints. If we had selected his slowest sprint, we would find the same thing reversed – the adjacent sprints would now be a bit faster. Choosing the most extreme result as your starting point – the fastest or slowest biases the whole analysis. One way this happens is because choosing the most extreme results exposes you to the undue influence of idiosyncratic factors which either speed up or slow down the sprinter on each occasion. For Usain, the track might be more or less slippery from sprint to sprint, there might be a tailwind or a headwind, or really anything. The very fast sprints are likely to have been bolstered by idiosyncratic factors that are less likely to be there on other sprints, while the very slow sprints are likely to have been impeded by other non-repeating idiosyncratic factors.
The Feinstein graph is constructed in the same way with undue emphasis on extreme results. Students are selected that score high or low on a test at one time, and then those scores are compared to ones taken at another time.
Other phenomena possibly due to regression toward the mean
Tall fathers tend to have sons that are slightly shorter than they are ... and tall sons tend to have fathers that are slightly shorter than they are.
An outstanding meal at a restaurant can bring us back on another occasion, when the meal is not so good as it was before. (Notice how we never go back to restaurants that disappointed us the first time, so we never experience regression in the opposite direction.)
An X-factor contestant who messes up one week, will probably come back the next week with a better performance. But one who performs outstandingly one week, is probably more ordinary the next.
A political party that wins by a landslide, wins by a smaller margin next time. A political party that is routed, makes a modest comeback next time.
The sophomore slump. An artist that produces an outstanding first album, comes back with one that is a bit of a disappointment. (Notice that only artists whose first album is exceptional get a second chance – this is the selection required for RTTM).
Child stars are usually less successful as adult stars. (But note that adult stars were probably less successful as child actors).
Great books make average movies.
For further information please contact:
Professor Daniel Read
Professor of Behavioural Science
Warwick Business School, University of Warwick
NB in France with academic colleagues week beginning 11th April on 0033160724145
Peter Dunn, Head of Communications
Communications Office, University House,
University of Warwick, Coventry, CV4 8UW, United Kingdom
Tel: +44 (0)24 76 523708 Mobile/Cell: +44 (0)7767 655860
PR52 11th April 2011