Miniproject on Citation Analysis

supervisor: RS MacKay, Mathematics R.S.MacKay@warwick.ac.uk

HEFCE plans to base future assessment of research quality and resulting funding for the Sciences (and perhaps Maths&Stats) in the UK on numbers of citations per paper normalised by the average for papers in the "field" (plus research student numbers and research grant income):
http://www.hefce.ac.uk/pubs/hefce/2007/07_34/
They are supported in this decision by a report from the Centre for Science Technology Studies, Leiden, available from the same website.
Apart from the confusion of quality with impact, a fatal flaw in my opinion is that the result will almost certainly be discontinuous across field boundaries: a paper near a field boundary could obtain a different score by moving it across the boundary.  

The project: Design a method of using citation data that has no artificial discontinuities, argue why it should be good, test it on real data from Thomson Scientific Web of Science (get there via http://wos.mimas.ac.uk/ ), and compare with the (extensive!) literature on the subject.

I suggest that the right direction is eigenvector methods.  This means solving a linear system of equations for a self-consistent measure of impact of papers based on the impact of the papers citing and (to some extent) cited by each paper.  As stated on 
http://www.eigenfactor.org/methods.htm
such methods "adjust for differences in citation patterns among disciplines". For some surveys, see
Langville AN, Meyer CD, A survey of eigenvector methods for web information retrieval, SIAM Review (2003)
Palacios-Huerta, Volij O, The measurement of intellectual influence, Econometrica 72 (2004) 963-977
Kleinberg JM, Authoratitive sources in a hyperlinked environment, JACM 46 (1999)
Curiously, the Leiden report gives these methods NO mention.

For journal paper citations, there is the big problem that the associated directed graph is highly reducible, because most citations go backwards in time, whereas the eigenvector methods make sense only if the graph is irreducible.  Thus some fixes are required.  One is to allow a random restart, as in
Del Corso GM, A google-like algorithm for ranking scientific publications, talk at Matrix Analysis and applications conference, Luminy, Oct 2007
but this seems to me should be terribly sensitive to the restart distribution.  Another is to use Kleinberg's method, which pays attention to the papers cited as well as those citing, but one question here is how much one could influence one's score by artificially citing some additional irrelevant papers (in principle journal referees and editors would pick this up, but a lot of errors are not picked up).  Another is whether it achieves a sensible adjustment for differences in citation patterns among disciplines.