Developing an evaluation framework for large-language model careers advice

Careers advice is a critical component of UK education and work transitions, particularly for disadvantaged groups. Around 33% of students currently use Large language model (LLM) tools for guidance, but policy makers, service managers, and practitioners lack evidence on how well they perform.

A new project, led by IER’s Associate Professor Peter Dickinson and funded by the Nuffield Foundation, will address this evidence gap. The project will develop and pilot a framework to evaluate LLM-powered careers advice tools against a human adviser baseline. Further education students and advisors from college around the UK will take part.

The advisory group, chaired by Huw Morris, University College London, met in-person at IER this month, bringing a wealth of experience from the Higher Education and Skills sector.

Peter Dickinson said:

‘We’re looking forward to the field-work phase of this work this autumn. Thank you to those who have already expressed interest in being involved”.

For further information see the IER project page. If you would like to get involved in the pilot please contact Stef Poole via email (Stefanie.Poole@warwick.ac.uk).