Skip to main content Skip to navigation

Warwick Institute for Employment Research

Trialling an Evaluation Framework for LLM-Powered Careers Advice - follow up

Expression of interest - follow-up questionnaire for careers advisers

How we will draw on the Careers Adviser pool of expertise

Once we have confirmed which FE colleges will be taking part, we will begin drawing from the pool for three strands of the study: the Panel Review, the Case Studies, and the randomised controlled trial (RCT).

The questions below will help us clarify which element or elements of the study you would ideally like to contribute to, alongside a few practical details. It should take around ten minutes to complete. Please return no later than 2nd September 2026

If you have any questions or would like to discuss any aspect of this evaluation please contact .

(all questions must be answered in order to be considered to take part in the study)

I understand that completing this form is not a guarantee of involvement in the study (required)

About you:

6. Which nation do you work in? (tick all that apply) (required)
7. Which regions do you have current, working labour market intelligence for? (tick all that apply) (required)
8. How long have you worked as a careers adviser? (required)
9. Highest careers guidance qualification held (required)
10. Are you on the CDI Register of Career Development Professionals? (required)
11. Which client groups do you currently work with? (tick all that apply) (required)

Which parts of the study would you like to be involved in?

The study has three strands, described below. Question 12 then asks which specific roles within those strands you would feel comfortable with. You may tick as many as you feel willing and able to do.

 

A. Panel Review

Careers advisers review the advice tools from the perspective of 16–19 year old college students with substantial careers questions. You would have unlimited access for one week and would be asked to explore the tool as much as you feel necessary to form a judgement. A scoring rubric will be provided. A post-intervention focus group will follow, covering the level of preparation and guidance required, how the rubric might be improved (scales, missing points, phrasing of specific items, low value-add items), and how long the process takes to do well.


 

B. Case Studies

Careers advisers role-play one or more from a set of ten client personas, for the purposes of receiving a careers advice session. Each interaction runs to a minimum of around 30 mins and a maximum of 60 mins. Each persona is applied to one of four interventions in a single interaction.

The four interventions are (i) a generic chatbot; (ii) a dedicated careers chatbot; (iii) a professional careers adviser session via video call; and (iv) a hybrid model, in which a human adviser has access to a dedicated careers LMI tool and uses it during the discussion with the young person. A total of 40 transcripts will be scored by two careers advisers, who are not involved in any of the role playing functions. We will arrange a post-intervention focus group with CAs to get feedback on their experiences.


 

C. Randomised controlled trial (RCT)

The RCT consists of a sample of 160 16-19 year old college students undertaking four careers interventions (see i-iv above) and then ranking their experiences across a range of measures.

Note: the careers adviser might be assigned to using AI in the interview they provide to students (you have professional discretion over how to use the AI tool, depending on what is appropriate for the student) - if that possibility would not be appropriate for you, please do not tick the relevant box. Carerers advisers will also review and score the advice transcripts with two advisers scoring each one on a ‘blind’ basis.  Careers advisers will review all the transcripts to check for any issues/concerns with the advice. Advisers will also score the transcripts against the scoring rubric whilst they are reviewing them.

Note: Participants will be able to access any other careers interventions throughout the trial, either ones they would normally use or others that may arise (e.g. attending a careers fair) throughout their participation, provided they also use the assigned intervention. 

The transcripts of participants who use a chatbot in their intervention will be reviewed by careers advisers and if there is an identified issue with the advice given a Careers Adviser will contact the research team who will contact the student to discuss it with them. All participants in the study will have the option of speaking to a careers adviser at the end of the trial, after the next steps and action survey have been completed.  As a result, all students should have careers support at least no worse than would have been anticipated and typically better than without the trial in place.


 
12. Which of the following specific roles would you feel comfortable with? (tick all that apply) (required)
13. Approximately how many hours could you commit across the study period? (required)
14. In which of the following periods would you have some availability? (tick all that apply) (required)
15. When could you usually take part (tick all that apply) (required)
16. Would you need your employer’s agreement to take part? (required)
17. Do you have reliable access to the following? (tick all that apply)
18. How would you describe your use of generative AI tools in your practice?
Privacy notice

Personal data will not be passed on to any third parties and will be kept securely in a folder at University of Warwick for the purposes of this research.

If you have any questions or would like to discuss any aspect of this evaluation please contact ier@warwick.ac.uk 

The University of Warwick is the data controller of any information you have entered on this form and as such is committed to protecting your data and rights in line with applicable data protection law (including the UK GDPR and the Data Protection Act 2018). Please review the privacy information (including privacy notices) provided by the University via the following link, for further information on your rights and how the University processes your personal data: Data Protection.

Spam prevention

Failure to load reCAPTCHA

reCAPTCHA is a utility used to verify you're not a robot filling out this form. Unfortunately this has failed to load correctly.

Please try reloading the page. If the problem persists, or if you are in a country which blocks Google products, please contact us by using the 'page contact' link at the foot of this page.

Let us know you agree to cookies

We use cookies to give you the best online experience. Please let us know if you agree to functional, advertising and performance cookies. You can update your cookie preferences at any time.

Cookie policy