Job portals have become a focal point for researchers and policymakers due to their cost-effectiveness and ability to offer comprehensive labour market data. At IER data group, we have been developing a vast vacancy database sourced from the main UK job portals since February 2019. Our project is funded by the UK Department for Education, enabling us to conduct groundbreaking research and analysis that drives change in the labour market.

Our process begins with the use of advanced web scraping techniques to automatically collect vast volumes of information from these portals. This raw data is then cleaned and standardised to ensure consistency and reliability. Each job vacancy is coded using various variables, including text patterns within the postings, geographical areas, required skills, and sector classifications such as the Standard Industry Classification (SIC). To further enhance our analysis, we use sophisticated tools like CASCOT to identify occupational groups based on the SOC2020 and ISCO 08 classification system. This enables us to categorise job titles accurately and efficiently.

Explore the resources below to learn more about this transformative project and its implications for research and policy in the field of labour market analysis:

Big Data and the shift towards real time labour market informationLink opens in a new window

ReWAGE Evidence Paper: Work, Wages and Employment in the UK's Hospitality SectorLink opens in a new window

Enhancing a labour market information database: LMI for AllLink opens in a new window

From Automation to Adaptation: Jobs at Risk of Exposure to AI in the UKLink opens in a new window

Green and Hydrogen Jobs in the MidlandsLink opens in a new window

Green Jobs and the Green Economy in YorkLink opens in a new window

Green Jobs and Skills in South YorkshireLink opens in a new window