ASSESSING OCCUPATIONS THROUGH ARTIFICIAL INTELLIGENCE: A COMPARISON OF HUMANS AND GPT-4

Abstract

Large language models (LLMs) such as GPT-4 have raised questions about the changing nature of work. Research has started to investigate how this technology affects labor markets and might replace or augment different types of jobs. Beyond their economic implications in the world of work, there are important sociological questions about how LLMs connect to subjective evaluations of work, such as the prestige and perceived social value of different occupations, and how the widespread use of LLMs perpetuate often biased views on the labor markets reflected in their training datasets. Despite initial research on LLMs’ world model, their inherent biases, attitudes and personalities, we lack evidence on how LLMs themselves evaluate occupations as well as how well they emulate the occupational evaluations of human evaluators. We present a systematic comparison of GPT-4 occupational evaluations with those from an in-depth, high-quality survey in the UK context. Our findings indicate that GPT-4 and human scores are highly correlated across all ISCO-08 major groups for prestige and social value. At the same time, GPT-4 substantially under- or overestimates the occupational prestige and social value of many occupations, particularly emerging occupations as well as stigmatized or contextual ones. In absolute terms, GPT-4 scores are more generous than those of the human respondents. Our analyses show both the potentials and risks of using LLM-generated data for occupational research.