annotations) are not random. Second, LLMs over-predict many of our codes. We can think of this as the LLM bringing the "pre-conceptions" it has learned from its training data to the annotation task. Consequently, LLMs are probably not suited for coding most qualitative data that requires nuanced and contextual analysis. This sort of analysis has traditionally been the province of anthropology and sociology but is increasingly being used by more quantitative fields such as economics and political science.

To analyze large-N qualitative data, such as those obtained from open-ended in-depth interviews, we show that a coding scheme based on a close-reading of transcripts by experts in qualitative analysis with a sub-sample of the full corpus of data is vital for interpretation and analysis. Firstly, high quality annotations are necessary in order to assess whether the LLM is introducing bias in its annotations. Secondly, these high quality annotations can then be used as a training set for smaller bespoke models. These bespoke models may be able to leverage LLMs through data augmentation, but importantly they are trained on context-specific data so researchers have better control, and an overview, of the information that is used. We suspect that these limitations will continue even as LLMs improve, and we encourage researchers using LLMs for annotation tasks to be aware of and check for bias.

# **References**

- Appadurai, A. (2004), ‘The capacity to aspire: Culture and the terms of recognition’, _Culture and Public Action, ed. Vijayendra Rao and Michael Walton, Stanford, California: Stanford University Press_ pp. 59–84.

- Ashwin, J., Rao, V., Biradavolu, M., Chhabra, A., Haque, A., Krishnan, N. and Khan, A. (2022), ‘A method to scale-up interpretative qualitative analysis, with an application to aspirations in cox’s bazaar, bangladesh’.

- Atari, M., Xue, M. J., Park, P. S., Blasi, D. and Henrich, J. (2023), ‘Which humans?’, _https://doi.org/10.31234/osf.io/5b26t_ .

- Bonikowski, B. and Nelson, L. K. (2022), ‘From ends to means: The promise of computational text analysis for theoretically driven sociological research’, _Sociological Methods & Research_ **51** (4), 1469– 1483.

- Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A. et al. (2020), ‘Language models are few-shot learners’, _Advances in neural information processing systems_ **33** , 1877–1901.

- Callard, A. (2018), _Aspiration: The agency of becoming_ , Oxford University Press.

- Dai, H., Liu, Z., Liao, W., Huang, X., Wu, Z., Zhao, L., Liu, W., Liu, N., Li, S., Zhu, D. et al. (2023), ‘Chataug: Leveraging chatgpt for text data augmentation’, _arXiv preprint arXiv:2302.13007_ .

- Detering, N. M. and Waters, M. (2018), ‘Flexible coding of in-depth interviews: A twenty-first century approach’, _Sociological Methods and Research_ **50** (2)), 708–738.

- Edunov, S., Ott, M., Auli, M. and Grangier, D. (2018), ‘Understanding back-translation at scale’, _arXiv preprint arXiv:1808.09381_ .

- Gilardi, F., Alizadeh, M. and Kubli, M. (2023), ‘Chatgpt outperforms crowd-workers for textannotation tasks’, _Proceedings of the National Academy of Sciences_ **120** (30), e2305016120.

- Kearns, M. and Roth, A. (2020), _The Ethical Algorithm_ , Oxford University Press.

11