in the context of related literature. Section 2 then very briefly introduces our dataset of annotated interview transcripts. Section 3 describes our approach to using LLMs for annotation (3.1) and the supervised NLP method introduced by Ashwin et al. (2022) which we refer to as iQual going forward (3.2). Section 4 then describes LLM-based out-of-sample performance in comparison to iQual (4.1) and then shows that LLMs introduce more bias and illustrates this could cause researchers to draw incorrect conclusions (4.2). Section 5 then concludes.

# **2 Data and Qualitative Analysis**

The interview transcripts, data collection and the qualitative coding process are explained in detail in Ashwin et al. (2022), so we restrict ourselves to a very brief description here. The population we sample are displaced Rohingyas based in the Cox’s Bazzar camp and local Bangladeshi residents. Along with a standard household survey including questions on demographics and economic conditions, the data include transcripts of 2,407 open-ended interviews with subjects on their aspirations for their eldest child. The interviews were conducted either in Bengali or in Rohingya which was then transcribed into Bengali, but we work with machine translations into English. The interviews take the form of an unstructured to-and-fro of question and answer (QA) pairs the interviewer and the subject. The interviews are on average 12.6 QA pairs long, with the average answer in each QA pair being 13.7 words long.

Based on a close reading of a subset of transcripts, and following a "flexible coding" process (Detering and Waters, 2018), a coding tree was developed including 25 potentially overlapping categories, 19 of which we focus on in this paper. A full description of each code along with examples are shown in Appendix A. Following Callard (2018) the distinction between aspiration and ambition was adapted within the context and nature of "dreams" parents expressed for their children. For example, concrete and measurable dreams for child (e.g wishing a child would become a doctor, teacher, entrepreneur, or specific educational goals) was used as a definition for ambition while intangible, value oriented goals (e.g wishing the child to live with dignity or be a good human being) was classified as aspiration. Aspirations, were divided into "Religious" and "Secular" . Ambition was divided into seven major categories – Education (further sub-coded into High, Low, Neutral and Religious), Job Secular, Marriage, Entrepreneurship, Migration, Vocational Training, and No Ambition. While ambition and aspiration came up at any point in an interview, "capacity to aspire" or Navigational capacity was restricted to discussions of what parents were planning or able to do to fulfill dreams for their children. Navigational Capacity was coded into seven sub-codes – Low and High "Ability", Low and High "Budget", Low and High "Information Awareness", and Reliance on God.

Of our sample of 2,407 interview transcripts, 789 are manually annotated by trained sociologists (co-authors on the Ashwin et al. (2022) paper) according to this coding structure. The annotations are defined at the level of QA pairs, allowing us to represent each annotation as a binary classification problem at the QA level.

# **3 Methods**

In this Section we first explain how we use LLMs to annotated our interview transcripts. We then briefly describe the iQual method which trains supervised models on our expert human annotations, as well as how we use LLMs for data augmentation in combination with iQual. We test three different LLMs- the closed-source ChatGPT (gpt-3.5-turbo) by OpenAI, as well as two open-source LLMs by Meta, the Llama-2 (13b) and its fine-tuned "chat" variant (Touvron et al., 2023).The base Llama-2 is pretrained on publicly available online data sources. The chat variant is then fine-tuned on publicly available instruction datasets and over 1 million human annotations. This fine-tuning is designed to make the model align with human preferences using techniques such as Reinforcement Learning with Human Feedback (RLHF). ChatGPT is also fine tuned using RLHFFor all three LLMs, our approach 3