over-predict a certain code for men but under-predict for women).

Figure 4: LLM models fail bias test much more regularly that iQual <!-- Start of picture text --> ChatGPT Llama−2 13B chat Llama−2 13B iQual + ChatGPT aug iQual<br>Aspiration: Religious<br>Aspiration: Secular<br>Ambition: No Ambition<br>Ambition: Job Secular<br>Ambition: Vocational Training<br>Ambition: Entrepreneur<br>Ambition: Education High<br>Ambition: Education Neutral p−value<br>Ambition: Education Low <1%<br>Ambition: Education Religious<br>Ambition: Marriage <5%<br>Ambition: Migration >5%<br>Capacity: Reliance On God<br>Capacity: Ability High<br>Capacity: Ability Low<br>Capacity: Budget High<br>Capacity: Budget Low<br>Capacity: Awareness Information High<br>Capacity: Awareness Information Low<br>−1 0 1 2 −1 0 1 2 −1 0 1 2 −1 0 1 2 −1 0 1 2<br>log F statistic<br>Annotation<br><!-- End of picture text --> _Note_ : This Figure shows the result of an F-test for a statistical association between the prediction errors of each model with the characteristics of the interview subject. Each model is shown as a separate panel, with each code shown along the vertical axis. The log F statistic of this test is shown along the horizontal axis with the color of the points indicating the statistical significance of the test statistic. The subject characteristics include refugee status; age and sex of eldest child; age, education and sex of interview subject, "refugee"; total number of children; household assets and income; and history of trauma experience. The LLM models display a bias much more frequently that the supervised models. The full results for each regression are shown in Appendix C Figure 4 shows these F statistics that test whether the prediction errors of each annotation approach are systematically related to the interview subjects’ characteristics, with the full regression in each case reported in Appendix C. The higher the (log) F statistic is, the stronger the evidence of bias. The color of the points indicates the level of statistical significance and each model is shown as a separate panel. We see that for iQual in the left-most panel, there is evidence of bias in only one of the 19 codes. While we should be cautious in interpreting results with this code, there is not much cause for concern. However, for the LLMs we find strong evidence of bias in many of the codes. This tells us that the prediction errors the LLMs make are not random and conducting analysis on the basis of its predictions is likely to result in misleading interpretations.

The F tests shown in Figure 4 tell us that there is some statistical association between prediction errors and subject characteristics. We can see concrete examples of how this can lead to misleading conclusions in Figure 5. This Figure shows estimated coefficients for regressions of the prevalence of an annotation in an interview on dummy variables for the subjects’ refugee status and the gender of their eldest child. So if the coefficient on refugee status is positive than this code appears more in interviews with refugees than in interviews with hosts. Six of the codes are shown as separate panels and the results based on the annotations of the five different models are shown for each coefficient, following the same color scheme as Figures 2 and 3, but with the coefficient based on only the expert annotations shown in black. The differences in the estimates across annotation methods are because the errors they make are not random with respect to refugee status and the gender of the eldest child.

9