”So, are you a different person today?” Analyzing Bias in Questions during Parole Hearings (2024)

Abstract: During Parole Suitability Hearings commissioners need to evaluate whether an inmate’s risk of reoffending has decreased sufficiently to justify their release from prison before completing their full sentence. The conversation between the commissioners and the inmate is the key element of such hearings and is largely driven by question-and-answer patterns which can be influenced by the commissioner’s questioning behavior. To our knowledge, no previous study has investigated the relationship between the types of questions asked during parole hearings and potentially biased outcomes. We address this gap by analysing commissioner’s questioning behavior during Californian parole hearings. We test ChatGPT-4o’s capability of annotating questions automatically and achieve a high F1-score of 0.91 without prior training. By analysing all questions posed directly by commissioners to inmates, we tested for potential biases in question types across multiple demographic variables. The results show minimal bias in questioning behavior toward inmates asking for parole.

## introduction during parole suitability hearings commissioners need to weigh different factors to evaluate whether an inmate's risk of reoffending has decreased sufficiently and therefore justify their release from prison before completing their full sentence. the interaction between the commissioners and the inmates is the key element of such hearings, which are largely driven by question-and-answer patterns. questions asked by commissioners to inmates, in particular, shape the entire conversation and guide the inmate's responses, demonstrating that these questions are far more than a mere formality. this is because different types of questions open spaces for different types of answers. while open questions give room for elaboration and justification, closed questions limit the answer space to either one of the alternatives given or "yes" or "no" in the case of polar questions. given this, we expect a positive relation between positive parole outcomes and the share of open questions in the hearings analysed. while biased outcomes in parole hearings have been studied by various disciplines, mainly focusing on the relationship between demographic variables and parole outcomes @xcite @xcite @xcite , only a few have looked further into the linguistic particularities of this kind of dialogue @xcite . to our knowledge, no previous study has investigated the effect of commissioners' questioning behavior during parole hearings on the reproduction of social inequality, despite their crucial role in the delivery of procedural justice. this study aims to contribute to this research gap by analyzing the questions posed to inmates during parole hearings from california. first we show that with the help of large language models, we are able to generate linguistically informed annotations automatically, allowing us to identify patterns in questioning style of parole board members, while simultaneously minimizing annotation cost and effort. drawing upon these annotations, we examine the relationship between question types and demographic variables collected from a sample of 102 parole-seeking inmates. the key questions of this study are the following: how do the types of questions asked during parole hearings relate to their outcomes? what is the relation between the types of questions posed and an inmate's racial background? are there any disparities in the share of questions posed related to an inmate's racial background and does this impact their chances of being released on parole? does the commissioners' gender bias their questioning behavior towards inmates of different races? ## parole suitability hearings in california the decision about whether an inmate no longer poses a severe risk to the public is based on documented information about the inmate's behavior, rehabilitation progress, and potential for successful reintegration into society. all documents relevant to the process are collected and reviewed by both the board of parole hearings and the inmate before the hearing. the hearings consist of an hour-long interview between both commissioners and the inmate, and are used to clarify issues that need to be addressed. moreover, it also allows the commissioners to assess the inmate's credibility and rehabilitation by interrogating the inmate directly. at the end of this interview, the commissioners make a decision on the inmate's parole eligibility. the committee might grant parole, deny it or postpone the decision to a later date. during the parole interview, the inmate is required to answer questions about various aspects of their life. the topics covered are their life prior to the crime that led to their life sentence, their criminal record, the crime itself, as well as their behaviour since incarceration and possible parole plans (such as job opportunities, housing, and community support). typically, the presiding commissioner conducts the initial and concluding parts of the hearing, addressing the pre-commitment factors and the decision, while deputy commissioners usually cover the post-commitment factors and parole plans. despite the structured nature of the hearing process, there is room for potential biases stemming from the different types of questions asked, consequently limiting the answer space available to the inmate. by analyzing these subtle linguistic strategies, we aim to shed light on this understudied area. ## results we prompted chatgpt-4o to annotate all questions directly posed to an inmate by any of the commissioners. out of 16,039 questions 9990 were annotated as closed (62%), 5385 as open (34%), and 664 as "other" (4%), typically consisting of clarification requests or cut-off questions. this distribution indicates that approximately two-thirds of the questions asked during the 102 selected parole hearings constrain the inmate's response to a predetermined format, while only one-third allow for a more open-ended reply. using the fully annotated dataset, we conducted a statistical analysis in order to answer the research questions posed in section 1. our first question addresses how the types of questions asked during parole hearings relate to the outcomes of those hearings. overall, more closed questions were asked in hearings where inmates were found to be eligible for parole (μ @xmath0.5 vs. μdenied = 89). in the case of open questions, the proportion was slightly higher in denied hearings (μ @xmath1. μgranted = 50.5). nevertheless, the results of a mann-whitney test showed a statistically insignificant relationship between question types and parole hearing outcomes. the second research question aims to analyze the relationship between the types of questions posed and an inmate's racial background. we are especially interested in whether there are any disparities in the share of questions posed related to an inmate's racial background and whether this impacts their chances of being released on parole. according to our dataset, the share of open and closed questions was higher for black inmates in comparison to white, hispanic, and inmates of other ethnic groups (see figure 1 ). given that the distribution of open and closed questions followed a normal and homogeneous distribution, we calculated an f-test to test for significant differences among racial groups and the share of posed ques-tions. figure 1 shows these differences were not statistically significant, for either open or closed questions. given the non-normal distribution and heterogeneity of "other" questions, we calculated a kruskal wallis test. this test yielded results at the 0.1 significance level, meaning that the share of "other" questions asked to black (μ @xmath2, μhispanic = 5, μother = 3; @xmath3.06, see figure 2 in appendix b). upon manual examination of the questions labelled as "other", we found that they primarily consisted of cut-off questions. the majority was incomplete utterances, due to inaudible content and interruptions (as marked and transcribed in the pdf files), or one-word clarification requests. for example, we found instances like "-know about the fight in november?" or "<@xmath4, which are challenging to be interpreted in isolation, as they heavily depend on the context. furthermore, black inmates experienced longer parole hearings, as measured by the page count of the corresponding pdf transcripts in our corpus, though this difference was also not statistically significant. we did not find evidence for an inmate's racial background influencing either the share of questions posed or their likelihood of being released. to investigate whether the commissioners' gender influences their questioning behavior towards inmates of different races, we investigated the types of questions posed by male and female commissioners during the hearings. we conducted a series of hierarchical linear regressions (see assessing the questioning behavior of male and female deputy commissioners on the inmate's race, we found no statistically significant relationship between the share of closed/open questions deputy commissioners (male or female) posed and the race of the inmates. however, we did observe a negative relationship between closed and open questions from deputy commissioners (regardless of gender) and the parole decision. this suggests that, on average, when parole is granted, the share of questions (open or closed) asked by deputy commissioners is lower (see we also conducted further analysis of the data to identify other factors influencing the share of open and closed questions in the parole suitability hearings within our dataset. with regard to open questions, we found significant effects related to the inmates' age, their age at the time of committing the life offense, and the number of years they have been in prison (see ## discussion and limitations this study aims to contribute to the research gap regarding the linguistic study of parole hearings, with a special focus on whether the types of questions asked lead to biased outcomes in parole suitability. based on our sample, we do not observe a relationship between the type of questions asked and the outcome of the hearings. contrary to our expectations, open questions do not seem to be positively correlated to a positive parole outcome, implying that board members might prioritize other criteria over the inmate's ability to provide persuasive answers. external factors to the conversation held in the hearing, such as an inmate's rehabilitation progress, low-risk scores in psychological evaluations, and potential for successful reintegration into society might have a bigger impact on parole eligibility than an inmate's articulating power. in addition, we do not find statistically significant evidence for disparities in the share of questions posed to inmates of different racial backgrounds, suggesting that the evidence regarding racial biases from the social sciences, might not be correlated with commissioners questioning behavior. the fact that female presiding commissioners ask fewer closed questions to white inmates, suggests that white inmates are given a higher chance to articulate themselves in these hearings. however, our findings do not indicate a significant effect on the parole hearing outcome. the low shares of all question types from deputy commissioners in granted parole hearings might reflect the supportive role they play, backing up the presiding commissioner. the presiding commissioners may have already gathered enough information to make a final decision, rendering further questions unnecessary. our findings regarding the inmate's age and their age at the time of the crime align with the findings from social sciences. older inmates were not only more likely to be found suitable for parole release, but were also given more often the chance to articulate themselves by being asked more open questions. similarly, inmates who were very young at the time they committed their life crime were also asked more open questions, allowing them to decide with how much detail they wanted to answer. these findings align with the board's engagement with the enactment of the elderly parole program and the youth parole program. limits to the generalizability of the findings lie in the small sample size, the incompleteness of the manually extracted metadata, and the short timeframe of data selection. due to the skewed sample (only two female inmates) we were not able to test for the gender-responsiveness of questioning patterns used by the board members. another limitation of our study is the lack of detailed content analysis of questions posed by the commissioners and the corresponding inmate responses, which might affect the outcome of the parole hearing. to address these limitations and to obtain more generalizable findings on potential question type bias, we plan to officially request metadata for a larger corpus of parole hearings. regarding the annotation via generative ai, we intend to implement a human-in-the-loop approach, where human oversight will complement chatgpt's output, ensuring greater reliability, through a combination of the ai's efficiency with the precision of human expertise. ## conclusion & outlook this study is the first to conduct an in-depth analysis of question patterns in spoken and transcribed parole hearing data, combining insights from social sciences and language technology. while our annotation approach using chatgpt, yielded very good results, our analysis, based on a sample of 102 parole hearings, did not reveal a significant correlation between the types of questions posed and parole outcomes. our findings also suggest, that racial disparities in parole hearings might not be correlated to a commissioner's questioning behavior or gender, but might be due to other factors discussed in section 7. in order to assess the complexity of the dialogical dynamics and further investigate possible relationships between question types, demographic variables and parole hearing outcome, we plan to expand our corpus. to generate more metadata for in-depth analysis, we consider developing information extraction techniques, such as those used by @xcite . as we are interested in the linguistic strategies employed in parole hearings, the next step is to analyze the content of questions and inmate responses to identify patterns of evasive and non-evasive responses and their potential impact on decisions made by the commissioners. this ongoing research will further bridge the gap between social sciences and computational linguistics, offering a more robust understanding of procedural justice in parole hearings.