Evaluating Biases in Context-Dependent Sexual and Reproductive Health Questions (2024)

Abstract: Chat-based large language models have the opportunity to empower individuals lacking high-quality healthcare access to receive personalized information across a variety of topics. However, users may ask underspecified questions that require additional context for a model to correctly answer. We study how large language model biases are exhibited through these contextual questions in the healthcare domain. To accomplish this, we curate a dataset of sexual and reproductive healthcare questions (ContextSRH) that are dependent on age, sex, and location attributes. We compare models’ outputs with and without demographic context to determine answer alignment among our contextual questions. Our experiments reveal biases in each of these attributes, where young adult female users are favored.

## introduction with the rise in accessibility of chat-based large language models (llms), the public increasingly uses them as question-answering systems for personalized answers. while many questions contain absolute, objective answers (e.g., when was benjamin franklin born?), some questions are contextual and when underspecified, will produce incorrect or incomplete answers that are dependent on missing information not stated in the question @xcite @xcite . these types of questions are especially prevalent in the healthcare domain, where an answer may depend on medical history or the user's attributes, such as age and biological sex. for example, the question "which is the best birth control method for me?" does not have a single correct answer and instead depends on both sex and age, among other factors. however, an llm's answer may not account for these factors and instead ## data we focus on contextual questions relating to sexual and reproductive health, as these topics are often stigmatized in american society @xcite , and can depend on age, location and sex. users may turn to llms for these questions since they can obtain information anonymously without potential societal and familial repercussions. we source our data from two public health question-answering websites: after collecting questions from both sources, we filter our dataset to contain context-dependent questions. we specifically focus on retaining questions that are dependent on a person's age, location, or sex, as these can often affect the answers to these types of questions. we label whether each question is dependent on one or more of our three attributes, with annotations verified by public health and gender studies researchers. the final contextsrh dataset contains 116 questions from planned parenthood and 71 from go ask alice foot_2 . of the 187 questions, 64 depend on sex, 106 on age, and 55 on location. public health and gender studies experts were consulted in determining which groups to analyze. we focus on milestone ages in the united states @xcite @xcite @xcite @xcite @xcite and aim to cover topics that relate to our broad range of ages such as puberty, contraception, and menopause. for sex, we study people who are assigned one binary sex at birth (female and male). while intersex and additional genders exist (e.g., transgender, twospirit), we focus our initial study on the binary female/male sex categories due to data constraints with plans to expand to other groups in the future. as our questions stem from u.s.-based websites, and reproductive health and sex education have state-level policies, we limit our locations to u.s. states. laws relating to parental consent, healthcare accessibility, and sex education in public school systems differ across states. recent years have seen more restrictive reproductive healthcare laws arise in traditionally conservative states. as such, we use warshaw and tausanovitch (2022)'s study on the ideological preferences of americans to select the three most conservative -wyoming (wy), idaho (id), south dakota (sd) -and liberal -massachusetts (ma), vermont (vt), hawaii (hi) -states foot_3 . we specify our evaluations to laws in effect at the end of 2023. ## biases in context-dependent health questions we hypothesize that asking context-dependent questions without stating the user's attributes as context will reveal biases in answering questions toward specific demographic groups. to analyze this, we: 1) probe the model with the original question from our dataset, 2) probe the model with the question and a demographic group as context for all groups within an attribute, and 3) compare the answers produced by the model for each group against the model's answer to the question without context (original answer). our model inferences use a temperature of 0. we evaluate gpt-3.5turbo foot_4 , llama-2-70b-chat @xcite , and llama-3-70b-chat 67 . when comparing model outputs against the original answer, we use sentencebert embeddings @xcite to embed each generated answer and measure the cosine similarity of each group's answer to the original answer. this is used as a measurement to then rank the answers with context across demographic groups. these measurements allow us to compare which demographic groups contain more contextually similar answers to the original answers (the answers may be more similar but are not necessarily equivalent). while sentence similarity may not completely capture bias, it can help capture instances in which answers from questions with context deviate greatly from original answers (e.g. can contain different or additional information). these significant deviations can indicate instances of bias, where original answers are more similar to answers given specific groups of context (e.g. female sex). with these sim-ilarity scores, we calculate (formulas in appendix a.3): in some cases, multiple groups were tied for the most similar answer and as such, the total percentage across all groups is over 100%. in our setting, we consider an unbiased model to be one that presents comprehensive answers pertaining to all relevant demographic groups (e.g. all sexes) when asked an underspecified question. when asked a question with demographic context, the model would tailor the answer to that demographic. however, an unbiased model would still have this demographic-specific information present in its answer. as there are many situations in sexual and reproductive health in which a user may not want to include further identifying information to maintain privacy, having a model that produces comprehensive answers pertaining to all demographic groups allows these users to maintain this privacy. our metrics can be seen as a version of demographic parity, where percent win indicates how often each group has the most similar answer to the original answer. in this case, an unbiased model would have equal values across all groups for an attribute. additionally, we perform a human evaluation of the models' answers to the original questions. annotators read a contextual question and model response and determine for which groups is the model correctly responding. for the sex-based questions, we use healthcare annotators on prolific foot_7 . for location and age-based questions, we manually annotate the results since location-based questions require knowledge of current laws in the u.s. age-based questions are dependent on either current u.s. laws or health-related context. however, we find that healthcare annotators on prolific contain the very biases we are examining in the models (e.g. don't associate pregnancy with individuals past age 40). as a result, we use internal annotation for these two lists of questions. two ## results when comparing sexes, we find that models provide female-leaning answers (example in figure 1 ). adding the male sex as context frequently pushes models to generate information regarding male anatomy rather than only female anatomy or highlight birth control related to males. though sexual and reproductive healthcare is relevant to both sexes, some of the topics are typically discussed more in the female context (e.g. birth control and yeast infections). this bias is consistent with sexual and reproductive health service provision globally, as they are typically focused exclusively on females. these societal biases may affect the quality of answers given to male users or suggest to male users that they do not have a role in reproductive health, as information in this space can drastically differ between the sexes. our qualitative human evaluation shows a strong alignment with quantitative results in the age and sex attributes, verifying this analysis. meanwhile, the location attribute has more variation, though massachusetts is highly favored in both. we verify the statistical significance of differences in similarity scores across groups with friedman's test for age and location and wilcox signedrank test for sex. differences across the age and location groups are statistically significant (@xmath0.05) for all models. differences between male/female are statistically significant for gpt-3.5-turbo and llama-3. we measure cohen's kappa @xcite inter-annotator agreement for the location and age annotations through binary label splits across groups and obtain agreement scores (gpt, llama-2, llama-3) of 0.78, 0.64, and 0.67 for location and 0.37, 0.48, and 0.53 for age. while location inter-annotator agreement is strong, age is lower likely due to the large number of categories (10) and age being a continuous variable. ## conclusion we studied how social biases may arise through underspecified contextual healthcare questions in chat-based llms, illuminating the types of questions that may be susceptible to bias. disparities exist among model answers for different groups across age, location, and sex attributes. therefore, it is crucial to ensure equality in models' answers in critical domains such as sexual and reproductive healthcare. future question-answering research can work toward providing comprehensive answers that are not tailored to certain demographics. this can then help ensure user privacy when asking sensitive questions by providing users with relevant knowledge without asking for additional information.