Constructing a Psychometric Testbed for Fair Natural Language Processing (2021)

Abstract: Psychometric measures of ability, attitudes, perceptions, and beliefs are crucial for understanding user behavior in various contexts including health, security, e-commerce, and finance. Traditionally, psychometric dimensions have been measured and collected using survey-based methods. Inferring such constructs from user-generated text could allow timely, unobtrusive collection and analysis. In this paper we describe our efforts to construct a corpus for psychometric natural language processing (NLP) related to important dimensions such as trust, anxiety, numeracy, and literacy, in the health domain. We discuss our multi-step process to align user text with their survey-based response items and provide an overview of the resulting testbed which encompasses survey-based psychometric measures and accompanying user-generated text from 8,502 respondents. Our testbed also encompasses self-reported demographic information, including race, sex, age, income, and education - thereby affording opportunities for measuring bias and benchmarking fairness of text classification methods. We report preliminary results on use of the text to predict/categorize users’ survey response labels - and on the fairness of these models. We also discuss the important implications of our work and resulting testbed for future NLP research on psychometrics and fairness.

## introduction psychometrics is the field of study concerned with the measurement of individuals' knowledge, abilities, attitudes, personality traits, and perceptions @xcite . in social science research, psychometric dimensions are latent constructs that are known to be important antecedents, moderators, mediators, and consequents for important humanistic behaviors and outcomes. for example, constructs such as threat severity and re- * authors listed alphabetically. sponse efficacy of protective mechanisms are critical psychometric measures of one's likelihood to avoid security threats @xcite . in behavioral health, psychometric dimensions such as health numeracy, subjective health literacy, trust in physicians, and anxiety visiting the doctor's office are known to effect various health and wellness outcomes such as future physician visits and allaround well-being @xcite . in electronic commerce, satisfaction with a website's functional, information, and visual design are correlated with purchase propensity and customer loyalty @xcite . similarly, many individualized financial behaviors can be partially explained by financial literacy and psychological traits @xcite . given the importance of psychometric dimensions for understanding behaviors and outcomes in various domains, rigorous data collection protocols and best practices have been developed over the years @xcite . the primary modes of collection involve surveys and interviews. while these techniques afford many benefits such as measurement control and robustness checks, they are not without their limitations. first, primary data collection facilitated through an administered survey can be time-consuming and invasive (often requiring 20-30 minutes of the respondents' time and attention). second, such primary data collection cannot occur in real-time. most surveys in field studies are conducted periodically at monthly or quarterly intervals. third, while surveys are a rigorous form of data collection, they are limited in their ability to account for data/observations outside the predefined measurement framework. effectively collecting and measuring relevant psychometric dimensions in a timely, unobtrusive, and open-ended manner could be invaluable in many real-world settings @xcite , including information retrieval and behavior modeling @xcite @xcite . in this paper we describe our efforts to construct a testbed for psychometric natural language processing (nlp). in the same vein as prior work on constructing language resources for sentiment, emotion, affect, and personality traits @xcite @xcite , and more recent work on modeling empathy and distress @xcite , we describe our approach and resulting testbed related to psychometric dimensions such as trust, anxiety, literacy, and numeracy in the health context. figure 1 presents a motivating example describing the goal of our work. given a well-established survey-based scale for "trust in visiting the physician's office," how can we obtain a similar score based on user-generated text? further, how do we ensure that our nlp-based scores are fair and unbiased? the resulting testbed is comprised of usergenerated text from 8,502 individuals for four key health-related psychometric dimensions of interest: trust in physicians, anxiety visiting the doctor's office, health numeracy, subjective health literacy. our construction method and testbed contribute to the nlp language resource literature in the following ways: in this section we describe the process taken to construct our psychometric nlp testbed. the key steps included identifying relevant psychometric dimensions of interest, finding suitable survey-based items to operationalize our latent constructs, assessing different prompts for text equivalency questions, and testbed construction validation. ## testbed results and summary statistics two rounds of data collection were performed using amt and qualtrics, respectively. in order to ensure high data quality, we followed best practices for crowd-sourced data collection including suitable compensation, validity checks, clear instructions, and manual inspection of the data @xcite . in each round, all responses were manually examined for quality assurance. a small proportion of responses were removed due to noisy text (e.g., failing to properly answer the questions), a failed validity check, or for responding too quickly (relative to the median response times). for both data collections, each participant was compensated five us dollars. in the first round, we collected a total of 4,262 usable responses via amazon mechanical turk (amt). in order to attain a second, more diverse set of responses, qualtrics was used to collect an additional 4,240 clean responses. based on quantitative and qualitative assessment of the data, participants seemed engaged in the task and thoughtful in their responses -the mean and median response times were 32.7 and 24.1 minutes, respectively (which are in the same ballpark as @xcite ). psychometric dimension question or prompt anxiety visiting the doctor (av) in a few sentences, please describe what makes you most anxious or worried visiting the doctor's office. subjective health literacy (hl) regarding all the questions you just answered, to what degree do you feel you have capacity to obtain, process, and understand basic health information and services needed to make appropriate health decisions? please explain you answer in a few sentences. trust in physicians (td) in a few sentences, please explain the reasons why you trust or distrust your primary care physician. if you do not have a primary care physician, please answer in regard to doctors in general. objective health numeracy (hn) in a few sentences, please describe an experience in your life that demonstrated your knowledge of health or medical issues. characteristic description unique users 8,502 text fields subjective literacy (sl) (per user) objective numeracy (hn) trust in physicians (td) anxiety visiting the doctor (av) age mean: 40.5 over 50: 28.7% race 65.5% white 28.7% black 5.8% other sex (female) 65.4% income (usd) 62.7% < $55k education 50.9% college grad or higher examples of usage of prescription drugs other behavior/ presence of primary care physician psychometric frequency of doctor visits dimensions smoking and drinking frequency response 32.7 minutes (mean) times 24.1 minutes (median) mean response 226 (sl), 221 (hn) lengths (chars) 218 (td), 179 (av) the most critical survey response items in the data were the ones corresponding to the four psychometric dimensions. following best practices from the social science literature, we constructed a single composite score for each of these dimensions by averaging across multi-item scales @xcite . the scores were scaled to a 0-1 range. figure 3 depicts the distribution of user responses for the four dimensions (hl, hn, td, av). we can see that for hl and td, the responses followed a skewed gaussian distribution. in contrast, av, and to a lesser extent, hn, were more uniformly distributed.