More than Votes? Voting and Language based Partisanship in the US S upreme C ourt (2023)

Abstract: Understanding the prevalence and dynamics of justice partisanship and ideology in the US Supreme Court is critical in studying jurisdiction. Most research quantifies partisanship based on voting behavior, and oral arguments in the courtroom — the last essential procedure before the final case outcome — have not been well studied for this purpose. To address this gap, we present a framework for analyzing the language of justices in the courtroom for partisan signals, and study how partisanship in speech aligns with voting patterns. Our results show that the affiliated party of justices can be predicted reliably from their oral contributions. We further show a strong correlation between language partisanship and voting ideology.

## introduction the study of partisanship and ideology has been an important topic in understanding the us legal system @xcite @xcite . most research has focused on justice voting patterns @xcite @xcite @xcite and behavior in court, e.g. the frequency of interruptions @xcite or questions @xcite . despite their core role in legal determination, the content in terms of whether oral arguments portray partisan values has received less attention @xcite . in political discourse in particular, word choice is nuanced to convey specific messages @xcite @xcite @xcite . for instance, republicans tend to prefer the term baby over fetus to emphasize their belief that human rights begin at conception @xcite @xcite @xcite 2000 2010 2020 0 beliefs and partisanship @xcite @xcite @xcite . in this paper, we ask to what extent do oral arguments reveal justice partisanship. we propose a framework to analyze partisanship in justices' oral arguments in the supreme court of the united states (scotus). we classify justices as democrat or republican based on the political party of their nominating president, following common practice in the law literature @xcite . we cast language-based partisanship prediction as a classification task and show that our models can reliably predict partisanship, suggesting that justices project their affiliations in court arguments. we go on to derive language-based partisanship scores from our models, and ask how does language-based partisanship align with established measures of voting-based ideology? we show that our scores correlate well with established voting-based measures (figure 1 ). equipped with this layer of validation, we move on to more nuanced analyses of language-based partisanship: (a) of the overall court, (b) of individual justices, and n cases 6,663 justices 35 turns 756,018 average #words per turn 27.24 instances 104,498 republican 62,497 (59.8%) democratic 42,001 (40.2%) ## methodology data we use a subset of the super-scotus dataset @xcite , which contains transcripts of scotus oral arguments from 1955-2019 @xcite . 2 we filter out cases where hearings stretch over more than one year. we additionally remove non-linguistic indicators, e.g., @xcite , and mask all person names with a bert ner model foot_1 @xcite in order to focus our models on linguistic indicators rather than mentions of individuals with potential party affiliations. 4 we concatenate all turns from an individual justice in a single court hearing, to derive multiple instances with a maximum of 510 tokens and a sliding window of 255 tokens. to retain only informative instances, we further remove those that have less than 50 tokens. we finally randomly separate the resulting instances into 10 folds for cross-validation. partisanship and polarization are dynamic phenomena, responding to shifts in topics and political landscapes. we equip our models with a notion of time by adding a special [year] tag to each instance, flagging its year of origin. we obtain the reference party affiliation for each justice as the party (republican or democrat) of the president who nominated the justice, which correlates strongly with the self-reported party of the justice and has been shown to be associated with the interest of the nomination party @xcite @xcite . the statistics of the resulting dataset are shown in ## affiliated party prediction we first evaluate our framework intrinsically, asking to what extent do oral arguments reveal justices' party affiliation? by predicting justices' affiliated party from their court contributions. to further investigate the impact of temporal language shift @xcite on our oral partisanship prediction task, we train and test our model with a chronological data split. we split the data into non-overlapping temporal spans, with the training covering all cases from 1955-2000 (@xmath0, development set @xcite @xcite @xcite @xcite @xcite , @xmath1, @xmath2. the results in a natural question is whether our model indeed captures partisanship, or rather language idiosyncracies of individual justices. to test this, we use our trained models to predict speaker identities. specifically, we extract the final layer embedding of the [cls] token and train logistic regression (lr) and support vector machine (svm) models for speaker prediction (i.e., a 35-way justice classification task). we compare against off-the-shelf bert-base-cased embeddings, with no fine-tuning. we use instances with year tags as input, based on the best-performing setup in ## justices' language reflects their voting we next ask how does language-based partisanship align with established measures of votingbased ideology? to do so, we derive a partisanship score from our predicted party affiliation probabilities. we then relate our language-based ideology scores to established measures of ideology obtained from justices' voting behavior @xcite @xcite . while partisanship and ideology are not identical, a strong correlation exists, particularly in the two-party system of the united states @xcite @xcite . having shown that our bert models capture partisanship from court arguments (section 3) and with a grounded assumption that partisanship is reflected in (reasonably) local linguistic choice @xcite , we obtain a language partisanship score of a justice by averaging the predicted party probability for each of the justices' instances (section 2). for example, to obtain the oral partisanship score of a given justice in a given year, we average the predicted party probability of all related instances from the given justice in the given year. similarly, the language partisanship score for a case (year) is calculated by averaging the predicted party probability of all instances from all justices in that case (year), respectively. we use mq scores foot_6 (martin and quinn, 2002), a widely used and validated measure of the ideology of a court or individual justices, derived from voting outcomes. mq scores are estimated with a dynamic bayesian item response model @xcite , which infers the latent 'ideal point' of a justice, i.e., their ideological standpoint on a unidimensional scale (liberal -conservative) based on their observed voting behavior and a prior encouraging a smooth change in ideology over time. court-level partisanship figure 1 compares the overall scotus ideology based on voting behavior (mq scores, orange) and language (blue) over time, from 1955 to 2019. we observe a strong correlation of both measures across time (pearson's @xmath3.611, @xmath4.5e-8), indicating that partisanship in oral arguments reflects the voting behavior in court. similar observations have been made for more overtly-partisan domains such as political discourse and votes in the us congress @xcite . justice-level partisanship we further investigate language partisanship and voting ideology at the individual justice level. justices in recent years have shown clearer alignment with party affiliations in terms of their voting patterns @xcite , and we ask whether this is reflected in their language, too. we present a case study of the eight most recently appointed justices in our data set. 9 figure 2 shows that republicanaffiliated/conservative justices (roberts, saalito, gorsuch, kavanaugh) are consistently separated by the neutral (thick, dashed) line from the democrat/liberal justices (ginsburg, breyer, kagan, sotomayor). this holds for both language-based partisanship scores (solid lines) and mq scores (dashed lines). for democratic justices, mq scores and language partisanship are aligned in tendency over their tenure, with a minimum of pearson's @xmath5.6 across justices. we do not observe such strong evidence in republican justices, indicating that conservative justices project their values less directly in their speech. language-based partisanship estimates tend to be more extreme than voting ideology. notable examples are roberts (republican) and breyer (democrat), who is known to be a pragmatist whose decisions are often guided by real-life consequences regardless of party-lines, 10 possibly explaining the disparity between partisanship and mq scores. @xcite observed that, over their tenure, justices drift away from their first-year preferences, but with no certainty in what direction they will move. to study if the same observation holds 9 we excluded the most-recently appointed justice clarence thomas due to data sparsity.the same comparison for all justices is provided in appendix d. for language partisanship, we compare in figure 3 how our eight most recent justices' voting ideology and language partisanship have drifted between their first year of tenure (small dots) vs. their entire tenure to date (large dots). the error bars indicate the standard deviation of partisanship scores (mq on the x-and language-based on the y-axis) over all years of service. for most justices, the overall trend exceeds the sd interval, suggesting that it goes beyond the typical year-to-year fluctuation. as before, mq and language scores align well (justices clustering in the bottom left and top right quadrants). compared to their first year, democrat-appointed justices (blue) statistically significantly tend to become more liberal in both mq scores and language partisanship over their tenure. the relative shift in language partisanship for republican-appointed justices (red) is less pronounced, in terms of both language and mq scores. case-level partisanship in high-profile cases, voting behaviors have more distinctively lined up along party lines in the more recent years @xcite . correspondingly, we might expect the gap between the average language partisanship in democrat (blue) vs. republican-affiliated (red) justices to increase over time. we test this by looking at 14 high-profile cases between 1962 and 2014 @xcite . 11 as shown in figwe average the language partisanship of justices that were nominated by the same party, i.e. republican (red) or democrat (blue). dashed lines denote the estimated linear regression for the corresponding party. ## discussion partisanship as affiliation along the liberalconservative political spectrum is a fundamental axis in the political discourse. automatically identifying partisan language through nlp techniques enables large-scale analysis of political and public discourse and a better understanding of divisions and polarization. while prior work has mostly focused on explicitly partisan text, e.g., congress speeches @xcite or news outlets @xcite , we study supreme court language, which, by definition, should be politically neutral, and ask does the language of justices in court reveal their party affiliation? we proposed an analysis framework showing that bert-based classifiers can reliably predict the affiliated party of justices based on linguistic signals in their sco-tus arguments. in line with research which has shown a correlation between conversational content and final voting in political discourse @xcite @xcite , we further asked to what extent lan-guage partisanship aligns with voting ideology. we derived language-based partisanship scores from our validated models and compared them with mq scores, an established ideology measure derived from voting patterns. we showed a strong correlation between language partisanship and voting ideology at the overall court, individual justice, and important case level over time (section 4), indicating that the oral arguments of justices do encode their political leanings. moreover, our work reveals nuanced differences in linguistic and voting behaviors, e.g., their respective tendency shift over justices' tenure (figure 3 ), which further demonstrates the importance of studying scotus from various perspectives. more broadly, we test the extent to which language representations from large language models capture nuanced socio-cultural phenomena, by comparing predictions against corresponding behavioral data sets. here we focus on partisanship, building on related approaches for stance @xcite and counterfactual approaches to investigate the effect of social speaker attributes like gender and seniority on language use in the courtroom @xcite . our proposed framework enriches the partisanship and ideology analysis with an additional dimension to voting behavior, the spoken text, where we show a strong correlation between voting and spoken language. we hope that this could further spur analyses from other dimensions, such as partisanship of advocates @xcite , and amicus curiae @xcite , and take important parts of the legal process, e.g., further opinionwriting @xcite , into consideration.