Legal and Political Stance Detection of SCOTUS Language (2022)

Abstract: We analyze publicly available US Supreme Court documents using automated stance detection. In the first phase of our work, we investigate the extent to which the Court’s public-facing language is political. We propose and calculate two distinct ideology metrics of SCOTUS justices using oral argument transcripts. We then compare these language-based metrics to existing social scientific measures of the ideology of the Supreme Court and the public. Through this cross-disciplinary analysis, we find that justices who are more responsive to public opinion tend to express their ideology during oral arguments. This observation provides a new kind of evidence in favor of the attitudinal change hypothesis of Supreme Court justice behavior. As a natural extension of this political stance detection, we propose the more specialized task of legal stance detection with our new dataset SC-stance, which matches written opinions to legal questions. We find competitive performance on this dataset using language adapters trained on legal documents.

## introduction the relationship between the supreme court of the united states (scotus) and american public opinion is complicated. some scholars debate normative questions as to whether the court's power of judicial review ought to obey democratic principles foot_0 @xcite . others investigate how scotus behaves in relation to the public and why @xcite . prior work in the field of american political science has consistently demonstrated an association between the partisan ideology of the court, as expressed through its decisions, and that of the public, as recorded through poll data @xcite . however, more recent work, particularly in light of the 2022 dobbs v. jackson decision, suggests a departure from this general pattern @xcite . this change in institutional behavior has profound social significance which calls for academic attention. this paper heeds that call by providing a new analytical perspective on sco-tus' democratic tendencies. despite extensive research confirming scotus' general responsiveness to public opinion, the underlying reasoning for this relationship is disputed. one hypothesis centers on strategic behavior: it posits that the court consciously acts in accordance with the public will in order to protect its constitutionally fragile claim to the power of judicial review @xcite . alternatively, the attitudinal change hypothesis contends that broader socio-political forces such as news media present confounding factors that influence both the justices and the public @xcite . in this paper, we gain new insight into these hypotheses by applying automated stance detection to a newly assembled corpus of supreme court written opinions and oral arguments. stance detection (i.e., automatically identifying the position of an author towards a given target statement) allows us to evaluate the implications of a justice's language. we use stance detection and related techniques to build two different textual indicators of ideology which we call issue-specific stance (iss) and holistic political stance (hps) respectively. we compare these indicators to existing social scientific metrics related to general public opinion (i.e. the stimson policy mood; stimson, 2018), supreme court justice ideology (i.e. the martin-quinn score; @xcite , and supreme court case salience (i.e. the clark case salience; @xcite . in addition, we build a supervised stance detection dataset, sc-stance, over a subset of supreme court written opinions. our dataset matches the text of the written opinion to a corresponding le-d 1 = once the court starts looking to the currents of public opinion regarding a particular judgment, it enters a truly bottomless pit from which there is simply no extracting itself. (rehnquist, 1992) d 2 = will this institution survive the stench that this creates in the public perception that the constitution and its reading are just political acts? (sotomayor, 2022) @xmath0. the main contributions of this work are as follows. (1) using stance detection, we formulate two distinct ideology metrics (i.e. holistic political stance and issue-specific stance) for sco-tus justices serving from 1955 to 2020. we find that justices who are responsive to public opinion tend to use language which correlates ideologically with their voting behavior. this provides new evidence in favor of the attitudinal change hypothesis. the repository of relevant code is publicly available through the following link: https://github.com/njbergam/scotus-public-stance . a common thread in many prior studies is the focus on court voting behavior or its reception in the public eye. in contrast, our work investigates how scotus presents its politics through its language. this approach takes advantage of the fact that the corpus of official scotus language is publicly available, relatively small, and well-structured. previous work in various fields demonstrates that there are concrete differences between the language used by people of different political ideologies. in psycholinguistics, @xcite suggests that the language of liberals tends to emphasize mental concepts, while that of conservatives uses more references to the body. nlp research has further investigated this concept through political ideology detection on two datasets @xcite : convote (i.e. congressional dialogue labeled with the political affiliation of the speaker) @xcite , and the ideological books corpus (i.e. sentences from political articles and books annotated for political cues) @xcite . ## sc-stance dataset 4.1 methods we describe the collection and characteristics of our new stance dataset, sc-stance, as well as the methods we apply to it. our dataset sc-stance was drawn from three sources: a dataset of full-text supreme court opinions through 2020 @xcite , the washing-ton university supreme court database @xcite , and the oyez website (urofsky, 2001). we started by collecting written opinions which had non-neutral holdings, as encoded in the sc database. we then automatically matched these opinion texts to the key legal question on the oyez website to obtain text-target pairs. since the questions on oyez are always phrased such that an affirmative answer is in favor of the petitioner, we used the winning party label foot_6 from the supreme court database, as well as the opinion type given in the kaggle dataset (i.e. majority, concurring, dissenting, etc.) to infer the stance that a given written opinion takes towards the legal question (e.g. if the winning party was the respondent, and the opinion type was dissenting, then the opinion affirms the legal question). the final dataset has 2708 labeled instances (1179 labeled pro, 930 labeled con). the average length of a target (i.e., the legal question) is 35 tokens and the average length of a text (i.e., the supreme court written opinion) is 5330 tokens. we show an example datapoint in in addition to providing a legal stance detection task, our dataset could provide an interesting passage retrieval task. most other legal information retrieval datasets map documents to other docu-case: school district of abington township v. schempp (id 1962-148), majority opinion. target: did the pennsylvania law requiring public school students to participate in classroom religious exercises violate the religious freedom of students as protected by the first and fourteenth amendments? text: once again we are called upon to consider the scope of the provision of the first amendment to the united states constitution which declares that "congress shall make no law respecting an establishment of religion, or prohibiting the free exercise thereof" [...] in light of the history of the first amendment and of our cases interpreting and applying its requirements, we hold that the practices at issue and the laws requiring them are unconstitutional under the establishment clause, as applied to the states through the fourteenth amendment. [...] label: pro (text affirms the target) ## conclusion using state-of-the-art nlp techniques, we gain new insight into a longstanding political science problem: the supreme court's relationship with public opinion. in our analysis of the language of supreme court justices, we leverage existing metrics of scotus behavior as well as stance detection datasets regarding political ideology. notably, we find a new source of evidence for the attitudinal change hypothesis of the supreme court, and we experiment with a competitive new model for legal language domain adaptation. this research sheds light on how stance detection allows us to interrogate the implicit opinions of static documents. this is a powerful use case of nlp for the social sciences, in that it allows for a large-scale, critical analysis of large bodies of text. of course, there is a long way to go in the field of stance detection, both generally and in specific linguistic domains such as the law. our contribution of sc-stance feeds into this goal, by providing semantically rich targets and a mix of legal and lay language. we emphasize this latter feature, in that quality textual understanding -for human and ai alike -is marked by a thorough comprehension of both colloquial and technical language formulation.