Super- SCOTUS : A multi-sourced dataset for the S upreme C ourt of the US (2023)

Abstract: Given the complexity of the judiciary in the US Supreme Court, various procedures, along with various resources, contribute to the court system. However, most research focuses on a limited set of resources, e.g., court opinions or oral arguments, for analyzing a specific perspective in court, e.g., partisanship or voting. To gain a fuller understanding of these perspectives in the legal system of the US Supreme Court, a more comprehensive dataset, connecting different sources in different phases of the court procedure, is needed. To address this gap, we present a multi-sourced dataset for the Supreme Court, comprising court resources from different procedural phases, connecting language documents with extensive metadata. We showcase its utility through a case study on how different court documents reveal the decision direction (conservative vs. liberal) of the cases. We analyze performance differences across three protected attributes, indicating that different court resources encode different biases, and reinforcing that considering various resources provides a fuller picture of the court procedures. We further discuss how our dataset can contribute to future research directions.

## introduction with the increasing attention to legal text processing, recent research has proposed various legal corpora, covering different sources, e.g., court documents @xcite @xcite @xcite or legal contracts @xcite @xcite , and tackling legal analysis with a diverse set of tasks, e.g., text classification @xcite , juridical output prediction @xcite , 2018; @xcite and case summarization @xcite . in the supreme court of the us (scotus), several procedures contribute to the court results and reasoning process @xcite . as illustrated in figure 1 , for instance, a writ of certiorari is needed for the petition of the appeal to the scotus. merits briefs in which petitioners and respondents lay out their arguments to the court, are required before the oral arguments take place, which in turn can clarify or elaborate on points made in the briefs. justices further discuss and decide the case, and afterwards issue opinions which explain the reasoning behind the final judgment. we present a comprehensive data set of scotus proceedings and meta data to enable holistic language analysis of the court. various scotus resources have been gathered from different phases of the ruling procedure and formulated in diverse tasks in order to analyze different perspectives of the court, e.g., voting @xcite @xcite , partisanship @xcite and topic prediction @xcite . however, each work comprises only a limited set of resources with a specific focus on one particular court perspective. in light of this observation, we take a step forward and create a multi-sourced scotus dataset, i,e., super-scotus (section 3). figure 1 illustrates the scope and structure of our dataset. we focus on the phases of the court hearing and post hearing, augmented with relevant metadata on the cases and justices. 2 our corpus connects publiclyavailable resources including oral arguments and various post-hearing annotations and summaries, including opinions and case summaries. to showcase the utility of our data set, we present a case study in section 4 where we ask how do different court documents reflect the direction of the final court decision (as conservative or liberal). we predict the decision direction from input documents from the phases of the court hearing (oral arguments) and post-hearing (syllabus). we consider three sensitive case attributes, derived from the meta data of our corpus, to analyse group disparity of our classifiers with respect to (1) issue area, (2) vote distribution, and (3) winning side. we observe that, compared to oral arguments, case direction is easier to decode from the syllabus. the group disparity across three protected attributes indicates that different court materials might encode different biases, suggesting the necessity of considering various resources for the full analysis of the supreme court. we finally point out potential directions and use cases for future work (section 5). in summary, our contributions are (1) a publicly available multi-sourced scotus dataset, comprising various court information from different procedure phases. (2) a case study, addressing the research question of how do the court materials reflect the case direction, overall and wrt. discrepancy for three protected attributes (3) a summary of the different layers of legal texts in the supreme court and a discussion for future research directions building on our dataset. ## related work the judiciary in court, as the base of the legal system, has been one of the key focuses in the legal domain @xcite @xcite @xcite . for the analysis of scotus, although various corpora exist, each corpus consists of a limited fraction of the court resources and analyzes a specific perspective of the court, e.g., topic prediction on court opinions @xcite . the lexglue corpus @xcite is a benchmark dataset for legal language understanding and formulates the scotus resource as a topic classification task. specifically, the scotus partition includes the opinion of the case, foot_1 and annotates the opinions with the issue area (topic), obtained from the supreme court database (scdb) @xcite , described in section 3. furthermore, @xcite considered the same scotus task but with a focus on the fairness of classification models. they investigated various debiasing methods in the context of the legal domain. however, their fairness study of scotus was limited to only one resource: opinion text. to enrich the utilization of conversational data, expanding from danescu- @xcite , convokit @xcite ) contains the transcripts of oral arguments of scotus cases between 1955 to 2019. 4 this partition is also included as one of the sources in our super-scotus dataset (section 3). the sc-stance dataset @xcite combines the scotus case questions annotated by the oyez website and the corresponding scotus opinion from a kaggle dataset @xcite to predict the political stands in court. they derived and labeled the political stands of the questions and opinions based on the winning side of cases from scdb, i,e., favoring petitioners or respondents. @xcite proposed a large legal corpus, called pile of law, with approx. 256g (growing) legal and administrative text which concentrates on the us legal system, with the aim to provide a comprehensive corpus for legal text without containing toxic or private content. this corpus also includes scotus opinions and the transcripts of oral arguments. @xcite annotated a dataset of approx. 436k us court opinions, including supreme and federal courts, with key passages and summaries. however, their dataset is not publicly available. as we discussed, current research only focuses on a limited partition of the scotus resources. a more comprehensive dataset that covers various court resources is needed in order to understand how various court materials from different phases contribute to and reflect the supreme court. to address this gap, we provide a publicly-available multi-sourced scotus dataset, connecting vari-ous resources in different phases of the court procedure and enriching the dataset with additional annotations from the post-hearing phase. ## the super-scotus dataset as shown in figure 1 , the court system comprises several stages. since most pre-hearing documents are not publicly available, our dataset includes the resources from the court hearing and post-hearing, listed in section 3.1. we detail our dataset construction in section 3.2. ## decision direction prediction from diverse sources as described in section 2, most nlp research focuses on only a single type of document (predominantly opinion documents @xcite ) and typically a single task, e.g., the classification of the overall case topic @xcite . here, we (a) propose a novel task of decision direction prediction which encodes information about case topic, justices' voting, and political leaning; and (b) consider this task across documents from the 'court hearing' and the 'post hearing' phase. finally, we leverage the rich meta-data of super-scotus to study biases in the respective sources. to demonstrate the utility of super-scotus, we study how different court materials reflect the decision direction of a case. decision direction is coded as a binary variable as conservative or liberal by legal experts for the scdb. both the case outcome and the issue area (case topic) are taken into account during this labeling. for instance, in the issue area of criminal procedure, a liberal label could indicate that the case outcome is 'pro civil liberties' or 'pro-underdog', and vice versa for conservative. for cases related to economy, a liberal label could indicate 'pro administrative action. 13 4.1 method data we consider two types of super-scotus documents, namely the oral arguments from the court hearing phase (oa), and the syllabus from post-hearing (sl), and predict case decision direction. we remove cases which (i) do not have labels for voting result, decision direction, and issue area; (ii) were discussed in more than one oral argument session; (iii) lack the syllabus or opinion; or (iv) include companion cases, where multiple cases are jointly discussed. the statistics of the result-ing dataset is shown in ## conclusion and future vision we presented a publicly-available multi-sourced dataset for the analysis of the us supreme court, namely the super-scotus dataset. specifically, we focus on various resources from the phases of the court hearing and post hearing, connecting court-released data, both text documents (opinions, syllabus) and structured data (voting outcomes, issue areas), and enriching it with various summaries from different legal platforms, including key questions/conclusions of the court from oyez and landmark case summaries from wikipedia. our dataset supports a variety novel of nlp tasks (section 5.1), where the empirical experiments in this paper only scratched the surface. to demonstrate the utility of super-scotus, we propose the challenging novel task of decision direction prediction, where labels encode the ideological direction most aligned with the case outcome (liberal vs conservative). we presented a case study using different court materials as input, namely oral arguments and syllabus, and analyzed performance discrepancies of the best resulting models regarding three attribute groups. we observed that model performance varies with different inputs. the group disparity analysis further shows the performance difference across attributes, indicating that different biases exist in models trained on different court documents and suggesting the importance of considering various sources in the analysis of the court.