2 The Data Revolution in Justice 27 examine how legally relevant and legally irrelevant factors affect legal outcomes. For example, Elliott Ash and Daniel L. Chen (2018) uses judges’ writings to predict the average harshness and racial and sex disparities in sentencing decisions. That work finds that the information contained in written opinions can improve significantly on naive prediction of punitiveness and disparity.

This information could be used to aid decision makers in ways that reduce bias in the system. For example, training programs could be targeted toward biased judges, either with the goal of debiasing or to help them learn how to use the hearing process to better advantage. Simply alerting judges to the fact that their behavior is highly predictable in ways that may indicate unfairness may also be sufficient to change their behavior. Informing judges about the predictions made by a model decision maker could help reduce judge-level variation and arbitrariness. Potential biases that have been identified in prior decisions or writing could be brought to a judge’s attention, where they could be subjected to higher order cognitive scrutiny.

Thus, data not only makes it possible to measure and understand judicial bias, but also provides different avenues to address these biases and reduce existing discrimination. Leveraging such opportunities may be essential to ensure that not only de jure equality exists, but also that there is de facto equality in the interpretation and application of the law.

# **Applications of Text as Data**

Recent innovations have generated new opportunities for empirical research on the delivery of justice. Court proceedings and rulings are now increasingly digitized, allowing the construction of large-scale data sets. The increase in text availability is particularly promising in the legal field, where legal documents are meticulously documented and play an essential role in judicial decisions. Additionally, computer scientists have developed a slate of machine learning tools that can produce interpretable data from unstructured text–including written judicial opinions–making it possible to analyze a quantity of text that would be far too large for humans to read.

The question of how to analyze texts has gained importance in social science research in recent years (Gentzkow, Kelly, and Taddy, 2017). In the past, the most common approach was qualitative, with either a deep reading of the text or a subjective coding of important themes (see Glaser, Strauss, and Strauss (2017) for an example of the latter approach). However, these approaches lack a rigorous method of replication (Ricoeur, 1981; DiMaggio, 1997) and more formal methods to analyze texts have been developed (Andrade, 1995; Mohr, 1998). Topic modeling discovers underlying topics and themes through an inductive method (Blei, Ng, and Jordan, 2003; Blei, 2012; DiMaggio et al., 2013; Mohr and Bogdanov, 2013). Another family of models learns the features of text that are predictive of some outcome, such as political ideology (Gentzkow, Kelly, and Taddy, 2017; Jelveh, Kogut, and Naidu, 2014; Elliott Ash, 2016; Elliott Ash, Morelli, and Van Weelden, 2015; Osnabr AŒgge,<sup> ̃</sup> Elliott Ash, and Morelli, 2020).

Recent approaches have gone beyond the traditional network or topic methods by mapping word relations into a high-dimensional vector space (Mikolov et al., 2013; Pennington, Socher, and Manning, 2014). This approach positions connected words close to each other in the space and can be used to recover relevant dimensions in language. A rich literature in computational social science has begun to apply these methods in many contexts (Spirling and Rodriguez, 2020).

to legal documents (Carlson, Livermore, and Rockmore, 2015; Leibon et al., 2018). Ganglmair and Wardlaw (2017) apply a topic model to debt contracts; Elliot Ash, MacLeod, and Naidu (2019) implement a syntactic parser to extract legal commitments and entitlements from union contracts. Daniel L Chen and Elliot Ash (2019) construct document embeddings for federal courts and show they recover differences between courts, over time, and across topics.