Automated Extraction of Sentencing Decisions from Court Cases in the H ebrew Language (2021)

Abstract: We present the task of Automated Punishment Extraction (APE) in sentencing decisions from criminal court cases in Hebrew. Addressing APE will enable the identification of sentencing patterns and constitute an important stepping stone for many follow up legal NLP applications in Hebrew, including the prediction of sentencing decisions. We curate a dataset of sexual assault sentencing decisions and a manually-annotated evaluation dataset, and implement rule-based and supervised models. We find that while supervised models can identify the sentence containing the punishment with good accuracy, rule-based approaches outperform them on the full APE task. We conclude by presenting a first analysis of sentencing patterns in our dataset and analyze common models’ errors, indicating avenues for future work, such as distinguishing between probation and actual imprisonment punishment. We will make all our resources available upon request, including data, annotation, and first benchmark models.

## introduction the legal world is rife with data, from constitu tions and national legislation to legal cases and court decisions. much of the legal data, however, comes in unstructured formats that pose critical challenges for extracting and analyzing it in sys tematic ways. in addition, different countries vary in their legal systems, norms and conventions, fur ther compounding the challenges in developing multilingual approaches @xcite . while legal nlp is gaining traction in recent years @xcite , relatively little attention has been given to lowresource settings outside of the english language, where the availability of tools * equal contribution. in this work, conducted as part of an ongoing collaboration with the association of rape cri sis centers in israel (arcci), we focus specif ically on the task of automated punishment ex traction (ape) in sexual assault cases in hebrew within israeli court sentencing decisions (see for mal task definition in section 2). punishment deci sions are of special importance as they constitute a prerequisite for many other downstream tasks in legal nlp and digital humanities, such as le gal prediction of judicial decisions @xcite and detecting biases in court decisions @xcite . ape is dif ficult in the israeli court system. this is due to the fact that sentencing decisions for criminal of fences are reported, in natural language idiomatic to the legal field, in the written sentencing deci sion. we focus on sexual assault cases due to the legal and public debate around claims of lenient punishments @xcite , that in the absence of systematic rigorous data collec tion cannot be empirically examined and assessed. this worldwide debate requires legal nlp meth ods in multiple languages and legal systems. 1 to address this challenge, we begin by curat ing a dataset of sexual assault sentencing decisions from the years 19902021 and manually annotate punishment in a subset of 100 cases with the use of legal experts in our team and in collaboration with arcci (section 3). following, in section 4, we use this data to build several models for the ape task, including rulebased and supervised meth ods, based on linguistically and semantically in formed features, setting first benchmark results on the ape task in hebrew. we thoroughly analyze our models' performance in section 5, finding that they are capable of extracting the correct punish ment in 68% of the cases,while the best model's av erage error is roughly 5 months, attesting to the dif ficulty of the task. based on our models, we find that in our data the median predicted punishment is 3 years, while more than a third of the punish ments are below 15 months. although these fig ures are obtained on a mediumsize corpus, using automatic measures which do not account for the type of offense, we note that they are well below the maximum punishments for sexual offenses as determined by the israeli legislator, which range between 27 years for indecent acts and sodomy and up to 20 years for aggravated rape. we conclude by analyzing common error pat terns in our models. for example, we find that models often tend to erroneously extract a proba tion imprisonment punishment instead of the ac tual imprisonment punishment. distinguishing be tween the two is left as an interesting avenue for future work. to the best of our knowledge, this is the first examination of automatic punishment extraction in the hebrew language. it includes data collection, annotation, and benchmark mod els. we hope it will spur further research into this important task. ## task definition we define the task of ape as the process of auto matically extracting the punishment from the sen tencing decision. in the israeli legal system, the punishment is given in a separate sentencing de cision, following a plea bargain or a guilty ver dict. in the sentencing decision, the court can impose different types of punishment: imprison ment, probation, or community service. in addi tion, the court can also impose fines and order the defendant to pay restitution to the victim. we consider all of the punitive elements mentioned as part of the ape process. however, in this work, we focus on the extraction of the actual imprison ment (i.e. jail time). given the text of a sentenc ing decision, we first need to distinguish between the different types of punishment (imprisonment, probation, community work, fines, etc.); then, we need to extract only the sentence that relates the duration of the actual imprisonment penalty, i.e. the number of months or years in prison imposed on the defendant. this is particularly challeng ing since the court decision often includes both the duration of the actual imprisonment, as well as the duration of the conditional imprisonment (i.e. probation). both are referred to in hebrew using the same term ``ma'asar'' (lit. imprisonment), and indicated by the same units of months and years. for example (translation and emphasis by the au thors): ``we impose on the defendant the fol lowing punishment: 48 months im prisonment, of which the defendant will serve 30 months actual imprison ment and the rest, 18 months, will be conditional imprisonment...(crimc 1124/04)''. in this case, the ape task is to extract ``30 months'' as the the actual imprisonment punish ment. this also exemplifies the typical linguistic difficulty of the task the noun ``imprisonment'' repeats three times, referring first to the total pun ishment imposed, then to the actual imprisonment, and then to the probation. ## a corpus of annotated sentencing decisions in israeli law this section describes the construction of our cor pus, which to the best of our knowledge is the first annotated legal corpus of sentencing decisions in sexual assault cases in hebrew. ֿ in section 3.1 we discuss the cases comprising the corpus, consist ing of 30 years of sentencing decisions in sexual offense cases, and in section 3.2 we present the manual annotation schema of the different types of punishment and the duration (in months and years) of the actual or conditional imprisonment which the courts imposed in these cases. ## models we present several models for predicting the im prisonment incurred in free text sentencing deci sions. the highlevel approach, depicted in fig ure 2 , is composed of two steps, following the hu man annotation process, described in the previous section. first, we identify sentences containing the imprisonment punishment (section 4.1), from which we extract the term itself, and normalize to number of imprisonment months (section 4.2). ## evaluation in this section we analyze the performance of the models described in section 4 in the different model recall precision f1 svm 0.9 0.86 0.88 rf 1 0.8 0.89 the supervised models present high accuracy in the sentence identification task. both super vised models in the supervised models' probability is not wellcalibrated. the ape task requires choosing one sentence from which the number of months of ac tual imprisonment is extracted in the next stage. this means that for each false positive sentence there is exactly one equivalent false negative. hence, in our case precision equals recall. in the rulebased approach we noticed that a differ ent threshold applies for each legal case, hence we scored them separately. our assumption was that the supervised models would score similar features to those used in the rulebased in a more accurate way. thus we attempted a similar local threshold approach, by performing argmax on the learner probability of all sentences from the same legal case, as defined in equation 1. however this was not the case, as observed in all models tend to confuse probation with actual imprisonment. error analysis in other error patterns. rulebased and rf er rors are similar, containing mostly references from past cases. this type of error includes sen tencing decisions either of similar crimes or past cases of the defendant (see inter-annotator agreement reveals the limitations of the sentence-level approach. we asked legal experts to evaluate the rulebased perfor mance by rating each sentence that the rulebased model predicted as whether it reflects the actual imprisonment or not. this is the same task the models were required to perform. we used co hen's kappa to measure the inter annotators agree ment for each pair, presented at error type % of examined cases example (english translation) rb svm rf probation 36% 37.5% 65% in addition, 18 months probation. ref to prev. case 46% -18% sentenced to 12 months actual imprisonment. fines -27.5% -a fine of 5,000 nis or 30 days imprisonment instead. procedural -17.5% -imprisonment starts at 31. there is room for improvement in extracting the actual imprisonment sentence. a post-hoc summary of the legal decision improves performance. nevo, the legal database we use, provides a ``mini ratio'', a posthoc sum mary of each decision in a few sentences, writ ten by nevo's editorial team. when we add this mini ratio to the annotation, it increases the super vised models' ability to extract the target sentence by about 50%, showing that shorter inputs lead to better generalization on smallscale datasets. identifying sentencing patterns in sexual assault cases in israel. using our best perform ing model, we present rough statistics regarding the punishments given in the past thirty years in figure 3 . the median sentenced punishment throughout all legal decisions of our data is 36 months, however we observe most commonly punishments are under a year. while the sen tencing decisions are generally available in legal search engines, annotating them is an expensive process. for this reason many statistical observa tions are hard to obtain. this demonstrates the po tential contribution of our task from a sociolegal point of view. most related to our work are those that bring to gether domain expertise with ml models to ex tract information from specialized texts. this is the case for @xcite who showed that con ditional random fields perform better than dnn for sentence border detection for the legal domain. while this is considered a closed problem in nlp they showed that this is not the case for legal texts. similar paradigms can be found in the medical field, where domain knowledge also plays a cru cial role. @xcite show that in some cases a rulebased approach achieves better per formances than svm. chalkidis et al. ( 2020 ) re cently introduced a language model finetuned su perficially for the english legal domain. taking a similar approach for the ape task is an interesting avenue for future work. ## conclusions in this work we created the first annotated cor pus of hebrew language sentencing decisions, fo cusing on sexual assaults. we compared a rule based approach with supervised learners using the unique attributes of the legal language for repre senting sentences. we found that the rulebased approach achieved best results with an average er ror rate of 5 months and accuracy of 68% in ex tracting the punishment sentence. our analysis shows that such research could focus on fine tun ing of the supervised models. while supervised learning models help us narrow down a full legal document to 2 5 sentences that include the pun ishment, further research can contribute in reach ing a single target sentence, which could also ben efit from our error analysis, especially regarding the probation sentences, perhaps targeting them separately in a prior task and using them as fea tures.