Summarization of G erman Court Rulings (2021)

Abstract: Historically speaking, the German legal language is widely neglected in NLP research, especially in summarization systems, as most of them are based on English newspaper articles. In this paper, we propose the task of automatic summarization of German court rulings. Due to their complexity and length, it is of critical importance that legal practitioners can quickly identify the content of a verdict and thus be able to decide on the relevance for a given legal case. To tackle this problem, we introduce a new dataset consisting of 100k German judgments with short summaries. Our dataset has the highest compression ratio among the most common summarization datasets. German court rulings contain much structural information, so we create a pre-processing pipeline tailored explicitly to the German legal domain. Additionally, we implement multiple extractive as well as abstractive summarization systems and build a wide variety of baseline models. Our best model achieves a ROUGE-1 score of 30.50. Therefore with this work, we are laying the crucial groundwork for further research on German summarization systems.

## introduction most modern summarization systems are built around a handful of datasets predominantly collected from newspapers. in recent years many researchers saw the need to introduce different datasets to the summarization domain such as arxiv/pubmed @xcite , xsum @xcite , or datasets from the legal domain like bigpatent @xcite and billsum @xcite . as the performance of data-driven systems is highly impacted by the quality and size of the dataset used, we introduce a new dataset for the german legal domain, more specifically a collection of court rulings. ## related work the literature differentiates two base approaches to text summarization: (1) extractive and (2) abstractive summarization. in the extractive case, sentences or tokens are copied and extracted from the given text and form a summarization. contrary, the abstractive summarization system is creating a summary word by word, thus it is not dependent on the words and information used in the verdict. this is why abstractive summarization is often referred to as natural language generation (nlg). most commonly, extractive summarization systems use a hierarchical document representation, i.e. words are combined into sentences, sentences are aggregated into documents. various different architectures with convolutional neural networks (cnn), recurrent neural networks (rnn), and attention mechanisms @xcite @xcite are used to extract the most important sentences. more recent approaches focus on transformer architectures @xcite @xcite and investigate the level of extraction @xcite @xcite ) by selecting words or n-grams instead of whole sentences. the most common way to tackle abstractive summarization is to use an encoder-decoder structure. hereby the encoder is aggregating the information from the text and the decoder is a conditional language model producing the words in the summary. hierarchical document representations are again important and similar modeling approaches are used @xcite @xcite . in abstractive summarization, there is also a shift towards transformers in recent years @xcite @xcite @xcite . many different techniques were found to be useful for summarization. pointer networks @xcite allow the selection of words from the input text and are used in extractive @xcite as well as abstractive summarization @xcite . some papers use templates or salient sentences in the word generation process @xcite @xcite . others try to discourage the system from repeating certain phrases by introducing special loss function such as coverage @xcite or novelty @xcite . especially long documents are challenging and thus multiple approaches try to tackle this problem e.g. by introducing multiple encoders to find a hierarchical representation of the text @xcite or introducing the discourse structure into a hierarchical attention model @xcite . ## pre-processing given the json format described above, we build a pre-processing pipeline specifically tailored to the summarization of german verdicts. sentences from the guiding principle, facts, and reasoning are first split into tokens on spaces and dots. to reduce the overall complexity, each token is then converted to lower case in order to decrease the number of unique tokens. this is mainly to remove possible mixed-cased words at the beginning of sentences as the meaning of a word is in our case clear given its context. afterwards norm placeholders in the text are replaced with a @xmath0, numeric tokens @xmath1, "..", etc.) @xmath2, to infer some semantic meaning for them via word embeddings. then any tokens within parentheses were removed, as in most cases these only contain references to other legal documents. afterward, all special characters and any enumeration symbols were removed, as german verdicts contain a lot of structural information. finally, each token is then mapped to a unique identifier. after all those processing steps, the whole dataset contains around one million unique tokens, thus we only use tokens that appear at least 100 times in the corpus resulting in a total number of tokens around 50.000. although this vocabulary size seems rather small, it is comparable to other methods @xcite and was necessary to reduce the hardware requirements for the neural networks. all tokens which were not selected are replaced by @xmath3. subword-based tokenization would be one option to removing unknown tokens, but we decided against it as only 2.7% of the text is replaced @xmath4. we pre-trained glove embeddings @xcite on those tokens. ## method the basic building blocks of our models, namely cnns, rnns, and the attention mechanisms can be defined by their transformations on a sequence of embeddings @xmath5, . . . , x n ; x @xmath6. for convenience, given a matrix @xmath7,: as the i-th row and m :,j as the j-th column vector. the general structure of both model types can be seen in figure 1 and will be further explained in the respective subsections. formally, a 1-dimensional cnn transforms a sequence @xmath8. we transform a sequence @xmath9: in this case σ is the sigmoid function and the hadamard product. w r , u r , w z , u z , w, u, b denote learnable parameters (definition as the py-torch implementation). the attention mechanism @xcite ) is defined as the weighted sum over the embeddings x 1 , . . . , x @xmath10: in those formulations @xmath11. ## performance analysis to evaluate a given system, we use the rouge metric @xcite which measures the percentage of n-gram overlap between a gold label summarization v @xmath12, . . . , w n ) and a created text @xmath13, . . . , c m ) with w i : @xmath14, . . . , n} and c i : @xmath15, . . . , m} being the words in the respective texts 3 . in when comparing the performance numbers with those reported at nlpprogress.com, we see the long historic evolution for summarization methods. the legal domain and especially the german language are under-researched in the summarization domain. a lot of work, such as creating appropriate language models is necessary to apply those methods to the german language. with certainty, we can say that there is a necessity to evaluate summarization models on diverse datasets. ## conclusion & future work in this paper, we introduced a new dataset for the summarization of german court rulings. we cre-ated multiple models specifically tailored to the german legal domain and tested strong baseline models which we want to improve in the future. nonetheless, there are still extensions and shortcomings which we want to address in future work. the vocabulary size needs to increase as german is a more morphologically rich language than english. it might be necessary to find more efficient ways to represent and generate compound nouns, the different verb forms, etc. as they drastically increase the size of the necessary vocabulary. consequently, the model complexity needs to be increased. this does not only include ways to better model the german language but also taking the structure of the verdicts more into account. court rulings have a more sophisticated structure which we did not entirely utilize in this research paper, i.e. each of the segments is again subdivided into other sub-segments and so on. embracing this more hierarchical view could make access to information easier and improve the understanding of the source document. in this research, we laid down the necessary groundwork for further research. in the future, we want to do more qualitative analysis, by letting legal experts evaluate our summarization models. summarization is far from being solved and in our opinion, it is important to introduce different texts and domains to summarization research.