G er D a LIR : A G erman Dataset for Legal Information Retrieval (2021)

Abstract: We present GerDaLIR, a German Dataset for Legal Information Retrieval based on case documents from the open legal information platform Open Legal Data. The dataset consists of 123K queries, each labelled with at least one relevant document in a collection of 131K case documents. We conduct several baseline experiments including BM25 and a state-of-the-art neural re-ranker. With our dataset, we aim to provide a standardized benchmark for German LIR and promote open research in this area. Beyond that, our dataset comprises sufficient training data to be used as a downstream task for German or multilingual language models.

## introduction there are few non-english datasets dedicated to natural legal language processing (nllp) or legal information retrieval (lir). to our knowledge, not a single dataset exists that provides a standardized benchmark for lir models on the german language. to this end we contribute gerdalir, a legal document retrieval dataset comprising a large document collection and corresponding queries forming a document ranking task. we provide a large amount of training data such that both unsupervised and supervised methods can be benchmarked. this also enables gerdalir to be used as a downstream task for german or multilingual language models. the task provided is a precedent retrieval task. as illustrated in figure 1 , we build gerdalir by extracting passages that reference other cases. for that we utilize 201,825 cases from the open legal information platform open legal data @xcite . we present baseline experiments on classic term-based retrieval methods, a semantic search approach based on word embeddings and a transformer-based re-ranker giving an orientation to other researchers using our dataset. in contrast to other lir datasets based on precedent case retrieval, gerdalir offers the following unique characteristics: large corpus size. with a total of 144k relevance labels for 123k query passages and a collection of 131k documents comprising over 3m passages, gerdalir is -to our knowledge -bigger than any other lir dataset. its size enables gerdalir to be used for full-ranking evaluation. german language. the german language is gerdalir's most prominent feature and fills a gap in the community. furthermore, in combination with the large training set, gerdalir can be used as a downstream task for german or multilingual language models, of which there are quite few. query passages. most other lir datasets based on precedent retrieval provide entire case documents to be used as queries. it may be unclear to which part of a given case relevant cases should be retrieved. as gerdalir provides query passages rather than whole documents, it better reflects a practical use case. the download links and descriptions to the format of gerdalir can be accessed via github foot_0 . ## other datasets there are several sources of lir datasets and tasks. in the following, we outline those based on precedent retrieval. coliee 2020 task 1. the competition on legal information extraction/entailment (coliee) is a workshop that annually provides a number of tasks in the areas of legal document retrieval, question answering and entailment. coliee 2020 task 1 @xcite ) is a re-ranking task that provides for training: "520 base cases, each with 200 candidate cases from which the participants must identify those that should be noticed with respect to the base case". from the total of 104,000 candidates, 2,680 are labelled as positive. for testing they provide 130 base cases, a total of 26,000 candidates and 646 positive labels. the case documents are written in english and originate from the federal court of canada. the workshop also provides tasks derived from the japanese jurisdiction including original japanese texts as well as english translations. sigmalaw. this dataset originates from the paper legal document retrieval using document vector embeddings and deep learning @xcite . it comprises 2,500 case documents and a citation graph indicating for each case which of the other cases are considered relevant. fire 2017 irled. the fire 2017 irled track presents a precedence retrieval task comprising 200 query cases, 2000 collection cases and 1000 positive relevance labels. the provided case documents originate from the indian supreme court, which uses the english language in their proceedings. ## dataset generation gerdalir is based on parsed references in german case documents taken from open legal data. although precedent cases have no binding effect in the german law system, references to prior cases are very common and arguably play a big role in supporting a line of argument. by definition, such referenced cases are relevant to the case at hand. with this in mind, the idea behind gerdalir's task is simple: passages containing one or more references to known cases become queries while the referenced cases are labelled as relevant. however, if a passage is used as a query, the document the passage originates from should not be used as an retrievable collection document. to achieve that, we classify case documents -as depicted in figure 1 -into the following classes: unknown documents belong to cases that we have seen references to, but are not part of open legal data. collection documents comprise the cases that will be indexed labels 115 , 360 14 , 570 14 , 394 for the retrieval task, which mainly consist of those that only refer to unknown documents. query documents are the documents from which queries are sampled. assigned to these are cases that contain references to collection documents. if a case document refers to other query cases, but not to collection cases, it is also classified as a collection document. the case documents are divided into passages along margin numbers. it regularly happens that the references to a passage follow with a margin number. those passages typically start with vgl. ("compare") or siehe ("see"). we use these and more indicator words in the beginning of a passage to detect such referential passages and assign their references to the previous passage, which is assumed to contain the corresponding statement or line of argument. the text describing the references, however, is not added to the passage, since we want models to rely on natural language rather than exploiting references or parts of them. for this reason, we attempt to replace any reference including those to statutes with a [ref] token. however, a small portion of references that we were unable to parse remain in the text. from the final text, we also remove any braced content, since they mostly contain comprehensively described references that are difficult to sanitize otherwise. after that we collect all passages from the query documents that are marked with references to one or more collection documents. these form a set of multi-sentence queries, each with at least one label to a relevant collection document. finally, we perform a 0.8/0.1/0.1 split on the queries for training, development and testing respectively. the resulting size of gerdalir's collection, the queries and the labels are summarized in ## baseline methods we conduct a series of baseline experiments demonstrating that gerdalir can be used to benchmark retrieval methods or to evaluate the language mod-els used by them. the resulting measures also serve as an orientation to other researchers using the dataset. the methods considered are described below. ## experiments in this section, we outline the experimental setup, describe the implementation of the methods described above, and briefly discuss the results. ## conclusion & future work we present gerdalir, a dataset filling the gap of a standardized ir benchmark for the german legal domain. we provide several baselines with which other researchers can compare their results. our experiments demonstrate the use of gerdalir as a downstream task for german or multilingual language models. in future work, we plan to investigate the importance of in-domain pre-training to neural lir models. we also intend to explore unsupervised methods that effectively leverage language models for domain-specific information retrieval, as well as approaches combining these with traditional term-based retrieval methods.