Searching for Legal Documents at Paragraph Level: Automating Label Generation and Use of an Extended Attention Mask for Boosting Neural Models of Semantic Similarity (2021)

Abstract: Searching for legal documents is a specialized Information Retrieval task that is relevant for expert users (lawyers and their assistants) and for non-expert users. By searching previous court decisions (cases), a user can better prepare the legal reasoning of a new case. Being able to search using a natural language text snippet instead of a more artificial query could help to prevent query formulation issues. Also, if semantic similarity could be modeled beyond exact lexical matches, more relevant results can be found even if the query terms don’t match exactly. For this domain, we formulated a task to compare different ways of modeling semantic similarity at paragraph level, using neural and non-neural systems. We compared systems that encode the query and the search collection paragraphs as vectors, enabling the use of cosine similarity for results ranking. After building a German dataset for cases and statutes from Switzerland, and extracting citations from cases to statutes, we developed an algorithm for estimating semantic similarity at paragraph level, using a link-based similarity method. When evaluating different systems in this way, we find that semantic similarity modeling by neural systems can be boosted with an extended attention mask that quenches noise in the inputs.

## introduction retrieving and understanding legal documents can take a considerable amount of time, even for legal professionals. searching a large database of previous cases (court decisions) that apply to a particular situation is an important part of legal work, as performed by lawyers and their legal assistants @xcite @xcite @xcite . such a retrieval system may not only help expert users with deep expertise in the domain but can also benefit non-expert users. the latter can, for example, try to develop a preliminary understanding of a situation, when selecting or approaching a lawyer, to better navigate the justice system @xcite @xcite . both expert and non-expert users may then be interested in a) identifying where the legal problem fits in regarding legal concepts, b) what legal actions are potentially relevant, and c) what were the outcomes of similar cases @xcite @xcite . the above problem can therefore be formulated as a specialized information retrieval task that involves searching through a corpus of legal documents, before ranking them according to their relevance. while the concept of relevance in this context is conceptually and practically complex, we will focus our attention on a key aspect, the semantic similarity between a query and a candidate hit @xcite . early attempts at developing such systems often involved the use of boolean expressions as queries, where a document ## related work understanding the central concept of relevance in information retrieval tasks can depend on the particular user, task and other contextual factors (van opijnen & santos 2017), hindering the emergence of a clear consensus on this topic. also, the construction of datasets that capture relevance as determined by human experts is challenging and expensive, leading to a paucity of such datasets in the public domain @xcite . therefore, algorithms that could help to estimate important aspects of relevance could therefore help to enable more research on the challenges outlined above. with semantic similarity as a key aspect of relevance, and the paucity of human expert annotations in the field, the question of the automated generation of labels that can approximate such human judgement poses itself. on a conceptual level, the computation of similarity between objects can not only be based on their context, but also on the link structure of a graph that describes relationships between those objects @xcite . such link-based similarity can often complement content-based similarity measures. the application of this principle can be found in an area that is related to our task, namely a field called context-aware citation recommendation, which builds on link-based similarities between scientific publications using citations between them @xcite . in the legal domain, citation networks can be described as linking case documents with relevant statutes, thereby generating a directed graph with two types of nodes (cases and statutes). the number of common out-citations can then be defined as bibliographic coupling @xcite ), indicating link-based similarity between two case documents @xcite @xcite . @xcite have noted that the notion of semantic similarity is domain-specific, and not completely defined, as they developed a framework for the comparison of content-based with link-based similarity methods. again, bibliographic coupling is used by calculating the number of common out-citations as a way of estimating semantic similarity between case documents. as semantic similarity modeling of text at paragraph level (beyond a ranking of hits that is solely based on exact lexical matches with terms in the query), is a key challenge in this domain, lessons learned from the comparison of systems that model semantic similarity at sentence level (in various domains) may be relevant. for example, senteval is an effort to evaluate sentence-to-vector encoders on a variety of tasks, including sentence similarity @xcite . @xcite describe a dataset of 47 pairs of indian supreme court case documents where similarity between each pair of documents is annotated on a scale from 0-10 by law experts, which the authors use to compare methods for modeling legal document similarity. however, it is important to keep in mind the many known domain-specific challenges (see the most widely used classic non-neural model is often referred to as a bag-of-words model, with every word in the modeled vocabulary represented as a separate dimension in a vector space. note that while such vector space models can enable a user to use natural language as a query, it will be unable to model an important aspect of semantic similarity, namely contributions based on word order @xcite . as semantically similar words are represented as distinct dimensions in that vector space, such similarities are not captured in bag-of-word models. these non-neural systems can provide useful baseline systems for comparisons with (neural) systems that aim to model these aspects of semantic similarity. neural systems for natural language understanding published in recent years have demonstrated promising abilities in this regard, in terms of their ability to model both word order aspects as well as other aspects of semantic similarities in natural language text @xcite . ## results and discussion the ability of the neural models to outperform the best non-neural baseline (boosted nnb) was then tested. the use implementation huface reached a performance of only 39.13%, so did not show a clear benefit over the boosted nnb baseline. the spacy implementation, however, achieved 42.41 %, a gain of more than 3 percentage points. this difference between the two use implementations may be explained by the fact that the huface variant is not the full use model, but a smaller, distilled version. a clearly improved performance over the boosted nnb was observed with germanbert, the model that was trained on 20% legal documents (cases and statutes in german language). this bertbased model achieved a performance of 44.38%. with the use of the extended attention mask, a maximal performance of 48.19% was reached at an optimal idf threshold of 2.0. for all tested bertbased transformers, the use of the extended attention mask resulted in clear improvements, at idf threshold values around 2 (compare fig. 5 ), while distilled, smaller models (of use and distilbert) did not perform well. in comparison, doc2vec achieved a performance of 45.47%, above nnb, but nowhere near the performance we saw with the extended attention mask-boosted germanbert. while both distilbert and germanbert were trained solely using german data, only germanbert had legal documents in the training data. note that this performance of germanbert was observed on a pre-trained model without any finetuning to the task. ## conclusions in the described semantic similarity modeling task, we found that the best neural model that was trained not only in the relevant language, but also in-domain data, offered a clear added value over the best non-neural model. this top performance was obtained by extending the use of the transformer attention mask to quench signals from less informative tokens (using idf scores), so they would not be considered in the model. the generated datasets of paragraphs extracted from swiss cases and statutes, as well as citations in cases that point to statutes, when published can facilitate further research in this domain and task, which currently lacks such datasets. with the linkbased automated labeling method we developed, the challenge of obtaining costly human expert annotations in this domain becomes less of an obstacle, aiding research into improved models (and their finetuning) that could help to reveal semantically similar paragraphs that may have been missed with the models we tested. therefore, with this new framework, a new generation of improved models can now be developed, for the task of searching swiss court decisions using a wide range of natural language queries that represent information needs from expert and even non-expert users. the ability to model semantic similarity beyond exact lexical matching could then help a larger audience of non-expert users who may struggle with query formulation, to better access, understand and navigate the legal system. ilias chalkidis, manos fergadiotis, prodomos malakasiotis, nikolaos aletras, ion androutsopoulos. 2020. legal-bert: the muppets straight out of law school. emnlp 2020 findings, acl: 2898-2904. yen-liang chen, yi-hung liu, and wu-liang ho. alexis conneau, and douwe kiela. 2018. senteval: an evaluation toolkit for universal sentence representations. arxiv:1803.05449 wilco draijer. 2019. the creation and integration of a legal information retrieval system without manual query construction. master thesis, leiden university. supervisor: suzan verbene. carole d. hafner. 1980. representation of knowledge in a legal information retrieval system. sigir '80: proc 3rd annual acm conf res dev inf retrieval. p 139-153 note that semantic similarity of both case paragraphs, as defined using this link-based similarity method using shared citations, goes beyond exact lexical matching for many of the key terms, except the term denoting the main topic, 'invalidität'. for comparison, the idf values of terms denoting key legal concepts in this case pair that contribute substantially to semantic similarity are shown below. additional terms and expressions that are specific to legal language and not likely to occur in other domains are shown in italics. also note how the statute texts are very concise and broadly understandable, which the case paragraph language is more about legal reasoning.