Detecting Relevant Differences Between Similar Legal Texts (2022)

Abstract: Given two similar legal texts, is it useful to be able to focus only on the parts that contain relevant differences. However, because of variation in linguistic structure and terminology, it is not easy to identify true semantic differences. An accurate difference detection model between similar legal texts is therefore in demand, in order to increase the efficiency of legal research and document analysis. In this paper, we automatically label a training dataset of sentence pairs using an existing legal resource of international investment treaties that were already manually annotated with metadata. Then we propose models based on state-of-the-art deep learning techniques for the novel task of detecting relevant differences. In addition to providing solutions for this task, we include models for automatically producing metadata for the treaties that do not have it.

## introduction legal documents typical use standardized forms and structures ("boilerplate language"). moreover, within a given domain, legal documents often follow model texts and templates resulting in shared norms, principles and dispute resolution mechanisms. however, faced with high-similarity texts, what matters most to lawyers are often textual differences. where does a contract deviate from an industry standard? how does a law differ from an international model law? and when are these differences legally relevant rather than just stylistic? our work seeks to detect such relevant differences between otherwise similar legal texts. it uses international investment treaties as a case study. ## related work there is a growing body of research on natural language processing and machine learning techniques for legal applications. the applications that focus on legal text processing can be divided by the type of text: court judgements and related types of texts on one side, and contracts, treaties, or statutes on the other side. the tasks addressed vary, from information retrieval from large amounts of legal text, to legal text summarization, legal named entity extraction, court judgement prediction, and more. pre-trained neural language models were developed for english texts, such as legalbert @xcite , as well as for a few other languages @xcite . common shared legal text mining tasks are exemplified by semeval-2023 task 6 legaleval: un-derstanding legal text foot_0 which has three subtasks: predicting the rhetorical roles of sentences (such as preamble, fact, ratio, arguments, etc.), legal named entity extraction, and court judgement prediction with explanation. similarly, the artificial intelligence for legal assistance (aila 2021) shared task at fire 2021 foot_1 included a rhetorical role labelling task continued from previous editions, and legal judgement summarization task. @xcite . finally, the competition on legal information extraction/entailment (coliee 2022) foot_2 included tasks relating to case law and statutory law such as a legal case retrieval task, a legal case entailment task, a question answering system based on relevant statutes from a database of japanese civil code statutes and entailment of a yes/no answer from the retrieved civil code statutes. the solutions used to solve these tasks involved classical information retrieval methods, while a few applied deep learning methods for retrieval @xcite . specifically relating to statutory law type documents, such as contracts, laws and treaties, there is a growing interest to automatically identify similarities between documents. use-cases include identifying where national laws implement international laws @xcite . in addition, researchers have attempted to assess to what extent legal texts copied from each other or from model agreements (ash and marian, 2019) @xcite . while these studies provide insights into document similarity, most legal scholars are interested not in how similar documents are but where and how similar documents differ, as discussed in @xcite . standard text difference detection algorithms (such as diff in linux/unix) are not able to detect which differences are relevant from a semantic point of view and which are not. therefore, our task is different from the tasks addressed in related work or in the shared tasks. we are also using a dataset of legal texts that has not been exploited before by computational methods. ## data the data used in the paper origins from the electronic database of investment treaties (edit) @xcite , which is a new comprehensive full-text database of international investment agreements (iias). it contains 3,786 international treaties. in edit, all articles with an article title have their structure category labeled through a manual assignment by experts. 71 different structure categories exist in the dataset. in addition, articles in the treaties were further classified into 144 content categories according to 702 different keywords. for the task we address in this paper, we need sentence pairs with high similarity to be classified into exhibiting a relevant, stylistic, or irrelevant difference. instead of asking human judges to label pairs of texts, we use the existing edit metadata to construct the labels we need for our training and test data. we first extracted sentence pairs with high similarity scores evenly from within each structure category. we then used the keyword-category mapping from figure 1 to automatically label a dataset of sentence pairs for our task. for each two highly similar sentences (with similarity score bigger than 0.965 and less than 0.98) 4 , we label them as only displaying a stylistic difference if both of them share a same set of content categories (according to the keywords they contain). in contrast, if the proportion of overlapping categories is less than one third of the sentence pair's content category union, we consider that one sentence discusses a subject or topic that is distinct from the other and that these two sentences thus have relevant differences. moreover, we also introduced less similar sentences (with a score less than 0.85) 5 as examples of sentences having irrelevant differences. via this method, we obtained a dataset with 12,968 sentence (article) pairs. 8,430 of them are sentences that have the same content categories and therefore are labelled as having only stylistic differences (no semantic or legal differences). 2,096 of the sentence pairs are labelled as containing relevant differences which would be the ones of interest to a legal researcher. we also introduced 2,442 sentence pairs which are less similar from each other and labeled as having irrelevant differences. this is done to better simulate common application scenarios 6 . we use this dataset of sentence pairs to train our automatic methods for detecting relevant differences. first, we keep aside 20% of the dataset for testing the models that we will train. figure 2 shows the distribution of the three classes in our dataset. 5 tasks the main task of this paper is to distinguish relevant differences from stylistic or irrelevant differences between similar texts, in order to facilitate legal research. this means to ignore differences that are too small and uninteresting from a semantic point of view. at the same time, sentences that are very different are not of interest since they are easy to identify (such sentences were not included in our dataset). to achieve our goal, a few preprocessing tasks (explained below) were be performed in order to build a dataset of similar sentences labeled for relevant differences, to allow us to train the models and evaluate them. figure 3 shows our workflow. ## limitations we limited our experiments to articles from legal treaties, though our techniques could be applied on any kind of legal texts or even wider, to any similar texts in general language or specific domains. though a significant barrier on experimenting with other kinds of texts is the lack of annotated data for training supervised classifiers. in our current experiments, we were able to use the already existing manual annotations in edit to produce training data of text pairs without the need for new manual work. another limitation is caused by the imbalance of the dataset for the structure category prediction. none of existing re-sampling methods seems appropriate to be applied on legal articles, as their structure is highly standardized. domain-specific re-sampling methods could be further investigated. ## conclusion and future work in this paper, we presented several deep leaning based models for the novel task of detecting semantically relevant differences between similar legal texts. in addition, we proposed an enhanced model that uses contextual information for the secondary task of predicting metadata (structure categories). we exploited a valuable legal resource that was not used before for computational analysis of this kind. we are making available our code on github and our datasets with training/test splits for reproducibility purposes 11 . we achieved very good results with the deep learning models that we considered as promising for our tasks, but there are other deep learning models that could be tried in future work. another direction of future research is to apply text entailment methods on the articles with relevant differences, to see if one entails the other. this could mean that one treaty was derived from the other one. we could apply this over multiple treaties to trace back the historical evolution of treaty writing. in case the entailment goes in both directions, one article entails the second one and the reverse holds too, this could be another filter to add on top of our best model for detecting relevant differences.