Contract Discovery: Dataset and a Few-Shot Semantic Retrieval Challenge with Competitive Baselines (2020)

Abstract: We propose a new shared task of semantic retrieval from legal texts, in which a so-called contract discovery is to be performed – where legal clauses are extracted from documents, given a few examples of similar clauses from other legal acts. The task differs substantially from conventional NLI and shared tasks on legal information extraction (e.g., one has to identify text span instead of a single document, page, or paragraph). The specification of the proposed task is followed by an evaluation of multiple solutions within the unified framework proposed for this branch of methods. It is shown that state-of-the-art pretrained encoders fail to provide satisfactory results on the task proposed. In contrast, Language Model-based solutions perform better, especially when unsupervised fine-tuning is applied. Besides the ablation studies, we addressed questions regarding detection accuracy for relevant text fragments depending on the number of examples available. In addition to the dataset and reference results, LMs specialized in the legal domain were made publicly available.

## introduction processing of legal contracts requires significant human resources due to the complexity of documents, the expertise required and the consequences at stake. therefore, a lot of effort has been made to automate such tasks in order to limit processing costs-notice that law was one of the first areas where electronic information retrieval systems were adopted @xcite . enterprise solutions referred to as contract discovery deal with tasks, such as ensuring the inclusion of relevant clauses or their retrieval for further analysis (e.g., risk assessment). such processes can consist of a manual definition of a few examples, followed by conventional information task legal si few-shot coliee + --snli ---multinli ---trec legal track + --propaganda detection -+ -thumos (video) -+ + activitynet (video) -+ + albayzin (audio) -+ -contract discovery (ours) + + + ## review of existing datasets ## competitive baselines solutions based on networks consuming pairs of sequences, such as bert in sentence pair classification task setting @xcite , are considered out of the scope of this paper since they are suboptimal in terms of performance-they require expensive encoding of all combinations from the cartesian product between seeds and targets, making such solutions unsuitable for semantic similarity search due to the combinatorial explosion (reimers and gurevych, 2019). because of the aforementioned problem and the fact that conventional classifiers require much more data than available in a few-shot setting, in this section, we describe simple k-nn-based approaches that we propose as baseline solutions to the problem stated. ## discussion the brief evaluation presented in the previous section has multiple limitations. first, it assumed retrieval of a single, most similar segment, whereas it appears that multiple clauses might be returned instead. however, we consider this restriction justifiable during a preliminary comparison of applicable methods. multiple alternative selectors may be proposed in the future. secondly, all the evaluated methods assume scoring with the policy of averaging individual similarities. we encourage readers to experiment with different pooling methods or meta-learning strategies. moreover, even the lm-based methods we had studied the most can be further studied in the proposed shared task. for example, only embeddings from the last layer were evaluated, even though it is possible that the higher layers may capture semantics better. finally, it is in principle possible to address the task in entirely different ways, for example, by per-forming neither segmentation nor aggregation of word embeddings at all, but by matching clauses on the word level instead, which may be an interesting direction for further research. we decided to take the most common and straightforward way, due to fact performed evaluations are to serve as baselines for other methods. ## related work there is a large and varied body of work related to information retrieval in general; however, following @xcite we consider the problem stated in an end-to-end manner, where the nearest neighbor search is performed on dense document representations. with this assumption, the main issue is to obtain reliable representations of documents, where by document we mean any selfcontained unit that can be returned to the user as a search result @xcite . we use the term segment with the same meaning wherever it aids clarity. many approaches considered in the literature rely on word embedding and aggregation strategies. simple methods proposed include averaging, as in the continuous bag-of-words (cbow) model @xcite or frequencyweighted averaging with the decomposition method applied @xcite . more sophisticated schemes include utilizing multiple weights, such as a novelty score, a significance score, and a corpuswise uniqueness @xcite or computing a vector of locally aggregated descriptors @xcite . most of the proposed methods are orderless, and their limitations were recently discussed by @xcite . however, there are also pooling approaches preserving spatial information, such as a hierarchical pooling operation @xcite . other methods of obtaining sentence representations from word embeddings include training an autoencoder on a large collection of unlabeled data @xcite or utilizing random encoders @xcite . despite its shortcomings and the availability of many sophisticated alternatives, the cbow model is a common choice due to its ability to ensure strong results on many downstream tasks. different approaches assume training encoders through document embedding in an unsupervised or supervised manner, without the need for explicit aggregation. the former include skip-thought vectors, trained with the objective of reconstruct-ing the surrounding sentences of an encoded passage @xcite . although this method was outperformed by supervised models trained on a single nli task @xcite , paraphrase corpora @xcite or multiple tasks @xcite , the objective of predicting the next sentence is used as an additional objective in multiple novel models, such as the universal sentence encoder @xcite . even though many transformer-based language models implement their own pooling strategy for generating sentence representations (special token pooling), they were shown to yield weak sentence embeddings, as described recently by @xcite . the authors proposed a superior method of fine-tuning a pretrained bert network with siamese and triplet network structures to obtain sentence embeddings. there were attempts to utilize semantic similarity methods explicitly in the legal domain, e.g., for a case law entailment within the coliee shared task. in a recent edition, @xcite used a bert model fine-tuned on a provided training set in a supervised manner, and achieved the highest f-score among all teams. however, due to the reasons discussed in section 4, their approach is not consistent with the nearest neighbor search, which is what we are aiming for. ## summary and conclusions we have introduced a new shared task of semantic retrieval from legal texts, which differs substantially from conventional nli. it is heavily inspired by enterprise solutions referred to as contract discovery, focused on ensuring the inclusion of relevant clauses or their retrieval for further analysis. the main distinguishing characteristic of contract discovery shared task is conceptual, since: for the purposes of providing competetive baselines, we considered the problem stated in an endto-end manner, where the nearest neighbor search is performed on document representations. with this assumption, the main issue was to obtain representations of text fragments, which we referred to as segments. the description of the task was followed by the evaluation of multiple k-nn-based solutions within the unified framework, which may be used to describe future solutions. moreover, a practical justification for handling the problem with k-nn was briefly introduced. it has been shown that in this particular setting, pretrained, universal encoders fail to provide satisfactory results. one may suspect that this is a result of the difference between the domain they were trained on and the legal domain. during the evaluation, solutions based on the language models performed well, especially when unsupervised finetuning was applied. in addition to the aforementioned ability to fine-tune the method on legal texts, the most important indicator of success so far has been the involvement of multiple, sometimes overlapping substrings instead of sentences. moreover, it has been demonstrated that the methods benefit substantially from the availability of a second example, and the presence of more leads to a decrease in variance, even when a simple similarity averaging scheme is applied. the discussion regarding the presented methods and their limitations briefly outlined possible measures towards improving the baseline methods. in addition to the dataset and reference results, legalspecialized lms have been made released to assist the research community in performing further experiments. the contract discovery dataset, edgar corpus, we crawled, and all the mentioned models are publicly available on github: https://github.com /applicaai/contract-discovery .