L awng NLI : A Long-Premise Benchmark for In-Domain Generalization from Short to Long Contexts and for Implication-Based Retrieval (2022)

Abstract: Natural language inference has trended toward studying contexts beyond the sentence level. An important application area is law: past cases often do not foretell how they apply to new situations and implications must be inferred. This paper introduces LawngNLI, constructed from U.S. legal opinions with automatic labels with high human-validated accuracy. Premises are long and multigranular. Experiments show two use cases. First, LawngNLI can benchmark for in-domain generalization from short to long contexts. It has remained unclear if large-scale long-premise NLI datasets actually need to be constructed: near-top performance on long premises could be achievable by fine-tuning using short premises. Without multigranularity, benchmarks cannot distinguish lack of fine-tuning on long premises versus domain shift between short and long datasets. In contrast, our long and short premises share the same examples and domain. Models fine-tuned using several past NLI datasets and/or our short premises fall short of top performance on our long premises. So for at least certain domains (such as ours), large-scale long-premise datasets are needed. Second, LawngNLI can benchmark for implication-based retrieval. Queries are entailed or contradicted by target documents, allowing users to move between arguments and evidence. Leading retrieval models perform reasonably zero shot on a LawngNLI-derived retrieval task. We compare different systems for re-ranking, including lexical overlap and cross-encoders fine-tuned using a modified LawngNLI or past NLI datasets. LawngNLI can train and test systems for implication-based case retrieval and argumentation.

## introduction this work proposes a new natural language inference (nli) benchmark lawngnli constructed from u.s. legal opinions via the caselaw access project (the president and fellows of harvard university, 2018) that have been largely cleaned of inline citations in order to read more naturally. foot_0 it follows the general three-label nli formulation: given a pair of texts (premise and hypothesis), the goal is to predict the label of whether the premise entails, is neutral toward, or contradicts the hypothesis. lawngnli's premises are especially long and are multigranular. its automatic labels derive from the dataset construction using (negation-based) contradiction and (similarity-based) neutralization algorithms. these labels exhibit an accuracy of 88.8% (94.7% for high-confidence human labels) on a subset with human-validated gold labels. examples are derived from actual inferences from past cases that judges wrote to apply usefully to new situations, relying on their expertise while avoiding crowdsourced labeling. we conduct two sets of experiments on lawngnli. first, regarding in-domain generalization to long contexts, we compare top longsequence and short-sequence nli models on lawngnli. models are transferred with and without intermediate fine-tuning on existing nli benchmarks. we find that absent fine-tuning directly, models fall substantially short of top performance on our long premises. this continues to hold true when we leverage lawngnli's multigranularity to control for domain shift, by further fine-tuning models on our own short premises. top performance on our long premises is by short-sequence models prepended with a standard retrieval method (bm25 @xcite ) filtering across each premise. absent their fine-tuning that uses long premises as inputs, however, these models would underperform on long premises at inference time. we provide evidence that this gap is likely robust to any deviations between the automatic and gold labels. thus a large-scale longpremise dataset like lawngnli is needed. second, regarding legal retrieval systems, we conduct a comparison of leading models by recall@k for selecting target cases via implication/nli-based retrieval (when a user provides arguments entailed or contradicted by target cases as queries), in our domain. implicationbased retrieval is an underexplored subtask (see, e.g., @xcite . our zero-shot panel comprises sentence-transformers @xcite lightweight bi-encoders pretrained using short-sequence nli, user web queries, and general semantic relatedness. while baseline models transfer reasonably to our domain zero shot, we further compare re-ranking with models finetuned using several previous nli datasets along with an adjusted retrieval version of lawngnli. future improvements can put more evidence within the range of human users' cognitive reach beyond the top result, including for law which could help make legal work more affordable and equitably accessible (see section 6). overall, our main contributions are: (1) a new nli benchmark with multigranular premises much longer than in most existing nli benchmarks across percentiles (see ## lawngnli dataset we construct lawngnli beginning with all citations with parentheticals in u.s. state and federal case opinions, via the caselaw access project (the president and fellows of harvard university, 2018). its hypotheses derive from the actual inferences from past cases applied by judges to decide between competing parties' positions across a large cross-section of u.s. opinions. thus we hope its examples capture the variety of reasoning underlying the useful arguments that legal practitioners aim to support or undermine during their daily work. ## experimental evaluation our experiments demonstrate two applications of lawngnli. foot_4 see implementation details in appendix section a.2. 1. generalization first, we evaluate whether models perform competitively on lawngnli's long premises, before and after fine-tuning on existing nli benchmarks and/or its short premises (with lawngnli's multigranularity, we can evaluate on long premises within the same domain). rq1: can top nli models approach top performance on lawngnli with long premises, absent fine-tuning directly with our long premises? rq1a: is our answer to rq1 robust to any error rate in lawngnli's automatic labels? 2. retrieval second, we evaluate retrieval and nli models on a retrieve-and-re-rank approach to nli-based (entailed or contradicted arguments as queries) case retrieval. rq2: how do models compare on implicationbased case retrieval? ## conclusion and future work this work presents lawngnli, a new nli benchmark with multigranular long premises, each containing a shorter version. experiments demonstrate some use cases. first, we show that leading nli models fall substantially short of competitive performance when generalizing to lawngnli with its long premises, even after fine-tuning using existing nli benchmarks and/or lawngnli with short premises (with the same domain and examples as the long-premise evaluation). unconfounded by domain shift, these results show the need for a large-scale long-premise dataset like ours at finetuning time. second, we show that leading lightweight retrieval models can reasonably handle implicationbased retrieval on lawngnli zero shot with both entailed and contradicted arguments as queries. we then compare re-ranking by lexical overlap and models fine-tuned using a modified lawngnli or several previous nli datasets. multiple other aspects of lawngnli are left for future study.