Agent-Specific Deontic Modality Detection in Legal Language (2022)

Abstract: Legal documents are typically long and written in legalese, which makes it particularly difficult for laypeople to understand their rights and duties. While natural language understanding technologies can be valuable in supporting such understanding in the legal domain, the limited availability of datasets annotated for deontic modalities in the legal domain, due to the cost of hiring experts and privacy issues, is a bottleneck. To this end, we introduce, LEXDEMOD, a corpus of English contracts annotatedwith deontic modality expressed with respect to a contracting party or agent along with the modal triggers. We benchmark this dataset on two tasks: (i) agent-specific multi-label deontic modality classification, and (ii) agent-specific deontic modality and trigger span detection using Transformer-based (Vaswani et al., 2017) language models. Transfer learning experiments show that the linguistic diversity of modal expressions in LEXDEMOD generalizes reasonably from lease to employment andrental agreements. A small case study indicates that a model trained on LEXDEMOD can detect red flags with high recall. We believe our work offers a new research direction for deontic modality detection in the legal domain.

## introduction a contract is a legal document executed by two or more parties. to sign a contract (e.g., lease agreements, terms of services, privacy policies, eula, etc.), it is important for these parties to precisely understand their obligations, entitlements, prohibitions, and permissions as described in the contract. however, for a layperson, understanding contracts can be difficult due to their length and the complexity of legalese used. therefore, a layperson often signs agreements without even reading them @xcite . having a system which can provide an "at a glance" summary of obligations, entitlements, prohibitions, and permissions to a contracting party (henceforth, "agent" foot_1 ), will be of great help not only to the agents but also to legal professionals for contract review. while existing language processing and understanding systems can be used for legal understanding, limited availability of annotated datasets in the legal domain due to the cost of hiring experts and privacy issues is a bottleneck. furthermore, the highly specialized lexical and syntactic features of legalese make it difficult to directly apply systems trained on data from other linguistic domains (e.g., news) to the legal domain. for an "at a glance" summary of contracts, we first need to identify the obligations, entitlements, prohibitions, and permissions present in the contract for a given agent. deontic modality is fre-quently used in contracts to express such obligations, entitlements, permissions, and prohibitions of agents @xcite . for instance, 'shall', 'shall not', and 'may' is used to express 'obligation/entitlement', 'prohibition', and 'permission' respectively in example (1) below. however, existing works for identifying such deontic modality types (henceforth "deontic types") either use rule-based @xcite @xcite or data-driven @xcite approaches, which cannot be directly used for our purpose. this is because rule-based approaches are not robust as they do not (in practice) capture the rich linguistic variety (e.g., use of nonmodal expressions in (2)) and ambiguity of modal expressions (e.g., 'shall' in (1a)). furthermore, annotated datasets used in the data-driven approaches do not consider multiple deontic types for a sentence and their association with the agent (e.g., (1a) is an instance of 'obligation' for the tenant and an 'entitlement' for the landlord). although, @xcite introduced a corpus with annotations for rights, obligations, and associated agents, it does not cover all the deontic types. moreover, different corpora consider different deontic types, lacking an accepted standard. in this work, we address these issues through the following contributions: (a) we present a linguistically-informed taxonomy for annotating deontic types in the legal domain, and use the taxonomy to build a corpus (lexdemod; §3) of english contracts with two types of annotations: (i) all deontic types expressed in a sentence with respect to an agent, and (ii) spans of modal triggers, i.e., expressions (e.g., bold-faced phrases in examples (1) and ( 2 )) that evoke the modal meaning; (b) we benchmark the corpus on two tasks: (i) agentspecific multi-label deontic modality classification ( §6), and (ii) agent-specific deontic modality and trigger span detection ( §7) using state-of-the-art transformer @xcite models, and (c) we perform transfer learning experiments ( §8) to investigate the generalizability of diverse modal expressions in lexdemod and a case study to detect red flags ( §9) in lease agreements. ## lexdemod dataset curation we first describe the dataset source ( §3.1) followed by pre-processing ( §3.2), annotation protocol ( §3.3), and the quantitative and qualitative analysis ( §3.4) of the collected dataset. ## proposed benchmarking tasks having established the rich variety and coverage of linguistic expressions for deontic modality in lexdemod, we benchmark the corpus on the proposed two tasks defined below: ## benchmarking setup we experiment with various pre-trained language models (plms) @xcite , which have shown state-of-the-art performance on natural language understanding tasks, to study their performance on our proposed tasks. we fine-tune these models for both the tasks on binary cross-entropy loss for 20 epochs each with a batch size of 8, and maximum sequence length of 256 using huggingface's transformers library @xcite . the model(s) with the best macro-f1 score on the dev set is used to report results on the test set. further implementation details are in a.6. we also partition the data according to the agent being conditioned to assess the performance of the trained models with respect to each agent. ## benchmarking multi-label classification comparison models. we experiment with three kinds of approaches for the agent-specific multilabel deontic modality classification task. all the above models are trained assuming trigger span information is not available and full context (i.e., sentence) is used. to understand the importance of agent conditioning, context, and trigger for this task, we additionally train the following models: (i) no-agent where special token for agent is not used during training; (ii) act-masked wherein everything in the context except the trigger span is masked using [mask] token to hide the context but retain the positional information of the trigger; (iii) at wherein only the tokens belonging to a trigger are used and multiple triggers are separated using a special token (e.g., [sep] @xmath0, and (iv) act wherein all the triggers are appended at the end of the context separated by a special token (e.g., [sep] @xmath1. evaluation measures. we report macro-averaged precision, recall, and f1 scores across all the types, calculated using sklearn library @xcite . we also report the accuracy of predicting all the classes correctly for a sentence. results and analysis. we report the results for multi-label classification task in as roberta-l performs the best on this task, we report the results for variants of this model to understand the importance of agent conditioning, context, and trigger, in the last block of manual inspection of deontic type-wise ( figure 4a shows the trend for roberta-l's f1 score as train data size varies indicating that the rate of increase in f1 decreases with additional data. ## benchmarking trigger span detection comparison models. we experiment with three kinds of approaches for the agent-specific deontic modality and trigger span detection task. ## beyond lease contracts to investigate if the diverse linguistic expressions used for expressing deontic modality is specific to a contract type, we collect annotations via amt using the same annotation protocol ( §3.3) for: (1) 470 sentences from 3 employment contracts in the ledgar corpus, and (2) 154 sentences from 4 rental agreement templates freely available at pan-dadoc. 13 we evaluate the performance of the best model (roberta-l) for both the tasks on these sentences and report the results in ## case study: red flag detection to investigate if our agent-specific deontic modality classifier is capable of identifying the red flags annotated by @xcite for lease agreements, we compare the predictions on the dev set from aleasebert, proposed by @xcite , and roberta-l model trained on lexde-mod dataset. for each sentence in the red flags dataset, we predict the deontic modality with respect to each of the agent alias mentioned in that sentence. if any one of the deontic types is expressed for any of the agents then we consider the prediction as positive otherwise negative. we find that (see ## conclusion and future work we introduced lexdemod for deontic modality detection in the legal domain which consists of diverse linguistic expressions of deontic modality. we propose and benchmark two tasks namely, agent-specific multi-label deontic modality classification, and agent-specific deontic modality and trigger span detection using transformer-based models. while evaluation results are promising, there is substantial room for improvement. we demonstrated the generalizability of the diverse linguistic expressions captured in lexdemod via transfer learning experiments to employment and rental lease agreements. the small case study on red flag detection using our data showed the usability of our dataset. we leave joint-modeling of the two tasks and using these identification models for generating "at a glance" summary of contracts for future work.