UO - L ou TAL at S em E val-2023 Task 6: Lightweight Systems for Legal Processing (2023)

Abstract: This paper presents the work produced by students of the University of Orlans Masters in Natural Language Processing program by way of participating in SemEval Task 6, LegalEval, which aims to enhance the capabilities of legal professionals through automated systems. Two out of the three sub-tasks available – Rhetorical Role prediction (RR) and Legal Named Entity Recognition (L-NER) – were tackled, with the express intent of developing lightweight and interpretable systems. For the L-NER sub-task, a CRF model was trained, augmented with post-processing rules for some named entity types. A macro F1 score of 0.74 was obtained on the DEV set, and 0.64 on the evaluation set. As for the RR sub-task, two sentence classification systems were built: one based on the Bag-of-Words technique with L-NER system output integrated, the other using a sentence-transformer approach. Rule-based post-processing then converted the results of the sentence classification systems into RR predictions. The better-performing Bag-of-Words system obtained a macro F1 score of 0.49 on the DEV set and 0.57 on the evaluation set.

## introduction the ever-increasing volume of legal documents in populous countries such as india, for example, shows that automated systems may prove to be highly beneficial. automating such processes reduces the need to perform unrewarding tasks such as manual information retrieval, thereby facilitating the work of legal professionals, while reserving more complex aspects of law and decisions for their intervention ; and shortens the overall time required for legal processing @xcite . two out of the three legaleval 2023 @xcite sub-tasks available were tackled: the rhetorical roles (rr) textual classification task (sub-task a) and the legal-named entities extraction (l-ner) task @xcite . systems for the sub-tasks were not developed in silo: some rr systems made use of data that was first pre-processed with l-ner system-generated annotations, which in some cases improved performance on the dev set by approximately 2 points 1 . some challenges encountered include (but are not limited to) handling complex and precise semantics developed over long spans of text in the rr sub-task; and understanding the subtleties distinguishing the five named entity types pertaining to persons (judge, other person, petitioner, respondent, witness) for the l-ner sub-task. special attention was placed on developing lightweight and interpretable systems. the express focus on being lightweight was mainly to curb the carbon footprint of machine learning systems and improve widespread usability as state-ofthe-art architectures such as transformers @xcite consume considerable computational resources and energy @xcite , release undesirable pollutants such as greenhouse gases, and is less deployable in a production environment, the latter contradicting the task's objective of enabling widespread and rapid processing of legal documents. a system is considered to be lightweight by the authors' definition if it is capable of completing its training in a maximum of several hours on a single machine without specialized hardware, such as gpus or tpus. this is the case for crfs and systems employing pre-trained models. with regard to interpretability, the aim is to provide a means for end-users (i.e., legal profession-als) to query the system on how a specific decision was derived. exposing the workings of the system would build trust for users if its explanations are deemed acceptable by the experts. in the event where they challenge the system's predictions as they are found to not be logical, the feedback received that can be used to identify and target specific system mechanics involved would also boost the robustness of the system. explainability is especially characteristic of crfs, while more complicated systems such as deep learning approaches that employ standard real-valued tensors (i.e., @xmath0...×dn ) are not as interpretable. the authors' participation was an attempt to test the capabilities of lightweight systems in achieving reasonable performance on specialized data, given a lack of no prior knowledge or competencies in the legal domain that could provide leverage in tailoring systems for legal processing. the code is available in a github repository ( https://github.com/louisesteve/legaleval2023 ). the rest of the paper is structured as follows: (1) presentation of the two sub-tasks (section 2); (2) systems overview in (section 3); (3) experimental setup (section 4); (4) results and error analysis (section 5); (5) conclusion and way forward (section 6). ## background the datasets used in both sub-tasks and the division into train, dev and test sets were provided by the legaleval organizing team as json files. ## experimental setup the train and dev data split correspond to the raw train and dev datasets provided by the organizers. during training, systems were trained on train and tested on dev, but for the final version, they were trained on both train and dev. ## results in this section, results on the dev dataset and evaluation results on the test dataset (as published on the codalab submission platform) are listed. an error analysis is also performed on each system. ## conclusion our participation in the legaleval task was motivated by the express aim of evaluating the competitiveness of lightweight systems, with the interpretability of the system as a bonus criterion. results obtained in the rr and l-ner tasks were modest, at 0.568 and 0.649 and placing 25th (out of 27) and 14th (out of 17) respectively. for the l-ner task, we found that (1) crfs may attain a reasonable performance level (macro f β=1 ≈ 0.74 on dev) when trained to annotate all the classes at once, but (2) they face issues when having to differentiate between very similar classes (e.g., between petitioner and respondent) and ( 3 ) it is not certain if they are really capable of generalizing across datasets (macro f β=1 ≈ 0.64 on test, 10 points below dev). for the rr task, we found that replacing named entity spans with their class slightly improves performance of rr systems to a certain extent. the synergy between the two sub-tasks should be capitalized on by deploying the systems in that order. with regard to the way forward, the l-ner system could potentially be improved in terms of its raw performance by incorporating an lstm before the crf, as shallow (and with rather few cells in width) lstms are considered to boost performance @xcite while keeping the model lightweight. this may go towards improving the detection of rhetorical roles for the bow rr system as well. however, this could possibly degrade both the highly-human interpretable ner and bow rr systems' interpretability. as for the sentence-transformers rr system, it could be considered 'lightweight' in its current form that makes use of an existing trained model available as a library. however, the obvious method of improving the system by fine-tuning it on legal documents would render the system a classic deep learning heavyweight system.