SC a LAR NITK at S em E val-2024 Task 5: Towards Unsupervised Question Answering system with Multi-level Summarization for Legal Text (2024)

Abstract: This paper summarizes Team SCaLAR’s work on SemEval-2024 Task 5: Legal Argument Reasoning in Civil Procedure. To address this Binary Classification task, which was daunting due to the complexity of the Legal Texts involved, we propose a simple yet novel similarity and distance-based unsupervised approach to generate labels. Further, we explore the Multi-level fusion of Legal-Bert embeddings using ensemble features, including CNN, GRU, and LSTM. To address the lengthy nature of Legal explanation in the dataset, we introduce T5-based segment-wise summarization, which successfully retained crucial information, enhancing the model’s performance. Our unsupervised system witnessed a 20-point increase in macro F1-score on the development set and a 10-point increase on the test set, which is promising given its uncomplicated architecture.

## introduction the domain of law demands sheer expertise and experience for a human to master, but it takes much more to teach a machine the same. legal nlp @xcite ) is advancing at a rapid pace, and the advent of transformers @xcite has widened the prospects of research in this area. however, the intricate nature of legal texts and the underlying complex relationships between entities make it difficult even for state-of-the-art language models like bert @xcite to capture the details effectively. to advance our understanding of the reasoning ability of llms in the legal domain @xcite , task 5 of semeval-2024 was proposed @xcite . the objective of this task is to discern the accurate responses to legal inquiries in u.s. civil procedure, as posited by the organizers. the questions and answers adhere to a multiple-choice question-answering model, with accompanying explanations provided to facilitate comprehension of the legal concepts associated with each question. we have also released the code on github 1 we delve into the foundational paradigms of machine learning, specifically focusing on supervised and unsupervised learning, to introduce innovative approaches and present a comprehensive comparative analysis. the explanation part of our dataset undergoes a two-level segment-wise summarization generated by t5 @xcite , which is consistently utilized throughout our investigation. within the framework of the supervised setup, we leverage a multi-level cnn fusion approach @xcite , integrating lstm and gru architectures. this amalgamation facilitates the extraction of ensemble feature representations from questions, answers, and summaries. additionally, a one-dimensional cnn model @xcite , is trained. we employ a manual grid search technique to determine the optimal threshold that maximizes the macro f1 score, contributing to the refinement of our model. in the unsupervised setup, we delve into the acquisition of diverse word representations such as word2vec and glove. the assessment involves computing the similarity between questionanswer pairs and answer-summary pairs, employing combinations like glove-cosine, transformer embedding-cosine, transformer embeddingeuclidean and word2vec-cosine. notably, the bestperforming supervised model achieved a macro f1 score of 66 % on the development set and 49.6 % on the test set. in contrast, the unsupervised approach yielded scores of 62 % (development) and 52.3 % (test). this outcome highlights a nuanced challenge related to generalization on the test set, prompting further exploration into the intricacies of model adaptability and robustness. ## background the dataset provided by the organizers comprises three sets: train set, dev set, and test set, containing 666, 84, and 98 data points, respectively. within the training and dev sets, each entry includes fields such as question, answer, explanation, label (with values of 0 or 1), analysis, and complete-analysis providing a detailed examination. the test set, on the other hand, only consists of question, answer, and explanation. the label, when equal to 1, signifies a correct answer, while 0 denotes an incorrect one. the explanation field provides context and background details for each question. ## related works legal texts pose a unique challenge for pre-trained transformers @xcite due to the inclusion of specialized terminology not commonly used in everyday language . as a result, leveraging pre-trained models like bert @xcite , roberta @xcite , and others becomes essential by training them on legal corpora to enhance their understanding of legal terminologies. notable examples of transformers tailored for legal contexts include inlegalbert @xcite , legal-roberta @xcite , and similar models. fine-tuning transformers, such as legal-bert @xcite , on available legal data has been proposed as an effective strategy to improve performance on test sets, as suggested by @xcite . this approach capitalizes on domain-specific knowledge encoded during pre-training, enhancing the model's proficiency in handling legal language nuances. in the domain of legal question answering (lqa), recent works have extensively discussed significant advancements and challenges. the comprehensive review by martinez-gil provides insights into the key works in lqa, outlining challenges and proposing future research directions. louis et al. (2023) @xcite shed light on the limitations of existing large language models (llms) in legal question answering, emphasizing the need for interpretability. ## system overview transformers like t5, as demonstrated in the work of @xcite , exhibit high efficiency in producing summaries for lengthy paragraphs. in this study, t5 was employed to generate segmentlevel summaries on explanation column using a two-step approach. the initial summary was created from the original text, with a segment length of 1000 tokens. these segment-wise summaries were then concatenated with spaces in between to form the first summary. subsequently, the second summary was generated from the first summary, employing a segment length of 300 tokens, and similarly concatenated to provide a comprehensive summary of the input text. these summaries were used for further applications in place of explanation. segment wise summary approach can be visualized as follows: 4.1 supervised models ## experimental setup we utilized google colab for training and testing our models, taking advantage of the t4 gpu provided by the platform. ## results the performance metrics of our models on the test set and development set are presented in despite the varying performance, most of our models consistently outperformed the baseline. analysis from observing tables 3 and 4 , it becomes evident that the number of accurate predictions substantially increases in the development set, relative to its total size. in the codalab leader-board we ranked 16 out of 21 teams, and in the overall laederboard we ranked 15 out 21 teams. ## conclusion and future scope the dataset presents challenges for models to grasp the intricate legal context, resulting in subpar performance of regular supervised models. unsupervised models heavily rely on embeddings, but available transformers inadequately capture the dataset's nuances. these models operate under the assumption of at least one correct answer per question; however, instances where all answers were labeled as incorrect hindered unsupervised model performance. future endeavors entail amalgamating these models into a unified super model. this super model would aggregate predictions from various models to yield a singular final prediction, enhancing overall performance and addressing the limitations of individual approaches. an alternative strategy involves leveraging siamese networks to learn similarity, addressing challenges encountered by unsupervised models when all answers for a particular question are labeled as incorrect (0). by employing siamese networks, we believe that the model can effectively capture nuanced similarities between question-answer pairs, and provide better predictions. exploring other kind of summarizers and using other transformers for summarization such bart @xcite ) may also increase the overall performance of all the systems used in this paper. data augmentation @xcite can also be implemented to get better word2vec and glove embeddings.