Contemporary LLM s struggle with extracting formal legal arguments (2025)

Abstract: Legal Argument Mining (LAM) is a complex challenge for humans and language models alike. This paper explores the application of Large Language Models (LLMs) in LAM, focusing on the identification of fine-grained argument types within judgment texts. We compare the performance of Flan-T5 and Llama 3 models against a baseline RoBERTa model to study if the advantages of magnitude-bigger LLMs can be leveraged for this task. Our study investigates the effectiveness of fine-tuning and prompting strategies in enhancing the models’ ability to discern nuanced argument types. Despite employing state-of-the-art techniques, our findings indicate that neither fine-tuning nor prompting could surpass the performance of a domain-pre-trained encoder-only model. This highlights the challenges and limitations in adapting general-purpose large language models to the specialized domain of legal argumentation. The insights gained from this research contribute to the ongoing discourse on optimizing NLP models for complex, domain-specific tasks. Our code and data for reproducibility are available at https://github.com/trusthlt/legal-argument-spans.

## introduction mining legal arguments is the task of identifying, classifying and structuring argumentative units in a legal document. early works on argument mining in the legal domain considered generic types of arguments, such as claim and premise @xcite . recent works have shifted towards a legal theory inspired typology of argumentation @xcite . finegrained argument schemes can help legal scholars in structuring and analyzing legal documents, allowing insights into form and strategy of the presented argumentative content. however, the larger inventory of categories, as well as the granularity and complexity make the task of identifying specific argument types more challenging. a layperson would find the categories difficult to comprehend and even domain experts typically require instructions, additional context and training to identify the arguments in text. despite this, the majority of existing works classify argument spans with a limited context in the given legal document (e.g., only a single paragraph). this design choice is mostly due to the limited input size of the used models. however, the quantitative legal analysis of @xcite clearly shows that the lack of contextual information inevitably leads to a drop in prediction performance. contemporary decoder-only llms offer two main advantages over the early encoder-based models, namely the capability to ingest very large input documents and the in-context learning (icl) abilities without the need of expensive fine-tuning. we hypothesize that these features would help us overcome the difficulties in mining legal arguments. however, the research community on legal argument mining lacks the empirical evidence whether llms can be used with more complex argumentation schemes. we aim to address this gap in this work. first, we look into how encoder-decoder and decoder-only models can be used to mine finegrained legal arguments in long documents. second, we investigate how additional information and context in the input affects the performance. we hypothesize that the additional context which llms are able to ingest will further boost the performance of the argument extraction and classification. we explore the potential of in-context learning and fine-tuning. we use the lam:echr dataset @xcite , which contains an expert-crafted legal argument scheme. we reformat the dataset and test it with the contemporary llms flan-t5 and llama 3, attempting to improve upon the state-of-the-art performance on the dataset. we also explore the in-clusion of additional context to the input paragraph, as well as supplying the models with annotation guidelines to further boost the understanding of the label categories. ## related work argument mining (am) describes the extraction and analysis of natural language into argumentative components to enable their study in a variety of applications and domains @xcite . traditionally, the process of mining arguments is often split into multiple subtasks considering argument components, roles and relationships @xcite . although these tasks are usually formulated as a classification problem, the emerge of llms has enabled the possibility of using new techniques, such as prompt-based extraction or generative approaches. one such approach is successfully implemented by @xcite , who extend the idea of generating a structured label format from input which was first introduced by @xcite to the task of argument mining. by fine-tuning flan-t5 @xcite for the different subtasks span identification, component classification and relation classification, they show the potential of this technique on the established argument mining datasets containing persuasive essays (aaec) @xcite , medical abstracts (abstrct) @xcite and comments on consumer debt collection practices (cdcp) @xcite . the key point of this technique is the application of the identification and classification jointly into the output text generated by the decoder. this way the decoder will generate a copy of the input sequence with in-text annotations and labels. it remains to be seen if this approach is transferable to a richer argument typology and longer contextualized input sequences, since the tested datasets contain short documents and broader argument schemes. a different approach is pursued by @xcite , in which a range of contemporary llms are fine-tuned for individual argument mining subtasks across the same datasets. contrastive to the previous approach, the desired llm output is in a structured format. the best performance is achieved by fine-tuning llama 3.1 8b @xcite which reaches state-of-the-art performance on all datasets and tasks. @xcite also investigate the influence of context, comparing a paragraph-level and an essay-level classification and find that in some cases an extended context can boost the performance. ## methodology our overall methodology is as follows. first, we cast the bio-tagged lam:echr dataset into a format which is more suitable to be passed to the llms of our choice. because of this, we also need to adjust the evaluation metric to suit the expected predictions better; see in detail in the next subsection. we then fine-tune flan-t5-xxl, trying to replicate the good performance the model demonstrated in the related in-text annotation ap-case id 001-61184 ## results ## discussion we can make several observations from our experiments. first of all, fine-tuning llms on the task of adding annotations and labels in-text is feasible even with a domain-specific legal dataset. we find that reformatting the task into in-text annotations is a possible avenue for the future of legal argument mining outside of traditional formats like bio even in complex and difficult annotation schemes. after just 1 epoch of training, the models are able to output the required formats with proper labels. this is an optimistic finding, given that newer and better llms are introduced at a high frequency. and even though the overall performance of our best trained model could barely keep up with the roberta baseline, it is possible that scaling to much larger models could outperform the baseline. using an llm instead of an encoder model could also open up the possibility to handle more difficult argument annotations (i.e., legal argument relations) through in-text classification. secondly, we hypothesized that including additional context or the full document would provide a better understanding of the short paragraph and help with argument classification. from the results, however, we can surmise that the actual effect is minimal. it is conceivable that we did not chose the ideal method to incorporate context and a more sophisticated method than merely adding previous paragraphs is needed to draw the llm's full potential. in a similar fashion, annotation guidelines did not boost the performance either, which also leads us to believe that the biggest struggle for llms is the interpretation and understanding of the actual labels. this could also imply that providing context and guidelines is not sufficient to understand the argument categories properly. despite the advancements of contemporary llms in related works, successfully classifying labels like premise and conclusion and even argument relations, the models we tested struggle with handling the nuances of a complex legal argument scheme. we can also hypothesize that due to the nature of the arguments, which are based on the formal meaning of the argument, encoder models might be better suited by design to pick up standardized formulations and keywords which are often used in judgment documents. fine-tuning has proven to be the best technique to ensure adherence to the correct output format, while icl is not enough for the complexity of the fine-grained legal argumentation scheme. on the other hand, fine-tuning is also computationally expensive and it is less cost-efficient to train and finetune an llm like llama 3.3 70b, compared to training a roberta model. future work could focus on better techniques for teaching llms the knowledge necessary to understand and apply complex argumentation schemes. the addition of annotation guidelines did not prove to be useful in our case, but it could still be helpful to incorporate them in a different way in future experiments, because the underlying idea of requiring instructions, examples and context to solve the task is still at the core of legal reasoning. it would also be interesting to experiment with an llm that has received extensive legal pretraining and is still able to be finetuned for specific tasks. another difficulty that needs to be overcome is how the training procedure can make up for the massively imbalanced class representation. a training dataset of higher quality and carefully selected, representative examples could be more beneficial than a larger amount of data. we believe that reasoning models or reinforcement learning training methods could also be leveraged to enable this in future work. ## conclusion our study finds that flan-t5 and llama 3 did not outperform the roberta baseline on average in fine-grained legal argument mining, despite using fine-tuning and enriching the prompts with context and annotation guidelines. however, the performance surpasses the baseline for some argument types, showing that instruction-tuned llms generally have the potential to learn legal argument classification. contrary to the findings of related works, there was no strong indication that adding context helps, but passing the entire document actually reduced the performance. we suggest that the exploration of long contexts in the legal domain should be explored further in this regard. underrepresented and unbalanced labels remain a challenge, highlighting the difficulty of the task. a better integration of the annotation guidelines could be a good future direction to teach llms the specific skills to understand and apply such a complex legal argument scheme. the specialized nature of legal argumentation and its connection to legal theory presents unique challenges that current llms struggle to understand, emphasizing the need for further research and potential domain-specific adaptations.