Transductive Legal Judgment Prediction Combining BERT Embeddings with Delaunay-Based GNN s (2024)

Abstract: This paper presents a novel approach to legal judgment prediction by combining BERT embeddings with a Delaunay-based Graph Neural Network (GNN). Unlike inductive methods that classify legal documents independently, our transductive approach models the entire document set as a graph, capturing both contextual and relational information. This method significantly improves classification accuracy by enabling effective label propagation across connected documents. Evaluated on the Swiss-Judgment-Prediction (SJP) dataset, our model outperforms established baselines, including larger models with cross-lingual training and data augmentation techniques, while maintaining efficiency with minimal computational overhead.

## introduction modeling legal texts have attracted lots of interest recently in two directions @xcite . the first is to gather large collections of legal text such as the multilegalpile corpus @xcite and train legal large language models (llms) such as @xcite . the second focuses on smaller, manually annotated and specialized datasets and benchmarks such as the swiss judgment prediction foot_0 (sjp) dataset @xcite , lexglue @xcite and lex-treme @xcite , and train smaller supervised models, mainly by finetuning bert-like models, sometimes applying cross-lingual transfer and data augmentation @xcite . general-purpose llms like chatgpt often perform poorly on legal tasks in zero and few-shot settings @xcite , though they can be useful as components in larger frameworks @xcite . specialized models, fine-tuned with supervised learning @xcite @xcite , require significant resources to improve performance, such as applying cross-lingual transfer, adapter-based fine-tuning, or tripling the dataset size with machine-translated documents @xcite . the suboptimal performance is likely due to the complexity of legal texts, which are long, dense, and filled with specialized terminology that generic pre-trained models struggle to understand. additionally, these models lack sufficient exposure to the contextual and nuanced nature of legal reasoning, requiring more domain-specific data to adapt effectively. in this paper, we hypothesize that transductive learning techniques @xcite are well adapted to legal judgment prediction (ljp) as it has been shown to work well in few-shot scenarios @xcite and on small training datasets @xcite . along these lines, we construct a single graph with all training (labeled) and test (unlabeled) documents as nodes, allowing a graph neural network (gnn) to learn from the entire dataset simultaneously. this approach leverages the relationships between documents for effective label propagation and contextaware classification, improving generalization by using both labeled and unlabeled data. it also captures domain-specific knowledge through connections like citations and shared terminology, adapts dynamically to the test set, and reduces overfitting by integrating test data into the learning process. our model ( §3) is a simple and efficient graphbased approach that achieves state-of-the-art results on the swiss judgment prediction (sjp) task @xcite without additional resources. it is also simpler than existing transductive graphbased models for document classificaiton @xcite . experiments ( §4) show it outperforms strong baselines from the literature and a new zeroshot saullm-7b baseline @xcite . graph-based methods in legal text graphbased models have been explored for legal judgment prediction, similar to our approach. @xcite use a graph network with heterogeneous text graphs and a gcn to predict outcomes, while ladan @xcite employs a graph neural network and attention mechanism to distinguish between confusing law articles. however, neither constructs a comprehensive graph for all documents, as we do. other methods focus on different tasks, such as legalgnn @xcite for legal recommendations, using a heterogeneous graph with user queries, and casegnn @xcite for legal case retrieval by modeling document-level relationships. ## method in this section we describe our architecture, also depicted in figure 1 . document encoder we begin by modeling documents as a graph, using the [cls] tokens extracted from a standard bert model @xcite (up to 512 tokens) to represent each document. while this approach leverages bert's document representation, our method is flexible and can easily incorporate other encoders that provide document representations. documents that are longer than bert context capacity are cut off. in contrast to our simple approach, some of the baselines we present in §4.2 handle long documents hierarchically or using larger models. delaunay graph to effectively model documents as a graph, we propose using a a delaunay graph @xcite . this kind of graph is particularly advantageous for information propagation by a gnn. it helps mitigating common challenges such as oversquashing @xcite )information loss due to bottleneck structures in the graph, and oversmoothing (oono and suzuki, 2020; cai and wang, 2020) -information mixing which can blur distinctions between nodes. in fact, delaunay graphs do not have tight bottlenecks and large cliques @xcite . additionally, delaunay triangulation correlates with improved homophily of the graph, meaning it better captures the similarity between connected nodes. in our approach, each document to be classified is represented as a node within this graph. to construct the graph, we employ a strategy similar to that used in @xcite . first, we perform a delaunay triangulation in a 2-dimensional feature space, where each [cls] token represents the document's embedding. since the [cls] to-ken is typically high-dimensional, we reduce its dimensionality using umap @xcite that preserves the local structure of data. delaunay graphs basically establish relationships between documents based on their distances in feature space. this operation is computationally efficient and scalable as we show in our experiments §4. ## conclusions this paper demonstrates that a transductive legal judgment prediction method, combining bert embeddings with delaunay-based gnns, significantly outperforms traditional inductive classification methods by effectively utilizing contextual and relational information between legal documents for more accurate label propagation and classification. in future work, we will study the necessity of retraining the model whenever a new batch of documents are to be classified. we will also explore semi-supervised training approaches to study the dependency of the performance on annotated data.