Aligning Language Models for Icelandic Legal Text Summarization (2025)

Abstract: The integration of language models in the legal domain holds considerable promise for streamlining processes and improving efficiency in managing extensive workloads. However, the specialized terminology, nuanced language, and formal style of legal texts can present substantial challenges. This study examines whether preference-based training techniques, specifically Reinforcement Learning from Human Feedback and Direct Preference Optimization, can enhance models’ performance in generating Icelandic legal summaries that align with domain-specific language standards and user preferences. We compare models fine-tuned with preference training to those using conventional supervised learning. Results indicate that preference training improves the legal accuracy of generated summaries over standard fine-tuning but does not significantly enhance the overall quality of Icelandic language usage. Discrepancies between automated metrics and human evaluations further underscore the importance of qualitative assessment in developing language models for the legal domain.

## introduction the development of language models (lms) for use in specialized, professional domains has the potential to create time-saving, value-adding processes. this may benefit various fields such as law, healthcare, and engineering, where much of the work involves analyzing and writing domainspecific texts and documents. this is particularly relevant in the legal domain. an analysis of the legal systems in the usa and germany from 1998 to 2019 reported a monolithic growth in these systems @xcite . massive volumes of text data are a byproduct of most modern legal systems @xcite , leading to an environment with an ever-increasing amount of source material. consequently, lawyers and attorneys must devote more time to analyzing and reviewing legal documents while preparing their casework, resulting in a growing workload in an already overburdened profession @xcite . a comprehensive awareness and understanding of relevant laws and precedents is paramount to success in legal arguments. therefore, the ability to quickly summarize legal sources may significantly reduce the time spent reviewing pertinent material @xcite . summaries can also serve as references for justifying claims and building cases. this is an area where generative lms can be particularly useful, by processing and analyzing the bulk of the text needed. in iceland, there are substantial requirements within the legal domain that the quality of text meets the linguistic standards of the domain, both in terms of domain-specific terminology and general icelandic language proficiency. consequently, lms must adhere to the professional standards of the domain in which they are applied. the legal domain is also characterized by a specialized vocabulary, particularly formal syntax, and semantics based on extensive domain-specific knowledge @xcite . this makes the task of aligning lms to the specific language of the legal domain a non-trivial issue. the most common method to enhance the capabilities of a pre-trained generative lm is instruction fine-tuning, where the model receives an instruction as input and the correct response as the target label @xcite . under this paradigm, the model is rewarded for correctly following the instruc-tions; however, this does not necessarily entail that it captures the linguistic nuances within the target texts. reinforcement learning from human feedback (rlhf) is one such method that uses algorithms and reward-based methods from reinforcement learning (rl) to directly optimize a lm based on data collected from human feedback @xcite , aiming to help the model align better with both subjective and complex texts. another more recent approach, based on the same principle, is direct preference optimization (dpo) @xcite , which optimizes the model by transforming the rl reward maximization problem into a more simple classification problem. though the complexity of the dpo method is less than that of rlhf, it is unclear which method is best suited to align lms for summarizing icelandic legal text. this paper addresses the following research question: rq: can preference training methods, such as dpo and rlhf, enhance the ability of lms to generate domain-specific icelandic texts that users prefer, compared to lms fine-tuned solely with supervised learning? we compared the quality of text summaries generated for the icelandic legal domain by models fine-tuned with preference training to those fine-tuned solely through supervised learning. our findings indicate that applying either rlhf or dpo on top of domain-specific pre-training and instruction fine-tuning can improve the legal accuracy of the generated summaries. however, no similar improvements were observed in the general quality of icelandic language usage. additionally, there were discrepancies between automated numerical evaluations and qualitative human assessments. ## background and related work transformer-based language models (lms) have become central to text generation and nlp tasks, largely due to their adaptability when fine-tuned on specific tasks @xcite . these models, typically containing billions of parameters @xcite , excel at few-shot or zero-shot tasks that previously required supervised fine-tuning @xcite . however, languages with smaller speaker populations, such as icelandic, face challenges due to limited representation in training data. efforts to address this include icebert, a masked lm for icelandic @xcite , and gpt-sw3, a multilingual model covering most nordic languages @xcite . these initiatives align with ongoing government initiatives in iceland to preserve the icelandic language midst the rapid advancements in language technology @xcite and with the government's partnership with openai. 1 recent lm advancements emphasize rlhf to improve performance. initial work by openai explored human feedback to refine rl reward functions for complex tasks @xcite . @xcite applied rlhf in nlp, training models for improved text summaries. @xcite extended this approach with instruct-gpt, producing outputs that were preferred over those from larger models like gpt-3. rlhftrained models have shown advantages in common sense reasoning and world knowledge @xcite . a more streamlined approach, direct preference optimization (dpo), optimizes the model directly via preference-based comparisons, showing similar performance to rlhf with faster results @xcite . given the powerful text processing capabilities of modern lms, numerous studies have explored their applications in the legal domain, including judgment prediction @xcite , statutory interpretation @xcite , legal reasoning @xcite , and using large models like chatgpt as proxy legal advisors @xcite . research has also assessed performance on legal exams to gauge legal reasoning capabilities @xcite . for domain-specific improvements, legal-bert @xcite demonstrates the advantages of pre-training a lm specifically for legal tasks, finding that additional domain-specific pre-training on legal corpora improved performance compared to using general-purpose bert. building on this work, @xcite developed italian legal-bert, which they used in experiments for legal text summarization @xcite . another italian research, the prodigit project @xcite , is a largescale initiative aiming to support tax lawyers by utilizing lms for summarization. in a similar line of work, @xcite applied a bartbased lm for summarization of dutch case verdicts. the lm-generated summaries were considered useful in human evaluations, although they still fall short of the quality of human-generated summaries. ## methods we selected two open-source models for experimentation in generated icelandic legal summaries. the first model, a 1.3b parameter version of gpt-sw3, has been pre-trained on nordic languages using the nordic pile, a large corpus of approximately 1.2 tb, containing data in swedish, english, norwegian, danish, and icelandic @xcite . the second model was a 7b parameter version of llama2 @xcite , mostly pre-trained on english texts. 2 with this setup, we compared the effectiveness of language-specific pre-training (gpt-sw3) to the general learning capacity of a larger model (llama2). to better understand the effect of pre-training on icelandic texts, we created a sub-corpus of the icelandic gigaword corpus (igc) @xcite that contained 10% of its data sampled at random. we then created a version of llama2 (called ice-llama2) that was pre-trained on this sub-corpus. all models were trained in three phases. in the first phase, the models were further pre-trained on domain-specific icelandic legal text (see section 3.1) and in the second phase, the models were fine-tuned to perform the supervised court case summarization task. after this training phase, the model able to produce the highest rouge score @xcite ) -a commonly used metric for summarization tasks -was used to create summaries for a pairwise comparison dataset. finally, in the third phase, this data was then used to perform preference training with dpo and rlhf. ## results to optimize training efficiency and make the best use of available resources, all models were trained using low-rank adapters (lora) @xcite . the first parameter to be tuned and analyzed during pre-training was the adapter rank value. during the first phase of further pre-training on the r dataset (icelandic court rulings only), increasing the rank consistently led to a lower loss. this suggests that increasing the number of trainable parameters helps the model to learn better from the training data. based on these findings, a relatively large adapter with a rank of 1024 was used for training llama2-7b on the igc sub-corpus data and a rank of 256 for phase one (further pre-training on the r data, court rulings only), and a rank of 128 for phase two (training for the summarization task on the rs data, court rulings and summaries). to assess the impact of phase one, all models were evaluated by calculating their perplexity scores on the test split of the dataset. ## conclusions we evaluated the effect of language-specific and preference training to enhance the ability of lms in generating icelandic legal text summaries, compared to lms fine-tuned solely with supervised learning. an analysis of the evaluation results reveals that models further trained using either dpo or rlhf can exhibit improved performance in domain-specific language generation compared to those solely fine-tuned through supervised instruction; however, not consistently. notably, this additional preference training did not lead to a general improvement in the quality of icelandic used in the generated text. this underscores the critical role of language-specific pre-training in establishing a robust foundation for language generation. a notable finding was the gap between rouge scores and expert preferences, suggesting earlier integration of human feedback could be beneficial. the dataset for pairwise comparison was based on responses with top rouge scores post finetuning. a more effective approach might involve gathering human feedback at this stage to identify which model is best suited for generating data for further training. however, this approach is constrained by the high costs associated with obtain-ing feedback from professional experts. while the expert feedback gathered provides valuable insight into their preferences, achieving significant improvements driven by human feedback will likely require additional resources and investment. future work should emphasize languagespecific pre-training on icelandic legal texts, focusing on laws, bills, and resolutions. this could enhance icelandic quality while expanding legal knowledge. leveraging newer models with extended context windows, such as the now available llama3 family, could enable better utilization of training data by processing longer rulings. this capability would allow the inclusion of more training samples, potentially leading to improvements in performance. in addition to this, retrieval-augmented generation (rag) @xcite ) could be used to give models access to external knowledge when generating summaries, helping to increase factual accuracy in the responses. moreover, a greater variety of lms should be evaluated, as well as a larger cohort of legal experts.