Patent Claim Translation via Continual Pre-training of Large Language Models with Parallel Data (2025)

Abstract: Recent advancements in large language models (LLMs) have enabled their application across various domains. However, in the field of patent translation, Transformer encoder-decoder based models remain the standard approach, and the potential of LLMs for translation tasks has not been thoroughly explored. In this study, we conducted patent claim translation using an LLM fine-tuned with parallel data through continual pre-training and supervised fine-tuning, following the methodology proposed by Guo et al. (2024) and Kondo et al. (2024). Comparative evaluation against the Transformer encoder-decoder based translations revealed that the LLM achieved high scores for both BLEU and COMET. This demonstrated improvements in addressing issues such as omissions and repetitions. Nonetheless, hallucination errors, which were not observed in the traditional models, occurred in some cases and negatively affected the translation quality. This study highlights the promise of LLMs for patent translation while identifying the challenges that warrant further investigation.

## introduction large language models (llms) demonstrate exceptional versatility because of their extensive pretraining, proving highly effective in various natural language processing tasks, such as summarization and question-answering. in the field of machine translation, closed llms like gpt-4 have been reported to achieve higher human evaluation scores than existing translation models @xcite . however, in the patent domain, transformer encoder-decoder based translation models remain the mainstream approach, and the translation capabilities of llms have not been sufficiently explored. the translation quality of patent documents has reached a sufficiently high level with conventional neural machine translation (nmt) methods, particularly for the main body of patent texts. however, patent claims remain a notable exception where translation quality is still problematic. patent claims are known for their extremely long and syntactically complex sentence structures, which pose significant challenges for traditional models. in addition, this study focuses on japanese-to-english translation, where a major obstacle is the significant difference in word order between the two languages. such structural divergence further complicates the translation of patent claims, especially in preserving the meaning and consistency across long sequences. in contrast, llms are believed to be capable of translating long sequences while maintaining global coherence and consistency. motivated by this potential, the present study investigates how effectively llms can translate patent claims, which represent the most difficult component in patent translation. to this end, we adopt the method proposed by @xcite , utilizing parallel patent data for continual pre-training and supervised fine-tuning (sft) to construct an llm specialized in patent claim translation. the performance of this llm is then compared with that of conventional transformer-based models, with translation quality evaluated using metrics such as bleu and comet. the results demonstrate that the llm statistically significantly outperforms conventional models, effectively addressing issues such as omissions, repetitions, and terminology inconsistency. however, the study also reveals llm-specific challenges, such as hallucinations, which are observed in specific cases that do not occur in conventional models. this study evaluates both the potentials and challenges of applying llms to patent translation, highlighting their effectiveness and identifying areas requiring further improvement. ## conclusion this study investigated the effectiveness of llms for patent claim translation through the application of continual pre-training and sft with domain-specific parallel data. the results demonstrated that llms, fine-tuned with patent-specific datasets, outperformed traditional transformer encoder-decoder based models in terms of bleu and comet scores, thereby highlighting their superior ability to handle the intricate sentence structures and technical terminology characteristic of patent documents. a notable improvement was observed in the reduction of common translation issues such as omissions and repetitions, highlighting the capacity of llms to better retain and reproduce the detailed content of the source text. furthermore, the experimental findings underscore the critical role of prompt design in enhancing translation performance, as improved prompts led to more accurate results. the study further showed the impact of data volume on the continual pre-training, indicating that substantial enhancements in translation performance can be achieved with relatively moderate data sizes. these findings provide a strong foundation for the potential of llms as a viable tool for high-quality patent translation tasks, contributing to advancements in the field of specialized machine translation.