Building Machine Translation Tools for Patent Language: A Data Generation Strategy at the E uropean Patent Office (2023)

Abstract: The European Patent Office (EPO) is an international organisation responsible for granting patents and promoting global cooperation in the intellectual property world. With three official languages (English, German, French) and a need to constantly access and manipulate information in multiple languages, machine translation is essential for the EPO. Over the last years we have developed internal machine translation engines, specifically for the translation of patent language. This article presents our data generation strategy: it describes our approach to the generation of parallel corpora of documents, training datasets of aligned sentences, and respective evaluation datasets. Details on the challenges and technical implementation are presented, as well as statistics of the training dataset generation process.

## introduction and background the mission of the european patent office (epo) is "to deliver high-quality patents and efficient services that foster innovation, competitiveness and economic growth." @xcite . the epo is an international organisation with english, french and german as official languages (article 14(1) of the european patent convention, (epc)) and, as a global player, it develops and promotes international cooperation at a worldwide level with organisations both inside and outside of the patent system @xcite . both its role as a patent granting authority and being a global stakeholder in the intellectual property world requires constant access, exchange and manipulation of information in a myriad of different languages, making machine translation an indispensable tool. not surprisingly, the most significant part of the machine translations performed concerns the translation of patent documents. a patent is a technical and legal document that gives inventors for a time-limited period the right to prevent others from creating, using, or selling their invention without their permission in the countries for which the patent has been granted. the basic legal requirements for a patent to be granted are, that the claimed invention is considered to be new and that it involves an inventive step in view of the state-of-the-art. according to the epc, "the state-of-the-art shall be held to comprise everything made available to the public by means of a written or oral description, by use, or in any other way, before the date of filing of the european patent application" (article 54(2) of the epc). as will be appreciated, this definition imposes no restriction on language, i.e. in order to assess the basic requirements of patentability, examiners need to be able to access information in any possible language. however, this is not the only use case for machine translation of patent documents. in the last years, the epo has invested heavily in the development of ai-based tools for improving the efficiency of the search process by providing the best possible set of documents to start the search for an invention @xcite , or by automatically classifying patent documents according to the cooperative patent classification (cpc) foot_0 . these tools rely on language models such as bert @xcite that our team trained from scratch on a corpus of patent text in english, thus requiring the translation of all incoming applications into english. the epo has a duty of confidentiality regarding unpublished applications, which makes the use of external translation providers difficult for these cases. furthermore, patents are written using peculiar syntactic structures and employ specific terminologies, creating a hurdle for off-the-shelf translation engines trained on generic text corpora. as part of its strategic plan 2023, the epo has hence dedicated a substantial effort to the development of machine translation tools, particularly focusing on the translation of patent language. in this article we present the strategy followed to create training and evaluation datasets for the training of our own neural machine translation models for the following languages, paired to english (en): german (de), french (fr), italian (it), dutch (nl), spanish (es), chinese (zh), japanese (ja), korean (ko) and russian (ru). these languages have been selected to cover 99% of the full-text patent documents in our internal document collections. ## identification of paired documents in order to generate a parallel corpus for training neural machine translation (nmt) models on patent language, we rely on the concept of patent family. a patent family is a collection of patent applications (or granted patents) covering the same technical content. patents are national legal rights, providing protection in a specific jurisdiction, e.g. a certain country. protection in different jurisdictions requires thus filing and patent prosecution in every one of them. however, as mentioned in the previous section, the date of filing of a patent application is decisive for the assessment of the novelty and inventiveness of the claimed invention. in order to simplify the process of protecting inventions in different countries, a series of international treaties (e.g. paris convention, or patent cooperation treaty) have been established, which among others, allow to use the filing date of the first filing (priority) for the assessment of patentability in all jurisdictions. the generation of our parallel corpus assumes that the text of patent applications or granted patents for the same invention in different jurisdictions is likely to be a human translation of the first filing. this is a reasonable assumption, since the basic principles of patentability are common to most national or regional patent laws. consequently, the text contents of two family members in different languages, e.g. a patent application in germany, in german, and a patent application in the us, in english, will largely overlap, i.e. comprise the same sentences in german and english, respectively. additionally, certain legal provisions require the human translation of a patent publication, e.g. article 65(1) of the epc confers member states the right to request a translation of the patent as granted into one of its official languages. based on these principles, a database of parallel corpora of documents for different language pairs has been created, in which pairs of documents are stored, one document being assumed to be a human translation of the other @xcite . ## identification of paired sentences we have seen how the concept of patent family is used to generate a parallel corpus of documents. however, we can only assume that the text contents of a pair will highly overlap. in general, it cannot be expected that sentences correspond to each other directly and in perfect order. this is why we employ a sentence alignment algorithm that identifies the sentence pairs that correspond between parallel documents. to do so, we chose the recently published vecalign @xcite because it does not require the availability of a (however rudimentary) initial translation engine as other methods do @xcite . instead, it relies on sentence embeddings, dense semantic vectors, that are generated by a multilingual pre-trained language model. these are used to assess the similarity of parallel sentences. we parameterise vecalign to generate alignments with a maximum sentence count of 2, allowing at maximum 1:1 sentence alignments, because this is the data we use for the training of our translation models. a huge benefit of working in an international organisation like the epo is that there is a high likelihood to identify a native speaker in the organization with a technical background for any of our languages of interest. to assess the alignment quality of 1:1 alignments created by vecalign, we compiled evaluation data sets that were used by internally recruited language experts to align sentences from parallel documents manually. ideally, vecalign would confirm these 1:1 alignments. as the only pre-processing, vecalign requires that documents are already split into sentences. we use different sentence splitters for different languages: the sentence-splitter package (sentencesplitter, 2023) is used for languages de, en, es, fr, it, nl, ru, the pysbd package @xcite is used for languages ja, zh, and the kss package (kss, 2023) is used for ko. for each language, the generation of data for the manual alignment starts with a large set of paired publication sections (e.g. description or claims of a patent publication). for these sections, suitable pairs are selected by: the remaining section pairs were subject to further selection criteria aimed at spreading the examples uniformly over different technical fields using cpc classification. target was 75 example section pairs per section (a-h in the cpc classification scheme); except for rare cases, this target was achieved. the selected sections were prepared for the human alignment task by splitting them into chunks of target size 50 sentences. this was done to reduce the mental load of the cross-lingual alignment task. reference section for the number of chunks was the english section: the number of chunks was determined by dividing the number of available sentences by the target size and rounding the result. the sentences of the parallel non-english document were divided into the same number of chunks. the chunks were brought into an order that alternated examples from different technical fields. the chunks were presented via a graphical user interface (figure 1 ) to our internal language experts. in our sentence alignment tool, annotators can map sentences from parallel chunks to each other if they are literal translations. additionally, they can annotate ocr and splitting errors. these examples were used to fine-tune the language specific sentence splitting, or to improve our internal text quality assessment tools. the manually aligned sentences were used as reference for the evaluation of vecalign. parallel chunks were aligned with vecalign, and the quality of the generated 1:1 alignments was scored with precision, recall and the f0.5 score, weighing precision higher as recall. we chose that weighted score over the typical f1 score because precision is our primary concern, as it measures how many (in)correct alignments vecalign created. only the fastest parameterisation of vecalign with maximum alignment size 2 was evaluated on all languages. this ignores the ability of vecalign to create many-to-one alignments in both directions. we observed in early evaluations that with higher maximum alignment sizes, recall decreases and precision increases slightly (both for 1:1 alignments). example: for de, with maximum alignment sizes 2, 3, 4, 5, recall develops as 0.98, 0.95, 0.95, 0.96, precision develops as 0.86, 0.93, 0.93, 0.93. even though precision slightly increases, processing time on average doubles, which on our corpus of 1.4 billion sentence pairs makes a difference of weeks in computing time. that is why we opted for the fastest parameterization. in vecalign, the semantic relatedness of text chunks is assessed based on dense vector representations generated by multi-lingual language models. the original version of vecalign used language-agnostic sentence representations (laser) @xcite . we made use of these embeddings in the evaluations of languages de, fr, it, ja, nl, and zh. later in the project, we also evaluated language-agnostic bert sentence embedding (labse) @xcite for generating embeddings and found that it required only 75% of the processing time while keeping the same performance. additionally, it is much easier to use and maintain. this is why for the last languages in this project (es, ko, ru) we switched to labse. the work in this project was structured in a linear fashion that did not allow us to go back to the first group of languages that were initially processed with vecalign and laser and process them again using labse. if this should be possible in the future, we will make this switch also for them. to combat the lower precision with maximum alignment size 2, and to be able to create even higher quality aligned data, we trained a machine learning model for each language that classifies a 1:1 alignment as generated by vecalign as 'good' or 'bad'. this is necessary even though vecalign produces something like a quality indicator, the alignment cost (the higher the cost, the worse the alignment). in length (src-tgt); (5) src character count; (6) tgt character count; (7) difference character count (src-tgt); (8) labse cosine similarity between sentences (only for languages es, ko, ru). sentence length is measured as whitespace-separated tokens, tgt language is always en. all classification models were trained in scikit-learn @xcite . four different learning paradigms were selected for comparison: linear support vector machine, logistic regression, k-nearest neighbors, tree-based models (random forest, extra trees). for all models except the tree-based learners, a scaling model was trained on the training data and applied to the test data. the available data was split into train/test 70%/30% stratified on the target value confirmed (or not). all classifiers were evaluated in multiple configurations in a grid search, making use of a 5-fold cross validation. the tree-based model outperformed all other classifiers on all language pairs and was chosen for our data generation pipeline. the final performance is reported in once the classifiers were trained, the document pairs stored in our parallel corpora database were processed using an etl pipeline to extract sentence pairs and to store them in a postgresql database, termed sentence-aligned corpora repository (sacr), ready to be used to generate training datasets. ## global evaluation dataset with the purpose of measuring the performance and benchmarking the trained models, global evaluation datasets (ged) have been created for each language pair; the careful selection of sentence-pairs for each ged is aiming to ensure high-quality translations. to generate these datasets, sentence-pairs were extracted from the sacr following the process described in the previous section. additionally, to the extraction, pre-processing and filtering steps described in section 4, the extracted data went through the following subsequent filtering steps: after these filtering steps, a dataset was generated by selecting sentence-pairs from the ranked list covering the following criteria: a minimum number of sentences of 400 for each of the above combined criteria was selected, with the purpose of ensuring the statistical significance of the evaluations. the global evaluation dataset consists thus of 8×3×2×400 = 19,200 sentencepairs per language-pair. the hashes of the sentence-pairs in the ged were stored, so that these sentences could be excluded in the training data generation process. our internally developed machine translation engines achieve the following scores: german/french-english ged bleu @xcite : 72.0/70.8, chrf @xcite : 84.9/85.8 as implemented in sacrebleu @xcite . ## datasets the following datasets have been made available with this publication: these datasets can be found here: https://huggingface.co/datasets/mwirth-epo/eponmt-datasets. ## conclusion this publication outlines our strategy for the creation of parallel datasets for the training and evaluation of patent-language specific machine translation models. first, our comprehensive approach to patent sentence alignment was detailed. we highlighted our approach to identify high quality sentence alignments from a pair of related patent documents. one major contribution of our work are the details on the development of a classification model that significantly improved precision over a vecalignonly-based alignment strategy. both the evaluation of the performance of vecalign and the training of the subsequent classifiers relied on a set of manually curated sentences pairs created by inhouse language experts, assisted by a visual interface developed in-house. the curated datasets are shared via a huggingface dataset repository. in the second part of the publication, the aligned sentence corpus created from confirmed sentence pairs was described, with emphasis on the different actions taken to ensure a desired level of sentence quality and technical field balance. details on the corpus were presented along with our approach of creating global evaluation datasets for each language pair. our geds for the language pairs german-english, and french-english are shared with this publication. for this language pair no pre-processing was required and no sentences were discarded in the filtering process due to language mismatch. it is our hope that this contribution provides a helpful insight for the interested reader into the motivations behind the efforts of the epo regarding the development of internal machine translation engines, and how the challenge of training and evaluation data creation is being addressed. detailed information on the training procedure, experiments, implementation, and quality assessments of our internal machine translation engines will be the scope of a separate article. in closing, we would like to emphasize that patents and their technical field-based classification scheme represent valuable multi-lingual resources, not only for the development of machine translation engines, but also other language processing applications.