Enhancing Scientific Discourse: Machine Translation for the Scientific Domain (2024)

Abstract: The increasing volume of scientific research necessitates effective communication across language barriers. Machine translation (MT) offers a promising solution for accessing international publications. However, the scientific domain presents unique challenges due to its specialized vocabulary and complex sentence structures. In this paper, we present the development of a collection of parallel and monolingual corpora from the scientific domain. The corpora target the language pairs Spanish-English, French-English, and Portuguese-English. For each language pair, we create a large general scientific corpus as well as four smaller corpora focused on the research domains of: Energy Research, Neuroscience, Cancer and Transportation. To evaluate the quality of these corpora, we utilize them for fine-tuning general-purpose neural machine translation (NMT) systems. We provide details regarding the corpus creation process, the fine-tuning strategies employed, and we conclude with the evaluation results.

## introduction the growth of scientific research across disciplines has intensified the need for efficient communication and international collaboration that transcends language barriers. while english is the dominant language of scientific publications @xcite , a substantial volume of valuable academic work is produced in other languages. according to a 2019 study @xcite , almost 40% of the articles of non-anglophone researchers are submitted in a language other than english. @xcite . machine translation (mt) offers a compelling solution, providing access to a vast pool of international research and fostering seamless collaboration among researchers worldwide. advancements in neural machine translation have significantly improved the quality of translations in various domains, largely driven by the application of the transformer architecture @xcite . transformers revolutionized nmt by enabling efficient parallel processing of entire sequences, leading to significant improvements in translation quality and fluency. however, the performance of nmt models often suffers when translating specialized domains due to the presence of specific terminology and sentence structures. translating scientific text presents unique challenges distinct from general language translation @xcite . scientific domains are characterized by: these factors can significantly impair the performance of generic nmt models, leading to mistranslations or loss of scientific meaning. to address these challenges, this paper presents the development of domain-specific corpora for scientific research and their application towards creating nmt models for four academic domains. we aimed at using the open-source nmt models which exhibit the best generic performance as base models and fine-tune them on these domainspecific corpora, so as to achieve translations that are not only fluent but also accurate and faithful to the scientific content. we focus on the spanish-english (es-en), french-english (fr-en), and portuguese-english (pt-en) translation directions, creating parallel corpora for each, alongside monolingual corpora for the respective languages (i.e., english, spanish, french, and portuguese). our corpora encompass a large general scientific corpus and smaller sub-corpora dedicated to the research areas of: we begin by presenting our dataset creation process which exploits the wealth of parallel titles and abstracts from bachelor and master theses, doctoral dissertations, and other scientific publications (such as published books and articles). next, we outline the methodology of fine-tuning pre-trained nmt models using the aforementioned datasets. finally, we evaluate the resulting models' performance by contrasting their output with the translations received from the original general-purpose nmt models as well as google translate. ## related work even though there is a plethora of parallel corpora in the ever-growing opus collection @xcite , there is a shortage of those which aim at addressing parallel data acquisition for the scientific and academic domains, relatively to the importance and challenges of scientific translation. in opus, there are 2 parallel datasets of note, namely capes @xcite and scielo @xcite . capes originates from the database of the theses and dissertations catalog (tdc) and contains approximately 1.2 million sentence pairs for the en-pt language pair @xcite , mined from theses and dissertation abstracts of students in post-graduate programs across brazilian universities. similarly, the scielo parallel corpus has been extracted through the use of the scielo database, which contains a broad range of open-access scientific articles. sci-elo consists of approxiately 3.3 million parallel sentences and metadata for english, spanish, and portuguese, some of which are trilingually aligned (i.e., en-es-pt) @xcite . both corpora are evaluated manually, as well as automatically by training and evaluating mt systems. in order to address the parallel data gap for scientific texts among underrepresented european languages, the scipar corpus @xcite was created and made publicly availabe via the elrc-share repository. foot_0 scipar contains 9.17 million sentence pairs in 31 language pairs and has been constructed from the titles and abstracts of bachelor and master theses, doctoral dissertations, and other scientific publications. it has been constructed through collecting, parsing, and processing metadata from 86 institutional repositories, digital libraries of universities, and national archives. the translations and open science project focused on building en-fr nmt systems for three pilot domains: (a) climatology and climate change, (b) neurosciences, and (c) human mobility, environment, and space. @xcite collected 316,701 parallel segments and 1,112 bilingual terms for these three domains, trained a generic nmt model from scratch, and the fine-tuned it on the collected datasets. the paper describes the manual and automatic evaluation and comparison that was conducted to determine domain-specific translation quality, while it also shows that adding data from scipar results in further improvements. ## dataset creation in this section, we outline the end-to-end pipeline used for mining high-quality parallel and monolingual corpora from the titles and abstracts of theses and dissertations. first, we detail the strategy used to process approximately 9.3 million records from 62 academic repositories to extract 11,700,912 sentence pairs. a detailed list of the repositories can be seen at ## domain adaptation for nmt using domain adaptation, we can enhance the ability of an existing mt system to translate documents from a very specific domain, through the utilization of an in-domain parallel corpus. to this end, we chose to use selected pre-trained models from opus-mt @xcite as our baseline systems. we chose these models because they are all based on state-of-the-art transformer-based neural machine translation architectures, and they were trained on freely available parallel corpora from the opus3 bitext repository. even though these models are not considered to produce the best quality for the selected language pairs, they provide a robust initial performance across a variety of language pairs. additionally, they have also been trained on general domain texts, making them ideal for our purpose, as they have seen little to no data from the scientific domain. since we decided to work on french, spanish and portuguese, we selected the open-source opus-mt models @xcite for the fr-en, es-en, and pt-en, translation directions as our base models. we aimed at using the largest available opus-mt models for each language pair and made use of the latest ones which where trained for the tatoeba challenge @xcite , an open project encouraging people to develop machine translation in real-world cases for many languages. in particular, we used the following models: all of these variants have been originally trained on subsets of the opus-mt corpus, along with additional back-translated data; a method which has been widely adopted by the research community to further improve translation quality @xcite . for fine-tuning the opus-mt models we used the marianmt framework @xcite , a decision driven by marianmt's efficiency, flexibility, and open-source nature, making it particularly suited for research purposes. fine-tuning was conducted using the curated corpora which resulted from the filtering process described in section 3.2, comprising parallel sentences from academic texts across our target domains: cancer research, energy research, neuroscience, and transportation research. these corpora were further enriched with general academic texts to provide a broader linguistic context, thereby mitigating the potential for overfitting to domain-specific jargon and syntax. in our domain adaptation experiments, we make use of all developer sets described in section 3.3 (7000 sentence pairs in total) so that the fine-tuning process does average of 4 domains not continue if the bleu or chrf scores do not improve for 5 consecutive checkpoints (every 500 steps). batch size is automatically determined by mariannmt to fit reserved gpu memory, while parameters are updated every 2 batches @xcite . during the fine-tuning phase, particular attention was paid to the balance between domainspecific and general academic texts, ensuring that the models retained their generalizability while enhancing their domain-specific performance. model hyperparameters were meticulously optimized through a series of experiments, with evaluation metrics such as bleu @xcite , chrf2++ @xcite @xcite , and comet @xcite used to monitor improvements in translation quality. ## results & evaluation in order to determine the improvement on translation quality for each specific domain of interest, we perform evaluation using all the 4 domainspecific test sets, as well as the general scientific test set (see section 3.3). to that end, the most widely-used benchmarks for the evaluation of automatic translations have been chosen, these being bleu, chrf2++, and comet. bleu measures how much overlap there is between the translated text and a reference translation in terms of n-gram phrases, while chrf2++ considers character-level accuracy and morphology in addition to n-grams. unlike bleu and chrf2++ which focus on wordlevel overlap, comet doesn't directly compare the words themselves. instead, it uses the xlm-r large as its backbone and estimates how well the translated text captures the meaning conveyed in the reference text. comet, in particular, has been shown to be superior to the other two metrics that we used, as it better correlates with human judgements and provides a robust way to differentiate high-performing systems @xcite . we use the sacrebleu toolkit @xcite for evaluation of bleu 8 and chrf2++ 9 as it ensures that they can be computed in a reproducible way, while the wmt20-comet-da 10 version was used for computing the comet score. in across all language pairs, adding general scientific text for fine-tuning on top of in-domain data (+ ft w/ ind & gsc) leads to further improvements compared to just using in-domain data, improving by +0.9 bleu and +0.8 comet. this suggests that general scientific data can act as a helpful supplement for domain adaptation, although it should be noted that their number is an order of magnitude larger than the in-domain data. while the fine-tuned systems achieve the best scores in the majority of cases, google translate remains highly competitive, producing the highest score in comet and outperforming all other systems in almost all metrics for fr-en. this indicates that even without specific domain adaptation, google translate offers strong translation capabilities for high-resource language pairs, even for niche domains like the ones selected for this experiment. ## conclusion our work presents the development of domainspecific corpora for scientific research and their application towards creating improved nmt models for four scientific domains: cancer research, energy research, neuroscience, and transportation research. we acquired 11.7m parallel sentences for three language pairs (en-es, en-pt, and en-fr) by processing the records from 62 academic repositories. our aim was to leverage the strengths of existing open-source nmt models while specializing them in these particular domains through fine-tuning. our findings indicate that fine-tuning generic nmt models with domain-specific parallel data leads to substantial improvements in translation quality for the targeted scientific domains. additionally, by including general scientific text alongside domain-specific data during fine-tuning of-wmt20-comet-da fers further enhancements, potentially by providing broader linguistic context. this work contributes to the ongoing effort to bridge the language gap in scientific research by developing nmt models that can accurately and fluently translate scientific text across various domains. looking ahead, there are a lot of ideas that we could implement in order to further improve our results. multi-domain adaptation holds promise for creating models adaptable to a wider range of scientific content. backtranslation, especially with domain-specific tagging and iterative approaches, offers another potential path for significant improvements. additionally, advanced parallel sentence filtering techniques, like those offered by bicleaner ai, can ensure high-quality training data. by incorporating these advancements, we can contribute to nmt models that effectively bridge the communication gap in the global scientific community.