Enriching Biomedical Knowledge for Low-resource Language Through Large-scale Translation (2023)

Abstract: Biomedical data and benchmarks are highly valuable yet very limited in low-resource languages other than English, such as Vietnamese. In this paper, we use a state-of-the-art translation model in English-Vietnamese to translate and produce both pretrained and supervised data in the biomedical domains. Thanks to such large-scale translation, we introduce ViPubmedT5, a pretrained Encoder-Decoder Transformer model trained on 20 million translated abstracts from the high-quality public PubMed corpus. ViPubMedT5 demonstrates state-of-the-art results on two different biomedical benchmarks in summarization and acronym disambiguation. Further, we release ViMedNLI - a new NLP task in Vietnamese translated from MedNLI using the recently public En-vi translation model and carefully refined by human experts, with evaluations of existing methods against ViPubmedT5.

## introduction in recent years, pretrained language models (lms) have played an important and novel role in developing many natural language processing (nlp) systems. utilizing large pretrained models like bert @xcite , xlnet @xcite , albert @xcite , roberta @xcite , @xcite bart @xcite , and t5 @xcite has become an effective trend in natural language processing. all these large models follow the transformer architecture proposed by @xcite with the attention mechanism. the architecture has been proven to be very suitable for finetuning downstream tasks leveraging transfer learning with their large pretrained checkpoints. before the emergence of large transformer lms, traditional wording embedding gave each word a fixed global * the first four authors contributed equally to this work representation. large pretrained models can derive word vector representation from a trained large corpus. this will give the pretrained model a better knowledge of the generalized representation of a trained language/domain and significantly improve performance on downstream finetune tasks. the success of pretrained models on a generative domain (bert, roberta, bart, t5, etc.) has created a path in creating more specific-domain language models such as codebert @xcite and cotext @xcite for coding languages, tabert @xcite for tabular data, biobert @xcite and pubmed-bert @xcite for biomedical languages. biomedical literature is getting more popular and widely accessible to the scientific community through large databases such as pubmed foot_0 , pmc foot_1 , and mimic-iv @xcite . this also leads to many studies, corpora, or projects released to further advance the biomedical natural language processing field @xcite @xcite . these biomedical domain models leverage transfer learning from pretrained models @xcite @xcite to achieve state-of-the-art results on multiple biomedical nlp tasks like named entity recognition (ner), relation extraction (re), or document classification. however, few studies have been on leveraging large pretrained models for biomedical nlp in lowresource languages. the main reason is the lack of large biomedical pretraining data and benchmark datasets. furthermore, collecting biomedical data in low-resource languages can be very expensive due to scientific limitations and inaccessibility. we attempt to overcome the issue of lacking biomedical text data in low-resource languages by using state-of-the-art translation works. we start ## related works the development of parallel text corpora for translation and use for training mt systems has been a rapidly growing field of research. in recent years, low-resource languages have gained more attention from the industry, and academia @xcite @xcite . previous works include gathering more training data or training large multilingual models @xcite . low-language mt enhances billions of people's daily life in numerous fields. nonetheless, there are specific domains crucial yet limited such as biomedical and healthcare, in which mt systems have not been able to contribute adequately. previous works using mt systems for biomedical tasks includes @xcite . additionally, several biomedical parallel @xcite have been utilized just for terminology translation only. pioneer attempts to train mt systems using a corpus of medline titles @xcite , and the use of publication titles and abstracts for both @xcite . however, none of these works targets low-resource languages. a recent effort to train vietnamese ml systems for biomedical and healthcare is @xcite . these, however, do not utilize the capability of mt systems, instead relying on manual crawling. therefore, this motivation has led us to employ mt systems to contribute high-quality vietnamese datasets that emerged from the english language. to the best of our knowledge, this is the first work utilizing state-of-the-art machine translation to translate both self-supervised and supervised learning biomedical data for pretrained models in a low-resource language setting. ## improvements on biomedical english-vietnamese translation through self-training to generate a large-scale synthetic translated vietnamese biomedical corpus, we first look into improving the existing english-vietnamese translation system in the biomedical translation domain. previous work from @xcite has shown that en-vi biomedical bitexts are very rare, even for large-scale bitext mining. therefore, we look into self-training to leverage the available monolingual english biomedical data. self-training approach has been experimented with in @xcite and utilized to improve translation on low-resource mt systems @xcite . the advantage of this method is that the source side of the monolingual corpus can be domain-specific data for translation. however, the shortcoming is that the generated targets can be low-quality and affect the machine translation performance. therefore, we start with the english-vietnamese machine translation model from @xcite , denoted bt a , which achieves stateof-the-art results on both en-vi biomedical and general translation domains. we use bt a to translate and generate a synthetic parallel biomedical dataset with 1m pairs of english-vietnamese biomedical abstracts from the pubmed corpus. the new 1m en-vi biomedical pairs are then concatenated with the current high-quality en-vi translation dataset from mtet @xcite and phomt @xcite , increasing from 6.2m to 7.2m en-vi sentence pairs total. to verify the effectiveness of our new selftraining data, we re-finetune the bt a model on this 7.2m bitexts corpus. we report the model performance on the medical test set from mtet and the general test set from phomt in the results show that our model outperforms existing machine translation systems in english-vietnamese translation by applying self-training. we obtain a significant gain of 6.61 bleu score (38.69->45.61) on the mtet medical test set. our model with self-training also achieves state-of-theart results on the phomt general domain test set by 0.53 bleu score (45.47->46.01). this shows that our approach not only improves the english-vietnamese translation performance in the biomedical context but also generalizes to general translation. we further discuss our self-training model performance on other translation domains in appendix a. ## vipubmed after developing a new state-of-the-art machine translation system for english-vietnamese translation in the biomedical domain in section 3, we apply the system, denoted bt b , on downstream translation to generate the first large-scale synthetic translated biomedical corpus for vietnamese. to ensure that our translated vipubmed dataset contains up-to-date biomedical research (for example, covid-19 diseases and covid-19 vaccines), we use the newest pubmed22 7 which contains approximately 34 million english biomedical abstracts published. the raw data is compressed in xml format. we then parse these structured xmls to obtain the abstract text with pubmed parser 8 @xcite . the machine translation model bt b is an encoder-decoder transformer based model with 512 token-length for input and output. therefore, 7 https://ftp.ncbi.nlm.nih.gov/pubmed/baseline 8 https://github.com/titipata/pubmed_parser we filter out english pubmed abstracts with more than 512 tokens. for fair size comparison with the unlabeled dataset of other health-related vietnamese pretrained models (discussed in section 7.2), we take a subset of 20m biomedical abstracts (20gb of text) for translation and leave a larger subset for future releases. we then translate the 20m english biomedical abstracts with the bt b model using 4 tpuv2-8 and 4 tpuv3-8. ## vimednli along with an unlabeled dataset for pretraining, we also introduce a benchmark dataset generated by translation and refined with human experts. we start with a natural language inference (nli) task as it is less prone to errors in biomedical entity translation compared to named-entity recognition (ner) or relation extraction (re) tasks. the process of creating the vimednli is shown in figure 2 . #3: abbreviation in english is wrong when translated to vietnamese. #4: the word "term" is misspelled as "herm". #5: the word "osteophytes" is misspelled as "sotesophytes". #6: the word "delirium" is misspelled as "delerium". ## vipubmedt5 with an unlabeled synthetic translated vipubmed corpus (section 4) and a benchmark vimednli dataset (section 5), we pretrain and finetune a transformer-based language model @xcite to verify the effectiveness of our approach in enriching vietnamese biomedical domain with translation data. we explain our model and the pretraining settings we applied in this section. ## scaling to other languages our novel way of utilizing a state-of-the-art nmt system to generate synthetic translated medical data for pretrained models is not limited to the vietnamese language and is scalable to many other low-resource languages. recent works focus on improving the quality of multiple low-resource nmt systems @xcite @xcite . these new state-of-the-art nmts make the approach discussed in this paper more practical to produce synthetic translated biomedical data, enriching the biomedical nlp research knowledge in multiple low-resource languages. ## conclusion we utilize the state-of-the-art translation model mtet to scale up the very low-resourced yet highly valuable biomedical data in vietnamese. namely, vipubmedt5, a t5-style encoder-decoder transformer pretrained on a large-scale translated corpus of the biomedical domain that demonstrated stateof-the-art results on both inference and acronym disambiguation in the biomedical domain. we also introduced vimednli, a machine-translated and human-expert refined benchmark in natural language inference to further grow the vietnamese suite of benchmarks and data in biomedical data.