Biomedical Parallel Sentence Retrieval Using Large Language Models (2023)

Abstract: We have explored the effect of in domain knowledge during parallel sentence filtering from in domain corpora. Models built with sentences mined from in domain corpora without domain knowledge performed poorly, whereas model performance improved by more than 2.3 BLEU points on average with further domain centric filtering. We have used Large Language Models for selecting similar and domain aligned sentences. Our experiments show the importance of inclusion of domain knowledge in sentence selection methodologies even if the initial comparable corpora are in domain.

## introduction this paper describes fjwu's submission to the biomedical translation task. this year the focus of our research was domain specific parallel corpus mining from wikipedia using large language models, we explored the potential of the mined sentences using two sentence selection schemes. neural machine translation (nmt) @xcite @xcite has witnessed great success over the years @xcite . nmt systems train on parallel corpora to produce translations that capture language intricacies and context with enormous precision as compared to the previous counterpart statistical machine translation (smt) systems. machine translation in the biomedical domain is becoming increasingly important due to the critical nature of medical scientific texts. the majority of these texts are published in english, and the goal of biomedical machine translation is to make them accessible in multiple languages. however, this is a complex undertaking due to the extensive nature of this field and the vast and diverse vocabulary it encompasses. this vocabulary includes specialized terms and non-lexical forms (such as dates and biomedical entities) that pose unique challenges. consequently, the quality of machine translation output fluctuates depending on the availability of biomedical resources tailored to each target language. availability of parallel corpora in reasonable amounts has greatly enhanced the performance of nmt systems, especially for the high-resource languages @xcite . however, its efficacy remains sub optimal for low-resource languages and domain-specific contexts @xcite @xcite . performance of nmt system degrades as soon as the application domain deviates from training domain. domain adaptation @xcite , transfer learning @xcite @xcite , model fusion @xcite , back translation @xcite , fine-tuning @xcite , data augmentation @xcite , data selective training @xcite , decoding strategies @xcite , zero-shot translation @xcite are some of the techniques used to address this issue. we will be focusing on domain adaptation using data augmentation and fine tuning. for this years submission we explore the potential of large-scale language models for extracting parallel sentences from wikipedia foot_0 . french-english parallel articles are scraped as detailed in section 4. for learning sentence embeddings of scraped bilingual data, rather than training encoders from scratch, we leverage the potential of llm in parallel sentence extraction from our bilingual scraped articles. we used lealla-large, a lightweight system developed by @xcite to compute the language-agnostic low-dimensional sentence embeddings for each sentence in the english and french parallel articles. potential parallel sentences are filtered based on the similarity scores. these sentence are then further domain filtered by comparing the closeness with medline titles embeddings computed using transformers minilm. our experiments show the importance of inclusion of domain knowledge in sentence selection methodologies even if the initial comparable corpora are in domain. our main contributions include: the paper is structured as follows: section 2 presents a brief overview of background and related work, section 3,4 elaborates the data collection pipeline, section 5 outline the nmt experiments and results, followed by the conclusion of this study. ## related work recent work on parallel sentence extraction has focused on lightweight end-to-end word-level and sentence-level embedding methods @xcite @xcite . these embedding-based approaches have gained success @xcite @xcite as these systems outperformed the large-distributed computationally intensive systems @xcite used to mine parallel documents. bilingual sentence embeddings, learned from dual-encoder models, have also been used effectively for parallel corpus mining @xcite . cross-lingual embeddings encode bilingual texts into a single unified vector space allowing nearest-neighbor search can be used to find potential translation candidates. these embedding approaches produce noisy matches that require a re-scoring step in order to obtain a clean parallel sentence retrieval as addressed by @xcite who explored using a bi-directional dual encoder with additive margin softmax @xcite which results in state-of-the-art performance for sentence filtering. multilingual sentence embedding approaches @xcite also show promising results. since language-specific models often demand extensive amounts of labeled data for training and can be limited by their language-specific parameters, language-agnostic sentence embedding @xcite @xcite @xcite align multiple languages in a shared embedding space, facilitating parallel sentence alignment that extracts parallel sentences for training translation systems. among them, labse @xcite achieved state-of-the-art performance on various bi-text retrieval. the problem of inference speed and computation overhead of large language models was addressed by @xcite who proposed learning leight-weight language-agnostic sentence embeddings (lealla) with knowledge distillation @xcite . they reported significant reduction in computation overhead and inference speed by providing language-agnostic low-dimensional sentence embeddings. we also use lealla in the second phase of our pipeline for parallel sentence alignemnent. our primary objective was to collect a comprehensive dataset from the biomedical domain, we explored wikipedia's key biological categories and selected those having a substantial volume of articles. a brief overview of the selected subdomains is given below: ## parallel corpus mining this section presents an overview of our parallel data creation pipeline. wikipedia has been extensively used as a data resource for corpus development @xcite @xcite @xcite . we also used wikipedia's inter language links to mine potential parallel sentences by exploring the potential of large language models for filtering the closet candidates. our data preparation pipeline involves three main steps; 1) domain specific web scraping, 2) candidate sentence scoring and filtering and 3) domain adapted filtering. ## translation performance on retrieved sentences we used transformer base @xcite architecture provided by fairseq @xcite wmt20 testset model fine-tuning model name llmfilter name biofilter b1 -19.52 s1 @xmath0.12 sb1 20.29 s2 @xmath1.41 sb2 20.29 s3 @xmath2.54 sb3 20.58 s4 @xmath3.78 sb4 21.11 b2 -38.71 l5 @xmath4.69 lb1 21.81 l6 @xmath5.57 lb2 21.88 l7 @xmath6.62 lb3 22.07 l8 @xmath7.36 lb4 22.43 as transformer_iwslt_en_de. the relu activation function was used in all encoder and decoder layers. we optimize with adam @xcite , set up with a maximum learning rate of 0.0005 and an inverse square root decay schedule, as well as 4000 warmup updates. all corpora were segmented into subword units using sentence piece @xcite with a vocabulary of 32k units. we share the decoder input and output embedding matrices. models are trained with mixed precision and a batch size of 4096 tokens on a single gpu. systems were trained until convergence based on the bleu score on the development sets. evaluation was performed using sacrebleu @xcite . scores are chosen based on the best score on the development set (medline 18, 19), and the corresponding scores for that checkpoint are reported on medline 20 test set. for fine-tuned systems, the process starts with models trained to convergence, based on bleu score on dev sets. training then resumes using a selected portion of the training corpus using the same parameters and criterion as for the base systems. baseline we trained a smaller model b1 on the biomedical texts provided by the wmt'23 organizers: edp, medline abstracts and titles @xcite , scielo @xcite and the ufal medical corpus 11 consisting of cesta, ecdc, emea (opensubtitles), pattr medical and subtitles. we used a bigger model b2 by @xcite trained on wmt14 general domain corpus and wmt and supplementary biomed data including b1 data. ## conclusion in this study, we explored the potential of large language models for parallel sentence extraction from domain-adapted bilingual corpus extracted from wikipedia. on our dataset, we experimented with two data selection schemes and assessed the nmt performance for the biomedical domain. our findings demonstrate that merely web-mining from indomain corpus is not sufficient to improve domainspecific nmt performance but there is also a need for further filtering out out-domain sentences to improve the domain-specific nmt systems. leveraging large language models to extract in-domain parallel sentences resulted in improved nmt performance by outperforming the baseline with 2 bleu points.