A Fast Method to Filter Noisy Parallel Data WMT 2023 Shared Task on Parallel Data Curation (2023)

Abstract: The effectiveness of a machine translation (MT) system is intricately linked to the quality of its training dataset. In an era where websites offer an extensive repository of translations such as movie subtitles, stories, and TED Talks, the fundamental challenge resides in pinpointing the sentence pairs or documents that represent accurate translations of each other. This paper presents the results of our submission to the shared task WMT2023 (Sloto et al., 2023), which aimed to evaluate parallel data curation methods for improving the MT system. The task involved alignment and filtering data to create high-quality parallel corpora for training and evaluating the MT models. Our approach leveraged a combination of dictionary and rule-based methods to ensure data quality and consistency. We achieved an improvement with the highest 1.6 BLEU score compared to the baseline system. Significantly, our approach showed consistent improvements across all test sets, suggesting its efficiency.

## introduction neural machine translation (nmt) has revolutionized the field of machine translation by utilizing deep learning algorithms to learn from large amounts of data and generate high-accurate translations @xcite . however, the success of nmt models heavily depends on the quantity and quality of data used for training. on low-resource language pairs, the nmt architectures perform poorly @xcite and are more sensitive to noisy data than statistical machine translation (smt) methods @xcite . therefore, access to vast cleaned corpus can significantly improve the performance of nmt models, allowing them to learn and produce more accurate translations @xcite . fortunately, very large text sources offer a massive collection of data for various types of content, including movie subtitles, stories, and ted talks. these resources have not been fully exploited for nmt training due to the lack of alignment between the source and target languages. furthermore, the parallel data which movie subtitles also could be noisy with poor accuracy @xcite . to address this challenge, wmt2023 introduced a shared task on parallel data curation for the estonian-lithuanian (et-lt) language pair, focusing on finding the best possible training data set within the web-crawled data to train a downstream nmt model @xcite . among the popular solutions, @xcite introduced using vecalign to embed sentences and compute the cosine similarity of sentence pairs. following this method, the shared task provides participants with extensive cosine similarity files and laser embeddings generated by the laser model @xcite . participants are tasked with identifying the most optimal parallel data to train the mt models. although this approach performs efficiently in many cases, @xcite has shown that the cosine similarity has several limitations. because the sentence representation in vector space could be impacted by word frequency. to tackle this problem, we build a pipeline to improve the quality of the parallel corpus. our contributions focus on: the related work is presented in section 2. the detail of our method is described in section 3, experi-ments and results are shown in section 4. finally, the analysis is presented in section 4.5. ## related work the wmt2023 shared task builds upon previous shared tasks focused on document alignment (wmt 16) and sentence filtering (wmt 18, 19, 20) @xcite @xcite . previously, several researchers proposed a method to align documents, such as @xcite used the phrase table to align in the search space and then fill in and refine alignments. moreover, @xcite employed the vecalign to gain the sentence embeddings. nevertheless, sentence alignments based on cosine similarity have some limitations because the cosine scores could be dense in the range of 0.5 to 1 @xcite . and with the same query sentence, the higher score could not determine the quality of the parallel sentence. in addition, for the filtering shared task, participants applied filtering rules to eliminate noisy data, including removing too long/short sentences, using language identification for source and target (kejriwal and koehn, 2020) or fine-tuning pre-trained models such as bert, xlm to re-score sentence pairs @xcite @xcite . besides, @xcite created artificially noisy data by generating inadequate and nonfluent translations. they used this noisy data to train a classifier to distinguish between high-quality and low-quality sentence pairs within a corpus containing noise. we found the related ideas from @xcite . both of these approaches only focus on the alignment rules and adopt the other pretrained models. junczys-dowmunt (2018) trained an nmt model to filter data and became standard for the high-resource case. nevertheless, when training an original nmt model with low or noisy resources, the nmt model could face certain limitations. in our work, we utilize the phrase table to compute edit distance and extract the superior sentences. furthermore, we introduce a pipeline to re-rank sentences based on their top-k cosine similarity scores and extract the best compact corpus for training purposes. the detail of our method is presented in section 3. ## experiments in this section, we describe the experimental setup for our system, including the data, training tools, and baseline system. ## analysis in this section, we delve deeply into our approaches and the scale of our data corpora. firstly, we conduct some experiments to find the best threshold when selecting the top-k highest cosine similarity score. for every source sentence, our approach involves selecting a single target sentence from a set of eight candidates based on the highest cosine similarity score provided. secondly, we conduct a statistical analysis to determine the number of sentence pairs that achieve the highest cosine score but are not considered parallel sentences. ## conclusion in conclusion, our study has provided valuable insights into the performance of different methods employed in our research on wmt2023 parallel data curation shared tasks. our findings reveal that while the laser baseline and using the dictionary method exhibited moderate and consistent performance across several metrics, the accumulative filtering approach, particularly when adopting lower threshold values (0.85, 0.8, and 0.75), demonstrated notable improvements in various aspects. notably, the selection of the threshold played a pivotal role in influencing performance outcomes. furthermore, our analysis also encompassed the identification of sentence pairs that exhibit parallel characteristics, even if they may not always possess the highest cosine similarity scores. in the future, further investigation and task-specific considerations will be essential in finding the smallest possible set of training data and achieving the highest result.