Ask Language Model to Clean Your Noisy Translation Data (2023)

Abstract: TTransformer models have demonstrated remarkable performance in neural machine translation (NMT). However, their vulnerability to noisy input poses a significant challenge in practical implementation, where generating clean output from noisy input is crucial. The MTNT dataset is widely used as a benchmark for evaluating the robustness of NMT models against noisy input. Nevertheless, its utility is limited due to the presence of noise in both the source and target sentences. To address this limitation, we focus on cleaning the noise from the target sentences in MTNT, making it more suitable as a benchmark for noise evaluation. Leveraging the capabilities of large language models (LLMs), we observe their impressive abilities in noise removal. For example, they can remove emojis while considering their semantic meaning. Additionally, we show that LLM can effectively rephrase slang, jargon, and profanities. The resulting datasets, called C-MTNT, exhibit significantly less noise in the target sentences while preserving the semantic integrity of the original sentences. Our human and GPT-4 evaluations also lead to a consistent conclusion that LLM performs well on this task. Lastly, experiments on C-MTNT showcased its effectiveness in evaluating the robustness of NMT models, highlighting the potential of advanced language models for data cleaning and emphasizing C-MTNT as a valuable resource.

## introduction neural machine translation (nmt) has witnessed significant progress @xcite @xcite in recent years, particularly with the introduction of transformer @xcite . despite their impressive performance on clean benchmarks @xcite @xcite , these models exhibit a noticeable decline in translation quality when exposed to noisy input. this hampers their performance in real-world scenarios, where human users unintentionally introduce misspellings and grammatical errors during text input @xcite . therefore, evaluating the robustness of nmt models against noisy inputs becomes crucial before their deployment. despite the importance of assessing the resilience of nmt models against noise, the available evaluation datasets remain limited. to the best of our knowledge, mtnt @xcite stands as one of the few well-established resources for evaluating nmt models' performance in the presence of noise. the noise distribution in mtnt closely resembles real-world use cases, but its applicability is constrained by the presence of noise in the target sentences. for instance, a french-english pair may appear as: "reeeeeeeeeeee les normies sur mon eiffel y'en a marre" ↔ "reeeeeee bored of the normies on my eiffel", which is not desirable. our expectation is that an effective nmt model is capable of translating a noisy source sentence into a clean target sentence. in order to enhance the applicability of mtnt for evaluating nmt models, we propose cleaning the target side of this dataset. in contrast to conventional cleaning approaches @xcite @xcite that typically involve filtering out undesirable sentences and retaining only high-quality ones, we aim to remove noise from sentences without reducing the overall sample number. language-aware rulebased approaches, which rely on predefined rules to eliminate noise, have been widely employed as a common method for such cleaning @xcite . while these methods can effectively remove certain types of noise, it becomes impractical to define rules for every possible noise source. moreover, some natural noise introduced by human input is not easily identified by rule-based approaches alone. another highly promising approach involves leveraging large language models (llms) @xcite . previous works have already demonstrated the effectiveness of llms in various tasks, including q&a @xcite ), text summarization @xcite , and data generation @xcite @xcite . however, applying an llm to clean the target sentences poses several challenges that need to be addressed diligently: in this paper, we propose to apply gpt-3.5 @xcite to clean the noisy target sentences in the mtnt dataset. our approach addresses the challenges of noise removal, semantic preservation, and alignment with the source sentence. inspired by the success of prompting methods @xcite @xcite , we design few-shot prompts to guide gpt-3.5 to clean the noisy target sentences in three ways: (1) utilizing information from both the source and target side for cleaning, (2) cleaning the target sentence independently, and (3) generating a clean target sentence by translating the noisy source sentence. through a comprehensive analysis, we demonstrate the effectiveness of our cleaning methods, particularly with methods (1) and (3). these methods successfully remove emojis, while considering their semantic meaning, and also rephrase slang, jargon, and profanities appropriately. the resulting datasets, named c-mtnt, exhibit significantly reduced levels of noise compared to the original sentences, while still preserving their semantic integrity. furthermore, our research highlights another remarkable potential of llms in generating highquality parallel data with limited monolingual resources. this finding has significant implications for low-resource domains and languages, where acquiring parallel corpora is often challenging. after the cleaning, we conduct human and gpt-4 (openai, 2023) evaluations. both evaluations draw the same conclusion that llm has a great capability for such a cleaning task, especially with method (1). in the end, we conduct comprehensive experiments to train nmt models on some noisy training datasets, and evaluate them on different evaluation sets, including clean benchmarks and the benchmark constructed with a rule-based noise removal method. c-mtnt consistently demonstrates better noise evaluation manner on the nmt models. to the best of our knowledge, this is the first study to apply llms in the context of cleaning data, specifically addressing the challenges outlined above. our findings highlight the potential of leveraging advanced language models for datacleaning tasks and emphasize c-mtnt as a valuable resource for evaluating the nmt model in real-world scenarios. ## data generation and cleaning the primary objective of this study is to harness the capabilities of llms in effectively removing noise and generating parallel language datasets to evaluate the robustness of nmt models against noisy input. in this section, we will give an overview of our data source mtnt and delineate our approach to llm interaction. ## analysis on c-mtnt in this section, we analyze the generated data to evaluate its quality and suitability as a noise evaluation benchmark. we compare our method to a rule-based baseline and quantitatively assess the level of noise present in the target sentences. in addition, we also compare the new target samples from c-mtnt to the original ones by evaluating their semantic similarity. ## machine translation experiments in this section, we further investigate the suitability of c-mtnt as a benchmark to evaluate nmt model's robustness against noisy input. let's reemphasize our expected nmt model: irrespective of whether the source sentence is clean or noisy, the model has the ability to generate a coherent target sentence that is free of errors or distortions. we first mimic the real-world scenario to train a set of nmt models on datasets that contain both noisy and clean source sentences but with only clean target sentences, then evaluate these models on c-mtnt and other benchmarks. ## related work the application of llms, such as gpt-3.5 (textdavinci-003) (openai, 2021), in downstream tasks has garnered significant attention and led to extensive research in this field @xcite . researchers have conducted comprehensive investigations to explore the capabilities of llms, and built upon their various applications @xcite @xcite . researchers have also observed the potential of llms to generate high-quality data, leading to a focus on expanding existing datasets, augmenting data, or generating entirely new data @xcite @xcite . these efforts have helped address the issue of data scarcity in various domains. however, it is crucial to note that the aforementioned research works lack at least one of the two novel aspects addressed in this paper. firstly, our research focuses on evaluating the robustness of nmt models to real-world noise introduced by human users. secondly, we explore the generation or cleaning of parallel data specifically for nmt purposes. these unique aspects of robustness evaluation and parallel data generation/cleaning contribute to the existing literature in a novel way. ## conclusion in this work, we propose three methods to apply llm to clean a noisy mt benchmark, mtnt, where natural noise exists in both source and target sentences. we aim to clean the target side of mtnt to make it more suitable as a benchmark in evaluating nmt models' robustness against noisy input. with a meticulous design of some few-shot prompts, we guide gpt to clean the noisy target sentence with only the noisy source sentence, only the noisy target sentence, or both noisy source and target sentences. by measuring the noise frequency in the cleaned target sentences, measuring the semantic similarity between noisy and cleaned target sentences, and evaluating with human annotators and gpt-4, we show that our proposed methods can effectively remove natural noise with llm, while preserving the semantic structure. our further investigation of the newly created benchmark, c-mtnt, on some trained models also shows its effectiveness as a noise evaluation benchmark for nmt models.