P ara Z h-22 M : A Large-Scale C hinese Parabank via Machine Translation (2022)

Abstract: Paraphrasing, i.e., restating the same meaning in different ways, is an important data augmentation approach for natural language processing (NLP). Zhang et al. (2019b) propose to extract sentence-level paraphrases from multiple Chinese translations of the same source texts, and construct the PKU Paraphrase Bank of 0.5M sentence pairs. However, despite being the largest Chinese parabank to date, the size of PKU parabank is limited by the availability of one-to-many sentence translation data, and cannot well support the training of large Chinese paraphrasers. In this paper, we relieve the restriction with one-to-many sentence translation data, and construct ParaZh-22M, a larger Chinese parabank that is composed of 22M sentence pairs, based on one-to-one bilingual sentence translation data and machine translation (MT). In our data augmentation experiments, we show that paraphrasing based on ParaZh-22M can bring about consistent and significant improvements over several strong baselines on a wide range of Chinese NLP tasks, including a number of Chinese natural language understanding benchmarks (CLUE) and low-resource machine translation.

## introduction a paraphrase is a restatement of meaning with different expressions @xcite . paraphrasing has been proven to be an effective data augmentation approach for many nlp tasks, ranging from linguistically controlled paraphrase generation @xcite @xcite , style transfer @xcite , to applications like low-resource machine translation @xcite and automatic mt evaluation @xcite . @xcite extract sentence-level paraphrases from multiple chinese translations of the same source texts, and create the largest chinese paraphrase bank (pku parabank) to date, which contains 509,832 pairs of paraphrased sentences. however, the amount of one-to-many sentence translation data constrains the size of pku parabank, and it cannot meet the requirement to train large chinese paraphrasers. inspired by @xcite and @xcite , we propose to relax the restriction that requires one-to-many translation data on the construction of large-scale chinese parabanks, by utilizing bilingual one-to-one translation data of larger scales and mt, and construct parazh-22m. specifically, we leverage the huge chinese-english machine translation data from wmt 2021 @xcite of 30.4m sentence pairs, apply strict rules to ensure the data quality, and translate the english side of the parallel corpus to chinese with the cutting-edge deep transformers and several approaches to ensure the translation quality. we pick the machine translated chinese sentences considering both diversity and semantic consistency, and pair with the corresponding original chinese references to form paraphrase pairs. compared to the pku parabank, parazh-22m is ∼40 times as large, involves a broader range of paraphrase phenomena and domains, and can support the training of large chinese paraphrasers. our main contributions are as follows: zhang et al. (2019b) extract sentence-level paraphrases from multiple chinese translations of the same source texts. their approach requires oneto-many translation data, which is hard to collect. instead, we try to relieve this restriction in chinese parabank construction, and build the parabank based on one-to-one sentence-level translation data. specifically, we translate the translation of chinese sentences in the parallel data back to chinese with the cutting-edge neural machine translation (nmt) technology, and construct the parabank by pairing the machine translated chinese sentences with the corresponding original chinese sentences. we suggest that the semantic consistency and quality of the parallel bank is ensured by the cutting-edge nmt algorithm, as the translation quality of advanced nmt methods is already close to that of translation agencies in high resource scenarios @xcite . the construction of parazh-22m can be divided into 4 steps: 1) data collection, 2) data processing and cleaning, 3) training of nmt models, and 4) paraphrase generation. ## evaluation of parazh-22m we compare the constructed parazh-22m with two existing chinese paraphrase datasets: pku paraphrase bank @xcite and chinese paraphrase from quora @xcite . chinese translations of the same source texts (written in english as well as other european languages). ## using parazh-22m in chinese nlp we examine the effectiveness of data augmentation based on parazh-22m on a number of chinese nlp tasks, including long/short text classification, natural language inference, keyword recognition from clue (a chinese language understanding evaluation benchmark) @xcite and the ccmt 2022 low-resource @xmath0, by paraphrasing the original training set. ## related work data augmentation via paraphrasing is beneficial for many nlp tasks, such as question answering @xcite , semantic parsing @xcite and machine translation @xcite , especially in low-resource scenarios. paraphrasing relies heavily on large scale paraphrase datasets. construction of english paraphrase data most paraphrase corpus construction studies are for english @xcite . given the development of nmt, @xcite leverage large amounts of bilingual parallel data to generate paraphrases via mt. @xcite add lexical constraints during nmt decoding to enrich the diversity. @xcite cluster over constrained sampling decoding candidates to generate diverse paraphrases. compared to wieting and gimpel (2018) and @xcite , we assign a higher priority to the quality of machine translation than the diversity to ensure the translation correctness and the semantic consistency, without using constrained decoding or sampling that hampers the translation quality. chinese paraphrase corpus to our knowledge, existing chinese parabanks are much smaller than large scale english parabanks. zhang et al. (2019b) extract sentence-level paraphrases from multiple chinese translations of the same source text, obtaining the pku paraphrase bank of 509,832 paraphrase pairs. @xcite translate the question retelling quora corpus into chinese with multiple mt engines, and construct a chinese parabank of 263,729 sentence pairs. compared to their work, parazh-22m is much larger and involves many domains. ## conclusion in this paper, we relieve the requirement of one-tomany translation data for the construction of chinese parabank, and construct a chinese parabank of 22m sentence pairs, parazh-22m, utilizing one-toone sentence-level parallel data and mt technology. parazh-22m involves many domains and is over 40 times as large as the previous largest pku chinese paraphrase bank. human evaluation on semantic consistency, fluency and sentential diversity shows the good quality of parazh-22m. we test the effects of data augmentation via paraphrasing based on parazh-22m on a wide range of chinese nlp tasks, including short/long text classification, natural language inference, keyword recognition, and low-resource machine translation. our experiment results show that paraphrasing based on parazh-22m is able to achieve consistent and significant improvements over several baselines in all evaluations, demonstrating the contribution of parazh-22m to chinese nlp tasks.