Improving Large-scale Paraphrase Acquisition and Generation (2022)

Abstract: This paper addresses the quality issues in existing Twitter-based paraphrase datasets, and discusses the necessity of using two separate definitions of paraphrase for identification and generation tasks. We present a new Multi-Topic Paraphrase in Twitter (MultiPIT) corpus that consists of a total of 130k sentence pairs with crowdsoursing (MultiPIT_crowd) and expert (MultiPIT_expert) annotations using two different paraphrase definitions for paraphrase identification, in addition to a multi-reference test set (MultiPIT_NMR) and a large automatically constructed training set (MultiPIT_Auto) for paraphrase generation. With improved data annotation quality and task-specific paraphrase definition, the best pre-trained language model fine-tuned on our dataset achieves the state-of-the-art performance of 84.2 F1 for automatic paraphrase identification. Furthermore, our empirical results also demonstrate that the paraphrase generation models trained on MultiPIT_Auto generate more diverse and high-quality paraphrases compared to their counterparts fine-tuned on other corpora such as Quora, MSCOCO, and ParaNMT.

## introduction paraphrases are alternative expressions that convey a similar meaning @xcite . studying paraphrase facilitates research in both natural language understanding and generation. for instance, identifying paraphrases on social media is important for tracking the spread of misinformation @xcite and capturing emerging events @xcite . on the other hand, paraphrase generation improves the linguistic diversity in conventional agents @xcite and machine translation @xcite . it has also been successfully applied in data argumentation to improve information extraction @xcite and question answering systems (gan and ng, 2019). 2. so so so so thankful it's finally friday. many researchers have been leveraging twitter data to study paraphrase given its lexical and style diversity as well as coverage of up-to-date events. however, existing twitter-based paraphrase datasets, namely pit-2015 @xcite and twitter-url @xcite , suffer from quality issues such as topic unbalance and annotation noise, 1 which limit the performance of the models trained using them. moreover, past efforts on creating paraphrase corpora only consider one paraphrase criteria without taking into account the fact that the desired "strictness" of semantic equivalence in paraphrases varies from task to task @xcite . for example, for the purpose of tracking unfolding events, "a tsunami hit haiti." and "303 people died because of the tsunami in haiti" are sufficiently close to be considered as paraphrases; whereas for paraphrase generation, the extra information "303 people dead" in the latter sentence may lead models to learn to hallucinate and generate more unfaithful content. in this paper, we present an effective data collection and annotation method to address these issues. we curate the multi-topic paraphrase in twitter (multipit) corpus, which includes mul-tipit crowd , a large crowdsourced set of 125k sentence pairs that is useful for tracking information on twitter, and multipit expert , an expert annotated set of 5.5k sentence pairs using a stricter definition that is more suitable for acquiring paraphrases for generation purpose. compared to pit-2015 and twitter-url, our corpus contains more than twice as much data with more balanced topic distribution and better annotation quality. two sets of examples from multipit are shown in figure 1 . we extensively evaluate several state-of-the-art neural language models on our datasets to demonstrate the importance of having task-specific paraphrase definition. our best model achieves 84.2 f 1 for automatic paraphrase identification. in addition, we construct a continually growing paraphrase dataset, multipit auto , by applying the automatic identification model to unlabelled twitter data. empirical results and analysis show that generation models fine-tuned on multipit auto generate more diverse and high-quality paraphrases compared to models trained on other corpora, such as mscoco @xcite , paranmt (wieting and gimpel, 2018), and quora. 2 we hope our multipit corpus will facilitate future innovation in paraphrase research. ## multi-topic pit corpus in this section, we present our data collection and annotation methodology for creating multi-pit crowd and multipit expert datasets. the data statistics is detailed in ## paraphrase generation paraphrase generation is a task that rewrites the input sentence while preserving its semantic meaning. since new data is generated on twitter every day, we introduce multipit auto , an automated continual growing dataset for paraphrase generation. we show that the model fine-tuned on multi-pit auto generates more diverse and high-quality paraphrases than other paraphrase datasets. ## other related work besides the several frequently used paraphrase datasets we mentioned above, here are a few other paraphrase corpora. the msr paraphrase corpus @xcite ) contains 5,801 sentences pairs from news articles, but it has a deficiency that skewing toward over-identification @xcite and having high lexical overlap @xcite . ppdb @xcite contains over 220 million phrase and lexical paraphrases without any sentence paraphrases. wikianswer @xcite consists of 18 million word-aligned question pairs. however, same as quora, wikianswer is restricted to only questions. in addition, the semantic textual similarity (sts) shared task @xcite measures the degree type definition generation example mauto paranmt good paraphrase type number of occurrences per generation: 1.53 1.22 add new add new phrases while keeping the meaning of given sentence. sent: relax, take it easy. gen: relax, take a deep breath, and enjoy the moment. 0.18 0.03 word syn substitute a word with its synonym (another word). sent: which is the best gre coaching centre in bangalore? gen: what is the best gre training centre . . . 0.39 0.54 phrase syn replace a phrase with synonym or expand a word to a phrase. sent: it looks goddamned foolish to put an oyster on the clam. gen: putting an oyster on a clam is a fucking joke. 0.28 0.16 structure use different sentence structures to express the same thing. sent: two big plates filled with some tasty looking food. gen: two big plates of food, and the food looks good. 0.28 0.23 simplification delete minor details or shorten phrases while maintaining the meaning of given sentence. sent: daniel farke sacked by norwich after first win of premier league season over brentford. gen: norwich sack daniel farke after first win of premier league season. 0.39 0.26 poor paraphrase type percentage in 200 generations: 26% 44% copy copy the given sentence. sent: did you have a good day today? gen: did you have a good day today? 14.5% 10% small change only have small changes such as changing article, tense, or prep. sent: fda approves new test that can detect coronavirus in 45 minutes. gen: the fda has approved a new test . . . 5.5% 18% hallucination add new info that is not commonsense or world knowledge. sent: a dog at a table wearing a birthday hat. gen: a dog wearing a birthday hat at a dinner party. 2.5% 0% miss info miss important info in the given sentence. sent: very sad though that the amazing aj and kai will be missing the final. gen: aj and kai will not be in the final. 1.5% 1% misinterpret misinterpret or contradict meaning of the given sentence. sent: why are most first basemen left handed? gen: why do most of the first basemen have left hands? 2% 14% bad grammar contain grammar error. gen: what is the best earphones for rs 3000? 0% 1% ## conclusion in this paper, we present the multi-topic paraphrase in twitter (multipit) corpus. our work surpasses prior twitter-based paraphrase corpora in topic diversity as well as the quality and quantity of annotation. experimental results demonstrate the necessity of defining paraphrases based on downstream tasks. our paraphrase generation evaluation shows that models trained on our corpus have better generation quality and generalizability compared to models fine-tuned on existing widely-used paraphrase datasets. we believe that multipit will facilitate further research in both paraphrase identification and paraphrase generation.