P ara T ag: A Dataset of Paraphrase Tagging for Fine-Grained Labels, NLG Evaluation, and Data Augmentation (2022)

Abstract: Paraphrase identification has been formulated as a binary classification task to decide whether two sentences hold a paraphrase relationship. Existing paraphrase datasets only annotate a binary label for each sentence pair. However, after a systematical analysis of existing paraphrase datasets, we found that the degree of paraphrase cannot be well characterized by a single binary label. And the criteria of paraphrase are not even consistent within the same dataset. We hypothesize that such issues would limit the effectiveness of paraphrase models trained on these data. To this end, we propose a novel fine-grained paraphrase annotation schema that labels the minimum spans of tokens in a sentence that don’t have the corresponding paraphrases in the other sentence. Under this setting, we frame paraphrasing as a sequence tagging task. We collect 30k sentence pairs in English with the new annotation schema, resulting in the ParaTag dataset. In addition to reporting baseline results on ParaTag using state-of-art language models, we show that ParaTag is especially useful for training an automatic scorer for language generation evaluation. Finally, we train a paraphrase generation model from ParaTag and achieve better data augmentation performance on the GLUE benchmark than other public paraphrasing datasets.

## introduction paraphrase identification and paraphrase generation are important problems in natural language processing. they are the essential tasks to verify whether machine learning models genuinely understand the text. a lot of research @xcite @xcite is devoted to identifying paraphrase between sentences, on various annotated datasets such as microsoft research paraphrase corpus g-1 it covers some of the same ground as spotlight but has nothing new to add. g-2 it revisits the same topic as spotlight, but does not say anything new. g-3 heavy rainstorms have caused floods in the area every year since 2003. g-4 since 2003, the area has been hit every year by heavy rainstorms and floods. the main reason is that a sentence could contain multiple pieces of information, i.e., thought groups, and two sentences may only partially overlap on a few thought groups. this fine-grained information cannot be captured by a single binary paraphrase label. for instance, we show a sentence pair labeled as paraphrase in mrpc dataset in the first row of to this end, we build a large-scale and diverse fine-grained paraphrasing dataset, paratag . for each sentence in a pair, the annotators must tag all spans that do not have the corresponding paraphrase in the other sentence. thus, we frame paraphrase detection as a sequence tagging problem, with a binary label for each word in each sentence. first, we hire labelers to re-annotate mrpc dataset with the fine-grained labeling schema. it turns out that actually 90% of the sentence pairs labeled as paraphrase in mrpc dataset are tagged with at least one phrase not being paraphrased by the other sentence. then, to improve the diversity of the dataset, we generate more sentence pairs by feeding paraphrases in mrpc as prompts to the language model gpt-3 @xcite . in total, we collect 30k sentence pairs and 840k word-level annotations in paratag dataset. we then fine-tune a variety of pre-trained language models on paratag and report the result. next, we use the above sequence labeling models @xcite trained on paratag as evaluators for natural language generation tasks like machine translation and summarization. different from rule-based heuristic methods such as bleu @xcite and unsupervised alignment-based methods such as bertscore @xcite , a sequence labeling model trained on the word-level annotations in paratag can provide a more accurate comparison between reference output and nlg model's output. on summarization and data2text benchmarks, we show that models trained on paratag outperform existing matching based nlg evaluator models by 13.0% and 6.0% respectively, and outperform bartscore @xcite trained with parabank @xcite by 3.8% and 2%. finally, we train a paraphrase generator based on sentence pairs in paratag for data augmentation. experiments show that the generator model trained on paratag can reach better data augmentation quality compared with models trained on existing paraphrasing datasets and further improves the bert-base @xcite model by 1.5% on glue dataset. we summarize our contributions in this work as follows: ## rethinking paraphrase dataset our work is motivated by a retrospect of the widely used mrpc paraphrasing dataset @xcite via a more strict standard of paraphrasing. specifically, we define that a pair of sentences are paraphrases only if they contain the same meanings for all semantic units. for instance, in the first row of we believe the sentence-level binary labeling schema is not suitable because the labelers may ignore some non-paraphrase phrases by hallucination, which leads to false positives, as in the example mentioned above. thus, we propose a novel annotation schema for fine-grained paraphrasing labels in section 4.2. to increase the data diversity, we leverage the strong generation capacity of gpt-3 to produce candidate sentence pairs. we then filter the generated data to make sure it is non-trivial to judge the paraphrasing relationship between sentences in each. we describe the details in section 4.1. in natural language generation (nlg) evaluation, one needs to compare a system prediction s with a reference r and output a score representing how well s aligns with r. a good evaluator model should produce scores highly correlated with human judgment. humans are sensitive to hallucinations, modifiers, and factual inconsistencies when judging a nlg system. thus, we believe that a model trained on fine-grained and diverse annotations can better capture these delicate aspects in a sentence, which is verified by detailed experiments in section 6. additionally, given the fine-grained word-level labels in paratag , we can select a subset of sentence pairs with the highest degrees of paraphrasing to train a data augmentation (da) model. section 6 shows that da models trained on paratag are more effective than those trained on other public paraphrasing datasets. ## paratag dataset in this section, we will introduce the details of our dataset annotation. we hire 4 native speakers, who come from an annotation company and are trained by a program manager based on our instructions, to annotate the dataset for 5 weeks with around $24k in total. we have two targets in building this dataset: 1) annotating fine-grained word-level paraphrasing labels between sentences, and 2) a high quality and diverse paraphrase corpus. thus, the labeling process is partitioned into the following three phases (figure 1 ). ## model this section introduces how to train models on the paratag dataset and apply them to various tasks. ## conclusion we propose a novel dataset paratag for a more fine-grained and diverse paraphrase dataset. we set baselines with state-of-art pretrained language models for the ease of future comparison. in addition, we utilize paratag to train nlg evaluators that better correlate with human judgment. finally, we are able to build a better data argumentation model from paratag for general nlu tasks.