Finding F riends and Flipping Frenemies: Automatic Paraphrase Dataset Augmentation Using Graph Theory (2020)

Abstract: Most NLP datasets are manually labeled, so suffer from inconsistent labeling or limited size. We propose methods for automatically improving datasets by viewing them as graphs with expected semantic properties. We construct a paraphrase graph from the provided sentence pair labels, and create an augmented dataset by directly inferring labels from the original sentence pairs using a transitivity property. We use structural balance theory to identify likely mislabelings in the graph, and flip their labels. We evaluate our methods on paraphrase models trained using these datasets starting from a pretrained BERT model, and find that the automatically-enhanced training sets result in more accurate models.

## introduction having high quality annotated data is crucial for training supervised machine learning models. however, producing large datasets with good labeling quality is expensive and labor intensive. most nlp datasets rely on labels provided by human annotators with varying skills and limited training and expertise. the label instances are also often based on ambiguous definitions and guidelines. to address this problem, we study automated techniques to improve datasets for training and testing. in particular, we focus on paraphrase identification task, which aims to determine whether two given sentences are semantically equivalent. the sentences and labels in a dataset can be viewed as nodes and edges of a graph. moving from single labeled sentence pairs to a graph provides a better understanding of the sentence relations of the dataset, which can be exploited to infer additional edge labels. in particular, since paraphrases are an equality relation, we can perform a transitive closure on the graph to infer additional labels. in addition, we use the notion of balance @xcite for signed graphs to identify conflicted relations. in the context of semantic relationships between pairs of sentences, any paraphrases of a given sentence cannot be a non-paraphrase of each other since they should all share an identical meaning. contributions. we show the benefits of representing sentence pair relations as a graph. we first construct a paraphrase graph with the original pairs and their relation labels from the quora question pairs (qqp) dataset @xcite following the structure of a signed graph. with the graph structure and the transitivity of paraphrases, we can automatically infer new sentence pair relations directly from the original dataset (section 3.1). in addition, we identify and correct likely mislabeled pairs based on violations of expected structural balance properties we expect a valid paraphrase graph to satisfy (section 3.2). we found 90 seemingly mislabeled sentence pairs in the qqp dataset. we show that fine-tuning a bert model on the augmented set improves its performance on both the original and augmented testing sets, decreasing the error rate from 10% to under 6% when testing on the augmented test set. we released the augmented qqp dataset and the implementation code. ( https://github.com/hannahxchen/automaticparaphrase-dataset-augmentation ) ## representing datasets as graphs a signed graph is a graph where each edge is labeled either positive or negative to indicate a relationship between the two connected nodes. for undirected graphs, this relationship is symmetric. a path is a set of connected edges with no repeated nodes, and a path with the last node connecting back to the first node forms a cycle. given semantic interpretations of the edge labels, all paths in a signed graph should have certain properties. structural balance. balance theory was proposed by @xcite to study interpersonal relationships in social psychology. the idea was generalized to signed graphs by @xcite . a graph is said to be balanced if the product of the edge signs in every cycle is positive. there are only two types of conditions exist in a balanced signed graph: paraphrase graph. the definition of a paraphrase remains ambiguous and varies by task, but the most common definitions are similar to the one from @xcite , which define paraphrases as sentences that convey the same meaning but are expressed in different forms. since this notion is a symmetric relation, we can form an undirected signed graph by linking the sentence pairs from the paraphrase dataset with their annotated relations. sentence pairs labeled as paraphrases are connected with positive edges; sentences labeled as non-paraphrases are connected with negative edges. a paraphrase cluster contains sentences connected with positive edges, and all sentences in the cluster should share the same meaning. figure 2 shows how a paraphrase graph is constructed from selected labeled pairs in the qqp dataset. ## improving datasets using graphs typically, training sets for paraphrase identification are constructed by using annotations for sentence pairs provided by human annotators. based on the semantics implied by the paraphrase and nonparaphrase labels, we can augment and correct the sentence-level paraphrase graph. our method infers labels based on transitivity (section 3.1), and identifies likely mislabelings based on expected graph consistency properties (section 3.2). ## experiments to understand the effectiveness and impact of our augmentation and correction methods, we compare the preforms of bert models fine-tuned to the paraphrase identification task on the original qqp dataset and three datasets derived using the graphbased methods from the previous section. ## related work the most closely related work from @xcite also applies paraphrase graphs to generate additional paraphrase and non-paraphrase pairs. similar to our method, they infer non-paraphrase pairs from sentences within different paraphrase groups, and use transitivity to find new paraphrase pairs. different from our work, they generate additional paraphrase pairs by pairing sentences to themselves, and reversing the order of each sentence pair. other than using structural balance, their method can only identify conflicted labels between pairs of sentences. in addition, they only apply data augmentation on the training sets and evaluate their models directly on the original testing sets. we infer additional data and identify conflicts for both training and testing, which illustrates the full potential of our data augmentation method. besides, @xcite propose a graphbased method to improve the quality of paraphrase generation. they represent phrases as nodes and translation similarities as edges from a bilingual parallel corpus, and infer paraphrases with the pivot based method, which finds phrases with the same translation. however, this method can only infer new paraphrases within a path length of two. @xcite use a simpler approach by generating new paraphrase pairs with the reflexive and symmetric property of paraphrases with no graph involved. the non-paraphrase pairs are sentences randomly selected from two different pairs, which can not be guaranteed to have a correct relation. ## conclusion in this paper, we show the benefit of representing datasets as graphs. we develop methods based on graph theory to automatically expand a paraphrase dataset and improve labeling consistency. our experiments show an improvement on the augmented-flipped testing set after correcting the conflicted labels in the original training set, and the combination of the two methods produce a model that gives the best performance across all testing sets.