PCC : Paraphrasing with Bottom-k Sampling and Cyclic Learning for Curriculum Data Augmentation (2023)

Abstract: Curriculum Data Augmentation (CDA) improves neural models by presenting synthetic data with increasing difficulties from easy to hard. However, traditional CDA simply treats the ratio of word perturbation as the difficulty measure and goes through the curriculums only once. This paper presents PCC : P araphrasing with Bottom-k Sampling and C yclic Learning for C urriculum Data Augmentation, a novel CDA framework via paraphrasing, which exploits the textual paraphrase similarity as the curriculum difficulty measure. We propose a curriculum-aware paraphrase generation module composed of three units: a paraphrase candidate generator with bottom-k sampling, a filtering mechanism and a difficulty measure. We also propose a cyclic learning strategy that passes through the curriculums multiple times. The bottom-k sampling is proposed to generate super-hard instances for the later curriculums. Experimental results on few-shot text classification as well as dialogue generation indicate that PCC surpasses competitive baselines. Human evaluation and extensive case studies indicate that bottom-k sampling effectively generates super-hard instances, and PCC significantly improves the baseline dialogue agent.

## introduction data augmentation techniques create artificial data mixed with the original data for improved performance. traditional data augmentation techniques in the language community include word-level perturbation such as synonym replacement, random insertion, random swap, and random deletion @xcite . sentence-level techniques such as round-trip translation @xcite exploits the use of machine translation models to translate the input sentence to another language before translating back to the source language which can be essentially treated as a form of paraphrasing. curriculum learning presents training instances in a meaningful order with increasing difficulties to neural models for a boost in performance. traditional curriculum learning @xcite @xcite @xcite categorizes the original training instances into different levels of difficulties to be gradually presented to the model where a core component called difficulty measure, which is usually defined as a numerical number where a bigger number indicates a more difficult sample. combining the merits of the above two mentioned techniques, curriculum data augmentation (cda) creates synthetic data with increasing levels of difficulties to be presented to our neural models. existing cda defines the ratio of the words perturbation as the difficulty measure for curriculums and a gradual course which increases the difficulty of curriculums when the training loss plateaus @xcite , which then ends when the most challenging curriculum ends. although existing cda is effective, yet there are several disadvantages. first, it employs word-level perturbation. this superficial operation keeps the augmentation to have a similar sentence structure as the original one. next, it employs random insertion, random swap, and random deletion for augmentation. although this can be durable as for text classification @xcite , this is not suitable for generation tasks, particularly when many words are perturbed, which can even easily break the sentence grammar. third, it uses a gradual course that only enters each level of difficulty once. a typical problem in neural network training called catastrophic forgetting @xcite can potentially happen in such a course, where the model might undesirably gradually forget some early learned knowledge. to mitigate the problems of word-level perturba-tion, we propose that paraphrasing can be a source of data augmentation, which provides diverse and grammatically correct augmentation. however, it is non-trivial to utilize paraphrase augmentation in a curriculum setting. inspired by the fundamental linguistic concept of mutual implication @xcite , we treat two sentences as a pair of paraphrases if they can infer each other. for example, 'i am glad to help you.' and 'let me help you out!' can be a pair of paraphrases, which provides a diverse change of the sentence structure suitable for the curriculum setting. we also employ textual similarity for our difficulty measures for the curriculum. higher scores indicate that two sentences are more textually similar to each other. specifically, we treat pairs with lower scores as more difficult instances to be presented in later curriculums. we propose a paraphrase candidate generator integrated with bottom-k sampling. traditional sampling methods such as top-k sampling @xcite and top-p @xcite sampling tend to generate easier paraphrases that have relatively high similarity scores. we propose bottom-k sampling to generate super-hard paraphrases for the later harder curriculums by pruning the most probable words. 1 this leads the generation towards a more grammatically and lexically diverse paraphrase sampling space with low textual similarity. to mitigate catastrophic forgetting, we propose to incorporate cyclic learning to pass through the curriculums multiple times. in summary, our proposed framework, called pcc: paraphrasing with bottom-k sampling and cyclic learning for curriculum data augmentation, makes three contributions: experimental results indicate that pcc surpasses competitive baselines on few-shot text classification as well as dialogue generation. human evaluation indicates that bottom-k sampling effectively generates grammatically and lexically rich paraphrases, and pcc significantly improves our baseline dialogue agent. to our best knowledge, this is the first time to apply cda on a generation task. takeaway overall, we present the effectiveness of paraphrasing as a curriculum data augmentation technique. the use of cyclic learning and bottom-k sampling further boosts performance. with some modifications, future works can treat pcc as a data augmentation framework and adapt it to other downstream tasks. future works can also leverage bottom-k sampling in generating textual outputs that are grammatically and lexically rich. ## experimental setup in our experiments, we define six curriculums ranging from 0 to 5. 0 represents the original data, and 1 and 5 represent the easiest and the most difficult curriculum respectively. 6 4.1 few-shot text classification task for the downstream application task for our experiments, we follow @xcite to conduct the task of few-shot, highly multi-class text classification @xcite , which typically has a large number of classes with only a few samples for each of the class. we use triplet loss, a loss computed with three elements, namely, an anchor a, a positive sample p, and a negative sample n. it origins from the vision community @xcite , which was later applied to language tasks @xcite , suitable for the few-shot setting. precisely, the learning objective is defined as: three datasets for the text classification task are used in our experiments, namely, huffpost @xcite , covid-q @xcite , and amzn @xcite . for space reasons, we leave their detailed dataset description in appendix b. ## conclusions we propose a novel framework that uses mutual implicative paraphrasing as a curriculum data augmentation technique. our proposed curriculumaware paraphrase generation module is composed of three components, a paraphrase candidate generator with a bottom-k sampling strategy for generating superhard paraphrases, a paraphrase filtering mechanism, and a difficulty measure. we propose a bottom-k sampling strategy to effectively generate super-hard instances with grammatical and lexical richness to be used for the later stages in curriculum learning. moreover, we propose a cyclic learning strategy that mitigates catastrophic forgetting. experimental results on the task of few-shot text classification as well as dialogue generation support our proposed methodology pcc's usefulness, surpassing several competitive baselines.