CIA ug: Equipping Interpolative Augmentation with Curriculum Learning (2022)

Abstract: Interpolative data augmentation has proven to be effective for NLP tasks. Despite its merits, the sample selection process in mixup is random, which might make it difficult for the model to generalize better and converge faster. We propose CIAug, a novel curriculum-based learning method that builds upon mixup. It leverages the relative position of samples in hyperbolic embedding space as a complexity measure to gradually mix up increasingly difficult and diverse samples along training. CIAug achieves state-of-the-art results over existing interpolative augmentation methods on 10 benchmark datasets across 4 languages in text classification and named-entity recognition tasks. It also converges and achieves benchmark F1 scores 3 times faster. We empirically analyze the various components of CIAug, and evaluate its robustness against adversarial attacks.

## introduction data augmentation is an effective tool for avoiding overfitting in model training in cases where there is an absence of sufficient training data @xcite . interpolative augmentation techniques, such as mixup @xcite , have shown an increase in model performance across various modalities, with further improvements gained by applying mixup at latent representation layers @xcite . current implementations of mixup select samples for interpolation at random, not leveraging the potential for adaptive selection techniques which have been shown to lead to better generalizability @xcite . in addition, these methods do not account for the spatial distribution of linguistic data, known to extend beyond the capacities of euclidean space @xcite . we propose ciaug foot_0 , a method which addresses these challenges by offering an augmentation procedure that selects samples in an adaptive fashion and is geometrically sound. ciaug's sampling strategy follows the idea that selecting easier mixing samples first and gradually increasing sample difficulty based on relative spatial position would generate more suitable synthetic inputs, resulting in better model training @xcite . this notion ties in with the framework of curriculum learning @xcite , where training data is presented in a similarly staggered way, increasing model capabilities @xcite . ciaug's selection strategy performs distance operations in hyperbolic space, applying insights about language data spatial distribution to its definition of 'similar samples' @xcite . we first train ciaug by sampling pairs of sentences similar to each other for mixing, then gradually as training progresses, we sample sentences that are dissimilar, following the curriculum learning strategy. through experiments on 10 benchmark datasets in english, arabic, turkish and german, we show that ciaug outperforms state-of-the-art models in classification and named-entity recognition tasks. we probe the effectiveness of ciaug in conjunction with different similarity measures and qualitatively evaluate it. we show that ciaug converges faster than traditional mixup while being both generalizable across tasks and languages, as well as more resilient to adversarial classification examples. ## methodology we illustrate ciaug's sample selection strategy in figure 1 . in this section, we first introduce mixup ( §2.1), and follow by formulating ciaug and its relative sample distribution component ( §2.2). ## experimental setup we evaluate ciaug on classification and ner tasks in various settings across 4 languages, and on glue datasets @xcite . ## results and analysis we present our main results in ## conclusion we propose ciaug, a novel mixup technique that uses curriculum learning, leveraging the relative spatial positions of the samples in the embedding space as a measure of complexity to signal curriculum learning.