DM ix: Adaptive Distance-aware Interpolative Mixup (2022)

Abstract: Interpolation-based regularisation methods such as Mixup, which generate virtual training samples, have proven to be effective for various tasks and modalities. We extend Mixup and propose DMix, an adaptive distance-aware interpolative Mixup that selects samples based on their diversity in the embedding space. DMix leverages the hyperbolic space as a similarity measure among input samples for a richer encoded representation.DMix achieves state-of-the-art results on sentence classification over existing data augmentation methods on 8 benchmark datasets across English, Arabic, Turkish, and Hindi languages while achieving benchmark F1 scores in 3 times less number of iterations. We probe the effectiveness of DMix in conjunction with various similarity measures and qualitatively analyze the different components.DMix being generalizable, can be applied to various tasks, models and modalities.

## introduction deep learning models, though effective for many applications are prone to overfitting in absence of sufficient training data. data augmentation techniques can efficiently use this limited training data @xcite . interpolationbased augmentation techniques such as mixup @xcite have shown improved performance across different modalities. mixup over latent representations of inputs leads to further improvements @xcite . however, mixup does not account for the spatial distribution of dataset samples, but choosing samples randomly for interpolation-based augmentation. while randomization in mixup helps, augmenting mixup's sample selection strategy with logic based on the similarity of the samples to be mixed can lead to improved generalization @xcite , 2020b). the relative spatial position of samples can be leveraged to produce more suitable synthetic inputs for training underlying models @xcite . further, natural language possesses hierarchical structures and complex geometries, which the standard euclidean space cannot capture effectively @xcite . hyperbolic geometry presents a solution in defining similarity between latent representations @xcite . we propose dmix, an adaptive distance-aware interpolative data augmentation method. instead of choosing random inputs from the complete training distribution as in the case of mixup, dmix samples instances based on the (dis)similarity between latent representations of samples in the hyperbolic space. furthermore, dmix performs interpolations with trainable pair-wise parameters derived from the spatial distribution of the samples rather than sampling mixing ratios randomly from standard distributions, making it adaptive for pair-wise interpolation. our contributions are: • we propose dmix, a novel adaptive distanceaware interpolative regularization method developed over the spatial distribution of dataset sampled in the hyperbolic space. ## methodology we present an overview of dmix in figure 1 . we first introduce interpolative mixup ( §2.1), and then formulate dmix by leveraging the relative sample distribution in the hyperbolic space ( §2.2). ## experimental setup we evaluate dmix on standard english, glue, and multi-lingual datasets in 4 languages ( ## conclusion we propose dmix, a novel data augmentation technique that interpolates samples intelligently chosen based on their hyperbolic distance in the embedding space. dmix achieves state-of-the-art results over existing data augmentation approaches on 8 standard and multilingual datasets in english, arabic, turkish, and hindi languages, requiring 3 times less number of iterations than random mixup. dmix being independent of the underlying model and modality, holds potential to be applied on text, speech, and vision downstream tasks. we compare the performance of dmix on standard english and glue datasets with additional baselines and interpolative augmentation methods like emix @xcite and ssmix @xcite .