DM ix: Distance Constrained Interpolative Mixup (2021)

Abstract: Interpolation-based regularisation methods have proven to be effective for various tasks and modalities. Mixup is a data augmentation method that generates virtual training samples from convex combinations of individual inputs and labels. We extend Mixup and propose DMix, distance-constrained interpolative Mixup for sentence classification leveraging the hyperbolic space. DMix achieves state-of-the-art results on sentence classification over existing data augmentation methods across datasets in four languages.

## introduction deep learning models are effective across a wide range of applications. however, these models are prone to overfitting when only limited training data is available. interpolation-based approaches such as mixup @xcite have shown improved performance across different modalities. mixup over latent representations of inputs has led to further improvements, as latent representations often carry more information than raw input samples. however, mixup does not account for the spatial distribution of data samples, and chooses samples randomly. while randomization in mixup helps, augmenting mixup's sample selection strategy with logic based on the similarity of the samples to be mixed can lead to improved generalization. further, natural language text possesses hierarchical structures and complex geometries, which the standard euclidean space cannot capture effectively. in such a scenario, hyperbolic geometry presents a solution in defining similarity between latent representations via hyperbolic distance. we propose dmix, a distance-constrained interpolative data augmentation method. instead of choosing random inputs from the complete training * equal contribution distribution as in the case of vanilla mixup, dmix samples instances based on the (dis)similarity between latent representations of samples in the hyperbolic space. we probe dmix through experiments on sentence classification tasks across four languages, obtaining state-of-the-art results over existing data augmentation techniques. ## methodology interpolative mixup given two data samples x i , x @xmath0, y @xmath1, mixup @xcite uses linear interpolation with mixing ratio r to generate the synthetic sample @xcite performs performs linear interpolation over the latent representations of models. let f θ (•) be a model with parameters θ having n layers, f θ,n (•) denotes the n-th layer of the model and h n is the hidden space vector at layer n for @xmath2, n ] and h 0 denotes the input vector. to perform interpolative mixup at a layer k ∼ [1, n ], we first calculate the latent representations separately for the inputs for layers before the k-th layer. for input samples x i , x j , we let h i n , h j n denote their respective hidden state representations at layer n, we then perform mixup over individual hidden state representations h i k , h j k from layer k as, the mixed hidden representation h k is used as the input for the continuing forward pass, ## conclusion we propose dmix, an interpolative regularization based data augmentation technique sampling inputs based on their latent hyperbolic similarity. dmix achieves state-of-the-art results over existing data augmentation approaches on datasets in four languages.we further analyze dmix through ablations over different similarity threshold values across the languages. dmix being data-, modality-, and model-agnostic, holds potential to be applied on text, speech, and vision tasks. 1 we obtain similar results for ttc and ghc.