H yp M ix: Hyperbolic Interpolative Data Augmentation (2021)

Abstract: Interpolation-based regularisation methods for data augmentation have proven to be effective for various tasks and modalities. These methods involve performing mathematical operations over the raw input samples or their latent states representations - vectors that often possess complex hierarchical geometries. However, these operations are performed in the Euclidean space, simplifying these representations, which may lead to distorted and noisy interpolations. We propose HypMix, a novel model-, data-, and modality-agnostic interpolative data augmentation technique operating in the hyperbolic space, which captures the complex geometry of input and hidden state hierarchies better than its contemporaries. We evaluate HypMix on benchmark and low resource datasets across speech, text, and vision modalities, showing that HypMix consistently outperforms state-of-the-art data augmentation techniques. In addition, we demonstrate the use of HypMix in semi-supervised settings. We further probe into the adversarial robustness and qualitative inferences we draw from HypMix that elucidate the efficacy of the Riemannian hyperbolic manifolds for interpolation-based data augmentation.

## introduction deep learning methods have improved the state-ofthe-art in a wide range of tasks. yet, when only limited training data is available, they are prone to overfitting @xcite . numerous data augmentation techniques have been proposed, which involve performing operations such as cropping or rotation @xcite , or paraphrasing @xcite individual examples. however, * equal contribution. semi-supervised learning methods, on the other hand, can leverage unlabeled data for training. several semi-supervised methods use interpolation based regularization methods over unlabeled samples to predict soft labels, and combine them with existing labeled samples to increase the overall training data @xcite . semi-supervised methods use consistency based regularization training @xcite which makes the model predictions robust to perturbations on unlabeled samples. however, current semi-supervised learning methods do not generalize across modalities or datasets. existing data-augmentation and semi-supervised learning methods operate in the euclidean space, which is a simplified representative geometry. representations across modalities inherently possess properties that the euclidean space is incapable of modeling, and can be better expressed using the more general hyperbolic space @xcite . the interference of sound waves is hyperbolic in nature, which generates hyperboloid waveforms @xcite . natural language text exhibits hierarchical structure in a variety of respects and embeddings are more expressive when represented in the hyperbolic space @xcite . data augmentation using möbius operations over images has shown more diversification and generalization compared to euclidean operations @xcite . performing interpolative operations over representations having complex geometry in the hyperbolic space can lead to more suitable representations for model training. building on prior research in limited data and data augmentation studies, and the hyperbolic characteristics of speech, text, and vision, we propose hypmix 1 : a model, data, and modality agnostic interpolative regularization method operating in the hyperbolic space. we further extend hypmix to semi-supervised settings, which is especially effective in extremely low resource environments. we probe the effectiveness of hypmix through extensive experiments over three different tasks for supervised and semi-supervised settings on benchmark and low resource datasets across speech, text, and vision in different languages with varying class label distribution. hypmix outperforms current state-of-the-art modality and task specific data augmentation methods across all the datasets for both supervised and semi-supervised conditions. our contributions can be summarized as: semi-supervised learning methods leverage unlabeled data which is typically available in larger quantities @xcite . consistency regularization methods for semi-supervised learning predict soft labels for unlabeled data and train models on different permutations of labeled and unlabeled data @xcite . @xcite uses a label guessing strategy on different augmentations of unlabeled data and combines it with labeled data for training models. however, these methods perform label prediction for unlabeled data using euclidean operations. hyperbolic learning has proven to be effective in representing information where relations among data points possess hierarchical and tree-like nature @xcite . learning in the hyperbolic space has been applied to various natural language processing @xcite @xcite , and computer vision tasks @xcite as well as graph @xcite , sequence @xcite , and financial @xcite learning. however, the ability of the hyperbolic space to model complex representations while performing interpolative operations across modalities is unexplored. figure 1 : overview of hypmix and mixh applied at layer k over hidden representations of input x i and x j . we perform the forward pass for the inputs upto layer k, and use the mixed representation for the continued pass. ## methodology: hypmix we first formulate the task and introduce the hyperbolic space @xcite and mixup @xcite ( §3.1). using the hyperbolic operations, we then introduce mixup in the hyperbolic space ( §3.2), and extend it to operate on the hidden state representations of neural networks. we call the resulting approach hypmix. an overview of the steps is presented in figure 1 . we formulate hypmix for both supervised ( §3.3) and semisupervised ( §3.4) methods. we test hypmix on classification tasks across speech, text, and vision.