A da PT : A Set of Guidelines for Hyperbolic Multimodal Multilingual NLP (2024)

Abstract: The Euclidean space is the familiar space for training neural models and performing arithmetic operations.However, many data types inherently possess complex geometries, and model training methods involve operating over their latent representations, which cannot be effectively captured in the Euclidean space.The hyperbolic space provides a more generalized representative geometry to model the hierarchical complexities of the tree-like structure of natural language.We propose AdaPT a set of guidelines for initialization, parametrization, and training of neural networks, which adapts to the dataset and can be used with different manifolds. AdaPT can be generalized over any existing neural network training methodology and leads to more stable training without a substantial increase in training time.We apply AdaPT guidelines over two state-of-the-art deep learning approaches and empirically demonstrate its effectiveness through experiments on three tasks over 12 languages across speech and text.Through extensive qualitative analysis, we put forward the applicability of AdaPT as a set of guidelines optimally utilizing the manifold geometry, which can be extended to various downstream tasks across languages and modalities.

## introduction using the euclidean geometric space for representing latent embeddings, performing mathematical operations, and training neural models is common and has proved to be effective across various tasks and modalities @xcite @xcite . this is mainly because the euclidean space is more convenient * equal contribution to use, and it is a natural generalization of the visual three-dimensional space. however, studies have shown that complex data types such as graphs and text exhibit a non-euclidean and complex nature, in which case the standard euclidean space may not be the most suitable geometric representation space @xcite . this has led to works defining neural models in the hyperbolic space using möbius operations of the riemannian geometry @xcite , outperforming standard euclidean methods across a variety of domains @xcite @xcite . the hyperbolic space has proven significantly effective for textual entailment tasks @xcite , as well as for interpolative augmentation for text and speech domains @xcite . these approaches consider a fixed radius of curvature for the poincaré ball model used as the hyperbolic representation during the course of training the models. they also use a default radius of curvature across the datasets and do not consider the specific extent of hyperbolic nature possessed by the dataset. the δ-hyperbolicity of a space is a measure of its tree-likeliness, indicating the extent of hierarchical or hyperbolic nature this space possesses @xcite . @xcite incorporate gromov's calculation of δ-hyperbolicity for a space and define the δ-hyperbolicity for a dataset. the δ-hyperbolicity of a dataset can be used to estimate the optimum radius of the poincaré disk in the hyperbolic space to represent the embedded dataset @xcite . this can lead to a more suitable riemannian manifold representation that can model complex geometries and latent representations of the dataset for performing mathematical operations and effectively training models. however, as the dataset level δ-hyperbolicity is determined using the latent embeddings given by the underlying encoder, the optimal radius of curvature changes as the weights of the base model are updated during the course of training. therefore, we hypothesize that a parameterized radius of curvature which is jointly optimized with the neural network training can effectively represent these embeddings at all steps of the training. we propose adaptive poincaré transfer (adapt), a set of guidelines that is based on initialization, parameterization and training of neural network, independent of model, dataset and modality, developed using standard möbius operations. adapt can be generalized over any existing neural model to equip it with the capabilities of the hyperbolic space in representing complex geometries, both at the input and the latent representation level. adapt is optimized for a dataset, as it operates in a riemannian space with a poincaré disk having a dataset-specific radius, and hence, it is the maximally suitable representation geometry. this radius of curvature is jointly optimized with the neural network training, enabling the model to 'adapt' to the dynamic latent representations of the input samples. to show the generalizability of adapt, we apply it over two existing stateof-the-art deep learning approaches, variational information bottleneck @xcite , which uses the information bottleneck principle on the latent representation, and ssmix @xcite , saliency-aware interpolation. through extensive experiments on datasets in 12 languages on sentence classification, natural language inference, named-entity recognition, and speech classification tasks, we present the improved performance of the existing methods when the proposed set of guidelines in adapt are followed, without any considerable increase in training time and resource requirements. by performing comprehensive qualitative experiments, we further analyze the effect of using adapt, and put forward its applicability for numerous multilingual language processing tasks leveraging the hyperbolic space. our contributions are: tifrea et al. (2019) propose the dataset level δ-hyperbolicity metric to empirically measure the tree-likeliness of the dataset. khrulkov et al. (2020) estimate the radius of curvature of the poincaré disk using the corresponding δhyperbolicity. these works, however, do not incorporate the dataset-specific hyperbolicity in training the underlying neural networks and use a constant curvature throughout the training process. regularization and data augmentation techniques are used for improving model generalization in the absence of required training data and avoiding model overfitting. variational information bottleneck (mahabadi et al., 2021) extends the information bottleneck principle to a neural training objective and is effective in training models in low resource settings suppressing irrelevant features and preventing overfitting. mixup @xcite techniques perform convex combinations over raw inputs or their latent representations @xcite to generate synthetic training data. saliency-aware interpolative regularization approaches @xcite have been introduced, which show performance improvement over randomized mixup methods. these methods function in the simplified euclidean space, which is unable to capture the complex characteristics possessed by the input samples or their latent representations. multilingual nlp is gaining widespread attention, but only a very small subset of languages are wellrepresented in progressing technologies and applications @xcite . techniques successful in the high resource scenario may not be effective for low resource languages that are of a different language family or very distinctive in linguistic and typological terms @xcite . a language agnostic set of guidelines can prove effective for wider research in multilingual nlp. ## adapt formulations we first formulate adapt using several model, modality, task, and dataset agnostic operations which we later use to effectively leverage the hyperbolic space over existing state-of-the-art methods ( §4). to give an overview of how initialization and parameterization work in adapt: (i) we first obtain the hyperbolicity i.e. the hierarchical tree-likeness of the dataset ( §3.2)(ii) this helps us obtain the poincaré ball radius of curvature for projection in the hyperbolic space to capture dataset's structure ( §3.3)(iii) finally, we propose trainable curvature to adapt to the dynamic nature of the encodings during training ( §3.4). we discuss the hyperbolic mathematical operations needed for adapt in section 3.1. ## conclusion, future work, limitations drawing inspiration from works showing that various datasets and their latent representations inherently possess hyperbolic characteristics and can be better represented in the hyperbolic space, we propose adapt, a data and task independent set of guidelines that can be applied over any existing neural network training method to maximally leverage the hyperbolic space. adapt obtains significant improvements over existing training methodologies on three tasks in 12 languages across text and speech without any computational overhead. as future work, we plan to extend adapt to multimodal and graph neural network training methods. though adapt is capable of utilizing the optimal representation space as it has a trainable curvature, it is difficult to theoretically claim when to use it purely based on the δ-hyperbolicity of the datasets as it is an underexplored area of research. we leave the deeper analysis of the hyperbolic space for nlp applications as future work.