Retrofitting Multilingual Sentence Embeddings with A bstract M eaning R epresentation (2022)

Abstract: We introduce a new method to improve existing multilingual sentence embeddings with Abstract Meaning Representation (AMR). Compared with the original textual input, AMR is a structured semantic representation that presents the core concepts and relations in a sentence explicitly and unambiguously. It also helps reduce the surface variations across different expressions and languages. Unlike most prior work that only evaluates the ability to measure semantic similarity, we present a thorough evaluation of existing multilingual sentence embeddings and our improved versions, which include a collection of five transfer tasks in different downstream applications. Experiment results show that retrofitting multilingual sentence embeddings with AMR leads to better state-of-the-art performance on both semantic textual similarity and transfer tasks.

## introduction multilingual sentence embedding (mse) aims to provide universal sentence representations shared across different languages @xcite @xcite . as an important ingredient of crosslingual and multilingual natural language processing (nlp), mse has recently attracted increasing attention in the nlp community. mse has been widely adopted to bridge the language barrier in several downstream applications such as bitext mining @xcite , document classification @xcite @xcite and natural language inference @xcite . prior work typically borrows fixed-size embedding vectors from multilingual neural machine models @xcite or trains siamese neural networks to align the semantically similar sentences written in different languages @xcite @xcite . despite the recent progress, the current evaluation of multilingual sentence embeddings has focused on cross-lingual semantic textual similarity (sts) @xcite or bi-text mining tasks @xcite . nevertheless, as pointed out by @xcite , the evaluation on semantic similarity may not be sufficient because better performance on sts does not always indicate better embeddings for downstream tasks. therefore, for a more comprehensive mse evaluation, it is necessary to additionally evaluate downstream tasks, which is largely ignored in recent work @xcite @xcite . in this paper, we collect a set of multilingual transfer tasks and test various existing multilingual sentence embeddings. we find that different methods excel at different tasks and the conclusions drawn from the sts evaluation do not always hold in the transfer tasks and vice versa. we aim to establish a standardized evaluation protocol for future research in multilingual sentence embeddings. to improve the quality of existing mse models, we explore abstract meaning representation (amr) @xcite , a symbolic semantic representation, for augmenting existing neural semantic representations. our motivation is twofold. first, amr explicitly offers core concepts and relations in a sentence. this helps prevent learning the superficial patterns or spurious correlations in the training data, which do not generalize well to new domains or tasks @xcite . second, amr reduces the variances in surface forms with the same meaning. this helps alleviate the data sparsity issue as there are rich lexical variations across different languages. on the other hand, despite that amr is advocated to act as an interlingua @xcite @xcite , little work has been done to reflect on the ability of amr to have impact on subsequent tasks. in order to advance research in amr and its applications, multilingual sentence embedding can be seen as an important benchmark for highlighting its ability to abstract away from surface realizations and represent the core concepts expressed in the sentence. to our knowledge, this is the first attempt to leverage the amr semantic representation for multilingual nlp. we learn amr embeddings with contrastive siamese network @xcite and amr graphs derived from different languages @xcite . experiment results on 10 sts tasks and 5 transfer tasks with four state-of-the-art embedding methods show that retrofitting multilingual sentence embeddings with amr improves the performance substantially and consistently. our contribution is three-fold. ## related work universal sentence embeddings our work aims to learn universal sentence representations, which should be useful for a broad set of applications. there are two lines of research for universal sentence embeddings: unsupervised approaches and supervised approaches. early unsupervised approaches @xcite @xcite design various surrounding sentence reconstruction/prediction objectives for sentence representation learning. @xcite exploit sentencelevel discourse relations as supervision signals for training sentence embedding model. instead of us-ing the interactions of sentences within a document, @xcite propose to learn the embeddings for texts of arbitrary length on top of word vectors. likewise, chen (2017); pagliardini et al. (2018); yang et al. (2019b ) calculate sentence embeddings from compositional n-gram features. recent approaches often adopt contrastive objectives @xcite @xcite @xcite @xcite by taking different views-from data augmentation or different copies of models-of the same sentence as training examples. on the other hand, supervised methods @xcite @xcite take advantage of labeled natural language inference (nli) datasets @xcite , where a sentence embedding model is finetuned on entailment or contradiction sentence pairs. furthermore, @xcite ; @xcite demonstrate that bilingual and back-translation corpora provide useful supervision for learning semantic similarity. another line of work focuses on regularizing embeddings @xcite @xcite to alleviate the representation degeneration problem. very recently, @xcite combine the strengths of amr metrics and embedding similarities for accurate and explainable sentence similarity rating. multilingual sentence embeddings recently, multilingual sentence representations have attracted increasing attention. @xcite ; @xcite ; @xcite propose to use encoders from multilingual neural machine translation to produce universal representations across different languages. chidambaram et al. (2019); wieting et al. (2019); yang et al. (2020); feng et al. (2020) fine-tune siamese networks @xcite with contrastive objectives using parallel corpora. @xcite train a multilingual model to map sentences to the same embedding space of an existing english model. different from existing work, our work resorts to multilingual amr, a languageagnostic disambiguated semantic representation, for performance enhancement. evaluation of sentence embeddings traditionally, the mainstream evaluation for assessing the quality of english-only sentence embeddings is based on the semantic textual similarity (sts) tasks and a suite of downstream classification tasks. the sts tasks @xcite @xcite @xcite @xcite calculate the embedding distance of sentence pairs and compare them with the human-annotated scores for semantic similarity. the classification tasks (e.g., sentiment analysis) from senteval (conneau and kiela, 2018) take sentence embeddings as fixed input features to a logistic regression classifier. these tasks are commonly used to benchmark the transferability of sentence embeddings on downstream tasks. for multilingual sentence embeddings, most previous work has focused on crosslingual sts @xcite and the relevant bi-text mining tasks @xcite . the evaluation on downstream transfer tasks has been largely ignored @xcite @xcite . nevertheless, as pointed out in @xcite in english scenarios, better performance on semantic similarity tasks does not always indicate better embeddings for transfer tasks. for a more comprehensive evaluation, in this paper, we collect a set of multilingual transfer tasks and test various existing multilingual sentence embeddings. we aim to establish a standardized evaluation protocol for future research in multilingual sentence embeddings. ## proposed method we first introduce how we learn amr embeddings and then describe the whole pipeline for enhancing existing sentence embeddings. ## evaluation benchmark to provide a more comprehensive evaluation of multilingual sentence representations, in addition to traditional semantic textual similarity tasks, we also introduce a set of downstream transfer tasks. ## conclusion this paper presented a thorough evaluation of existing multilingual sentence embeddings, ranging from traditional text similarity measurement to a new variety of transfer tasks. we found that different methods excel at different tasks. we then proposed to improve existing methods with universal amr embeddings, which leads to better performance on all tasks.