GRI : Graph-based Relative Isomorphism of Word Embedding Spaces (2023)

Abstract: Automated construction of bi-lingual dictionaries using monolingual embedding spaces is a core challenge in machine translation. The end performance of these dictionaries relies on the geometric similarity of individual spaces, i.e., their degree of isomorphism. Existing attempts aimed at controlling the relative isomorphism of different spaces fail to incorporate the impact of lexically different but semantically related words in the training objective. To address this, we propose GRI that combines the distributional training objectives with attentive graph convolutions to unanimously consider the impact of lexical variations of semantically similar words required to define/compute the relative isomorphism of multiple spaces. Exper imental evaluation shows that GRI outperforms the existing research by improving the average P@1 by a relative score of upto 63.6%.

## introduction bilingual lexical induction (bli) aims at the construction of lexical dictionaries using different mono-lingual word embeddings. automated construction of bilingual dictionaries plays a significant role, especially for resource-constrained languages where hand-crafted dictionaries are almost non-existent. it is also a key tool to bootstrap the performance of many down-streaming applications, e.g., cross-lingual information retrieval @xcite , neural machine translation @xcite . the most prevalent way for the construction of cross-lingual embeddings is to map the monolingual embeddings in a shared space using linear and/or non-linear transformations, also known as mapping-based methods @xcite @xcite . a core limitation of the mapping-based methods is their reliance on the approximate isomorphism assumption, i.e., the underlying monolingual embedding spaces are geometrically similar. this severely limits the applicability of the mapping-based methods to closely related languages and similar data domains. this isomorphism assumption does not hold, especially in case of domain-mismatch and for languages exhibiting different characteristics @xcite @xcite . other dominant factors identified in the literature that limit the end performance of bli systems include: (i) linguistic differences (ii) algorithmic mismatch, (iii) variation in data size, (iv) parameterization etc. similar to the supervised models, the unsupervised variants of bli are also unable to cater to the above-mentioned challenges @xcite . instead of relying on embedding spaces trained completely independent of each other, in the recent past there have been a shift in explicitly using the isomorphism measures alongside distributional training objective @xcite . in order to control the relative isomorphism of monolingual embedding spaces, these models use existing bilingual dictionaries as training seeds. however, one core limitation of these models is their inability to incorporate the impact of semantically relevant tokens into the training objective. this severely deteriorates the relative isomorphism of the resultant cross-lingual embedding space. this phenomenon is illustrated in figure 1 for english and ukrainian languages. for example, for the english language, we often use terms {"terrible", "horrible"} within the same context without a significant change in the meaning of the sentence. for these terms, corresponding terms in the ukrainian language {"страшни ´й", "жахливо"} may also be used interchangeably without a significant change in the context. likewise, for the bottom row in figure 1 , the words {"good", "great", "excellent"} are semantically related words in the english language, with {"вiдмiнно", "чудово", "добре"} as corresponding semantically related words in the ukrainian language. to address these challenges, in this paper we propose a novel framework named: graph-based relative isomorphism (gri). gri uses attentive graph convolutions to pay attention to semantically related tokens, followed by using isomorphism metrics to inject this information into the model training. later, it combines the isomorphism loss with the distributional training objective to train the complete model. we argue gri offers a better alternative for bli, as it allows injecting information about the semantic variations of tokens in the training objective, which is a more natural setting in order to control the relative isomorphism of linguistic data. an immediate benefit of the proposed model is obvious in the domain-mismatch settings, where attentive graph convolutions mechanism by gri offer the ability to unanimously analyze and/or model similar concepts represented by lexically varying terms across different corpora. this is also evident by a relatively stable performance of gri for both domain-sharing and domain-mismatch settings (section 6.1). we summarize the core contributions of this paper as follows: ## related work due to limited space, we primarily categorize the related work on relative isomorphism of crosslingual embeddings into: (i) mapping to shared space, and (ii) joint training. mapping to shared space. these models aim to find a linear and/or non-linear transformation for pre-trained word embeddings in order to map them to a shared space. these models rely on the assumption that the embedding models share similar structure across different languages @xcite , which allows them to independently train embeddings for different languages and learn mapping functions to align them in a shared space. supervised variants in this regard use existing bilingual resources, such as parallel dictionaries @xcite @xcite . the unsupervised variants use distributional matching @xcite @xcite . these models have also been applied to the contextualized embeddings @xcite . joint training these models put additional constraints on model learning, i.e., a hard or soft cross-lingual constraints in addition to the monolingual training objectives. similar to the mappingbased models, early works in this domain include the supervised variants relying on bilingual dictionaries @xcite @xcite . recently, the unsupervised approaches have gained attention because of their ease of implementation. for instance, @xcite analyzed the performance for concatenated monolingual corpora with shared vocabulary without any additional cross-lingual resources. results show that this setting outperforms many carefully crafted alignment based strategies for unsupervised machine translation. other unsupervised approaches with good results on benchmark data sets include zero-shot cross-lingual transfer by @xcite and cross-lingual pre-training by @xcite . @xcite proposed isovec that introduces multiple different losses along with the skip-gram loss function to control the relative isomorphism of monolingual spaces. a major limitation of these methods is their inability to incorporate the lexico-semantic variations of word pairs across different languages in the model training, which severely limits the end performance of these models. ## background in this section, we discuss the notation and mathematical background of the tools and techniques used in this paper. ## discussion in this sub-section, we perform a detailed analysis of the performance of gri. specifically, we analyze: (i) domain mis-match settings (ii) impact of attentive convolutions, (iii) isometric metrics, and (iv) error cases. ## conclusion and future directions in this paper, we propose graph-based relative isomorphism (gri) to incorporate the impact of lexical variations of semantically related tokens in order to control the relative isomorphism of crosslingual embeddings. gri uses multiple different isomorphism losses (guided by the attentive graph convolutions) along with the distributional loss to perform bli in a performance-enhanced fashion. experimental evaluation shows that gri outperforms the existing research on bli by a significant margin. some probable future directions include: (i) extending the concepts learned in this research to contextualized embeddings, and (ii) augmenting the gri to focus more on preserving lexicosemantic relations.