Mitigating Semantic Leakage in Cross-lingual Embeddings via Orthogonality Constraint (2024)

Abstract: Accurately aligning contextual representations in cross-lingual sentence embeddings is key for effective parallel data mining. A common strategy for achieving this alignment involves disentangling semantics and language in sentence embeddings derived from multilingual pre-trained models. However, we discover that current disentangled representation learning methods suffer from semantic leakage—a term we introduce to describe when a substantial amount of language-specific information is unintentionally leaked into semantic representations. This hinders the effective disentanglement of semantic and language representations, making it difficult to retrieve embeddings that distinctively represent the meaning of the sentence. To address this challenge, we propose a novel training objective, ORthogonAlity Constraint LEarning (ORACLE), tailored to enforce orthogonality between semantic and language embeddings. ORACLE builds upon two components: intra-class clustering and inter-class separation. Through experiments on cross-lingual retrieval and semantic textual similarity tasks, we demonstrate that training with the ORACLE objective effectively reduces semantic leakage and enhances semantic alignment within the embedding space.

## introduction parallel datasets play a pivotal role in enhancing neural machine translation (nmt) performance @xcite . however, acquiring high-quality parallel texts is challenging, especially for lower-resourced languages where monolingual data is more abundant @xcite . in this context, effective approaches for mining parallel data are essential for applying nmt in practical scenarios @xcite . *work done during internship at naver cloud. 1 our code and models will be released at publication. chinese-english sentence pairs. figure 1( @xcite shows substantial overlap between semantic and language-specific representations. this overlap is effectively mitigated by the proposed oracle method, as shown in figure 1(b) . recent approaches to this problem utilize crosslingual sentence embeddings @xcite generated by multilingual pre-trained encoders such as multilingual bert @xcite , mbert) or xlm-roberta @xcite , xlm-r). these embeddings aim to align semantically similar sentences across languages into a unified latent space, facilitating the extraction of pseudo-parallel pairs @xcite . however, @xcite and @xcite demonstrate that embeddings of parallel sentences from these encoders form clusters by language rather than by semantics. building on this, they attempt to disentangle language-specific information from sentence embeddings, thereby distilling language-agnostic semantic embeddings. in order to achieve this, two premises need to be considered. given parallel sentence, (1) how well are the semantic representations aligned? (2) how well are the language-specific representations separated? prior works have primarily focused on the former, leaving the latter question underexplored. figure 1 (1) intra-class clustering for bringing related components closer in embedding space, (2) interclass separation for ensuring unrelated components to be distant. s and t represent source and target sentence input respectively. ŝm : source semantic representation; ŝl : source language representation; tm : target semantic representation; tl : target language representation. pus pair, indicating that while semantics are wellaligned with previous disentanglement methods, there is still substantial overlap between languagespecific and semantic information (figure 1a ). we define this issue as semantic leakage, which undermines the effectiveness of cross-lingual embeddings in accurately mining parallel pairs. by constraining orthogonality between semantic and language representations, we facilitate a clearer separation of language-specific information in the embedding space (figure 1b ). in this work, we introduce oracle (orthogonality constraint learning), a training objective aimed at enforcing orthogonality between semantic and language-specific representations. our goal is to render these two representations independent to each other, thus ensuring their clear differentiation in the embedding space @xcite . oracle consists of two key components: intra-class clustering and inter-class separation. as shown in figure 2 , intra-class clustering aligns related components more closely, while inter-class separation enforces orthogonality between unrelated components. our method is designed to be simple and effective, capable of being implemented atop any disentanglement methods. we explore a range of pre-trained multilingual encoders (laser @xcite , infoxlm @xcite , and labse @xcite ) to generate initial sentence embeddings. subsequently, we train each semantic and language multi-layer perceptrons (mlps) with or-acle to disentangle the sentence embeddings into semantics and language-specific information. experimental results on both cross-lingual sentence retrieval tasks @xcite and the semantic textual similarity (sts) task @xcite demonstrate higher performance on semantic embeddings and lower performance on language embeddings with oracle. this suggests that our method not only resolves semantic leakage but also enhances semantic alignment ( §6). our analysis further reveals that oracle leads to robust performance in challenging scenarios such as code-switching ( §7.1). to summarize, our contributions are threefold: (1) we make the first attempt to address the issue of semantic leakage, wherein a substantial amount of language-specific information is leaked into semantic representations. ## oracle the two key ingredients of oracle are intraclass clustering ( §4.1) and inter-class separation ( §4.2). we reformulate the losses originally derived in dream and meat and impose additional constraints to ensure orthogonality between semantic and language embeddings. following the setup introduced in section 3.1, oracle also uses semantic (mlp m ) and language mlp (mlp l ) to extract semantics (ŝ i m , ti m ) and language-specific information (ŝ i l , ti l ) for each language. 4.1 intra-class clustering (l ic ) l ic aims to bring relevant representations closer in the multilingual embedding space. as shown in figure 1a , we notice that previous methods lack a constraint to enforce language embeddings to be clustered within themselves. this causes the language-specific information to leak into the semantics, making it difficult to capture the underlying semantics of the sentence. we constrain this by imposing pairwise cosine distances of each language embeddings: ## conclusion we explore the issue of semantic leakage, which we define as when language-specific information is leaked into the semantic representations, across various multilingual encoders and objective functions. addressing this issue is crucial for achieving disentangled semantic and language representations, which is a cornerstone for effective parallel mining. we introduce oracle, a simple and effective training objective designed to enforce orthogonality between semantic and language embeddings. through comprehensive evaluations, we demonstrate that integrating oracle not only improves semantic alignment but also ensures clear separation of language representations, as evidenced by embedding space visualization. further, we conduct detailed analysis to understand the roles of the two key components of oracle: intra-class clustering and inter-class separation. while our study primarily focuses on integrating oracle with dream and meat, our method is easily adaptable to various frameworks, offering promising avenues for future work.