Unsupervised Geometric and Topological Approaches for Cross-Lingual Sentence Representation and Comparison (2022)

Abstract: We propose novel structural-based approaches for the generation and comparison of cross lingual sentence representations. We do so by applying geometric and topological methods to analyze the structure of sentences, as captured by their word embeddings. The key properties of our methods are”:” (a) They are designed to be isometric invariant, in order to provide language-agnostic representations. (b) They are fully unsupervised, and use no cross-lingual signal. The quality of our representations, and their preservation across languages, are evaluated in similarity comparison tasks, achieving competitive results. Furthermore, we show that our structural-based representations can be combined with existing methods for improved results.

## introduction word embeddings are driven by distributional concepts, i.e. words can be described by their surrounding words. for example, the words "dog" and "cat" often occur in similar contexts, and therefore their embeddings are expected to be nearby, and to have similar distances to other words. such similarities are inherent to the real world (e.g., a dog is similar to a cat), and therefore should be language-agnostic. for that reason, the embedding spaces of different languages are expected to be near-isomorphic @xcite . notably, @xcite demonstrated that high degree of isomorphism can be reached with sufficient monolingual resources. this assumption has enabled various applications at the word level, e.g. generating cross-lingual word embeddings by mapping monolingual vector spaces @xcite . in this paper, we take a step further, leveraging the approximate isomorphism between monolingual spaces at the sentence level. considering each sentence as a point cloud (made by its word embeddings), our key argument is that these point clouds retain geometric and topological structures that should be preserved across languages. therefore, they have the potential to enable languageagnostic sentence representations. we investigate different approaches for extracting and utilizing such structures. firstly, we devise a geometric approach, based on the intra-distances of the word embeddings in a sentence. secondly, we explore a topological approach, borrowing methods from topological data analysis (tda). briefly, tda provides algebraic-topological methods to extract global structural information from shapes. these methods are coordinate free and invariant to isometries @xcite , which is highly desired in our setting. our main goal is to employ these structure-based features to generate novel cross-lingual sentence representations, in a fully unsupervised manner. in order to evaluate the cross-lingual nature of our representations, we experiment with similarity comparison tasks, including bilingual sentence retrieval @xcite and machine translation quality estimation @xcite . our contributions can be summarized as follows. (1) proposing the novel concept of exploiting the isomorphism of word embedding spaces at the sentence level. (2) devising fully unsupervised methods for cross-lingual sentence representation, based on geometric and topological approaches. (3) providing measures of similarity for the new representations. (4) evaluating the extent to which these representations are preserved across languages, via downstream similarity comparison tasks. ## isometry of word embedding spaces measuring and utilizing the similarities between word embedding spaces, is a well-studied topic in nlp. in this context, a standard assumption is that monolingual word embedding spaces are approximately isomorphic. a common use for such near-isomorphism is to search for a linear trans-formation between the embedding spaces of different languages @xcite @xcite . other studies argue that a better practice is to consider orthogonal transformations @xcite . these transformations have been used to induce bilingual dictionaries @xcite , as well as cross-lingual transfer learning @xcite . in fact, mapping-based approaches have become a prevalent way to learn cross-lingual embedding spaces. the isomorphism assumption is also used in fully unsupervised settings, including unsupervised bilingual lexicon induction @xcite and unsupervised machine translation @xcite . here, the alignment between the monolingual embedding spaces cannot be achieved by mapping pre-existing bilingual dictionaries. instead, it is achieved either by using adversarial training @xcite or by comparing the distribution of similarities or distances of the word embeddings across languages @xcite . as explained by @xcite , the isomorphism of embedding spaces can be extended to isometry, using normalization techniques. as the isometry of word embedding spaces becomes the premise for a large variety of methods, the following question arises: can we leverage the isometry of word embeddings at the sentence level? our approach is to take the embedding of a sentence to be the word-by-word embedding, resulting in a point-cloud (finite collection of points). assuming nearly-isometric word embeddings, one would expect that the geometric and topological structures of these point clouds are preserved across languages to some extent. for this reason, we devised methods to extract such structural information from sentences and provide means to compare the structures of different sentences. ## related work various studies of unsupervised cross-lingual sentence representations rely on aggregation of either mapped word embeddings or contextualized word embeddings from pre-trained multilingual models @xcite @xcite . these studies often rely (implicitly or explicitly) on the isometry assumption between word vector spaces, for constructing cross lingual mappings. however, they do not utilize the isometry for generating sentence representations. closer to our work are studies using structural similarities between languages. both @xcite and @xcite exploit the preservation of geometric structures between monolingual vector spaces for cross lingual mapping of word embeddings. however, neither refer to geometric structures of sentences. finally, several studies have used topological approaches in nlp related tasks, such as word sense disambiguation @xcite ) and text visualization @xcite . tda methods were also used to generate sentence and document representations. @xcite was the first to introduce the concept of topological text representation. built on this idea, recent studies designed various methods for document representations by computing persistent homology (see section 6.2) over their word embeddings. these methods were evaluated on tasks such as document classification and discourse analysis @xcite @xcite . most related to our work is @xcite , where persistence diagrams were used to represent documents and sentences. their final representation and comparison of sentences are quite different than ours, and achieved negative results in classification and clustering tasks. to the best of our knowledge, no previous study used topological-based approaches in cross-lingual tasks. ## sentence distance matrix in this section we present the fundamental element of our pipeline -the sentence distance matrix (sdm). representing sentences by point clouds, the geometric information about the sentence can be encoded by the pairwise distances between the words. formally, let @xmath0, ..., x n ) be a collection of word embeddings. we define sdm x to be the n × n matrix whose entries are given by (sdm x ) i,@xmath1, x j ), where dist can be any metric in the embedding space. the motivation for using sdms, is that in the hypothetical case where x and y represent equallength sentences 1 , with parallel words, and in languages with perfectly-isometric word embeddings, we have sdm @xmath2. realistically, while translated words are not always parallel, they are ex-(a) 'dog', 'stood', 'floor', 'cat', 'sat', 'mat' (b) 'perro', 'paró', 'suelo', 'gato' , 'sentó', 'alfombra' pected to be semantically related. in addition, while perfect-isometry does not exist, we do expect to have near-isometric embeddings. thus, we expect translated sentences to have sdm x ≈ sdm y , implying that the sdm is a good candidate to represent and compare the structure of sentences. we demonstrate the resemblance between the sdms of sentences in different languages in the following example. the spanish sentence: el perro se paró en el suelo y el gato se sentó en la alfombra, is a translation of the following english sentence: the dog stood on the floor and the cat sat on the mat. the sdms of these sentences are presented in figure 1 . the resemblance between the english sentence and its spanish translation is apparent through their sdms. defining suitable metrics to compare between sdms (see sections 5 and 6), will enable us measure similarity between sentences in different languages, without any supervised or bilingual signal. this measurement can be useful in many nlp tasks, such as machine translation quality estimation @xcite , parallel corpus filtering @xcite , parallel corpus mining @xcite and cross-lingual plagiarism detection @xcite . in addition, such multilingual representations can be useful in cross lingual transfer learning @xcite . ## toy example to demonstrate the potential of sdms, we start with a simple experiment. as we argued earlier, the sdms of sentences and their translations should be similar, especially if the translation uses parallel words (word-by-word translation). this suggests that sdms can achieve high performance in a the experiment settings are as follows. we use the english-italian dataset provided by @xcite and @xcite . the dataset contains monolingual word embeddings trained with word2vec using the cbow method with negative sampling @xcite foot_1 . we apply length normalization and mean centering to all embeddings. in addition to the embeddings, the dataset also contains a bilingual dictionary, split into a training set of 5,000 word pairs and a test set of 1,500 word pairs, both are uniformly distributed in frequency bins. we use the training bilingual dictionary, and removed repetitions foot_2 , which resulted in a dictionary of 3,281 word pairs. we generate a bilingual corpus of 1,000 artificial sentence pairs using random sampling from the bilingual dictionary, such that each sentence is made of a sequence of 20 random words, and its parallel sentence is made of the translations of these words. next, we calculate the sdm of each sentence (a 20 × 20 distance matrix), using the euclidean distance. in order to measure the distance between the english and italian sentences, we use the frobenius norm of the difference between their respective sdms. this results in a 1000 × 1000 distance matrix, the first 100 × 100 block of which is presented in figure 2 . the sentences have the same order in both languages. therefore, the distance between each sentence and its translation appears in the diagonal. one can easily notice that the diagonal tends to contain the lowest values, supporting our intuition that translations should have the closest sdm to their source sentences. to provide quantitative evaluation we used the mean accuracy measure. we count how many times the correct translation of a source sentence is retrieved, and report mean precision@k for @xmath3, 5, 10, by repeating the experiment 10 times. the results are provided in ## methods in this section, we describe how to utilize sdms for sentence representation in the realistic case, where parallel sentences in different languages may differ in length and word ordering. section 6.1 describes a direct approach -interpolating the sdms of parallel sentences to have the same size, enabling a direct matrix comparison. section 6.2 describes a vastly different approach -extracting topological structure information from the sdms using tda. ## experiments and results in this section we want to examine the preservation of our new representations across languages, by evaluating their performance in real-world tasks. in particular, we will focus on tasks that are based on similarity between parallel sentences. note that the proposed methods are fully unsupervised, in the sense that they do not use any task-specific training or cross-lingual data. we will evaluate the effectiveness of our methods as well as their combination with other unsupervised methods (i.e. without increasing the level of supervision).