Assessing the Similarity of Cross-Lingual S eq2 S eq Sentence Embeddings Using Low-Resource Spectral Clustering (2025)

Abstract: In this work, we study the cross-lingual distance of machine translations through alignment of seq2seq representations over small corpora. First, we use the M2M100 model to collect sentence-level representations of The Book of Revelation in several languages. We then perform unsupervised manifold alignment (spectral clustering) between these collections of embeddings. As verses between translations are not necessarily aligned, our procedure falls under the challenging, but more realistic non-correspondence regime. The cost function associated with each alignment is used to rank the relative (machine) similarity of one language to another. We then perform correspondent alignment over another cluster of languages, this time using FLORES+ parallel NLLB model embeddings. Our experiments demonstrate that the representations of closely-related languages group closely, and are cheap to align (requiring < 1000 sentences) via our strategy.

## introduction assessing the similarities and differences between languages, that is, comparative linguistics, requires the consideration of historical factors, vocabulary, phonology, and written script @xcite ; @xcite ; @xcite . computational linguists adopting lexicostatistical techniques can study language distances by measuring the evolution of cognates @xcite . comparative analysis which operates purely at the word level, such as ranking levensthein distances (a string-edit metric) @xcite , has been both widely used and disputed @xcite . in parallel, the machine learning community recognized the need for sentence-level processing to produce high-quality translations. the attention mechanism, common to transformer-based language models @xcite , considers the semantic contribution of all tokens (word/sub-word units) in an input to develop an output. the quality of machine translation has drastically improved in recent years due to the advent of attention-based sequence-to-sequence (seq2seq) models which intake sentences in a source language and output a corresponding translation in a target language @xcite ; @xcite . sharp improvements in multilingual training strategies have resulted in so-called many-to-many translation models that can accept many source-target language pairs. many-to-many translation models, such as the m2m100 @xcite and nllb @xcite , can accept pairs from 100 and 200 widely-spoken languages, respectively. given a specified source language, the m2m100 and nllb models tokenize an input and pass it along several attention layers which encode the specified sentence(s) to real-valued embeddings @xcite . such representations produced by deeper encoder layers are thought to embody abstract semantic meaning critical to developing coherent, high-quality output in the decoding phase @xcite ; @xcite ; @xcite . intuitively, we would expect that closely related languages produce similar representations. if we regard sentences as concepts, language generation benefits from the alignment of closely-related concepts @xcite . we expect the syntactical and figurative structure of sentences to align more closely among related languages, thus we want to investigate whether many-to-many transformer representations are capturing this dynamic. in this work, we propose a low-resource strategy for assessing how a many-to-many machine translation model encoder groups languages. first, we collect the sentence representations over a common corpus across a cluster of slavic, indo-aryan/dravidian, romance languages, scandinavian, turkic/mongolic, and bantu languages. for this paper, we use the mean pooling of hidden states over the entire sequence to get a sentence-level representation, in line with other works @xcite ; @xcite . for one group of language families, the common corpus is the book of revelation (bor). for the other group, the common text is a collection of parallel (i.e., correspondent) sentences from the flores-200 dataset @xcite . we validate our method over this correspondent dataset to verify alignment is working as expected in a naive setting. in comparison to the work of @xcite which uses an irreproducible web crawl to generate hundreds of thousands to tens of millions of parallel sentence pairs @xcite , our resource is low-resource: we only @xmath0. we treat each language's set of embeddings as a discrete manifold. then, we perform a pairwise manifold alignment via spectral clustering @xcite and use the associated cost to produce an ordering of machine-lingual similarities. for the bor corpus, since translations are not necessarily verse-aligned, we are performing alignment without correspondence -a much more challenging regime, and realistic scenario for ultra low-resource languages. our similarity rankings over both bor and flores+ closely correspond to established analyses in comparative linguistics @xcite along with a few sharp deviations that may indicate the preference of m2m100 and nllb to occasionally place representations of less related languages close to one another. ## comparison algorithm the semantics of a language, referring to its meaning and how words and phrases convey ideas, often follow distinct patterns based on the relationships between words, contexts, and usage. these patterns can be observed in how words group together, how similar meanings emerge in different contexts, or how words with similar meanings are often used in comparable syntactic structures. spectral clustering von luxburg (2007); law et al. (2017) can be applied to identify these semantic patterns by analyzing the structure of a similarity matrix constructed from the relationships between words or phrases. we follow the method of @xcite , referred to as manifold alignment without correspondence, and describe this process explicitly in section 2.1. we must use a "without-correspondence" strategy as variations in translations (in our case, of the christian bible), can produce different verse-orderings and shuffled semantics which prevents a verse-to-verse (1-1) correspondence between two languages. in our case, we used the heat kernel similarity on a suggestion by @xcite for language comparison. by representing sentences as real-valued vectors in high-dimensional space using encoder embeddings (e.g., m2m100/nllb representations), we can calculate pairwise similarities, which are then used to create a graph where nodes represent vectors in these representations, and the edges are given quantitatively by their similarity matrix. the spectral clustering algorithm then partitions this graph into clusters by projecting these vector representations onto a set of vectors given by solutions to a generalized eigenvector solution (see section 2.2). this method hopes to potentially reveal similarities between machine representations of languages by comparing these projections, which are closely related to the clusters. in particular, we examine the square sum of the first d eigenvalues as defined by the general eigenvector equation as given in section 2.1. chowdhury et al. ( 2021 ) also used a graph-based approach to study the similarities between languages. they created graph laplacians between given languages at the word level. our method considers language at the sentence level and, instead creates a joint graph based on their combined information. motivated by @xcite , we opt for the combined graph approach due to a belief that we can measure the distance between two languages by considering spectral data associated to a submanifold derived from a combination of data from the graphs of both languages. ## experiments in this section, we produce a ranking of (machine) language distances. we review our distances against prevailing comparative linguistics theory. ## conclusion in this work, we study how seq2seq translation models group languages together. we conduct this assessment by extracting m2m100 and nllb hidden representations of sentences of various languages over small, common corpora. we observe that the embedding manifolds of closely related languages likely contain similar structures as they, on average, do not incur high spectral clustering costs. in contrast to @xcite , we @xmath1, thus framing our method as a low-resource strategy.