The Secret is in the Spectra: Predicting Cross-lingual Task Performance with Spectral Similarity Measures (2020)

Abstract: Performance in cross-lingual NLP tasks is impacted by the (dis)similarity of languages at hand: e.g., previous work has suggested there is a connection between the expected success of bilingual lexicon induction (BLI) and the assumption of (approximate) isomorphism between monolingual embedding spaces. In this work we present a large-scale study focused on the correlations between monolingual embedding space similarity and task performance, covering thousands of language pairs and four different tasks: BLI, parsing, POS tagging and MT. We hypothesize that statistics of the spectrum of each monolingual embedding space indicate how well they can be aligned. We then introduce several isomorphism measures between two embedding spaces, based on the relevant statistics of their individual spectra. We empirically show that (1) language similarity scores derived from such spectral isomorphism measures are strongly associated with performance observed in different cross-lingual tasks, and (2) our spectral-based measures consistently outperform previous standard isomorphism measures, while being computationally more tractable and easier to interpret. Finally, our measures capture complementary information to typologically driven language distance measures, and the combination of measures from the two families yields even higher task performance correlations.

## introduction the effectiveness of joint multilingual modeling and cross-lingual transfer in cross-lingual nlp is critically impacted by the actual languages in consideration @xcite . characterizing, measuring, and understanding this cross-language variation is often the first step towards the development of more robust multilingually applicable nlp technology @xcite @xcite . for instance, selecting suitable source languages is a prerequisite for successful cross-lingual transfer of dependency parsers or pos taggers @xcite @xcite . in another example, with all other factors kept similar (e.g., training data size, domain similarity), the quality of machine translation also depends heavily on the properties and language proximity of the actual language pair @xcite . in this work, we contribute to this research endeavor by proposing a suite of spectral-based measures that capture the degree of isomorphism @xcite between the monolingual embedding spaces of two languages. our main hypothesis is that the potential to align two embedding spaces and learn transfer functions can be estimated through the differences between the monolingual embeddings' spectra. we therefore discuss representative statistics of the spectrum of an embedding space (i.e., the set of the singular values of the embedding matrix), such as its condition number or its sorted list of singular values. we then derive measures for the isomorphism between two embedding spaces based on these statistics. to validate our hypothesis, we perform an extensive empirical evaluation with a range of crosslingual nlp tasks. this analysis reveals that our proposed spectrum-based isomorphism measures better correlate and explain greater variance than previous isomorphism measures @xcite . in addition, our measures also outperform standard approaches based on linguistic information @xcite , the first part of our empirical analysis targets bilingual lexicon induction (bli), a cross-lingual task that received plenty of attention, in particular as a case study to investigate the impact of crosslanguage variation on task performance @xcite . its popularity stems from its simple task formulation and reduced resource requirements, which makes it widely applicable across a large number of language pairs @xcite . prior work has empirically verified that for some language pairs bli performs remarkably well, and for others rather poorly @xcite . it attempted to explain this variance in performance by grounding it in the differences between the monolingual embedding spaces themselves. these studies introduced the notion of approximate isomorphism, and argued that it is easier to learn a mapping function @xcite between language pairs whose embeddings are approximately isomorphic, than between languages pairs without this property @xcite . subsequently, novel methods to quantify the degree of isomorphism were proposed, and were shown to significantly correlate with bli scores @xcite @xcite . in this work, we report much higher correlations with bli scores than existing isomorphism measures, across a variety of state-of-the-art bli approaches. while previous work was limited only to coarse-grained analysis with a small number of language pairs (i.e., < 10), our study is the first large-scale analysis that is focused on the relationship between quantifiable isomorphism and bli performance. our analysis covers hundreds of diverse language pairs, focusing on typologically, geographically and phylogenetically distant pairs as well as on similar languages. we further show that our findings generalize beyond bli, to cross-lingual transfer in dependency parsing and pos tagging, and we also demonstrate strong correlations with machine translation (mt) performance. finally, our spectral-based measures can be combined with typologically driven language distance measures to achieve further correlation improvements. this indicates the complementary nature of the implicit knowledge coded in continuous semantic spaces (and captured by our spectral measures) and the discrete linguistic information from typological databases (captured by the typologically driven measures). ## quantifying isomorphism with spectral statistics following the distributional hypothesis @xcite , word embedding models learn the meaning of words according to their co-occurrence patterns. hence, the word embedding space of a language whose words are used in diverse contexts is intuitively expected to encode richer information and greater variance than the word embedding space of a language with more restricting word usage patterns. the difference between two monolingual embedding spaces may also result from other reasons, such as the difference between the training corpora on which the embedding induction algorithm is trained, and the degree to which this algorithm accounts for the linguistic properties of each of the languages. while the exact combination of factors that govern the difference between the embedding spaces of different languages is hard to figure, this difference is likely to be indicative of the quality of crosslingual transfer. this is particularly true when the embedding spaces are used by cross-lingual transfer algorithms. our core hypothesis is that the difference between two monolingual spaces can be quantified by spectral statistics of the two spaces. ## related work and baselines we now provide an overview of prior research that focused on two relevant themes: 1) measuring approximate isomorphism between two embedding spaces, and 2) more generally, quantifying the (dis)similarity between languages, going beyond isomorphism measures. the discussed approaches will also be used as the main baselines later in §5. ## experimental setup the conducted empirical analyses can be divided into two major parts. first, we run large-scale bli analyses across several hundred language pairs from dozens of languages, comparing the correlation of spectral-based isomorphism measures ( §2.2) and all baselines ( §3) with performance of a wide spectrum of state-of-the-art bli methods. second, we run further correlation analyses with performances in cross-lingual downstream tasks: dependency parsing, pos tagging, and mt. we first provide the details of the experiments that are shared between the two parts, and then provide further specifics of each experimental part. ## analyses and results the results are summarized in tables 1 and 2 . the first main finding is that our proposed spectral-based isomorphism measures are strongly correlated with performance across all tasks and settings. 12 in fact, they show the strongest individual correlations with task performance among all isomorphism measures and linguistic distances alike. the only exception is the mt task, where our measures fall short of typ (see econd-hm systematically outperforms cond-hm on 2 of 3 bli datasets and 2 of 3 downstream tasks, validating our assumption that discarding the smallest singular values reduces noise. additionally, svg shows greater stability across tasks and datasets than econd-hm. a general finding across all tasks is that our spectral measures are the most robust isomorphism measures: they substantially outperform the widely used baselines gh and is. as stepwise regression discerns between overlapping and complementing variables (see §4.3), another finding indicates that our spectral measures are complemented by linguistically driven language distances. indeed, their combination achieves very high correlation scores. the results demonstrate this across all tasks and settings (see bottom rows of the tables). for instance, when combining spectral measures with the linguistic distances, the regression model reaches outstanding correlation scores up to @xmath0.91 on panlex bli ( concerning individual tasks, we note that our spectral-based measures outperform the baselines additional results and analyses are provided in appendix b. they further demonstrate that our measures also indicate transfer quality of different target languages for a given source language, and transfer quality of source languages for a given target language, for the tasks discussed in this paper. ## further discussion and conclusion this work introduces two spectral-based measures, svg and econd-hm, that excel in predicting performance on a variety of cross-lingual tasks. both measures leverage information from singular values in different ways: econd-hm uses the ratio between two singular values, and is grounded in linear algebra and numerical analysis @xcite , while svg directly utilizes the full range of singular values. we suspect that the use of the full range of singular values is what makes svg more robust across different tasks and datasets, compared to econd-hm that shows greater variance, as observed in our results above. while the spectral methods are computed solely on word vectors from wikipedia, the results in the downstream tasks are computed with different sets of embeddings (e.g., multilingual embeddings for dependency parsing), or the embeddings are learnt during training (for pos tagging and mt). we believe that this discrepancy does not constitute a shortcoming of our analyses, but rather the opposite: spectral-based methods maintain their high correlations in the downstream tasks as well, and this supports the notion that these measures might indeed capture deeper linguistic information than mere similarities between embedding spaces. our use of effective rank in improving the condition number (via effective condition number) is also inspired by recent work that aimed to automatically detect true dimensionality of embedding spaces. however, previous work has taken an empirical approach by simply tuning embedding dimensionality to evaluation tasks at hand @xcite @xcite . our intention, on the other hand, is to extract the true embedding dimensionality directly from the embedding space. another recent study @xcite employed perturbation analysis to study the robustness of embedding spaces to noise in monolingual settings, and established that it is also related to effective dimensionality of the embedding space. all these inspired us to replace the standard matrix rank with effective rank when computing the condition number, and to introduce the statistic of effective condition number in §2.1. our study is also the first to compare language distance measures that are based on discrete linguistic information @xcite with measures of isomorphism (i.e., our spectral-based measures, is, gh), which can also be used as proxy language distance measures. our findings, suggesting that it is possible to effectively combine these two types of language distance measures, call for further research that will advance our understanding of: 1) what knowledge is captured in monolingual and cross-lingual embedding spaces @xcite @xcite ; 2) how that knowledge complements or overlaps with linguistic knowledge compiled into lexical-semantic and typological databases @xcite @xcite ; and 3) how to use the combined knowledge for more effective transfer in cross-lingual nlp applications @xcite . the differences in embedding spaces of different languages do not only depend on linguistic properties of the languages in consideration, but also on other factors such as the chosen training algorithm, underlying training domain, or training data size and quality @xcite @xcite . in future research we also plan an in-depth study of these factors and their relation to our spectral analysis. we believe that the main insights from this study will inform and guide different cross-lingual transfer learning methods and scenarios in future work. these might range from choosing source languages for transfer in low-data regimes, over monolingual word vector induction guided by the spectral statistics, even to more effective hyperparameter search. a is and gh isospectrality (is) after length-normalizing the vectors, @xcite compute the nearest neighbor graphs using a subset of the top n most frequent words in each space, and then calculate the laplacian matrices lp 1 and lp 2 of each graph. for lp 1 , the smallest k 1 is then sought such that the sum of its k 1 largest eigenvalues k 1 @xmath1. the same procedure is used to find k 2 . they then define @xmath2, k 2 ). the final is measure ∆ is then the sum of the squared differences of the k largest laplacian eigenvalues: ∆ = k @xmath3. the lower ∆, the more similar are the graphs and, consequently, the more isomorphic are the two embedding spaces. gromov-hausdorff distance (gh) it measures the worst case distance between two metric spaces x and y with a distance function m as: it computes the distance between the nearest neighbors that are farthest apart. the gromov-hausdorff distance then minimizes this distance over all isometric transforms x and y: computing gh directly is computationally intractable in practice, but it can be tractably approximated by computing the bottleneck distance between the metric spaces @xcite .