Measure and Evaluation of Semantic Divergence across Two Languages (2021)

Abstract: Languages are dynamic systems: word usage may change over time, reflecting various societal factors. However, all languages do not evolve identically: the impact of an event, the influence of a trend or thinking, can differ between communities. In this paper, we propose to track these divergences by comparing the evolution of a word and its translation across two languages. We investigate several methods of building time-varying and bilingual word embeddings, using contextualised and non-contextualised embeddings. We propose a set of scenarios to characterize semantic divergence across two languages, along with a setup to differentiate them in a bilingual corpus. We evaluate the different methods by generating a corpus of synthetic semantic change across two languages, English and French, before applying them to newspaper corpora to detect bilingual semantic divergence and provide qualitative insight for the task. We conclude that BERT embeddings coupled with a clustering step lead to the best performance on synthetic corpora; however, the performance of CBOW embeddings is very competitive and more adapted to an exploratory analysis on a large corpus.

## introduction languages evolve throughout time: for many words, their usages along with their frequent collocations and associations can change, revealing the evolution of the society @xcite . however, all languages do not evolve identically: the impact of an event, the influence of a trend or thinking, can differ between communities. moreover, languages do not evolve independently; some words can be inherited and borrowed between languages. for example, cognates -words that have the same etymological origin and similar meaning in two languages -can sometimes diverge into false friends, due to particular features of one language and its associated culture and history. a more specific example is the russian word "ukrop", meaning "dill". it started to be used by russian people as an ethnic slur-a pejorative term-to talk about ukrainian soldiers at the beginning of the russian-ukrainian conflict @xcite . then, ukrainian people started to use it to designate their own patriots, in a positive way. analysing the evolution of this word can lead to a better understanding of the evolution of the conflict; on the contrary, without suitable tools and methods to detect the divergence in its usage and connotation between communities, one might draw spurious results when analysing texts of this period. diachronic semantic change detection is an emerging field in natural language processing, building upon the growing number of digitised texts with temporal metadata publicly available in various languages. it opens new perspectives of improvement for downstream tasks (using timeaware word representation for tasks ranging from text classification to information retrieval in temporal corpora) or for socio-linguistic and historical linguistics analysis @xcite . the goal of this paper is to extend the analysis of lexical semantic change across two languages, aiming at estimating the degree of diachronic semantic divergence between a word and its translation across time in a bilingual corpus. we propose an experimental framework to learn word representations that are comparable across both time and languages, and to detect and classify semantic divergence in a bilingual setting. we compare: (i) diachronic word embeddings, which allow static embeddings such as cbow @xcite to drift through time, and (ii) contextualised embeddings, relying on a pre-trained multilingual lan-guage model (m-bert, @xcite . 1 we also propose an anchored-alignment strategy to tackle the bilingual setting for non-contextual embeddings. then, we suggest a metric to measure the divergence of word usage between two languages, the bilingual divergence. given the lack of a bilingual dataset annotated with semantic divergence, we generate a corpus of synthetic semantic drift across two languages using eurosense @xcite , a sense-disambiguated and aligned bilingual corpus. to do so, we define a set of monolingual and bilingual semantic change scenarios and evaluate our different approaches on them. finally, we apply our systems to newspaper corpora in two languages, english and french, covering the same time period, from 1987 to 2006. we classify all words of a bilingual lexicon into the scenarios defined for the synthetic drift generation. to sum up, we extend the most appropriate methods from the literature of diachronic semantic change to build a framework for the measure of semantic divergence across languages (sections 3 and 4), for which we propose a definition of the task, a measure of semantic divergence (section 5), and a process to evaluate the presented methods (section 6). these methods average all the senses of a word into a unique vector at each time step. pre-trained language models such as bert @xcite allow each occurrence of a word to have a contextualised vector representation. these models, pre-trained on large datasets, improved the state of the art on numerous nlp tasks. similarly, contextualised embeddings can be applied to semantic change detection @xcite using several aggregation techniques to measure the degree of semantic change of a word from all its contextualised representations over time. however, these methods are still outperformed by non-contextualised embeddings for this task @xcite . ## diachronic words embeddings before presenting systems based on contextualised embeddings, we introduce two methods using noncontextualised ones, as they are known to perform best for the task of semantic change detection @xcite . we use the continuous bag of words (cbow) architecture of word2vec @xcite ; we apply two different training methods to train it in a diachronic way. then, we describe an anchored-alignment method to obtain bilingual diachronic word embeddings. ## contextualised embeddings to challenge the systems based on aligned cbow embeddings, we use m-bert, the multilingual version of bert @xcite . it is trained on wikipedia content on 104 languages, without any additional multilingual mechanism nor language identifier. applying a pre-trained multilingual model on a bilingual temporal corpus enables immediate comparison without requiring any alignment. each sequence is labelled with the time it was written and its language. we extract contextualised representations for each token of a sequence by summing the top four hidden layers of the pre-trained model. bert representation relies on a system of wordpieces; if a word is divided into several wordpieces, we take the average of all the wordpiece embeddings as representation for the word. to sum up all the information about a word from the set of contextual embeddings of all its occurrences in a time slice, we experiment with two aggregation techniques: averaging and clustering. averaging : proposed by @xcite , this method averages all the token embeddings of a word for each time period and each language. we end up with a set of time-specific and languagespecific vector representations of a word. they can be compared using the cosine distance (shoemark et al., 2019). 3 clustering: this method, first used by @xcite , groups the set of token embeddings of a word into types of usages. we apply a clustering algorithm, k-means, to all the embeddings of a word and its translation, on all the time periods jointly. then, we compute the normalised distributions of clusters, for each language and period. more precisely, for a given word, we extract the number of tokens in each cluster and for each pair (period, language); we normalise it by the total number of occurrences of the word in the corpus. we obtain the probability distributions of the usages of this word at each time slice and in both languages. these distributions can be compared between two periods or two languages using the jensen-shannon divergence (jsd, @xcite . ## drift measures after applying the described systems to a bilingual corpus divided into t time slices, for a given target word in a given language l, we obtain either a sequence of t embeddings u (t) l in each language (for cbow and m-bert with averaging), or a vector of t cluster distributions c (t) l (for m-bert with clustering). we compute the distance between representations: the cosine distance between noncontextual embeddings and the jsd between clus3 we define the cosine distance as (1 -cosine similarity). ters distributions. in a bilingual setting, drift measures can be computed for each word pair (one word and its translation). first, we compute the distance inside each word pair at each time step. we call it the bilingual distance: s (t) @xmath0, t, l 1 , l 2 ) for @xmath1, 2, . . . , t . second, the temporal drift of this distance is measured similarly to the monolingual drift, either incrementally or inceptively. the distance is the norm between the bilingual distance s (t) b at two time steps, measuring the divergence of the usage of a word and its translation. we call it bilingual divergence. for example, the incremental bilingual divergence is computed as follows: various information can be extracted from the vector of bilingual divergence of a word d b : the trend (no trend i.e. stable distance between a word and its translation, decreasing i.e. convergence, or increasing i.e. divergence), the degree of divergence (e.g. by summing all its elements), and the speed of divergence (by estimating the slope). ## synthetic drift generation the study of semantic change faces the issue of evaluation, as few labeled corpora exist for this task. recent initiatives from the nlp community start to produce more annotated data @xcite ; however, no corpus is available for bilingual analysis. consequently, we generate a corpus of bilingual synthetic semantic change, following common practice in the literature of monolingual semantic change detection @xcite . it allows us to control exactly the shape and degree of semantic change in the corpus and thus gain a deeper understanding of the impact of each modeling choice. to create synthetic semantic change, common practice involve to merge two words that do not share a common sense, creating a pseudo-word; then, generate synthetic change by controlling the proportion of sentences using each of the two original words in the successive time slices of a temporal corpus @xcite . however, as advised by schlechtweg and schulte im walde (2020), it is preferable to use the natural polysemy of words for the synthetic drift to be as close as possible to reality: instead of controlling the proportion of sentences containing two unrelated words merged as a pseudo-word, we use sentences containing several senses of a unique word. to this end, we need a bilingual sense-annotated corpus with consistent annotations between languages @xcite . the eurosense corpus 4 @xcite is derived from the europarl corpus, a large public corpus of proceedings of the european parliament. it has a full and a refined version. we use the latter to build our synthetic corpus; it is half the size of the first one but more reliable. the framework babelnet @xcite is used for annotation. eurosense contains parallel text in 21 european languages. we focus on the two languages with the highest amount of annotations in the refined corpus: english and french. an example of aligned sentences in these languages can be found in ## experiments on synthetic data we compare the accuracy of our systems on the synthetic corpora generated in the previous section. ## experiment on newspaper corpora we analyse the semantic divergence of wordtranslation pairs in a bilingual corpus of news articles. our goal is to classify all words of a bilingual lexicon into the semantic divergence scenarios defined in section 6.1. ## discussion in this paper, we define an experimental framework to measure and classify the semantic divergence of a word and its translation in a bilingual corpus. we compare different kinds of word embeddings on various bilingual divergence scenarios generated in a synthetic corpus. we apply our conclusions to a bilingual newspaper corpus to identify words undergoing different types of semantic divergence. bert embeddings coupled with a clustering step lead to the best performance on synthetic corpora. the performance of cbow embeddings is nevertheless very competitive, and more adapted to an exploratory analysis on a large corpus. there is a large margin for future work; be it in terms of quality of diachronic bilingual representation, metric to measure semantic divergence, and evaluation method. our evaluation focuses on the trend of the drift, but its degree and its speed can also be quantified and analysed. in addition, the underlying bilingual representation learning approach is key for the detection of drifts. the transformations applied to create a cross-lingual word embedding space might result in information loss or generation of spurious drifts in the embeddings. to compare word embeddings with the purpose of detecting semantic divergence, the anchored alignment method presented here is not the only option; promising candidates are temporal referencing @xcite and the global anchor method @xcite . a limitation of our work is the use of an injection to define word pairs. in his general linguistics course, de @xcite states that there is no bijective relationship between words in different languages. the different meanings and uses of a word in a language cannot have a perfectly identical equivalent in another language. moreover, as noted by @xcite , a word can have synonyms in one language while the word bearing the same meaning in another language has none; in that case, the usage of the word in the first language is divided into all its synonyms. another limitation is evaluation with synthetic data. this method is common in monolingual semantic change analysis, but there is no guarantee that the generated phenomenon is similar to realworld data. for example, a degree of freedom is the shape of the synthetic drifts generated. in this paper, we used logarithmic and linear shapes; but some literature hint that a logistic shape is also a good match for semantic drift @xcite . furthermore, in real data the granularity (the size of the periods used to divide the corpus) might have an important impact on the shape of the semantic evolution. finally, as we build all bilingual scenarios from combinations of two monolingual scenarios, the flaws of the monolingual scenarios are inherited by the bilingual scenarios. it can potentially multiply the noise by propagation of uncertainty. we wished to overcome the limitations of synthetic evaluation with the application on real corpora, but more thorough interpretation would be necessary for a solid qualitative evaluation. to perform quantitative evaluation on real data, an annotated dataset similar to the ones for monolingual semantic change (e.g. @xcite ) would be necessary. however, the annotation task would be even more complex than for monolingual data. an easier entrance point towards annotating data for this task could be loanwords and cognate words. overall, this is a challenging task and we hope to attract more people to work on it in the future.