A W ord N et View on Crosslingual Transformers (2023)

Abstract: WordNet is a database that represents relations between words and concepts as an abstraction of the contexts in which words are used. Contextualized language models represent words in contexts but leave the underlying concepts implicit. In this paper, we investigate how different layers of a pre-trained language model shape the abstract lexical relationship toward the actual contextual concept. Can we define the amount of contextualized concept forming needed given the abstracted representation of a word? Specifically, we consider samples of words with different polysemy profiles shared across three languages, assuming that words with a different polysemy profile require a different degree of concept shaping by context. We conduct probing experiments to investigate the impact of prior polysemy profiles on the representation in different layers. We analyze how contextualized models can approximate meaning through context and examine crosslingual interference effects.

## introduction wordnet @xcite is a manually created database that relates the words of a language to concepts. concepts are represented through synsets, based on a weak synonymy relation, whereas explicit semantic relations between synsets place these concepts in a semantic space. words of a language can be positioned in that same space but this can become complex when they are ambiguous. a polysemous word such as "star" can be represented in several positions of this space depending on its meaning. word embeddings @xcite place words in a semantic space as well based on the dimensions of the vector that was derived when learning to predict their context words. static word embeddings can be interpreted as an average across contexts, even when words occur with different meanings. for our example, this means that "star" would be positioned somewhere in between celebrity and synonyms for the concept celestial body as a compromise across contexts. more recent pre-trained transformer-based language models (ptlm) such as bert @xcite and roberta @xcite ) capture a more nuanced relationship between words and concepts by not only representing the vocabulary through embeddings but also distinguishing contexts: the word "star" will be represented differently depending on the context in which it occurs. from an abstract point of view, these contextsensitive representations approximate a relation between words and concepts. @xcite investigates this relationship by measuring the impact of contextualization on the representation of meaning through the layers of ptlms, showing that representations of tokens in contextualized models deviate from their static initialization. the research by @xcite is limited to monolingual models, which leaves open what relationship between tokens or words and concepts is captured in cross-lingual models where words and concepts are shared across languages. in cross-lingual language models (xptlms) such as xlm-roberta @xcite , the challenge of contextualizing concepts is even more complex because of the additional cross-lingual ambiguity. the same word can be mapped to the same or to different concepts across languages. for example, the dutch word "star" is an adjective meaning inflexible whereas the translation for the english "star" corresponds to "ster" in both meanings. the dutch language, therefore, adds ambiguity to the word-concept relationship of "star". as most xptlms use a shared vocabulary for all languages, the variation in meaning across languages can simply be interpreted as different contexts for a word that needs to be encoded in the representations of the model. in most multi-lingual wordnet databases, crosslingual ambiguity is underrepresented because they are commonly build using the expand-method @xcite . this means that the english representation of concepts is maintained and crosslingual links are established by mapping the vocabulary of the new language onto the existing concept taxonomy. this approach hampers research to the universality of concepts in wordnet models @xcite but it has been applied widely because of its clear practical advantages over the alternative merge-method that requires intense manual labor. xptlms can be constructed in different ways as well, which partially mimics the difference between the expand and the merge approach: 1) expanding a monolingual ptml with static lexical embeddings for target languages while freezing the other layers @xcite or 2) training a model from texts from all languages @xcite so that all languages contribute conceptual representations as contexts (a merged approach). in this paper, we argue that xptlms provide new opportunities to move beyond the conceptual limitations of multilingual wordnet databases built through the expand method. we provide empirical evidence for the impact of languages on a shared conceptual xptlm for both the lexical and conceptual levels by measuring to what extent sharing tokens in xptlms has a positive or negative impact on representing concepts and to what extent the contexts in which these words occur compensate for any disturbances in the token representation. in other words: to what extent is the representation of "star" a compromise across all language meanings and to what extent is it defined by the cross-lingual contexts in which it occurs? xptlms use a shared vocabulary for all languages to exploit semantic commonalities across languages (cognate effects). however, cross-lingual differences caused by semantic drift @xcite can contribute to semantic interference @xcite . more specifically, we will address the following questions in our experiments: in order to investigate the above questions, ideally a large sense-tagged parallel corpus would be required to identify a representative set of concepts shared across languages. existing corpora @xcite are however small and have skewed sense distributions. another problem is that it is hard to determine the best level of granularity for identifying concepts associated with a word in contexts and they may not be distinguishable empirically through existing models @xcite . instead of multilingual corpora with word-net senses or all contextualized contexts, we, therefore use a controlled set of semantic classes as the representation of concepts following the work of @xcite . entity types such as person, organization, and location can be seen as coarse-grained concepts for which large datasets exist. we use the xlent dataset @xcite which contains 160 million aligned entity pairs in 120 languages paired with english. we investigate how well the entity in this data are distinguished by contextualized models in contextualized layers. our contributions are: the paper is further structured as follows. in the next section 2, we describe related work, especially on semantic probing of distributional models. after that, we describe in section 3 our methodology, which is based on @xcite but applied to multilingual models. the dataset that we use is described in section 4 and our experimental results are described in section 5. we discuss the results and conclude in section 6. ## related work analyzing the representational structure of contextualized models has become an essential means towards developing more transparent and interpretable ai models, for example in the black-boxnlp workshop which is reaching its 5th edition this year @xcite . however, only a limited amount of research has been done to investigate the relationship between the vocabulary of such models and the degree of context dependency of the concepts that are associated with the words in the vocabulary. in @xcite , this relationship is investigated by measuring the impact of contextualization on the representation of meaning through the layers of language models. this study indicates that 1) contextualized models do enhance the meaning compared to the static initialization of the token and 2) there is no finite and discrete set of representations (thus concepts) for single tokens across concepts. @xcite show that it is possible to transfer an english transformer to a new language by freezing all the inner parameters of the network and learning a new set of embeddings for the new language through masked language modeling. this works because the frozen transformer parameters constrain the resulting representations to become aligned with english. this approach does not adapt the concept representation established for the original language english. it only learns the token embedding using the english concept model and is thus comparable to the multilingual wordnet expand model (which uses a single english concept space and learns token mappings to another language). it is not possible to learn new concepts from another language nor adapt biases learned from the english data. phenomena of semantic drift across languages @xcite can therefore not be captured and it remains unclear how the addition of languages affects the conceptual distribution beyond the performance on the downstream tasks. for analyzing how a contextual language model captures the relationship between a word and a concept, we can use word sense disambiguation as a proxy task. the task evaluates model performance in associating an ambiguous word with the correct concept from the possible concept inventory. for example, the word "state" could represent a 'government' or the concept corresponding to the wordnet synset called "a way something is". one limitation of using such an approach is the granularity of the sense category. wsd categories are often too fine-grained and allow only limited abstraction @xcite . we opt for a task on a higher abstraction level and apply semantic class-based probing to quantify the contextualization capability of a language model using wiki-pse in line with @xcite . wiki-pse contains tokens used in contexts corresponding to different semantic classes. for example, the word 'apple' can refer to a technology company corresponding to the 'organization' class or it can refer to a fruit belonging to the 'food' class @xcite . a concept-tagged dataset can be used to investigate relationships between a word form and a concept in a language model in a simplified setup: word forms are limited to entity names and their semantic classes define the concept inventory. probing has been established as a tool to test whether linguistics information is encoded in language model representations @xcite @xcite . @xcite train a classifier to predict sentence characteristics such as length, semantic information, and word order from sentence representation. higher performance in the classification task indicates that information about the measured property is encoded in the embedding. @xcite extend the probing tasks to a wider range of linguistic phenomena such as coreference, semantic relations, and entity information. @xcite introduced edge probing and establish a standard format to quantify the availability of linguistic structure in pre-trained language models using various nlp benchmark tasks. our work follows @xcite in that we use sentence probing to measure the relationship between a word, its context, and the corresponding concept. we extend this approach to various multilingual models instead of english bert. we present pilot experiments to explore the utility of using semantic class probing with these multilingual models. ## methodology to analyze how language models capture the relationship between words and concepts, we identify words that illustrate edge cases for the relation between concepts and contexts: 1) a monosemous (mono) relation between a word and a single concept, 2) balanced polysemous relations between a word and multiple concepts, and 3) skewed polysemous relations where one concept is dominant in language use. we expect that the patterns in concept distribution are reflected in the probing performance of the cross-lingual models. our approach represents only a rough approximation of the set of concepts related to a word as well as the distribution of concepts in language use. the actual range of concepts is unknown and is the result of the pretraining of the model. our pilot experiments, therefore, explore whether the large-scale annotations of xlent can serve as a proxy for probing word-concept relationships in multilingual models. we assume that such data provide sufficient information on the relation between words and concepts to measure the degree of ambiguity and the capability of models to identify concept relations from contexts. we hypothesize that our observations for a selected set of words can be generalized to a larger sample, which should be tested in future research. in the probing setup, the model representation built during pretraining is not changed and can be tested for its capacity to represent a concept in target contexts at different layers. we assume that the lexical initialization in the first layer will reflect the prior ambiguity of the word in the pretraining data and that the integration of context will adjust the representation toward the target concept in higher layers. we expect the following observations for the respective profiles: in our experiments below, we report on the results for skewed and balanced ambiguous words in english and across the language english, dutch, and german. our code is publicly available at https://github.com/cltl/ probing-cross-linqual-model . ## data set and experiment for our experiments, we use entities and their respective semantic class as a proxy for a more general notion of words and concepts due to data available for many languages with a controlled number of concepts in the form of entity types as semantic classes. specifically, we select a sample from xlent which contains 160 million entity mentions annotated with 10 classes in 120 languages @xcite . we describe the selection procedure in the following subsections. ## results we first examine our probing setup for resolving conceptual ambiguity in english entities and next conduct additional analyses to examine the effect of shared tokens across multiple languages on conceptual ambiguity. lastly, we conduct tests across typologically related and distant languages to check if a relationship learned between context and concept in one language is relevant for another language. ## conclusions in this paper, we investigated to what extent polysemous profiles play a role in establishing a relation between words and concepts. we focused on english but we also investigated words shared across languages in cross-lingual pre-trained language models. we selected representative cases for concept distributions from a large dataset of entity mentions as ambiguity profiles. our probing experiments indicate that prior probabilities of polysemy profiles are reflected in the lexical initialization and that context is integrated for disambiguation in higher layers. our cross-lingual results indicate that sharing of tokens and contexts across languages has an influence on probing accuracy. our experiments are restricted to five languages: english, dutch, german, arabic, and amharic. in future work, we will extend our experiments to more languages. we plan to investigate the impact of optimizing the probing classifier with crosslingual training data. training on the data of other languages extends the fund of concepts in the classifier, which is comparable to an expand model for multilingual wordnets. our method is limited by the annotations in contexts. it is therefore difficult to extend it to other words and concepts than entity names. nevertheless, the entity results can be seen as a proof of concept to develop more sophisticated methods for analyzing concept relations in multilingual models. when more sense-tagged data becomes available, this method can also be applied to other words and concepts.