What do BERT Word Embeddings Learn about the F rench Language? (2024)

Abstract: Pre-trained word embeddings (for example, BERT-like) have been successfully used in a variety of downstream tasks. However, do all embeddings, obtained from the models of the same architecture, encode information in the same way? Does the size of the model correlate to the quality of the information encoding? In this paper, we will attempt to dissect the dimensions of several BERT-like models that were trained on the French language to find where grammatical information (gender, plurality, part of speech) and semantic features might be encoded. In addition to this, we propose a framework for comparing the quality of encoding in different models.

## introduction with over 95,000 citations (and counting) since its publication in 2019, bert @xcite ) can be considered one of the most prominent large language models (llms) architectures in the current state of the art, finding applications in fields ranging from text to image generation @xcite to protein structure prediction @xcite , still showing competitive results @xcite . we can attribute one of the reasons for such a successful and wide-range usage of bertlike models to the word embeddings (multidimensional word representations) they produce. however, with ever-growing model sizes, the interpretability of dimensions of the learned representations is still a complex task. while the number of parameters is constantly growing, the performance of the models is not improving as rapidly, and we are facing diminishing returns with the increased model and training data scale @xcite . additionally, the size of available data does not grow at the same rate as the model hyperparameters and all possible training data for language models might be exhausted between 2030 and 2040 @xcite . therefore, shifting the research focus from constantly increasing the training datasets towards understanding more about existing models, their parameters (and how they can be improved) might become necessary in the nearest feature. in this paper, we propose an intrinsic metric evaluating the quality of information encoding (infenc) and suggest a framework that allows the identification of the best dimension candidates of word embeddings that potentially encode target information (in our experiments: grammatical number and gender for french nouns and adjectives, part of speech (pos) for french nouns, adjectives, and verbs, and semantic information for french nouns). ## related works the explainability and interpretability of llms have become a growing interest for researchers. one of the approaches to the problem is to learn an explainable distributional word embedding model linking each feature to a word (for example, @xcite . in this work, the representations were learned from a co-occurrence matrix, which allowed for high interpretability of the embeddings. however, the representations were underperforming on similarity tasks such as ws-353 in comparison to other distributional representation models such as glove @xcite and bert-like embeddings @xcite . another approach to interpretability is to probe pre-trained models @xcite @xcite . torroba hennigen et al., 2020 focus on intrinsic probing that aims not only to identify if a linguistic feature is encoded in representations but additionally how the information is encoded. according to this defi-nition, our work remains in the scope of intrinsic evaluation. torroba @xcite propose a method to efficiently detect the most relevant subset overall features whereas we pre-select an initial subset and compute a score for each pre-selected feature. @xcite explore the usefulness of word embeddings to predict features at the sentence level (length, depth of the syntactic parsing tree, etc.). the sentence embeddings are obtained by merging word embeddings (by sum, maximum, minimum, and average operators). the authors do not explore which features encode linguistic information. for ravichander et al., 2021, the objective is also not to dissect word embeddings in order to find features encoding linguistic information, but to study the usefulness of linguistic information for a classification task at sentence level. contrary to these works, our objective is not to explore linguistic features at the sentence level, but at the word level. ## models in our work, we compared word embeddings of 10 different bert-like models working with the french language, details about which can be found in ## experimental protocol as mentioned above, our approach consisted of combining word embeddings with grammatical/semantic information, extracted from humanannotated sources where the final list of tested features consisted of grammatical gender for nouns and adjectives, grammatical number for nouns and adjectives, pos for nouns, adjectives and verbs, and semantic categories for nouns. in order to use the annotations, a decision was made to create a binary vector for each feature: 1 was assigned to if a word possessed a certain feature and 0 -otherwise foot_5 . due to the relatively small sizes of the datasets of annotated word embeddings, we opted to incorporate 5-fold cross-validation @xcite instead of the traditional train/test split. ## results following the protocol above, we calculated infenc for all listed models for encoding quality of grammatical gender (for adjectives and nouns), grammatical number (for adjectives and nouns), pos, and semantic supersenses. it is worth noting that for observed experiments, subsets of dimensions appear to achieve higher accuracy in the vast majority of cases. ## discussion we believe that identifying dimensions with grammatical or semantic features can be used both during training models, combined with reinforcement training, ensuring a better quality of encoding of information, and as a way to find the best-suited model for the downstream task, as well as evaluating the effect of fine-tuning on the encoding of the target features. due to time and resource limitations, our experiments were limited in terms of studied features and language which we hope to address in further work. we believe that the research could benefit from incorporating other semantic and grammatical features. however, additionally, we propose the following topics for discussion. ## conclusion in this paper, we proposed an intrinsic metric in-fenc and a framework allowing the extraction of stable dimensions that potentially encode grammatical or semantic information from word embeddings of bert-like models trained on the french language. our findings include: