The Hidden Folk: Linguistic Properties Encoded in Multilingual Contextual Character Representations (2023)

Abstract: To gain a better understanding of the linguistic information encoded in character-based language models, we probe the multilingual contextual CANINE model. We design a range of phonetic probing tasks in six Nordic languages, including Faroese as an additional zero-shot instance. We observe that some phonetic information is indeed encoded in the character representations, as consonants and vowels can be well distinguished using a linear classifier. Furthermore, results for the Danish and Norwegian language seem to be worse for the consonant/vowel distinction in comparison to other languages. The information encoded in these representations can also be learned in a zero-shot scenario, as Faroese shows a reasonably good performance in the same vowel/consonant distinction task.

## introduction subword and character sequence information is crucial in state-of-the-art neural language models @xcite because it improves their generalization and robustness capabilities @xcite . additionally, character-level features are beneficial for morphologically rich and low-resource languages @xcite . however, there is a lack of interpretability methods for character-based models. only a few approaches have tried to understand the linguistic information encoded in character embeddings and cross-lingual approaches must be evaluated more rigorously by considering typology and linguistic distance @xcite . therefore, in this work, we analyze how much phonetic information is encoded in contextualized multilingual character embeddings from the canine language model @xcite . based on six nordic languages, we extract phonetic features for characters in context through unsupervised grapheme-to-phoneme alignment and design a set of probing tasks. we explore the character representations in two different evaluation scenarios, a traditional train/test split scenario and a leave-one-letter-out scenario. we find that phonetic information on a global level (e.g., vowel and consonant detection) is encoded accurately in the character representations and more mixed results are achieved on lower-level probing tasks such as consonant voicing and manner, or vowel height and roundness. we also see that this information is transferred to the related zero-shot language faroese. our code is available online foot_0 . ## related work we discuss previous work in this area by focusing on existing multilingual character language models and the interpretability of these models. ## contextualized character embeddings we extract character embeddings (in the context of full words) from the canine model. in this section, we present the multilingual data and the embedding extraction. data since character-level features are important for morphologically rich and low-resource languages @xcite , we choose a set of six nordic languages for our experiments: danish (da), swedish (sv), norwegian (nb), finnish (fi), icelandic (is) and faroese (fo). five of the languages are included in the training data of the character language model (da sv, nb, fi and is). additionally, we use faroese to test performance of multilingual zero-shot embeddings. the starting point for extracting character embeddings is a frequency list for each language (see ## phonetic feature extraction in order to extract phonetic features from characters, we need to know how a specific letter should be pronounced. we do that by aligning the characters with the string of phones as given by wikipron. as the number of letters and phones may not match, we align them using m2m-aligner @xcite , an unsupervised model that is based on expectation-maximization. then, we use the ipapy toolkit to obtain phonetic features for each phone. we describe this process below. pipeline we use the aligner to obtain phonetic features for characters in all six languages. first, we obtain a dictionary for each language from wikipron (lee et al., 2020), foot_3 and then, we align graphemes to phones using m2m-aligner. 5 wikipron includes both phonemic and phonetic representations of words, which they refer to broad and narrow, respectively. as the model for extracting features works at a phonetic level, we use the phonetic representations (narrow). in the next step, we use the ipapy foot_5 toolkit to extract phonetic features for each phone. finally, the ipa features are merged with the canine character representations. this process results in one dataset per language, consisting of 6067 characters in 899 words for danish, 700 characters in 135 words for faroese, 4698 characterss in 745 words for finnish, 268 characters in 43 words for icelandic, 302 characters in 57 words for norwegian, and 312 character in 58 words for swedish. the reason for the drastic decrease in samples for some of the language is the small size of the pronunciation dictionaries. ## probing in this section, we describe the probing tasks and the evaluation scenarios that we devise to test the extracted character embeddings. we design 23 tasks to investigate the phonological knowledge encoded in the multilingual character representations. the tasks are split into three categories: global features (e.g., is this character a vowel or not?), consonant features (e.g., is the manner of articulation of this consonant plosive or not? foot_6 , and vowel features (e.g., is this vowel pronounced as a rounded vowel? 8 the probing tasks have the structure f : @xmath0, where given a set of character representations x, we want to find the best mapping f that relates x to a set of target features y using a supervised logistic regression classification model. as discussed by @xcite , it is important to take into account the expressivity of a probing task, since overly expressive probes, i.e., too many possible mappings for f : @xmath1, does not reveal much about the internal feature representations. therefore, we test the all probing classifiers in two evaluation scenarios: (i) 80/20, a random 80% training and 20% test split of the data, and (ii) lolo, a leave-one-letter-out training and test split. the 80/20 setup implies that the same characters (in different contexts) can appear in the train and test split, resulting in a simpler probing task, whereas the lolo setup ensures zero-shot learning for the specific character of which all representations in all contexts are held out during training. ## results to understand the implicit phonetic information encoded in contextual representations, we present weighted f1-scores for a selection of features. 9 most of the individual probing tasks show f1-scores above 0.5, which means that the models generally perform better than a random baseline. in figure 2 , we observe the results for two global features, where we predict whether a character is a vowel (top) and whether a letter is a consonant (bottom). in each plot, we report the results of both evaluation scenarios. faroese performs very similar @xcite ˚lod p ])(rounded) 9 see appendix a.4 for the full results. figure 3 shows the lolo performance of each model for all characters and languages. the models were trained to predict whether the contextual character embedding had the label global_type_vowel, meaning that the character in this specific context is pronounced as a vowel. taking a closer look at danish and norwegian, for instance, the f1-scores are relatively similar. this was expected given the similarities between the two languages in the written form. although the results are similar, if we zoom in to specific character, we observe that the performance for vowels is systematically worse for danish, which is reasonable given the complex nature of the danish vowel system @xcite . ## conclusions & future work in this work, we design a probing mechanism to better understand information encoded in contextual character representations, for which we use two validation mechanisms. the first one, where the training data is divided into training and testing set, and the second one where we train one model for each character. the reasoning behind this is that a letter representation, even though it occurs in a different context, would have a similar representation and therefore, it would involve a kind of data leakage to the testset. we can imagine the example of the letter "a" in the words "tram" and "gas" in english. the contexts are rather different, but we would expect the representation of the letter "a" to be relatively similar, as it is pronounced similarly. therefore, the lolo evaluation allows us to test the representations of a character in any context in a zero-shot scenario. following this methodology, we observe that contextual character representations encode some phonetic information, supported by the probing classifier performances. this result is in line with similar works performed on less complex representations, such as word2vec, ppmi+svd or an rnn encoder-decoder @xcite , where they found that in some languages these representations showed distributional properties. we use wikipron data as linguistic knowledge and we align graphs and phones using an automatic aligning mechanism. we validate the aligner for danish on manually aligned data, but other languages may have their own specific challenges. we use a wide range of phonetic probing tasks to accommodate language-specific particularities. it is crucial that the number of positive and negative instances is checked when probing each phonetic feature. as expected, many pronunciationrelated features do not occur in specific languages, and thus, this results in phonetic features with only one class, which will not shed any light on such phonetic properties. hence, future research should address the design of these probing tasks as much as the results. finally, since some of the datasets were relatively small, using a grapheme-to-phoneme model could increase the number of instances. besides, these analyses could be extended to more languages to cover the full spectrum of orthographic depth.