Interpreting Character Embeddings With Perceptual Representations: The Case of Shape, Sound, and Color (2022)

Abstract: Character-level information is included in many NLP models, but evaluating the information encoded in character representations is an open issue. We leverage perceptual representations in the form of shape, sound, and color embeddings and perform a representational similarity analysis to evaluate their correlation with textual representations in five languages. This cross-lingual analysis shows that textual character representations correlate strongly with sound representations for languages using an alphabetic script, while shape correlates with featural scripts. We further develop a set of probing classifiers to intrinsically evaluate what phonological information is encoded in character embeddings. Our results suggest that information on features such as voicing are embedded in both LSTM and transformer-based representations.

## introduction on the one hand, writing is an essential form of human communication. writing systems and orthographies differ across languages and impact our reading behavior. psycholinguists have extensively studied the effect of orthographic depth, i.e., the transparency of grapheme-to-phoneme mappings, on reading acquisition as well as skilled reading @xcite . on the other hand, the wide range of crosslinguistic diversity is still a major challenge for natural language processing (nlp) and for the study of language more generally @xcite , especially on sub-word levels @xcite . this increases the importance of cross-lingual analyses of character-level language models (lms), because anglocentrism in linguistic research is not only prevalent in nlp, but also in (reading and) orthography research @xcite . character-based language models have gained significant attention in recent years in languages with latin scripts, since they contain meaningful information on various linguistic levels and enhance the robustness of models. @xcite suggest that character lms provide a more human-like account of sentence processing, which assumes a larger role of morphology, phonotactics, and orthographic complexity than was previously thought. moreover, including character and sub-character information in lms for asian scripts is a standard practice. despite of this recent attention, work focusing on getting a deeper understanding of character representation is scarce @xcite , in particular regarding the comparison between languages and different types of scripts. the goal of this work is to improve our understanding of learned character representations, for better interpretability of the models. like other neural network based models, character-level lms can be seen as black-box methods and reveal limited insights about the causes for their predictions @xcite . we investigate the information encoded in character embeddings by comparing them to perceptual representations. such representations we design by mimicking features of human language processing, from reading, writing and speaking, by the creation of embeddings based on the shape of characters, the sound (phonological features derived from grapheme-to-phoneme mappings) and color (elicited in the form of graphemecolor mappings from synesthetes). ## related work character-level information in lms. including character-level information in lms of languages with latin scripts has become a common practice in nlp in recent years. this has been the case for different tasks, such as language modeling @xcite , part-ofspeech tagging @xcite , morphological inflection @xcite @xcite , named entity recognition @xcite , machine translation @xcite . character-level information can enhance the models by providing background knowledge in the form of the underlying structures of words in a language @xcite . @xcite showed how combining character-and word-level information in pretrained lms improves not only the performance but also the robustness of the model. for certain languages, it is standard practice to include sub-token information in lms, which happens naturally due to the compositional structure of their orthographies. this is the case for east asian languages such as korean and japanese (e.g., @xcite . korean lms are often trained on jamos (i.e., letters, as opposed to syllables), the smallest unit of the korean script @xcite . this reduces the vocabulary size and injects syntactic and semantic information to the model that is difficult to access with conventional characteror token-level units @xcite . recently, @xcite showed that a korean bert model using sub-character information requires less training data than previous models. similarly, japanese lms also benefit from sub-character information @xcite . ## character representations we train three types of character embeddings based on textual input: count-based ppmi embeddings, and embeddings learned by lstm and transformer language model. ## representational similarity analysis in order to analyze the relation between the learned character representations and the three perceptual representations -sound, shape, and color -we first compute the pairwise distances between characters of a single model/representation type to analyze how similar the model's representations for each character are to each other foot_2 . for each pair of experimental conditions, the spatial correlation is calculated between the distances of all characters of a language. figure 3 shows the pearson correlations between the character distances of all embedding types. the figure also includes a baseline, where the correlation between random distances and the distances of the respective character representations is computed. we correct the significance results by applying the bonferroni correction for multiple comparisons. as expected, the textual character representations show high correlation amongst each other for all five languages. the correlations between the textual embeddings and the perceptual representations show that even though the first are purely trained on written language, they still learn to encode certain inherent characteristics of human language processing and production. as a general pattern, the textual character representations correlate strongly with sound representations, moderately with color representations, and not at all with the shape representations (with the exception of korean, discussed below). japanese character embeddings behave differently. for instance, the correlation with the sound representations is weaker than for the other languages, which might be due to the syllabic nature of the japanese script. in the following, we discuss the results for each of the perceptual embedding types in detail. ## probing task except for japanese, the results show that the neural embeddings correlate the most with the perceptual sound representations. to get a closer look at the information that may be encoded in the dense embeddings, we design a probing task in which classifiers are trained to predict whether certain distinctive features are present given character embeddings as input. ## conclusion in this work, we attempted to understand the information encoded in character-level representations. we obtained two main types of embeddings: text-based embeddings and perceptual embeddings. while the first type of representations (ppmi, lstm, and transformer) were trained from raw text data, perceptual representations were obtained from sources mimicking human language, i.e., pronunciation dictionaries, synesthesia data and shape visualizations. we have performed representational similarity analyses between these types of embeddings for five different languages. besides, we defined and trained models to predict certain phonological distinctive features in order to interpret the embeddings. we found interesting patterns in the representational similarity analysis as a simple first approach for intrinsic character embedding evaluation. while clearly outperforming a random baseline in most cases, the strength of the correlations vary between scripts. for instance, the strong correlation between korean character embeddings and shape representations provides positive evidence of the suitability of this approach. further research is required to dissect the differences between character lms: while the lstm embeddings showed stronger correlation with color, the transformer embeddings were superior when compared to sound representations. the inclusion of additional languages and scripts will be helpful to identify more generalizable insights. these perceptual representations could be used as pre-trained representations. it might be the case that they contribute differently for different tasks. for instance, sound representations would be expected to be useful for tasks revolving around phonology, such as grapheme-to-phoneme conversion, or shape representations could be relevant for predicting orthographic errors. the phonological probing tasks show promising results, especially with respect to interpretability. besides, this methodology is applicable to any language with sufficient raw data and a pronunciation dictionary, and could potentially shed light in measuring the phonological difficulty of certain languages. in future work, we will focus on the development of more sophisticated probes, for instance, multitask networks with shared layers across tasks. moreover, the labels of the probing task were given from using the sound embeddings retrieved from the most frequent phoneme mapping. had we focused the analysis on contextual character embeddings instead, that would allow us to distance ourselves from this paradigm as we would be able to analyse character and sound embeddings in the context they occur in. finally, we stress the need for further intrinsic evaluation methods for character representations. the high impact of orthography on human language learning is an adamant argument to consider the cross-linguistic diversity of writing systems more carefully in the development of nlp models.