Do Word Embeddings Capture Spelling Variation? (2020)

Abstract: Analyses of word embeddings have primarily focused on semantic and syntactic properties. However, word embeddings have the potential to encode other properties as well. In this paper, we propose a new perspective on the analysis of word embeddings by focusing on spelling variation. In social media, spelling variation is abundant and often socially meaningful. Here, we analyze word embeddings trained on Twitter and Reddit data. We present three analyses using pairs of word forms covering seven types of spelling variation in English. Taken together, our results show that word embeddings encode spelling variation patterns of various types to some extent, even embeddings trained using the skipgram model which does not take spelling into account. Our results also suggest a link between the intentionality of the variation and the distance of the non-conventional spellings to their conventional spellings.

## introduction word embeddings play a key role in many nlp systems. unfortunately, embeddings are opaque: it is difficult to interpret the individual vector dimensions and to understand which factors contribute to the relational similarity between words nearby in space. there has been an increasing interest in understanding what word embeddings are encoding, for example, to understand their impact on the performance of a wide array of nlp systems @xcite . moreover, this is especially important when embeddings are used as research objects themselves, e.g., to study biases in society over time @xcite . the body of work on analyzing embeddings has focused primarily on semantic and syntactic properties @xcite @xcite . instead, we propose a new perspective on word embeddings by asking how spelling variation is encoded. in social media, linguistic variation is abundant @xcite . for example, nonconventional spellings for nothing are nothin, nuthin, noooothing, nithing, and so on. much work in nlp has focused on normalizing spelling variation @xcite . similarly, recent work by @xcite aims to push embeddings of misspelled words closer to the embeddings of their conventional forms. however, these approaches can remove valuable social @xcite and semantic @xcite signals. crucially, many non-conventional spellings are not misspellings: by deliberately deviating from conventional spelling norms, writers create social meaning @xcite . for example, a certain spelling may be used to evoke intimacy or to index a certain region. capturing spelling variation patterns is therefore important for many applications, e.g., when analyzing social phenomena @xcite . to illustrate, figure 1 shows skipgram embeddings of nothing and spelling variants projected on a 2d plane using t-sne (van der @xcite . this figure clearly suggests structure in the embedding space as forms with different types of spelling variation are pulled apart. on the middle right side we find lengthened forms (e.g., noooooothing), which are often used for emphasis, and at the top we find forms that appear to be misspellings (e.g., nithing) and the conventional form (nothing). in the lower left corner, we find instances of g-dropping (e.g. nothin) and forms reflecting dialect pronunciation (e.g., nuffin, which presumably reflects a dialect pronunciation of the voiceless interdental fricative /θ/). because spelling is not taken into account by skipgram, this suggests that these forms are used in different contexts. indeed, research has associated a range of contextual factors with the use of specific types of spelling variations @xcite . we therefore go beyond looking at whether non-conventional forms are close to their conventional form in the embedding space. instead, we ask whether embeddings capture fine-grained patterns of spelling variation, such as information about the type of spelling variation (e.g., whether the final 'g' is dropped, or whether the vowels are omitted). why does spelling variation matter? if embeddings are supposed to represent language use, we should also expect them to represent meaning associated with the choice of a specific spelling. nonconventional spellings often carry social meaning, e.g. reflecting and constructing social identities. but, crucially, this has been neglected in nlp at large and in representation learning more specifically, ignoring the fact that in language it doesn't only matter what is said, but also how it's said. if the relationship between embeddings of conventional and non-conventional forms is meaningful, then this opens up a range of opportunities of using embeddings for exploring questions in computational social science and sociolinguistics. moreover, there are many downstream applications that would benefit (e.g., community detection, modeling conversation dynamics). and finally, nlp needs to move away from focusing almost exclusively on so-called "standard language", resulting in systems that do not work well for many social groups. treating spelling variation as a meaningful phenomenon is one step in this direction. ## related work nlp research on spelling variation has mostly focused on text normalization @xcite and automatically extracting lexical variants @xcite . studies within computational sociolinguistics have analyzed the patterns and functions of spelling variation, e.g., in twitter @xcite , and research has suggested a deep connection between phonological and spelling variation @xcite . furthermore, @xcite observed that spelling variation also personalizes and informalizes sms messages. we look at spelling variation but focus on how spelling variation patterns are encoded in embeddings. there has been much interest in analyzing neural representations and more broadly neural networks @xcite . besides word embeddings, representations for higher-level units such as sentences have also been analyzed @xcite . our study differs from work in this space not only because it is based on spelling but also because it is based on comparing forms with the same referential meaning. in contrast, many evaluation studies (e.g., @xcite ) use pairs that are not semantically (cat:kitten) or grammatically (argue:argument) equivalent. our study is more similar to @xcite , who used paraphrase pairs to investigate formality information in embeddings, and @xcite , who focused on three language variety pairs. we are not aware of studies focusing on spelling. ## datasets this section describes the datasets and the types of spelling variation we use in our analyses. ## experiments we now examine whether spelling variation is represented in a structured way in the embedding space using the selected pairs (section 3.2). no intrinsic evaluation method is perfect-we therefore present three different types of analysis that complement each other: analogy tests (section 4.2), a similarity analysis (section 4.3), and diagnostic classifier tests (section 4.4). our goal is not to achieve the highest performance. instead, by analyzing the performance-and how this differs between spelling variation types-we obtain new insights on what embeddings capture about spelling variation. ## conclusion deviations from conventional spelling norms often carry social meaning @xcite . depending on the application, it may thus not be desirable to map all spelling variants to the same point in the embedding space, especially in the areas of computational social science @xcite and computational sociolinguistics @xcite . this paper is a first step towards analyzing how spelling variation is captured in word embeddings. we presented three analyses in which we looked at seven types of spelling variation. taken together, we found that even embeddings trained using the skipgram modelwhich does not take spelling into account-encode spelling variation patterns to some extent. moreover, our analysis of the cosine similarities between non-conventional and conventional forms for different types of spelling variation suggest a link between the intentionality of the variation and the distance to the conventional forms. a possible explanation is that intentional spelling variation might occur in more specific contexts; future work should investigate this more deeply. we also note that although nonconventional spellings generally have the same referential meaning as their conventional form, they can sometimes also mark specific new meanings of words @xcite . as future work, contextual embeddings @xcite and other spelling variation types can be explored. we also plan to investigate other classifiers and feature representations to capture the interaction between two embeddings for our diagnostic classifiers @xcite .