Towards Augmenting Lexical Resources for Slang and A frican A merican E nglish (2020)

Abstract: Researchers in natural language processing have developed large, robust resources for understanding formal Standard American English (SAE), but we lack similar resources for variations of English, such as slang and African American English (AAE). In this work, we use word embeddings and clustering algorithms to group semantically similar words in three datasets, two of which contain high incidence of slang and AAE. Since high-quality clusters would contain related words, we could also infer the meaning of an unfamiliar word based on the meanings of words clustered with it. After clustering, we compute precision and recall scores using WordNet and ConceptNet as gold standards and show that these scores are unimportant when the given resources do not fully represent slang and AAE. Amazon Mechanical Turk and expert evaluations show that clusters with low precision can still be considered high quality, and we propose the new Cluster Split Score as a metric for machine-generated clusters. These contributions emphasize the gap in natural language processing research for variations of English and motivate further work to close it.

## introduction current research in natural language processing has a fundamental gap: we lack strong resources for variations of english other than formal standard american english (sae), including slang and other dialects. african american english (aae) and slang, in particular, are both very common genres of modern english, especially on social media. this lack of lexical resources prevents us from creating technology for analyzing dialects like these, which evolve quickly on social media. we already have lexical resources for understanding sae (e.g., the dictionary of affect in language for sentiment, wordnet for synonyms and antonyms, and conceptnet for a variety of word relations), but equivalent resources for nonstandard english do not exist @xcite @xcite . the small wordnet-like resource for slang, slangnet, contains only 3000 words (compared to the 150000+ words in wordnet), and there are no resources for aae at the time of this study @xcite . to develop tools for analyzing nonstandard english, we need lexical resources that can provide clues about the meaning of new words that appear. we also need approaches for evaluating whether the derived representations are accurate or not. in this paper, we present a comparison of methods that combine clustering algorithms and word embeddings to group unknown words in the semantic space. we explore the use of agglomerative and k-means clustering on glove and word2vec embeddings and brown's clustering on bigrams to create semantically related clusters of words which could then be used in downstream tasks. the overall goal is to create clusters of semantically related words found in two datasets with large amounts of aae and slang, but we first conduct preliminary exploration on the smaller, more standard brown corpus. this corpus gives us a quick way to evaluate combinations of clustering algorithms and word embeddings, given that it uses a standard, formal dialect of english on which our algorithms can produce more directly this work is licensed under a creative commons attribution 4.0 international license. license details: http: //creativecommons.org/licenses/by/4.0/ . after clustering words in the brown corpus, we need an automatic way to determine if the clusters are accurate. we use wordnet synsets and conceptnet "relatedto" relations as the gold standards for sets of semantically related words and calculate the precision and recall of machine-generated clusters, but we note that even sae is better evaluated using human judgements. after evaluating these measures for all combinations of clustering algorithms and word embeddings on the brown corpus, we decide to move forward with glove embeddings and k-means clustering as the optimal combination on the two experimental corpora. the first corpus, twitteraae, was collected by the slang lab at the university of massachusetts at amherst and slightly resembles sae @xcite . the gang violence dataset, collected by the safe lab at columbia university, is the second corpus; it contains aae combined with hyper-local slang that make the text especially difficult to understand @xcite . we refer to the two aae corpora as the experimental datasets. by clustering unknown words with related known words, our approach can expand existing resources for automatically learning slang and aae words. throughout our work, we learn that established lexical resources like wordnet and conceptnet need to be augmented to support slang and aae: these resources are unable to serve as gold standards for words that have never been seen before. we also provide results from human evaluations on subsets of our machine-generated clusters. an evaluation on amazon mechanical turk (amt) over the clusters produced by the brown corpus shows that our machinegenerated clusters are of high quality even when precision and recall scores may disagree. an expert from the columbia university school of social work also evaluated a small set of example clusters from the experimental datasets because the turkers would not be familiar with the slang in these corpora. our efforts in combining clustering algorithms with word embeddings, automatically calculating precision and recall based on established lexical resources as gold standards, and verifying our results with human annotators show that we have made much progress towards methods for inferring the semantic meaning of slang and aae, but much work remains to expand lexical resources for these variations of english. this work makes the following contributions for analyzing slang and aae: ## related work researchers have already started to work on applications of natural language processing for social media and nonstandard english. one approach includes adapting pretrained word embeddings for target domains, like social media. recently, @xcite generated a lexicon and set of domain-specific word embeddings that were automatically induced from an unlabeled section of the gang violence dataset that is used in this work. @xcite 's work on fine-tuning bert embeddings for early modern english and twitter supports the viability of using domain-adaptive fine-tuning for social media. costa bertaglia and volpe nunes (2016) propose an unsupervised, scalable, and language-and domain-independent method for learning word embeddings for brazilian portuguese. other unsupervised methods include @xcite 's label propagation framework to induce domain-specific sentiment lexicons using seed words and @xcite 's graph-based word sense disambiguation. fine-tuning pretrained word embeddings is a promising foundation for developing completely unsupervised algorithms for semantics. in our work, we focus on simple methods using word embeddings to create clusters of semantically related words, an approach that enables interpretability of results as well as information about the meaning of new words that are just beginning to appear with low frequency. advancements in nlp methods related to nonstandard english could inform research in sociolinguistics and dialectology, which typically use other methods @xcite . timestamps on social media datasets also allow for observing and predicting the evolution of language. robust systems for tracking the appearance of new words, association of new meanings with existing words, and disappearance of old words can help us understand how, when, and why language changes. @xcite show that language evolution in computer-mediated communication reflects geographic proximity, population size, and racial demographics. @xcite find that linguistic dissemination is a strong predictor of the longevity of a new word while social dissemination is not. change in online language is driven by social dynamics and sociocultural influence, and using natural language processing techniques on large social media datasets yields important results for sociological studies @xcite . along with tracking the evolution of language, our methods can help with the use of word embeddings for lexical discovery @xcite . our work aims to embrace the constantly evolving nature of language and improve the representation of meanings of new words introduced over time. ## corpora and linguistic tools this paper presents clusters of vocabulary from three corpora: brown corpus, twitteraae, and the gang violence dataset. we also use wordnet and conceptnet to automatically calculate precision and recall of automatically generated clusters. we present example sentences from each corpus below. brown corpus "he's all right, craig," rachel said. twitteraae whoever tryna do this tax thing to get more bread let me know gang violence people just .i.p dis [emoji] [emoji] i ## manual evaluation with amazon mechanical turk to evaluate the quality of a cluster, amazon mechanical turkers are given a machine-generated cluster of words and asked to split the cluster into subclusters of semantically similar words (a cluster that requires no splits would yield one subcluster: itself). semantic similarity includes synonymy or relevance in the same context. red and blue, for example, are not synonyms but they are relevant in the context of colors, so they would be considered semantically similar and clustered together. turkers were given the following instructions (below) with three examples (full instructions with examples are provided in the appendix): you will be given a list of several words. these words will be separated with a space-there are no phrases or compound words. the goal is to group words into as few groups as possible by semantic similarity. words may be grouped together if they have similar meaning or would make sense appearing in the same context with at least one other word in the cluster. groups of singleton words are acceptable, but not every cluster should be singleton words. we also define our own metric, the cluster split score (see equation 3 ), for evaluating mechanical turk annotations. the intuition is simple: a high-quality cluster would not need to be split into multiple clusters because all of the words should be similar to each other. this can be considered a top-down hierarchical approach. a high-quality cluster would have a low number of splits. clusters with less than three words are omitted from this evaluation task and 300 clusters are randomly sampled without replacement from the remaining. the 300 clusters are divided into three batches of 100 clusters, and each batch is evaluated by three turkers. we then take the mean number of splits as the raw score for each cluster. we instruct the turkers to perform the task on clusters from the brown corpus because these words are more familiar and the clusters are more easily interpretable. although the task showed only slight agreement (fleiss' @xmath0.046, 0.066, 0.084 for each batch), the low number of mean splits indicate that the clusters are of higher quality than the automated precision scores may imply (see figure 2 ). this task would benefit from rigorously defining "semantic relatedness" and selecting knowledgeable annotators, especially for tasks including slang and aae. after mechanical turk evaluations are complete, we calculate the cluster split score for each cluster: @xmath1. we divide the number of words by the number of splits because we would like to reward large clusters remaining intact while penalizing clusters being divided. this number then becomes the input for the logistic equation, which squashes the range of the scores between 0 and 1, with 1 indicating the highest quality; this ensures that cluster split scores can be compared for clusters of different sizes and encourages smoother steps between scores. the automated precision and css are reported in figure 2 . we also ask an expert from the columbia university school of social work to evaluate a small subset of clusters from both experimental datasets. of the 14 gang violence clusters, he agrees with the machine generation 4 times and splits the cluster in two parts 4 times, for a mean css of 0.96. of the 12 twitteraae clusters, he splits the cluster in two parts 12 times and never agrees with the machine-generated cluster, for a mean css of 0.83. the high mean cluster split scores given by the expert show that machines can automatically generate high-quality clusters. below, we present sample clusters from the gang violence and twitter-aae corpora with the expert's annotations. each line shows one machine-generated cluster that has been split into subclusters by the expert. these clusters show many variations of english, like influences from french and spanish and alternative spellings for existing words. some words may be completely unfamiliar, which is why we motivate continued efforts in expanding lexical resources for slang and aae. gang violence { harry, mila, naya, malik, louis, payne, lou }, { hacked, dmed }, { lux } twitteraae { shootah, bakk }, { hyz }, { bizz } twitteraae { hungova, shleeep }, { haself } if the automated precision scores from conceptnet and cluster split scores from amt both reliably evaluated cluster quality, then we would see that precision and cluster split scores are directly correlated. this, however, is not the case in figure 2 . the scores are skewed to the top left, showing that clusters with low precision often receive a high cluster split score. this further shows that current lexical resources need to be expanded to become more inclusive of slang, aae, and other variations of english. ## conclusion and further studies natural language processing resources lack representation from slang and nonstandard english that would allow us to reliably and automatically evaluate our methods. in this work, we used glove embeddings and k-means clustering to cluster semantically similar words in the brown, twitteraae, and gang violence corpora. mechanical turk and expert observation showed promise in the quality of automatically generated clusters and that these clusters can reveal semantic relatedness between unknown words. in contrast, precision and recall scores from wordnet and conceptnet do not currently reflect this. we can improve this problem by expanding lexical resources for slang and aae. automatic clustering with simple parameters already show great potential for automatically learning the meanings of new words. this paper presents tools for automatically and manually evaluating these new resources for slang and aae that can be created or expanded in future work. context-dependent training methods for sense disambiguation would make clusters more robust. in addition, another important area for improvement is using domain-adaptive fine-tuning to include words that do not have pretrained embeddings and adjusting embeddings for words that may have taken on a new meaning. machine-generated clusters can also be used to track the evolution of language and learn new words as they appear online. this is especially important in the age of social media since new slang terms appear so quickly. existing words can even take on new meanings; old data for existing words can then affect results if the meaning of the word has changed. the words "terrible" and "terrific," for example, used to be synonyms-our results would certainly be different if we still assumed that. with improved word embeddings and resources for evaluation, we can build highly interpretable clusters with an algorithm as simple as k-means clustering. these tools will help natural language processing will become more inclusive of all variations of language, not just the formal, standard features that historically gathered more attention.