What is on Social Media that is not in W ord N et? A Preliminary Analysis on the T witter AAE Corpus (2021)

Abstract: Natural Language Processing tools and resources have been so far mainly created and trained for standard varieties of language. Nowadays, with the use of large amounts of data gathered from social media, other varieties and registers need to be processed, which may present other challenges and difficulties. In this work, we focus on English and we present a preliminary analysis by comparing the TwitterAAE corpus, which is annotated for ethnicity, and WordNet by quantifying and explaining the online language that WordNet misses.

## introduction natural language processing (nlp) tools and resources have usually been developed for major and standard varieties of language. maintaining and updating these resources is expensive @xcite , but recently open-source methodologies are being used to update the english word-net @xcite . however, well-known state-of-the-art tools in data processing that are used in industry and offer semantic analysis, such as nltk @xcite or knowledgebased word sense disambiguation tools like ukb, still rely on princeton wordnet @xcite , which has not been updated for a long time. on the other hand, nlp tools are being used nowadays in many industrial, marketing and social analysis projects and mainly social media text is being used. it is well known that social media may present several challenges when it comes to data processing, since, among others, non-standard varieties of languages or slang are used. other problems related to this kind of texts are the length of the produced texts (sentences are rather short) and the difficulty of identifying useful information in order to separate it from what is not useful, such as hashtags, mentions or user-names @xcite . in this paper, we explore the coverage of sociolects in wordnet to see what information stateof-the-art knowledge-based nlp tools may miss when analysing social media. in this preliminary analysis we use the twitteraae corpus @xcite , which contains geographic and census information. exactly, we follow this methodology: a) we select a corpus with geographical/sociological information; b) we extract a sample of each group and we preprocess it; c) we compare it against an nlp resource, in our case, word-net, and carry out a quantitative and qualitative analysis of the differences. as a contribution of this preliminary study, we want to raise awareness of, on the one hand, how much linguistic diversity can be found in common data sources and, on the other hand, how risky it may be to use generic nlp tools and resources to process these diverse linguistic registers (colloquial, informal, internet language...) and sociolects for which the tools were not designed. this paper is structured as follows: in section 2 we present the corpus we have used and its preprocessing, in section 3 we compare quantitatively and qualitatively the most frequent lemmas in the corpus to wordnet, in section 4 we discuss our findings and expand upon some issues that concern the whole project. finally, in section 5 we conclude and outline the future work. ## corpus selection and preprocessing the dataset used in this project was the publicly available twitteraae corpus @xcite , which consisted of 59.2 million publicly posted geolocated tweets (f. @xcite . these data were collected in the united states in 2013. each message was annotated considering the u.s. census block-group geographic area from which it was posted, meaning that ethnicity and race information associated with that district was taken into account. four different covariants were established for the annotation: (non-hispanic) black, hispanic (of any race), (non-hispanic) asian, and (non-hispanic) white. this grouping reflects the main categories observed in the us census data, removing some smaller categories like "native hawaiian and other pacific islander", and thus naturally as limited in nuance as the census categories may be (e.g. the census report groups all black identities together, whether the respondents are african-american, african, caribbean...). the terminology used in this paper will therefore reflect this simplified classification. for every user, the demographic values of all their tweets within the dataset were then averaged resulting in a length-four vector. the demographic data associated to each user and their corresponding tweets were then used by @xcite to develop a mixedmembership probabilistic model that linked linguistic features with ethnicity. this model, the twitteraae model, assigned ethnicity values to the messages according to census data, giving each ethnicity a different proportion based on the "weight" of that ethnicity in the area where the tweet was written. if a tweet was assigned a proportion of more than 0.8, that meant that the tweet had a strong association with one of the previously mentioned demographic groups. a small sample of example messages belonging to each of the four covariants can be found in ## comparison to wordnet in this section, we present a quantitative analysis of the lemmas found in each of the subcorpora and in wordnet. on the one hand, we have analyzed the number and percentage of lemmas and unique lemmas not found in wordnet. on the other hand, we have calculated the intersection of the subcorpora with the white subcorpus. moreover, in the qualitative analysis we present the commonalities and the specifics of each subcorpus. ## discussion this preliminary experiment has allowed us to see that general nlp tools' performance on online, colloquial speech is suboptimal, especially with texts written by users outside the white demographic according the annotations of the corpus. we used the spacy lemmatizer and ner tool, which are very popular nowadays, but even these modern tools had issues with some phenomena of internet speech: new terms, alternative spellings, new named entities and disregard for capitalization. we have seen that wordnet was developed with standard english in mind and has not been updated for many years, so it fails to account for "modern" terms (we are considering tweets from 2013 relatively modern), online slang and diverse dialects. interestingly, we saw that wordnet includes many multiword expressions (over sixty thousand), but the trend online seems to go in the opposite direction: expressions are shortened into acronyms (e.g. "lol", "omg"), and even single words are shortened (e.g. "bday, "txt"). as vast amounts of text are produced online daily, and this is of interest to businesses and researchers, there are initiatives that try to better deal with the type of language used online. for instance, colloquial wordnet @xcite aims to be a more modern version of wordnet, one that includes popular, colloquial terms used online and slangnet @xcite gathers slang words and neologisms from the internet structured like in wordnet. it would certainly be worthy of study whether these resources recognize twitter lexicon better; in our study, however, we did not perform any analysis using colloquial wordnet, due to the difficulty in extracting its list of lemmas, at least in comparison with the easy method available in wordnet (a line of python code suffices and returns text with no inconvenient tags) and slangnet is not available. wordnet does not include certain parts of speech, such as prepositions; it only includes "open class words". nonetheless, as we have seen, internet users create new versions of "closed class words" (e.g. "eht" as a synonym of "at") or create words that merge words from both classes (e.g. "lemme", meaning "let me"). a deeper analysis of the words from our corpus belonging to or containing pos not present in wordnet would be valuable, to consider whether such words should be added to semantic databases, or whether lemmatizers should be adapted to extract the standard form when processing new variants. though the focus of this project was on englishlanguage text, it is important to emphasize the large number of tweets written in languages other than english, especially in the case of the asian subcorpus. therefore, any toolkit employed to process tweets from the us will need to include language detection and analysis tools for languages other than english -processing only english leaves many users behind and reduces the validity of any conclusions that might be extracted from analyzing tweets. future studies in this area, when possible, should also analyze a larger section of the twit-teraae corpus. it is important to have a large corpus size to prevent a single user's repetitive lexicon from distorting the results. alternatively, this type of users could be detected as part of preprocessing and their tweets excluded. even though the corpus has been very useful and is relatively modern, considering how fast language can change online, it would be necessary to replicate the methodology of @xcite to annotate more recent tweets. the methodology could also be applied to other languages for which nlp tools and demographic data are available (e.g. to analyze dialects of spanish or german). the resulting datasets would be very valuable for sociolinguistic studies, but also to assess nlp tools' inclusivity -are nlp tools leaving some groups of people behind? nonetheless, it must be noted that, as any model, the one employed to annotate the dataset used here showed some inaccuracies. though the sociolinguistic validation performed by @xcite proved it quite accurate for aae, classification in other categories seemed more problematic (e.g. the large number of spanish tweets in the asian category). this may be due partly to the even larger diversity of asian and hispanic groups, which makes classifiying people into four categories overly simplistic at times (e.g. where do brazilians go, being racially very diverse and culturally close to the hispanic demographic but outside it?). problems may also have arisen due to the source of the data used to build the model: the us census is known to undercount minorities. 6 even though race and ethnicity are self-reported, the way the data are aggregated is problematic for some groups, such as middle-eastern populations, afro-latinos or portuguese speakers. 7 moreover, the way the census data were linked to the tweets may have also introduced some inaccuracies: geolocation may not have been perfectly exact, 8 and it may sometimes have been false, given the large number of internet users who connect through vpn. 9 finally, we would like to emphasize the same message that @xcite leave at the end of their paper: african americans are underrepresented in the computer science community, which makes it much harder for their voices to be taken into account. this conclusion is also valid for the hispanic demographic, though for the asian demographic there seems to be adequate representation. foot_8 ## conclusion and future work in this paper, we have carried out an analysis of a corpus with geolocated tweets and we have compared the lemmas used to wordnet. as the corpus contained text from social media, we have dealt with non-standard language and we have seen that it still presents a challenge for mainstream nlp resources, which may put them at risk of leaving behind some speakers and varieties. as a result of this study, we encourage linguistic work on different registers and non-standard varieties. in the future, we plan to expand the analysis to a bigger sample of the corpus and apply this methodology to study other languages e.g. spanish with corpus de referencia del español actual (crea) corpus. moreover, we are preparing a list of candidate synsets to propose to the english wordnet @xcite following the open source and collaborative initiative. moreover, we would like to study the possibility of adding register/ geographical information to synsets as e.g. @xcite are proposing for the swiss variety of german. analysing other twitter tokens such as hashtags or mentions that were left out in the preprocessing could lead also to other studies.