Towards Better Inclusivity: A Diverse Tweet Corpus of E nglish Varieties (2024)

Abstract: The prevalence of social media presents a growing opportunity to collect and analyse examples of English varieties. Whilst usage of these varieties is often used only in spoken contexts or hard-to-access private messages, social media sites like Twitter provide a platform for users to communicate informally in a scrapeable format. Notably, Indian English (Hinglish), Singaporean English (Singlish), and African-American English (AAE) can be commonly found online. These varieties pose a challenge to existing natural language processing (NLP) tools as they often differ orthographically and syntactically from standard English for which the majority of these tools are built. NLP models trained on standard English texts produced biased outcomes for users of underrepresented varieties (Blodgett and O’Connor, 2017). Some research has aimed to overcome the inherent biases caused by unrepresentative data through techniques like data augmentation or adjusting training models. We aim to address the issue of bias at its root - the data itself. We curate a dataset of tweets from countries with high proportions of underserved English variety speakers, and propose an annotation framework of six categorical classifications along a pseudo-spectrum that measures the degree of standard English and that thereby indirectly aims to surface the manifestations of English varieties in these tweets.

## introduction in many respects, there is a lack of linguistic and geographic diversity in nlp research. whether that be in authorship identity, conference location, or target language @xcite , it can be seen that wealthy, english-speaking nations are overrepresented. even within these overrepresented nationalities, there exist subgroups, usually correlated with socioeconomic factors, who are not * these authors contributed equally to this work fairly represented @xcite . a natural consequence is that the international proliferation and rapid advancement of language technologies serve the interests of the privileged. one major influence on the fairness of modern nlp systems is data availability. our research targets the geographic diversity of nlp datasets and specifically seeks to introduce a new corpus that captures a range of english varieties. english's status as a global lingua franca -as a consequence of historical colonialism and the current social influence of the western world -has meant that it has been adopted as an official language or, at least, it has become widely spoken in many countries across the world @xcite . in each culture, english takes on a different flavor to match the identities and needs of its speakers. in some cases, english is blended with other regional languages, forming pidgins and creoles. 1 in others, english takes over as a dominant language and phonetic indicators like accents or remnantal discourse markers become the main differentiators for that variety of english. sometimes, english becomes a complementary language, children learning it alongside one or more other distinct languages, regularly code-switching according to contextual circumstances or to take advantage of nuances in connotation of different languages @xcite ; each manifestation evinces a unique narrative about a speaker's cultural and socio-economic background. with such a range of englishes spoken, it seems a missed opportunity that widely-used english data sets significantly overrepresent samples from america and great britain relative to their respective proportional constitution of global english speakers @xcite . as a direct consequence, language technologies are predominantly trained with western varieties of standard english and make algorithmic assumptions based on standard english grammar. since the usage of english varieties are coupled tightly with geography and demography, there are numerous social and ethical implications of unrepresentative data. @xcite demonstrates there are accuracy biases in state-of-the-art speech recognition tools favoring accents and dialects most prevalent in training corpora. this reduces the effectiveness of, e.g., auto-captioning tools reducing the accessibility of global spoken media. blodgett and @xcite shows that language identification performs worse on aae than white-aligned english. this is notable because language identification is often an early filtration step in online text processing pipelines. for example, sentiment analysis research may discard anything labeled non-english including the higher proportion of false negatives for tweets of english varieties. this research aims to address this need to diversify data by producing a corpus of tweets posted from english-speaking countries across asia and africa, as well as english tweets from the united states and the united kingdom. our work leverages the availability of geo-tagged and informal written texts provided by twitter to create a dataset permeated by conversational dialectal featuresfeatures which were historically limited to the spoken medium. furthermore, the recruited annotators come from the international student body of new york university which consists of multilingual individuals. thus, these annotators are capable of extracting insights drawn from their experience in communities where a select english variety is commonplace. we propose a new annotation system that caters to social media tweets of english varieties, and assess the linguistic diversity of our corpus by examining its distribution across 6 categories and frequently-used words or phrases in each location. we apply common language identifiers such as langid.py, spacy, and google translate api to highlight their performance disparities compared to tweets of non-western english varieties. in section 2, we examine related nlp work on underserved english varieties, and outline some other attempts at addressing unrepresentative corpora. section 3 elaborates on the choice of twitter as a data source, explains the data collection approach and provides a brief linguistic variation analysis on our data. section 4 describes the annotation process including: a summarised set of annotation guidelines and its justification (section 4.1), inter-annotator agreement benchmarks (section 4.2) and the recruitment and training of annotators (section 4.3). section 4.5 evaluates the resulting corpus. in section 5, we investigate biases in several off-the-shelf language identification tools towards non-western english varieties. section 6 discusses potential applications and the continuation of this research. we also acknowledge several limitations of our corpus. ## related work the need to address the underrepresentation of english varieties in corpora has been recognised and approached in a number of ways ranging from manual collection to automatic generation. @xcite develops a rule-based method to translate standard american english (sae) sentences to african-american english (aae). this tool -codeswitch -relies on a list of 20 deterministic substitutions that aim to preserve "textual accuracy i.e., the original structure, intent, semantic equivalence and quality of a text." these are derived from the translations and annotations of 3 aae speaking workers. an independent set of aae speakers generally believed that the machine generated output of the tool resembled human translations. the researchers note that a deterministic substitution method lacks contextual awareness and, consequently, some of the machine generated text were indeed identified as most likely machine generated. such a tool simultaneously overgeneralises and undergeneralises the use of aae. not only would the rules be reflective of the patterns of the 3 aaespeaking workers, it assumes that every instance of a 'translatable' sequence would be converted into an aae form. similarly, the substitutions lack the inclusion of more complex varietal features such as idiomatic phrases or regional-specific lingo. nevertheless, codeswitch is a highly scalable and rapid way to augment current sae datasets with an english variety, leveraging most of the existing labels since substitutions preserve the overall syntactic qualities of the text and thereby reducing the labordemanding need to create and label new datasets. the international corpus of english (ice) project @xcite relies on the efforts of global linguistic research teams to produce english corpora for regions and countries where english is an official first or additional language. they curate and annotate various spoken and written texts like press editorials, news broadcasts and classroom lessons. this process is significantly labour-intensive and requires specialised local researchers. these corpora are much more suited towards qualitative sociolinguistic investigations due to the limited sample sizes. furthermore, each corpus is composed predominantly of formal texts so most comparative observations would be made on the standard national varieties of english as opposed to the more dialectal and informal varieties that our investigation is targeting. cook and brinton (2017) create national web corpora which are also largely composed of formal texts. they scrape websites which have nationaltop level domains (e.g. .au, .ca) corresponding to the countries. the authors conduct some frequency comparisons between each national corpus and are able to surface some terms unique to a country (e.g. canadianisms like "heritage language" or "pot light"). this process is more automated but the resultant corpora similarly lack the informal variations we are studying. finally, the twitter aae corpus by blodgett and o'connor (2017) most closely resembles the ones we are developing. it is a corpus of african american-aligned english tweets categorised based on probabilistic assumptions about tweet authorship. a tweet was placed in the african americanaligned corpus if it contained terms that were more frequently observed in tweets posted from areas with high concentrations of african-americans. we make a comparable, albeit less statistically complex, geographic assumption in the creation of our corpus that tweets from a particular country are more likely to exhibit features of a given variety. ## data we use twitter as a source for our data for two primary reasons. first, its international popularity and accessible api allows for simple data collection, due to the large volume of data which can be filtered based on location. foot_2 secondly, despite its written mode, the informal social media context means that some samples will be able to capture the differences between english varieties that would usually be found in casual spoken conversations. these differences include: code-switching, non-standard spellings to reflect phonetic variation and regional lexical markers. ## annotation the priority in designing annotation guidelines is ensuring that the classifications are (a) meaningful so annotators are able to confidently differentiate between labels whilst also coming to independent agreement about the classification of a text and (b) useful so the tweets in different categories contain extractable and informative features. the two primary researchers iteratively refine the annotation criteria, alternating between collaborative and independent labeling. they regularly apply interannotator agreement metrics to measure incremental improvements. the goal is to use class labels to isolate english varieties, by first distinguishing broadly between english and non-english words or sentences, and then distinguishing standard written english from english varieties. the following list has been taken directly from @xcite and reflects the guidelines we sought to follow in the creation of the annotation guidelines and throughout the annotation process: ## nlp tools on english varieties we evaluate three language identifiers -langid.py, spacy-langdetect, and google translate api -for detecting english vs. non-english tweets in our corpus. in our annotation framework, tweets in categories 1-3 should be classified as english without question. for category 4, code-switching, we randomly sampled 100 tweets from accra, islamabad, manila, new delhi, and singapore, and manually determined the proportion of english words, including both standard and informal language. our analysis reveals that 82% of the code-switched tweets had at least half of their content in english. thus, we consider code-switched tweets as english. ## future work while the initial annotation process was laborintensive, it was essential for us to develop a thorough annotation framework. this serves as a potentially valuable resource in the continuing labeling process, ensuring that the labeled data is consistent and of high quality. nevertheless, the limited amount of labeled data in our corpus poses a significant challenge to achieve sufficient coverage and accuracy in many nlp tasks. to address this issue, we plan to explore active learning techniques or large language models (llms) such as gpt-3 to expand our corpus. the labeled portion of our corpus will serve either as seeds to train a machine learning model that can then identify similar, yet unlabeled, data points in the remaining corpus, or as a baseline to evaluate annotations produced by llms. moreover, our corpus experiments focus on a fundamental nlp task: language identification, specifically identifying english vs. non-english tweets. we demonstrate the biases in pre-trained language models towards non-western english varieties. however, we recognize the need to broaden the scope of our investigation to other nlp tasks, with the goal of developing more robust, inclusive and accurate nlp models. for example, our corpus could be useful in training sentiment analysis models to analyze the sentiments of writers from different linguistic backgrounds. finally, we are committed to expand our corpus to include a wider range of global english-speaking communities. while our current corpus covers several countries in asia and africa, we recognize that there are many more english varieties around the world with linguistic characteristics different from those in our corpus. we believe that this continued expansion of our corpus will benefit researchers studying subfields of linguistics like sociolinguistics and corpus linguistics and enable the development of robust nlp models that are better suited to the needs of diverse linguistic communities. ## conclusion we have presented a diverse tweet corpus of english varieties and an annotation framework to label tweets. we analyse linguistic indicators of these english varieties and demonstrate that, despite a superficially independent relationship between some 'adjacent' labels, the classifications exist along a spectrum which intuitively measures distance from formal, standard english. we train several linguistically-diverse and geographicallyappropriate individuals to annotate a collective 3564 tweets. our experiments also show that there exists bias towards english varieties in off-the-shelf language identification tools when evaluated on our diverse corpus.