Investigating A frican- A merican V ernacular E nglish in Transformer-Based Text Generation (2020)

Abstract: The growth of social media has encouraged the written use of African American Vernacular English (AAVE), which has traditionally been used only in oral contexts. However, NLP models have historically been developed using dominant English varieties, such as Standard American English (SAE), due to text corpora availability. We investigate the performance of GPT-2 on AAVE text by creating a dataset of intent-equivalent parallel AAVE/SAE tweet pairs, thereby isolating syntactic structure and AAVE- or SAE-specific language for each pair. We evaluate each sample and its GPT-2 generated text with pretrained sentiment classifiers and find that while AAVE text results in more classifications of negative sentiment than SAE, the use of GPT-2 generally increases occurrences of positive sentiment for both. Additionally, we conduct human evaluation of AAVE and SAE text generated with GPT-2 to compare contextual rigor and overall quality.

## introduction african american vernacular english (aave) is a sociolinguistic variety of american english distinct from standard american english (sae) with unique syntactic, semantic, and lexical patterns @xcite . millions of people from predominately black communities in the united states and canada use variants of aave on a daily basis. although aave has historically been used in spoken contexts, the growing use of social media has encouraged aave in written media for which nlp models are increasingly being used. recent work in natural language generation (nlg) has introduced gpt-2, a transformer-based language model that generates high-quality, coherent text when prompted by arbitrary input @xcite . however, gpt-2 displays bias * equal contribution. in this paper, we examine the bias of gpt-2 text generation against aave features. we create a new dataset of aave/sae content-controlled pairs by retrieving aave tweets and employing human translators to obtain their sae counterparts. by doing so, we isolate aave syntactic structures and lexical items. we then prompt gpt-2 with the first segments of each aave/sae pair. the generated text is compared to its corresponding original second segment by bleu, rouge, and sentiment scores. additionally, we provide human evaluation for the generated text based on context and quality. thus, our contributions include: ## dataset our dataset consists of tweets identified as having at least 99.9% confidence of using aave lexical items by the twitteraae dataset @xcite . we then obtain the sae equivalent of each of these tweets by employing amazon mechanical turk (amt) annotators for a total of @xmath0. the average length of the original aave tweets is about 21 words, and the average length of the sae counterparts is about 22 words. these samples are intended to be used as a test set for probing neural language model-based text generation. we use the terms "first segment," "second segment," and "generated segment" to refer to the different sections of each aave/sae sample throughout this paper. a visualization of these partitions can be seen in figure 1 . @xcite collects aave tweets by using a distantly supervised mixed-membership model on samples that are geolocated to african-american blockgroups, as defined by the u.s. census data. the tweets have been filtered to ensure conversational language and verified as aave on the basis of aave-specific lexical item inclusion, phonological phenomena in orthographic variation, and syntactic construction. from twitteraae, we randomly sample tweets that contain at least 15 words and have a posterior probability of being demographically-aligned to aave of at least 99.9%. we remove hashtags as they are social media-specific occurrences and emoji since we expect them to have disproportionate influence on sentiment scores. ## sentiment analysis we use a sentiment analysis pipeline from huggingface foot_0 to evaluate the sentiment of our samples. the pipeline uses distilbert-base-uncased-finetuned-sst-2-english foot_1 , which is pretrained on movie reviews from the stanford sentiment treebank @xcite . in addition to the distilbert sentiment classifier, we use vader, which is a lexicon and rule-based sentiment analysis tool that is attuned to social-media specific sentiment intensity @xcite , and textblob foot_2 , which does not have documentation on its implementation. however, we justify our use of the latter through its widespread use as an off-the-shelf sentiment classifier, such as in @xcite . the distilbert sentiment classifier restricts classifications to either positive or negative, with degrees of confidence ranging from 0 to 1; we translate this to a -1 to 1 negative-to-positive scale. from vader we use the compound score, and from textblob the polarity; both metrics are normalized and weighted and thus also range from -1 to 1. vader and textblob scores include 0.0, or neutral, while the distilbert sentiment classifier does not. we average the latter two in baseline as a baseline, we compare the sentiment of each aave original second segment to its respective sae original second segment. we observe that the pretrained sentiment analysis models categorize aave as more negative than sae, despite having the same intent. aave has 157 (7.7 % percent) more negative instances than it does positive when using distilbert and 37 (1.8 % percent) more negative and neutral instances when using the vader-textblob average. the vader-textblob averages appear to be less biased against aave than distilbert. ## quality of generated text we use bleu, rouge, and human evaluation scores to determine the difference in the quality of rouge-1 rouge-2 rouge-l 0 2 4 6 •10 -2 rouge score sae aave figure 3: rouge scores for text generated by gpt-2. ## conclusion through this work, we highlight the need for aave-inclusivity in nlg models, especially those perceived as state-of-the-art. to this end, we provide a new evaluation of nlg models by comparing gpt-2's behavior on sae and aave. in addition, we present a new dataset consisting of intent-parallel aave/sae tweet pairs, which can be used in future works studying sae and aave context quality likely mg sae 48.7% 54.5% 37.3% aave 36.7% 32.8% 42.1% tie 14.6% 12.7% 20.6%