Disambiguation of morpho-syntactic features of A frican A merican E nglish – the case of habitual be (2022)

Abstract: Recent research has highlighted that natural language processing (NLP) systems exhibit a bias againstAfrican American speakers. These errors are often caused by poor representation of linguistic features unique to African American English (AAE), which is due to the relatively low probability of occurrence for many such features. We present a workflow to overcome this issue in the case of habitual “be”. Habitual “be” is isomorphic, and therefore ambiguous, with other forms of uninflected “be” found in both AAE and General American English (GAE). This creates a clear challenge for bias in NLP technologies. To overcome the scarcity, we employ a combination of rule-based filters and data augmentation that generate a corpus balanced between habitual and non-habitual instances. This balanced corpus trains unbiased machine learning classifiers, as demonstrated on a corpus of AAE transcribed texts, achieving .65 F _1 score at classifying habitual “be”.

## introduction linguistic discrimination has adversely affected the lives of marginalized populations for centuries, including racially marginalized groups in the united states. in spite of extensive research on linguistic discrimination @xcite , many nlp systems inherit the linguistic biases that exist between humans. for example, preliminary studies into the performance of automatic speech recognition (asr) systems uncovered a performance bias against african american speakers @xcite . this problem was confirmed most recently by @xcite who found that the average word error rate (wer) for white american speakers was significantly lower as compared to the average wer for african american speakers among five prominent asr systems from such companies as google, amazon, and apple. this performance gap is rooted in two related issues. first, the linguistic differences between african american english (aae) and general american english (gae) include distinctive features in their morphosyntactic structures. second, incorrect inferences in nlp systems are often caused by the scarcity of certain linguistic features when training, and the many unique features in aae have a relatively low probability of occurrence. this paper describes work that overcomes the data scarcity issue for a specific feature unique to aae: the habitual "be". as the name suggests, this morphologically invariant form of "be" communicates habitual action. disambiguating habitual "be" from non-habitual "be" is difficult for two prominent reasons. first, the form is isomorphic with the other uses of "be", such as the infinite use in "i want to be...". second, habitual "be" is relatively rare even in corpora of aae. our work addresses both these issues. it uses a rule-based method that capitalizes on morphosyntactic differences to eliminate a portion of non-habitual "be" instances and it uses a method of data augmentation that increases the ratio of habitual "be" instances. the resulting balanced data can then be used to train classifiers to tag "be" instances as habitual or non-habitual. foot_0 2 related work distinguishing habitual "be" and non-habitual "be" usage is a word sense disambiguation (wsd) problem because it involves identifying the meaning of words in context @xcite . most successful wsd algorithms make use of contextual embeddings @xcite , but some feature extraction algorithms, such as the ims algorithm by @xcite , have a comparable level of performance although comparatively much simpler. the ims algorithm uses a support-vector-machine (svm) with simple contextual features, such as word form or part-of-speech (pos) tags, and weighted average of embeddings. similarly, our disambiguation pipeline makes use of the pos tags of the surrounding words. this helps avoid the limited amount of annotated aae data which could lead to sparse word vectors and unreliable embeddings. data augmentation techniques that generate synthetic, or artificial, language in the training data often improve nlp applications when the training corpus is small or when a certain feature occurs rarely @xcite . our approach follows previously successful examples of data augmentation methods that combine a language model @xcite with a thesaurus @xcite or word embeddings @xcite . these methods identify substitutes for words in the data and insert them into synthetic strings that include the target feature. the "be" verb has various functions. this includes several types of non-habitual use, as shown in appendix a. the use of habitual "be" is a prominent, distinct, and well-researched morphosyntactic feature in aae. habitual "be" is a morphologically invariant form of the verb that encodes the habitual aspect, as shown below @xcite . syntactic contexts serve as important cues for disambiguating "be" as habitual or non-habitual. @xcite show that asr systems not only fail to recognize habitual "be" more often than non-habitual "be" but, when habitual "be" is present in an utterance, the surrounding words are also incorrectly recognized, particularly preceding words. these findings reveal a strong dependency between habitual "be" and its syntactic context. failure to reflect this dependency in a language model could lead to a less accurate and biased system. even in an aae corpus, habitual "be" is relatively rare. this imbalanced distribution poses a challenge for designing a non-biased nlp system because most classifiers tend to be biased towards the majority class. the ambiguity and scarcity of habitual "be" presents two obvious approaches to a solution: (i) incorporate more habitual "be" instances in the data, (ii) manually disambiguate habitual and nonhabitual "be" before training. each approach poses a challenge. for (i), simply collecting more data is extremely impractical, as the habitual "be" is naturally rare. for (ii), hand-coding is unsuitable for the scale of the data needed. our study addresses these challenges with a rulebased filter based on syntactic cues and with a data augmentation technique. together the filter and data augmentation increase the ratio of habitual "be", providing a more balanced training set for the model and allowing for a more fine-grained language model. ## methodology the first novel task towards training classifiers to disambiguate habitual "be" is to address the ambiguity of the invariant form by eliminating as many non-habitual "be" instances as possible. the second task is to increase the proportional occurrence of habitual "be" in the training data. we undertake these two tasks and incorporate them into a pipeline, shown in figure 1 . first, the entire corpus is run through a pre-trained nltk tokenizer and pos tagger trained using the penn treebank project. to eliminate as many non-habitual "be" instances as possible, a rule-based filter identifies determinate instances of non-habitual "be". with these removed, we increase the proportional occurrence of habitual "be" by augmenting the proportion of habitual "be". finally, we combine the filtered habitual "be" instances back into a now balanced dataset and use that dataset to train an ensemble model for classification. as discussed in section 5.1, the habituality of each instance is known and allows accurately creating rules and training the classifiers. ## experiment unbiased nlp systems should successfully disambiguate instances of habitual "be". we implemented our system on a corpus of aae speakers after training it our filtered and balanced corpus. ## conclusion our goal was to develop a pipeline which aids the creation of models unbiased against african american english. we proposed and tested a combi- the hand-crafted rules we used took into consideration morphosyntactic patterns that are unique to aae and correlate with habitual "be" usage. this allowed us to filter out most non-habitual "be" instances. we then found that word2vec and word-net augmentation methods were able to adequately imitate aae structure and balance the proportion of habitual "be" instances. together the filtering and the augmentation resulted in more balanced data with which to train the classifiers. in the future, with an increased amount of natural speech and more advanced classification algorithms, it is possible that the classification performance could be even higher. however, due to limited data, we treated the entire coraal corpus without regard to several interesting factors that should be considered. for example, we did not regard the geographic location or origin of the speaker. further analysis of our model's performance with respect to regional sub-varieties of aae would be an interesting avenue to explore. this exploration might refine the hand-crafted rules. also, our pipeline makes use of the pos tags of the surrounding words, similar to @xcite , but it does not include the surrounding words themselves or their embeddings as features because the limited data would have led to sparse word vectors and unreliable embeddings. we feel it should be easy to adapt our pipeline to other unique aae features such as the completive "done" @xcite . although we expect feature-based models to tend to perform better at low-resource settings than deep learning, we plan to compare our results against state-of-the-art neural models such as the transformer @xcite . the increase in scores we were able to achieve with these simple methods serves as a proof-ofconcept that systems based on similar syntactic filtering and data augmentation approaches have the potential to improve the performance of other aae-focused nlp systems and provide enough data for more advanced feature representations. a appendix: types of non-habitual "be"