Corpus-Guided Contrast Sets for Morphosyntactic Feature Detection in Low-Resource E nglish Varieties (2022)

Abstract: The study of language variation examines how language varies between and within different groups of speakers, shedding light on how we use language to construct identities and how social contexts affect language use. A common method is to identify instances of a certain linguistic feature - say, the zero copula construction - in a corpus, and analyze the feature’s distribution across speakers, topics, and other variables, to either gain a qualitative understanding of the feature’s function or systematically measure variation. In this paper, we explore the challenging task of automatic morphosyntactic feature detection in low-resource English varieties. We present a human-in-the-loop approach to generate and filter effective contrast sets via corpus-guided edits. We show that our approach improves feature detection for both Indian English and African American English, demonstrate how it can assist linguistic research, and release our fine-tuned models for use by other researchers.

## introduction linguistic features-such as specific phonological, syntactic, or lexical phenomena that may be associated with a language variety-are widely used by sociolinguists to quantify linguistic variation between speakers through feature frequency measurements @xcite @xcite , even if subject to certain limitations @xcite . since manual annotation is limited due to the required expert human labor, automatic methods are a valuable alternative @xcite @xcite . however, accurately detecting morphosyntactic features (e.g. figure 1 ) remains an open challenge, especially in informal genres such as transcripts and social media, and in low resource nonstandard languages. we explore fine-tuning pretrained language models (lms) for utterance-level classification of a feature by training on a contrast set-a small collection of positive and negative examples that are highly similar-as recently introduced by @xcite . our work makes the following contributions: ## results and analysis baselines. we compare our approach (which we refer to as cgedit) to several baseline methods, all of which take the same seed set of positive examples p then add negative examples to complete the training set. examples in p were sourced from @xcite for inde and crafted by the authors for aae. manualgen: the approach used in @xcite . this method involves manually generating negatives by modifying positive examples so they are (1) semantically-similar mainstream american english versions, and (2) do not have the feature (see figure 2 ); see discussion in §4.1. next, we also test two methods to completely automatically generate negative examples: autogen: this approach automatically generates negative examples by dividing a positive example p into n-grams and shuffling the n-grams. autoid: automatic identification randomly chooses unlabeled examples from target corpus c as the negatives. the assumption that unlabeled examples are negatives with class label noise underpins contrastive learning @xcite and pu learning methods @xcite . overall results. better performance on aae corpora may be due to a few variables: a higher number of aae features means a larger total training set; larger aae corpora mean more target corpus n-grams; the selected aae features may be easier to distinguish or more prevalent than the inde ones. discrepancies between coraal and fwp are likely due to different feature prevalences. results by feature. feature difficulty is similar across approaches; invariant features are easier to detect (i.e. focus itself in inde; finna in aae), while features with long-distance dependencies are more difficult (i.e. double object construction in aae). see appendix c for complete results. ## replicating prior sociolinguistic work we recreate three recent studies of coraal where original authors manually annotated aae morphosyntactic features and analyzed correlations between feature frequency and speaker metadata (i.e. gender, region, socioeconomic status). we used the combined manualgen + cgedit model and classify & count (cc, summing hard classifications @xcite ) to calculate per-speaker feature frequency. 6 the same subsets of features and coraal were used as in previous work when possible; detailed results are in appendix c. @xcite annotated 35 morphosyntactic features in 150 utterances. we confirm their conclusions that average feature frequency was lowest in rochester, followed by dc, then princeville; and lower among male speakers than female. cukor-avila and balcazar (2019) looked at 3 features over 14,506 utterances. they qualitatively found considerable variation in feature use between speakers, even when within the same age group. we confirm this quantitatively: standard deviation between speakers within an age group is larger than standard deviation between age group means. grieser (2019) examined 14 features over 18,553 utterances. we confirm findings that age and so-cioeconomic status are negatively correlated with feature use. grieser also found that being male was weakly correlated with feature use; interestingly, our results agree when we look at all 17 features or all of coraal, but not when we look at the same feature and data subsets as grieser. this may indicate how small sample size (in terms of both features and datasets) can skew results. see figure 3 for average frequencies of our 17 features in all 152,069 utterances of coraal, broken down by several social factors of the speaker. feature detection at this scale is only possible with automatic methods, and allows researchers to draw more reliable conclusions about language use. ## discussion and future work we propose a corpus-driven and manually-filtered approach to generate contrast sets for morphosyntactic feature detection in low-resource language varieties, which may be useful for novel sociolinguistic analysis in future work. this approach may be extendable to datasets with other nonstandard language varieties (e.g. ice with 14 english varieties @xcite , qadi with 18 arabic varieties @xcite , corpus del español with 21 spanish varieties @xcite , or masakhane's african language collection, currently under development @xcite ), in addition to social media corpora, which are largely unlabeled and could benefit from automatic methods. additionally, while we only examined automatic identification of noisy negatives, future work might explore automatic identification of reliable negatives by using an apt word representation and distance function to obtain unlabeled examples which are least similar to the positives @xcite . other extensions might consider adding manual filtering to an automatic identification approach, such as filtering through and identifying the nearest unlabeled examples that are true negatives, instead of identifying reliable (e.g. distant) negatives. ## ethical considerations and broader impact our objective is to expand the linguistic coverage of nlp tools to include marginalized language varieties, so that they may also benefit from the linguistic analysis made possible by methodological innovation. we hope to aid both sociolinguistic and corpus linguistic researchers studying nonstandard language use. since language varieties, including the ones examined in this study, may correlate with the national origin or ethnicity of the speaker and linguistic feature frequency may correlate with social factors, such as gender or socioeconomic status, there is a risk of automatic feature detection being used to infer personal information about a speaker @xcite @xcite . our study has sought to show that there is a correlation between language use and social factors, but does not support any claims about the accuracy or ethics of using linguistic feature frequency to predict a given social factor. there is not a one-to-one mapping of feature frequency to ethnicity, socioeconomic status, or any other social factor. two speakers with the same set of social factors may exhibit different feature frequencies; life circumstances do not deterministically produce linguistic competence. in addition, linguistic competence does not deterministically produce feature frequency. every speaker has the ability to style-shift and thus use linguistic features to varying degrees for a given context, exhibiting a range of feature frequencies throughout their spoken interactions @xcite . there are many factors that may influence observed feature frequency, including pragmatic context, register, topic, relationship between the speakers, relationship to one's own identity, and so on. this complex relationship between language production and external factors should be considered when using this technology. denmark. association for computational linguistics.