Perceptions of Language Technology Failures from S outh A sian E nglish Speakers (2024)

Abstract: English NLP systems have empirically worse performance for dialects other than Standard American English (SAmE). However, how these discrepancies impact use of language technology by speakers of non-SAmE global Englishes is not well understood. We focus on reducing this gap for South Asian Englishes (SAsE), a macro-group of regional varieties with cumulatively more speakers than SAmE, by surveying SAsE speakers about their interactions with language technology and compare their responses to a control survey of SAmE speakers. SAsE speakers are more likely to recall failures with language technology and more likely to reference specific issues with written language technology than their SAmE counterparts. Furthermore, SAsE speakers indicate that they modify both their lexicon and syntax to make technology work better, but that lexical issues are perceived as the most salient challenge. We then assess whether these issues are pervasive in more recently developed Large Language Models (LLMs), introducing two benchmarks for broader SAsE Lexical and Indian English Syntactic understanding and evaluating 11 families of LLMs on them.

## introduction previous studies in natural language processing have identified performance disparities between standard american english (same) and other english dialects @xcite @xcite @xcite @xcite . however, the degree to which these empirical discrepancies affect user experience is not well understood. this leaves open the question of whether reducing these gaps would have a noticeable and desired impact on the speakers of these dialects. prior work, focused on the perspectives of african-american english speakers on automatic speech recognition @xcite , has shown that directly asking subcommunities about their experiences with technology surfaces common problems and perceptions. our work aims to extend this understanding to sase speakers. sase encompasses the varieties of english spoken in south asia @xcite . while subvarieties (the largest being indian english and pakistani english) are defined by differing regions and local languages, they share common features @xcite and are often discussed in the context of englishes in the south asian diaspora @xcite @xcite . our respondents are located in the united states and bilingual in english and at least one other south asian language -corresponding most closely with sociolinguistic studies of indian english speakers in california @xcite . our work conducts user-centric analyses of the use of language technology in sase, which has a large globally-spread speaker community @xcite @xcite and previous empirical explorations in nlp research @xcite @xcite @xcite @xcite . we aim to understand both the degree of impact on sase speakers and how they interact with technology, adapt their speech, and express desired levels of support for their dialects. furthermore, we assess which frustrations are unique to sase speakers by comparing to a control survey of same speakers. our quantitative results indicate that sase speakers are significantly more likely to recall instances of language technology failures overall. furthermore, responses about which specific technologies cause failures indicate that phonological variation, such as accents, may cause more universal issues for users, while written tech fails more uniquely for sase speakers. sase speakers indicate that they modify both their lexicon and syntax to make technology work better (e.g., avoid dialectal syntax, local terminology or "slang", and codeswitching), but that lexical issues are perceived as the most salient challenge. we then develop two new benchmarks to assess the relevance of these challenges to state-of-the-art text-based nlp systems. as a lexical benchmark, we create a multiple choice evaluation of sase terms and their definitions, covering 317 loanwords and 724 stand-alone sase terms scraped from wiktionary @xcite . as a syntactic benchmark, we create a minimal pair evaluation, similar to @xcite ; augmenting the 110 minimal pairs exhibiting grammar specific to indian english introduced by @xcite with synthetically generated negative examples of syntax that has not been attested in indian english or pakistani english @xcite . as a control, we construct corresponding evaluations in same for each benchmark. in summary, we contribute the following: ## survey design to explore the wants of sase speakers within the realm of language technology, we extend prior usercentric surveying research @xcite to understand user experiences with dialect usage and technology. surveying directly addresses a gap in current nlp research, as we construct questions that allow us to analyze how empirical nlp failures with dialect data @xcite @xcite impact user perceptions and interactions. the survey results themselves serve as a foundation for constructing a robust failure framework, essential for guiding research to align with user needs and benchmarking models effectively against real-world user experiences. our survey aims to (1) quantitatively assess the differences in language technology failures between sase and same speakers, and (2) gather qualitative feedback on user experiences and adaptations to better understand whether failure modes correspond to dialect usage. before the study began, respondents were informed that "the purpose of this study is to understand how people use language to interact with technology." the survey starts with closed-ended questions to establish the occurrence and types of technology failures in english. this leads to open-ended questions exploring user perceptions and behavioral adaptations. by integrating both question types, we aim to minimize question ordering bias @xcite and gain a comprehensive understanding of the aspects of a dialect that users employ or avoid when engaging with technology. participants have the option to withdraw from the survey at any stage. we present the full survey in appendix c. ## survey results we analyze survey results by contrasting sase and same speakers' quantitative responses and exploring the wants and frustrations of sase speakers through qualitative responses (shown here with the notation 'px', shorthand for 'participant x'). ## benchmarking llms on challenges while some survey respondents mention extremely recent services such as chatgpt, most reference widely adopted technologies like customer service chat bots, search engines, and translation software. the connection between state-of-the-art research systems and those our respondents interact with on a regular basis is unclear. therefore, we seek to empirically understand how the types of variation respondents report affects llms; a specific technology which is a recent focus for nlp research. while benchmarks exist for hindi-english codeswitching @xcite , syntactic differences @xcite , and phonetic variation @xcite , existing benchmarks do not cover all of the reported challenge categories and most notably omit standalone lexical variation, which is the largest issue mentioned by our respondents. to address this, we develop new intrinsic benchmarks tied to each of our challenge categories. first, we create an intrinsic assessment of lexical understanding from wiktionary (meyer and gurevych, 2012; ylonen, 2022), covering 724 stand-alone terms representing challenge #1 discussed in section 4.2.1 and 317 loanwords from other south asian languages representing challenge #2 discussed in section 4.2.2. to assess challenge #3 discussed in section 4.2.3, we create a minimal pair syntactic language modeling evaluation in the style of @xcite with 110 sentences aligned between same and indian english @xcite augmented with aligned negative examples with syntax not attested in sase using rule-based transformations @xcite . we evaluate 8 series of open-source language models across all 3 assessments of reported challenges. for challenges #1 and #2, we additionally evaluate on models from 3 industrial llm providers but are unable to evaluate them on challenge #3 due to limitations in their apis. we show our results in figures 2 and 3 . prompts used across all language models are provided in appendix d.