Lost in Translation: Benchmarking Commercial Machine Translation Models for Dyslexic-Style Text (2025)

Abstract: Dyslexia can affect writing, leading to unique patterns such as letter and homophone swapping. As a result, text produced by people with dyslexia often differs from the text typically used to train natural language processing (NLP) models, raising concerns about their effectiveness for dyslexic users. This paper examines the fairness of four commercial machine translation (MT) systems towards dyslexic text through a systematic audit using both synthetically generated dyslexic text and real writing from individuals with dyslexia. By programmatically introducing various dyslexic-style errors into the WMT dataset, we present insights on how dyslexic biases manifest in MT systems as the text becomes more dyslexic, especially with real-word errors. Our results shed light on the NLP biases affecting people with dyslexia – a population that often relies on NLP tools as assistive technologies, highlighting the need for more diverse data and user representation in the development of foundational NLP models.

## introduction dyslexia is one of the most common learning disabilities, estimated to affect 10% to 17% of the english-speaking population @xcite . while dyslexia primarily affects one's ability to process and produce textual information @xcite , it can lead to long-term social, emotional, and economic challenges such as lower peer acceptance, poor self-image, lower educational attainment, and reduced employment opportunities @xcite . rapid development and adoption of neural language technologies, such as chatgpt, make them an important part of today's information ecosystem and a promising assistive tool for people with dyslexia @xcite . however, most existing neural language models have been developed and evaluated over typical text (e.g. wikitext @xcite , common-crawl foot_0 ), with little consideration of dyslexia use cases. the fairness and accessibility of neural language technologies for users with dyslexia remain largely underexplored. to better understand nlp systems' ability to serve dyslexic users, we perform a systematic audit of mainstream machine translation (mt) services using real and synthetic dyslexic text. our results show all audited services -including advanced llms -struggle with dyslexic-style input text, making substantially more lexical and semantic mistakes in their translations. by varying the quantity and types of dyslexic-style errors injected into the original text, we also observe a nearly linear relationship between the number of injected dyslexia errors and the degradation in performance for all services, especially for real-word errors such as homophone confusion @xcite . our contribution to nlp fairness and accessibility research is twofold: 1) our findings reveal disparities in the performance of commercial mt systems when translating dyslexia-style text; 2) our data augmentation technique for generating synthetic dyslexia data provides a valuable instrument for further investigating the potential sources and mechanisms behind such disparities in typically "black-box" systems, especially when real dyslexia datasets are scarce. as an early exploration in nlp fairness for dyslexia, our work invites further investment and attention from nlp researchers and commercial companies to develop accessible and fair nlp models in collaboration with people with dyslexia -a community deeply impacted by and highly experienced with language technologies. ## method for our exploratory audit, we selected the machine translation (mt) task because it is well-defined, with well-established metrics and benchmarking datasets, as well as popular consumer-facing applications such as google translate foot_1 . we also limited our initial benchmarking to the translation from english to french-two well-resourced languages in machine learning-to reduce potential confounding factors related to language resource scarcity. to address the data limitation, we leveraged and modified the wmt14 (en2fr) @xcite test dataset by injecting synthetic dyslexic-style errors in the english language source text. we also supplemented the synthetic dyslexia dataset with a small set of real dyslexic text collected from reddit. using the synthetic and real dyslexic data, we benchmark the performance of four mainstream mt services in both lexical and semantic dimensions. ## discussion our results uncover potential disparities in the quality of mt services for people with and without dyslexia. as part of the cloud infrastructure, these services have been ubiquitously adopted as a foundation for many other digital products and services. our work shows how typical dyslexic writing errors could lead to the degradation of sota mt services. even advanced llms, which have been regarded as a solution for dyslexia, struggle with real-word errors from homophones and our confusion set. while llms are better than other services in terms of lexical and syntactic mistakes, they do still produce semantic divergence when translating dyslexic text, and such divergence could be even harder for users with dyslexia to notice, resulting in higher user risk and potentially worse experience in the long term. ## limitations and future work although we were able to experiment with a wide variety of configurations with the quantities and types of dyslexic writing errors, our synthetic datasets are nevertheless limited in their ability to capture the full heterogeneity of dyslexic writing. like any other neurodivergence, dyslexia affects people differently: the way it manifests in writing differs across individuals and situations. disability simulations have been criticized to reinforce stereotypes and further exclude people with disabilities from the research process @xcite . our data augmentation approach should not be applied as a replacement for real dyslexic text. more authentic data from people with dyslexia are required to better represent this community in data in order to develop fair and accessible nlp models for dyslexia. researchers should prioritize the collaboration and involvement of people with dyslexia in future work in this direction. our audit is limited to a few publicly available, commercial mt services, without covering the full landscape of mt models and systems. while we prioritize mt services and products -such as google translate and chatgpt -as they have been ubiquitously deployed and used by millions of people every day, including people with dyslexia, extending the scope of evaluation to more open sourced, academically developed mt models will potentially provide even deeper insights into the inner workings of mt systems in relation to dyslexia. while prompting chatgpt with dyslexiaspecific context could improve its translation quality for dyslexic text, we made a deliberate decision in this study to audit llms without the use of prompt engineering. this decision was motivated by two major reasons: 1) as a foundational model and part of the broader ai infrastructure, many third-part applications and products are built on top of chatgpt without necessarily providing access to prompt customization; 2) prompting requires a certain level of ai literacy and imposes additional cognitive demands -placing the burden on dyslexic users, a population already facing challenges in language-related tasks, rather than on the systems designed to support them. this misplaced responsibility reinforces existing equity gaps instead of addressing them. while we recognize prompt engi-neering can serve as a workaround for the current performance disparities between dyslexic and nondyslexic text, our work calls for a paradigm shift in nlp generalizability research -one that ensures non-typical users are not treated as edge cases, but included by default. lastly, we also look forward to extending our methodology to other communities and application domains, making it easier to audit a wide range of ai models and services using synthetic data about marginalized, sensitive populations. ## ethical considerations positionality statements. while we sought and benefited from valuable input from members of the dyslexia community, this study was designed and executed by authors who do not self identify as having dyslexia. the last author, who self-identifies as neurodivergent and as having a speech-related disability, has worked with the dyslexia community and published research on ai-powered writing support for people with dyslexia @xcite . nevertheless, without lived experiences of dyslexia, we acknowledged the risk of misrepresenting this community and their interests in the context of nlp technologies. ## conclusion we developed a systematic method to inject typical dyslexic writing errors into standard nlp datasets, showing the promise to increase the representation of dyslexic text in nlp systems in an efficient, privacy-preserving way. our synthetically generated data captured three distinct yet common dyslexic writing patterns, allowing us to benchmark the gap in mt service performance in these controlled, simulated dyslexia conditions to detect and diagnose mt's "hidden" biases against dyslexia -a community deeply impacted by nlp technologies. our results show lexical and semantic divergence in the translations over dyslexic text, especially when real-word errors are present. by measuring mt's performance disparities between dyslexic and nondyslexic input text, our work sheds light on the potential user experience challenges for dyslexic users of everyday nlp tools, and calls on the research community to address the equity gap affecting this population.