Mixed Orthographic/Phonemic Language Modeling: Beyond Orthographically Restricted Transformers ( BORT ) (2023)

Abstract: Speech language pathologists rely on information spanning the layers of language, often drawing from multiple layers (e.g. phonology & semantics) at once. Recent innovations in large language models (LLMs) have been shown to build powerful representations for many complex language structures, especially syntax and semantics, unlocking the potential of large datasets through self-supervised learning techniques. However, these datasets are overwhelmingly orthographic, favoring writing systems like the English alphabet, a natural but phonetically imprecise choice. Meanwhile, LLM support for the international phonetic alphabet (IPA) ranges from poor to absent. Further, LLMs encode text at a word- or near-word level, and pre-training tasks have little to gain from phonetic/phonemic representations. In this paper, we introduce BORT, an LLM for mixed orthography/IPA meant to overcome these limitations. To this end, we extend the pre-training of an existing LLM with our own self-supervised pronunciation tasks. We then fine-tune for a clinical task that requires simultaneous phonological and semantic analysis. For an “easy” and “hard” version of these tasks, we show that fine-tuning from our models is more accurate by a relative 24% and 29%, and improved on character error rates by a relative 75% and 31%, respectively, than those starting from the original model.

## introduction recently, large language models (llms) have shown notable success in capturing information across several linguistic layers, developing rich representations of syntactic and semantic structures within their hidden layers @xcite . this is accomplished through the use of self-supervised techniques, in which llms are pre-trained on large corpora to perform generic, contrived tasks. with a well-designed task, the model can make the most of vast quantities of unlabeled text, gleaning the structural patterns of the language(s). for example, in masked language modeling (mlm), a model is trained to restore partially obscured text to its original form. following this relatively generic pre-training task, the resulting model can then be used as a starting point, and its weights fine-tuned (or its architecture augmented with additional output layers) in a task-specific manner. crucially, the task-specific training can be accomplished using orders of magnitude less data than is required for the original pre-training step. this approach is well-suited to many linguistic tasks, particularly those that rely on syntax and semantics. however, tasks that also depend on explicit representation of phonology are under-served by the current paradigm. examples of such tasks include: analysis of code-switched language; processing ambiguous and noisy output from automated speech recognition systems; handling of names and neologisms; and analysis of clinical language samples in the context of communication disorders. there are two underlying reasons for this issue, the first of which involves input representation. as with any computational model, llms require language to be encoded into a numerical form, and a model's choice of encoding technique has a profound impact on its functionality. today's llms typically rely on sub-word representations such as wordpieces @xcite or byte-pair encoding (bpe) @xcite , which strike a balance between vocabulary size and semantic precision while allowing unrestricted input and avoiding the limitations arising from a fixed word vocabulary. such tokenization schemes are generally optimized for representing textual input for word-level processing, based on the distributional properties of a training corpus, and as such in practice the most prevalent segments in a tokenized input sentence represent entire words or large word fragments, with sub-word token fragment boundaries only incidentally co-occurring with morphological boundaries. while it has long been known that neural language models are able to capture implicit information about phonology from orthography @xcite , the extent to which this occurs will depend on the degree to which the model's unit of representation maps to the writing system in question's representation of phonology. furthermore, the selfsupervised pre-training techniques at the foundation of llm tend to work (roughly) at a word-level or word-fragment scope; a task like mlm has little to gain from learning the sound relationships between words, so we have no reason to expect these models to adapt to phonetic tasks as well as they do semantic ones. 1 the second underlying reason for llms' phonological naïveté is data-related. phonology is typically expressed in written form using a writing system such as the international phonetic alphabet (ipa) or arpabet. one advantage of the tokenization schemes used by llms is that they are, at a technical level, able to represent arbitrary character input sequences: sequences of previously-unseen characters (e.g., a sequence of ipa glyphs representing a phonemically-transcribed utterance) simply resolve to a sequence of single-character tokens (rather than lumping together into sub-word units as do more commonly-seen sequences of characters). more recently, new techniques have emerged for training "token-free" llms that operate at the character level (e.g. canine @xcite )) yet retain many of the semantic benefits of word-level models; however, their pre-training remains exclusively orthographic in nature, and thus will only learn phonology indirectly (and in a way that is mediated by the specifics of the writing system in question). furthermore, and most importantly, the data used to pre-train an llm incidentally contains little ipa content if any at all. as such, regardless of the modeling strategy used, the resulting embedding of these tokens is likely to be uninformative from the a phonological perspective. previous attempts to augment llms with phonemic information have focused on speech-centered applications, and have emphasized phonology at the expense of orthography @xcite @xcite . for many applications however, particularly including clinical applications in speech-language pathology, both are crucial, as expressive and receptive language depend on both phonology and semantics. in this paper, we introduce bort, an llm that accepts a mixture of english pronunciations in ipa and english orthography, and demonstrate its use on a task motivated by a real-world clinical problem: analysis of speech errors made by individuals with aphasia following a stroke. in §3, we create the bort models by extending the pre-training of an existing llm, bart @xcite . our self-supervised task focuses on a novel ipa-to-orthography translation task: given a document, we transform some words into ipa, then train the model to restore the orthography. hypothesizing that we could bolster the pre-trained models with two additional transforms, we experiment with configurations that include spelling and noise transforms. in §4, we evaluate the utility of bort by fine-tuning to two clinically-motivated tasks: a) an "easy" task, another mixed ipa/orthography to orthography translation task, but in the context of aphasic speech; and b) a "hard" task, in which the model must predict the intended word for several types of word errors, including phonologically and semantically related errors. we make our pre-trained models available for anyone to download and fine-tune. foot_1 2 background 2.1 speech language pathology speech language pathologists (slps) work to diagnose and treat speech and language disorders. language disorders typically include breakdowns in the linguistic system that supports the abilities to activate the semantic representation of a concept; retrieve its lexical/syntactical representation; and, encode its phonological or orthographic form. speech disorders include deficits that may stem from underdeveloped or faulty perceptual representations of speech sounds; difficulties specifying a motor plan for the articulatory gestures required for producing a word; and/or executing the motor plan. such disorders often co-exist, and may be developmental (affecting primarily pediatric populations, as in the case of specific language impairment), or acquired and seen primarily in adult populations (e.g., dementia, aphasia, dysarthria). in addition, they might affect different modalities including spoken (e.g., anomia) or written output (e.g., agraphia). the use case described in the present work focuses on aphasia, a disorder in which an individual has an impairment to one or both of their expressive or receptive language abilities. in expressive language, this may take the form of difficulties in word retrieval or production; these difficulties may involve the inability to produce a word, the production of an unintended word, or the mispronunciation of a produced word. aphasia typically follows an injury to be brain (such as a stroke or a traumatic brain injury), though it may also be a sign of certain neurodegenerative conditions. to arrive at a diagnosis, clinical professionals typically elicit productions from patients, and then, based on the relationship between the intended target and the realized unexpected or atypical production, they draw inferences regarding the nature of the cognitivelinguistic or motoric deficits of the patient. inherent in this diagnostic process is identifying what was the intended word of a speaker. that requires a clinician to combine multiple sources of information including semantic, phonemic, and/or orthographic information. consider for example the response /b@gaen@/ when a stroke patient is asked to name the picture of an apple. a clinician, naturally, will recognize the phonological similarity of the production to the candidate intended word /b@naen@/ ("banana") and given the semantic similarity of "banana" and "apple," the clinician will arrive at the conclusion that most likely the speaker's intended target was "banana." this step is critical in the diagnostic process across clinical populations and disorders. therefore, the development of a robust computational tool that can combine multiple sources of information to predict a speaker's intended words during a paraphasic speech event is of great clinical significance. there exist several settings in which tools such these may find applications. there are a variety of highly accurate and informative assessment techniques that are regularly used in research settings but rarely used in clinical practice due to the large amount of effort they require for delivery and scoring @xcite @xcite @xcite ; automation has the potential to streamline this process greatly, thereby enabling their clinical use. additionally, automation of this sort would be a key part in many telemedicine and remote assessment scenarios, which is an area of great clinical interest @xcite as there exist major challenges around access to care for many individuals in need of speech and language services @xcite . use cases such as these currently are limited by two categories of technical barrier. the first is the need for robust automated speech recognition algorithms able to accurately process the disordered speech characteristic of individuals with speech and language disabilities, and produce detailed phonemic transcriptions; this is needed given the impracticality of detailed manual transcription in a fast-paced clinical setting. the second category is a lack of specialized algorithms designed to process the resulting data to identify features of clinical interest, for example in speech error classification @xcite . both types of technology are necessary, and neither on their own are sufficient, to leverage nlp in this clinical domain. the present work, by design, only addresses the second of these categories; however, it is important to note that the first is an area of very active research @xcite @xcite @xcite , with major strides being made in recent years. a second setting of use for automation in the analysis of language produced by people with aphasia (pwa) is that of aphasiological research. standard research practice typically results in the creation of recorded sessions with participants (for example, in a discourse elicitation task), which are then transcribed to a very high degree of accuracy by specially-trained research staff, for use in analysis. often, this transcription is done at a mixture of orthographic and phonemic levels, with particular phonemic attention paid to clinically-relevant phenomena such as neologisms (non-word productions), mis-pronunciations, etc. there exist large databases of such transcripts, for example talkbank and its many sub-projects (see https://talkbank.org ; @xcite , and automated analysis of these datasets is extremely valuable from a scientific perspective. notably, in this scenario, one need not assume the existence of an asr system robust to disordered speech in order for automation to be useful, as the data are transcribed as part of their collection and data management process. however, the specifics of this transcription process tend to be very closely linked to the scientific needs of the research team conducting the study, and while the amounts of data generated tend to be far more than humans can conveniently analyze by hand, they tend to be relatively small in comparison to the datasets commonly used in natural language processing. as such, techniques such as transfer learning have become crucial tools in this space. ## conclusion in §3, we pre-trained bort to accept a mixture of orthography and ipa. during training and validation, we used one to three different transforms of words (pronunciation, spelling, noise) and trained the model to restore the words to their original form. we directly evaluated the four models trained on pronunciation using a test set of the wikipedia data by testing the cer of their performance at restoring a word from its pronunciation. in §4, we evaluated all six models (and the baseline) by further fine-tuning them to restore words from productions in aphasic speech. this allowed us to evaluate the applicability of our mixed orthography/ipa llm to a clinical task. our best bort configurations achieved high accuracy and low cer rates for the "easy" fine-tuning task, with the fine-tuned accuracy as high as 90%. this indicates we were able to successfully produce a llm that can accept both orthography and ipa. moreover, observing differences between accuracy and cer revealed that even when our fine-tuned models incorrectly predicted the target word, they still may have found close-by words. for instance, considering the "easy" task, the best performing model picked the wrong target 10% of the time, but it achieved an average cer of just 5.7%. this indicates that there were likely instances where even though the llm predicted the wrong target, it picked a similar word with overlapping letters with the target word. future work will improve on these pre-trained models. in §3.1 we hypothesized bart's generative architecture and denoising task were advantages for our use case. with model selection, though, we find certain tradeoffs, and we would like to test whether these advantages outweigh unique functionalities found in other models. in particular, models designed to operate at a character level (e.g. canine, @xcite could overcome other limitations of bort (e.g. our explicit demarcation of phonemic and orthographic regions), and is perhaps generally well-suited for the task at hand. an additional area of future work will consist of exploring alternative training strategies. in the present work, we were quite strict with regard to preparing the test split, withholding the most frequent english words because they appeared in our aphasiabank evaluations. as our focus turns more toward downstream tasks, we will update our holdout methods to prioritize a stronger pre-trained model, holding out only as much data as needed for validation. further, seeing how our models did not train for an entire pass through the wikipedia data, we will adjust the training schedule (e.g. a ramp-up in the learning rate) and task configuration (e.g. word transform rates) to ensure we get the most out of the pre-training stage. for our aphasia-specific application, we see room to improve the noise transform with more strategic approaches. phoneme errors could be made more realistic with statistically or linguistically informed approaches (e.g. replacing phonemes with similar phonemes). to better prepare a model for semantic errors, whole-word replacements could be made with semantically similar words.