Structured abbreviation expansion in context (2021)

Abstract: Ad hoc abbreviations are commonly found in informal communication channels that favor shorter messages. We consider the task of reversing these abbreviations in context to recover normalized, expanded versions of abbreviated messages. The problem is related to, but distinct from, spelling correction, as ad hoc abbreviations are intentional and can involve more substantial differences from the original words. Ad hoc abbreviations are also productively generated on-the-fly, so they cannot be resolved solely by dictionary lookup. We generate a large, open-source data set of ad hoc abbreviations. This data is used to study abbreviation strategies and to develop two strong baselines for abbreviation expansion.

## introduction text normalization refers to transformations used to prepare text for downstream processing. originally, this term was reserved for transformations mapping between "written" and "spoken" forms required by technologies like speech recognition and speech synthesis @xcite , but it is now used for many other transformations, including normalizing informal text genres found on mobile messaging and social media platforms (e.g., @xcite . spans of text may require different kinds of normalization depending on their semiotic class @xcite and the requirements of the downstream application. for example, cardinal numbers such as 123 need to be normalized to a spoken form (one hundred twenty three) for speech processing, but this is not necessary for many text processing applications. the class of abbreviations has received particular attention. highfrequency, highly-conventionalized abbreviations, like those used for units of measure (e.g., ml, lbs) or geographic entities (e.g., ak, nz) are often expanded using hand-written grammars (e.g., @xcite , possibly augmented with machine learning systems for contextual disambiguation (e.g., @xcite . in this study we are interested in a different subclass of abbreviations, those which are neither frequent nor conventionalized. we refer to these as ad hoc abbreviations. such abbreviations are particularly common on those communication channels which demand or favor brevity, such as mobile messaging and social media platforms @xcite @xcite . unlike conventionalized abbreviations, ad hoc abbreviations are an open class, generated on-the-fly. unfortunately, there is little annotated data available to study ad hoc abbreviations as they occur in natural text. to remedy this, we provide a new open-source data set-derived from english wikipedia-designed specifically to collect sentences with ad hoc abbreviations. we also provide two strong baseline abbreviation expansion systems, one finite-state, one neural, and find that abbreviations in context can be expanded with human-like accuracy. both baselines use a noisy channel approach, which combines an abbreviation model, applied independently to each word in the sentence, and a language model enforcing fluency and local coherence in the expansion. ## data our goal was to construct a data set consisting of english abbreviated/expanded sentence pairs as shown in figure 1 . in much of the previous work, these were created by finding text that contains likely abbreviations, and then asking annotators to disambiguate. however, @xcite reports that this disambiguation task results in poor inter-annotator agreement. therefore, we instead choose to generate data using a task in which annotators generate abbreviated text rather than disambiguate it. furthermore, since there are many ways to abbreviate any given sentence, one can easily collect multiple annotations per sentence. ## generative story we approach the problem of abbreviation expansion as an instance of the noisy channel problem that has been applied to a wide range of problems including speech recognition @xcite and spelling correction @xcite . we first describe the generative process that produces the abbreviated sentence: given a, which we assume has passed through this noisy channel, our goal is to recover e. we can naturally express this as a conditional model using bayes' theorem. p(e), the probability of the expanded sequence, is naturally expressed by a language model over such sequences. for p(a | e), we make the simplifying assumption that the abbreviation of each token-or indeed, whether it is abbreviated at all-is independent of all other tokens. this allows us to approximate p(e | a) as the product under these assumptions, a model for abbreviation expansion is parameterized by an expansion language model p(e) and a per-token abbreviation generation model p(a | e). below we describe methods for constructing language models and abbreviation models and show how these are used to infer the expanded sentence for a sentence containing abbreviations. strategy example % delete final e @xmath0.0 delete other final letter @xmath1.3 delete 2 final letters @xmath2.6 delete 3 final letters @xmath3.2 delete 4 final letters @xmath4.6 (total) 17.6 delete all vowels @xmath5.2 delete all but word-initial @xmath6.9 delete all but first vowel @xmath7.3 delete all but final vowel @xmath8.8 delete other vowel subsets @xmath9.1 (total) 68.3 delete all vowels & other @xmath10.7 delete duplicated consonants @xmath11.0 delete non-duplicated consonants @xmath12.2 other @xmath13.3 (total) 14.2 ## models we propose two baseline systems for noisychannel decoding, one that relies on weighted finite-state transducers and one that uses a neural network language model. ## experiments we perform experiments with both pipelines described above, the finite state and neural pipelines described in section 4, and compare their performance with human participants attempting to expand the same abbreviated text. the training set described in section 2 is used to train the abbreviation models. language models are trained using the concatenation of the training set with 2.7m additional sentences from wikipedia as described in subsection 2.1. the development set was used to tune the markov order of the finite-state components, and to ablate the subsequence model heuristics. final evaluations are conducted on the test set. the full vocabulary consists of all 75k wordtypes appearing in the language model training set, simulating a general-domain normalization task. ## ethical concerns the proposed technology is intended as a component of other speech and language processing systems. we note that abbreviation expansion systems have some small potential for abuse beyond those of the larger systems they might be integrated into. for instance, this technology could be used to defeat abbreviation as a strategy for circumventing algorithmic state censorship. the data is drawn from english wikipedia text and was produced by a team of professional annotators based in the united states; its use to disambiguate abbreviations generated by other englishspeaking communities would likely introduce bias. ## conclusions we introduce a large, freely-available data set for ad hoc abbreviation expansion, describing the validating the annotation paradigm used to develop it. using this data set, we find that ad hoc abbreviation expansion can be performed at human levels of accuracy using noisy channel models. the finite-state pipeline described above has been integrated as an optional module for google text-tospeech synthesis engines. in future work we will survey abbreviation and abbreviation expansion beyond english. it is expected that abbreviation strategies may differ substantially across languages and scripts. indeed, while they are integral features of some languages, particularly in informal genres, others appear to use few if any abbreviations at all.