Truecasing G erman user-generated conversational text (2020)

Abstract: True-casing, the task of restoring proper case to (generally) lower case input, is important in downstream tasks and for screen display. In this paper, we investigate truecasing as an in- trinsic task and present several experiments on noisy user queries to a voice-controlled dia- log system. In particular, we compare a rule- based, an n-gram language model (LM) and a recurrent neural network (RNN) approaches, evaluating the results on a German Q&A cor- pus and reporting accuracy for different case categories. We show that while RNNs reach higher accuracy especially on large datasets, character n-gram models with interpolation are still competitive, in particular on mixed- case words where their fall-back mechanisms come into play.

## introduction truecasing is a natural language processing task that consists of assigning proper case to all words within a text where such information is not available. for many natural language applications, it is an important pre-processing step, shown to be useful in downstream tasks such as named entity recognition @xcite , automatic content extraction @xcite and machine translation @xcite . for languages with complex casing rules (e.g., german), determining part of speech (pos) or named entity becomes almost impossible without casing information: in this example, the token 'fliegen' in (1) is a noun and therefore should begin with a capital letter, while in (2) it is a verb and is hence lowercased. truecasing is particularly hard in the german language, as it follows multiple capitalization rules, some of which we describe below. for instance, all nouns (including both common nouns and proper nouns) and nominalized verbs must be capitalized. furthermore, german adjectives should be capitalized when used as substantive (e.g., bei grün), while not being confused with their true adjective form (which is not capitalized). another common ambiguity is the formal pronoun sie, capitalized in all of its form (as opposed to informal sie which must be lowercased). truecasing becomes particularly important in dialog systems that aim to correctly interpret the natural language of user queries. such systems often rely on automatic speech recognition (asr) output, often quite noisy and non-grammatical as users frequently engage with their devices using highly colloquial speech. this output is used by downstream modules such as spoken language understanding (slu) or entity resolution (er). another use case is displaying such output, e.g., on a device with screen, where casing information is necessary for readability and customer experience. in this paper, we investigate different approaches to truecasing german user queries in the context of a voice-controlled dialog agent. we focus on informational and world knowledge queries, in which the users ask the device questions about, e.g., recent events, famous people, general world knowledge etc. in particular, we pit a rule-based approach against two learnt models, a character n-gram one and a neural one. we report the results across different case categories and discuss the benefits and the drawbacks for each of the approaches. this paper is organized as follows: in section 2, we provide an overview of related work on text truecasing. next, we describe our experiment (section 3), giving the overview of the truecasing approaches and datasets used. in section 4, we present the results of the experiment and provide a detailed error analysis. finally, we discuss the results and outline future directions of research (section 5). ## related work the simplest approach to truecasing is to use unigram tagging, converting all tokens to their most frequent form in the training data. it is typically used as a baseline in capitalisation experiments which improve over this straightforward approach (e.g. @xcite ). most of the earlier approaches to truecasing rely on statistical modeling and work mostly at word level. for instance, one of such approaches was proposed by @xcite who used a trigram language model and sentence level context to predict the most probable case sequence in a sentence. next, @xcite proposed a crf-based probabilistic model, which exploited bilingual information and was able to improve over monolingual methods on machine translation output, while @xcite , @xcite used an approach based on maximum entropy models. benefiting from larger computational power, the more recent approaches advocate modelling at character level. @xcite first proposed to use a character-level rnn for truecasing, demonstrating the superiority of a character-level approach to word-based approaches. they show that the benefits of such approaches over wordbased are that they are able to better generalise on "unseen" words and they also perform better on mixed-case foot_0 . following this argument, we compare their architecture with based large character n-gram models with various smoothing. ## results and error analysis in the following, we present a comparison of the approaches on the the q&a corpus and the leipzig corpus. improvements for each of the approaches relative to the tag-based lm baseline are presented in ## conclusion and future work in this paper, we presented a study on truecasing, a common task in natural language processing, either as a pre-processing step in a larger pipeline (f.e. in mt), or as a post-processing one (f.e. when displaying the output of an spoken language system on screen). comparing rule-based, n-gram and neural models, we showed that, while state of the art methods such as dnns unsurprisingly reach the highest accuracy, standard n-gram models are still competitive, in particular on mixed-case words, where their fall-back mechanisms come into play. they also fair well in noisier corpora (such as the q&a corpora). truecasing, as a part of text normalization, is peculiar in that its bulk can be solved simply by a few hand-written rules, with however a long tail of very difficult cases such as acronyms, unseen words. finding a proper balance between the flexibility of neural approaches, and the controlled, more interpretable behaviour of fst-based systems, remains an open and challenging problem @xcite , sproat and jaitly (2016), @xcite ).