Character alignment methods for dialect-to-standard normalization (2023)

Abstract: This paper evaluates various character alignment methods on the task of sentence-level standardization of dialect transcriptions. We compare alignment methods from different scientific traditions (dialectometry, speech processing, machine translation) and apply them to Finnish, Norwegian and Swiss German dialect datasets. In the absence of gold alignments, we evaluate the methods on a set of characteristics that are deemed undesirable for the task. We find that trained alignment methods only show marginal benefits to simple Levenshtein distance. On this particular task, eflomal outperforms related methods such as GIZA++ or fast_align by a large margin.

## introduction in recent research, a wide range of character transduction tasks @xcite have been studied, such as modernization of historical spellings, correction of non-standard spellings in user-generated content, lemmatization, or grapheme-to-phoneme conversion (g2p). while most work aims at creating and improving generative models that produce the target representation given its source representation, we focus in this paper on the task of aligning characters when both representations are given. character alignment is a key step in the training pipeline of certain character transduction models such as those based on the statistical machine translation (smt) paradigm. other lines of research have been concerned with finding distances between strings, e.g., to compare different dialectal pronunciations (dialectometry) or to identify cognate pairs in corpora of related languages. while most research in these areas focuses on finding the optimal distance metric for a given task, we rather look at the alignments produced by these distance metrics here. indeed, character alignments are a by-product of distance computations and readily available. in most cases, both character transduction and distance computation are performed at word level, i.e., one word at a time. however, we argue that it is beneficial to carry them out at sentence level (if appropriate corpora are available) to enable contextual disambiguation, to avoid relying on preexistent tokenization and to capture assimilation effects at word boundaries. in this work, we focus on sentence-level standardization of dialect transcriptions. we compare character alignment methods from different scientific traditions and apply them to corpora of transcribed dialectal speech from three languages, namely finnish, norwegian and swiss german. in the absence of gold alignments, we evaluate the alignment methods on a set of characteristics (e.g., the amount of vowel-to-consonant alignments) that are deemed undesirable for dialect-tostandard character alignment. ## data we use existing dialect corpora from finnish, norwegian and swiss german for our experiments: skn -finnish the samples of spoken finnish corpus (suomen kielen näytteitä, hereafter skn) (institute for the languages of finland, 2021) consists of 99 interviews conducted mostly in the 1960s. it includes data from 50 finnish-speaking locations, with two speakers per location (with one exception). the interviews have been transcribed phonetically on two levels of granularity (detailed and simplified) and normalized manually by linguists. we use the detailed transcriptions here. foot_0 ndc -norwegian the norwegian dialect corpus @xcite was compiled between 2006 and 2010 in the context of a larger initiative to collect dialect data of the north germanic languages. typically, four speakers per location were recorded, and each speaker appears both in an interview with a researcher and in an informal conversation with another speaker. the recordings were transcribed phonetically and thereafter semi-automatically normalized to the bokmål standard. 2 archimob -swiss german the archimob corpus @xcite consists of oral history interviews conducted between 1999 and 2001. it contains 43 phonetically transcribed interviews, but only six of them were normalized manually. we only use these six documents for our experiments. some quantitative information about the datasets is given in ## discussion our evaluation of character alignment methods is based on a set of "undesirable characteristics" of the task. in this section, we would like to discuss some issues arising from this experimental setup. in swiss german and finnish, a common pattern is the lack of final n in the dialectal pronunciation. for swiss german müesse / müssen, two solutions are available: (a) a one-to-many alignment containing both e-e and e-n, and (b) leaving n unaligned. although both options can be considered linguistically equivalent, our evaluation favors solution (a). in the opposite direction, the same argument holds for the suprasegmental symbols in the skn corpus. the transcription systems of norwegian and swiss german are based on conventional orthography and render some phonemes by multiple characters (e.g. norwegian sje / ikke). it is unclear how alignment errors inside such multi-character graphemes should be evaluated. alignment can be performed left-to-right or right-to-left. for norwegian aein / en, the former yields ae-e and the latter i-e. although symmetrization minimizes the effects of alignment direction, its impact on the evaluation scores is not entirely clear. despite these yet unsolved questions, we believe that our evaluation provides interesting insights into the performance of character alignment methods for sentence-level dialect-to-standard normalization.