Comparative Error Analysis in Neural and Finite-state Models for Unsupervised Character-level Transduction (2021)

Abstract: Traditionally, character-level transduction problems have been solved with finite-state models designed to encode structural and linguistic knowledge of the underlying process, whereas recent approaches rely on the power and flexibility of sequence-to-sequence models with attention. Focusing on the less explored unsupervised learning scenario, we compare the two model classes side by side and find that they tend to make different types of errors even when achieving comparable performance. We analyze the distributions of different error classes using two unsupervised tasks as testbeds: converting informally romanized text into the native script of its language (for Russian, Arabic, and Kannada) and translating between a pair of closely related languages (Serbian and Bosnian). Finally, we investigate how combining finite-state and sequence-to-sequence models at decoding time affects the output quantitatively and qualitatively.

## introduction and prior work many natural language sequence transduction tasks, such as transliteration or grapheme-to-phoneme conversion, call for a character-level parameterization that reflects the linguistic knowledge of the underlying generative process. character-level transduction approaches have even been shown to perform well for tasks that are not entirely characterlevel in nature, such as translating between related languages @xcite . weighted finite-state transducers (wfsts) have traditionally been used for such character-level tasks @xcite . their structured formalization makes it easier to encode additional constraints, imposed either 1 code will be published at https://github.com/ ryskina/error-analysis-sigmorphon2021 это точно ಮನ #@xmath0 ತು 3to to4no mana belagitu техничка и стручна настава tehničko i stručno obrazovanje figure 1 : parallel examples from our test sets for two character-level transduction tasks: converting informally romanized text to its original script (top; examples in russian and kannada) and translating between closely related languages (bottom; bosnian-serbian). informal romanization is idiosyncratic and relies on both visual (@xmath1, while translation is more standardized but not fully character-level due to grammatical and lexical differences ('nastava' → 'obrazovanje') between the languages. the lines show character alignment between the source and target side where possible. by the underlying linguistic process (e.g. monotonic character alignment) or by the probabilistic generative model (markov assumption; @xcite . their interpretability also facilitates the introduction of useful inductive bias, which is crucial for unsupervised training @xcite . unsupervised neural sequence-to-sequence (seq2seq) architectures have also shown impressive performance on tasks like machine translation @xcite and style transfer @xcite . these models are substantially more powerful than wfsts, and they successfully learn the underlying patterns from monolingual data without any explicit information about the underlying generative process. as the strengths of the two model classes differ, so do their weaknesses: the wfsts and the seq2seq models are prone to different kinds of errors. on a higher level, it is explained by the structure-power trade-off: while the seq2seq models are better at recovering long-range dependencies and their outputs look less noisy, they also tend to insert and delete words arbitrarily because their alignments are unconstrained. we attribute the errors to the following aspects of the trade-off: language modeling capacity: the statistical character-level n-gram language models (lms) utilized by finite-state approaches are much weaker than the rnn language models with unlimited left context. while a word-level lm can improve the performance of a wfst, it would also restrict the model's ability to handle out-of-vocabulary words. controllability of learning: more structured models allow us to ensure that the model does not attempt to learn patterns orthogonal to the underlying process. for example, domain imbalance between the monolingual corpora can cause the seq2seq models to exhibit unwanted style transfer effects like inserting frequent target side words arbitrarily. search procedure: wfsts make it easy to perform exact maximum likelihood decoding via shortest-distance algorithm @xcite . for the neural models trained using conventional methods, decoding strategies that optimize for the output likelihood (e.g. beam search with a large beam size) have been shown to be susceptible to favoring empty outputs @xcite and generating repetitions @xcite . prior work on leveraging the strength of the two approaches proposes complex joint parameterizations, such as neural weighting of wfst arcs or paths @xcite or encoding alignment constraints into the attention layer of seq2seq models @xcite @xcite . we study whether performance can be improved with simpler decodingtime model combinations, reranking and product of experts, which have been used effectively for other model classes @xcite , evaluating on two unsupervised tasks: decipherment of informal roman-ization @xcite and related language translation @xcite . while there has been much error analysis for the wfst and seq2seq approaches separately, it largely focuses on the more common supervised case. we perform detailed side-by-side error analysis to draw high-level comparisons between finitestate and seq2seq models and investigate if the intuitions from prior work would transfer to the unsupervised transduction scenario. ## tasks we compare the errors made by the finite-state and the seq2seq approaches by analyzing their performance on two unsupervised character-level transduction tasks: translating between closely related languages written in different alphabets and converting informally romanized text into its native script. both tasks are illustrated in figure 1 . ## methods we perform our analysis using the finite-state and seq2seq models from prior work and experiment with two joint decoding strategies, reranking and product of experts. implementation details and hyperparameters are described in appendix b. ## results and analysis tables 2 and 3 present our evaluation of the two base models and three decoding-time model combinations on the romanization decipherment and related language translation tasks respectively. for each experiment, we report character error rate, word error rate, and bleu (see appendix c). the results for the base models support what we show later in this section: the seq2seq model is more likely to recover words correctly (higher bleu, lower wer), while the wfst is more faithful on character level and avoids word-level substitution errors (lower cer). example predictions can be found in our further qualitative and quantitative findings are summarized in the following high-level takeaways: #1: model combinations still suffer from search issues. we would expect the combined decoding to discourage all errors common under one model but not the other, improving the performance by leveraging the strengths of both model classes. however, as tables 2 and 3 show, they instead wfst seq2seq figure 6 : highest-density submatrices of the two base models' character confusion matrices, computed in the russian romanization task. white cells represent zero elements. the wfst confusion matrix (left) is noticeably sparser than the seq2seq one (right), indicating more repetitive errors. # symbol stands for unk. comparing the error rate distribution across output words for the translation task also supports this observation. as can be seen from figure 7 , the seq2seq model is likely to either predict the word correctly (cer of 0) or entirely wrong (cer of 1), while the the wfst more often predicts the word partially correctly-examples in ## conclusion we perform comparative error analysis in finitestate and seq2seq models and their combinations for two unsupervised character-level tasks, informal romanization decipherment and related language translation. we find that the two model types tend towards different errors: seq2seq models are more prone to word-level errors caused by distributional shifts while wfsts produce more characterlevel noise despite the hard alignment constraints. despite none of our simple decoding-time combinations substantially outperforming the base models, we believe that combining neural and finitestate models to harness their complementary advantages is a promising research direction. such combinations might involve biasing seq2seq models towards wfst-like behavior via pretraining or directly encoding constraints such as hard alignment or monotonicity into their parameterization @xcite . although recent work has shown that the transformer can learn to perform character-level transduction without such biases in a supervised setting @xcite , exploiting the structured nature of the task could be crucial for making up for the lack of large parallel corpora in low-data and/or unsupervised scenarios. we hope that our analysis provides insight into leveraging the strengths of the two approaches for modeling character-level phenomena in the absence of parallel data. input kongress ne odobril biudjet dlya osuchestvleniye "bor'bi s kommunizmom" v yuzhniy amerike. ground truth kongress ne odobril b d et dl osuwestvleni "bor by s kommunizmom" v noȋ amerike. kongress ne odobril bjudžet dlja osuščestvlenija "bor'by s kommunizmom" v južnoj amerike. wfst kongress ne odobril viu d et dl a osu sq estvleni y e "bor # b i s kommunizmom" v uuz n ani amerike. kongress ne odobril viu d et dl a osu sč estvleni y e "bor # b i s kommunizmom" v uuz n ani amerike. reranked wfst kongress ne odobril vi d et d e l a osu sq estvleni y e "bor # b i s kommunizmom" v uuz n ani amerike. kongress ne odobril vi d et d e l a osu sč estvleni y e "bor # b i s kommunizmom" v uuz n ani amerike. seq2seq kongress ne odobril b y udivitel no s kommunizmom" v n y ȋ amerike. kongress ne odobril b y udivitel'no s kommunizmom" v južn y j amerike. reranked seq2seq kongress ne odobril b d et dl osuwestvleni e "bor by s kommunizmom" v n y ȋ amerike. kongress ne odobril bjudžet dlja osuščestvleni e "bor'by s kommunizmom" v južn y j amerike. product of experts kongress ne odobril b i d et dl a osuwestvleni y e "bor by s kommunizmom" v uuz n nik ameri kongress ne odobril b i d et dlja a osuščestvleni y e "bor'by s kommunizmom" v uuz n nik ameri