Lenient Evaluation of J apanese Speech Recognition: Modeling Naturally Occurring Spelling Inconsistency (2023)

Abstract: Word error rate (WER) and character error rate (CER) are standard metrics in Speech Recognition (ASR), but one problem has always been alternative spellings: If one’s system transcribes adviser whereas the ground truth has advisor, this will count as an error even though the two spellings really represent the same word. Japanese is notorious for “lacking orthography”: most words can be spelled in multiple ways, presenting a problem for accurate ASR evaluation. In this paper we propose a new lenient evaluation metric as a more defensible CER measure for Japanese ASR. We create a lattice of plausible respellings of the reference transcription, using a combination of lexical resources, a Japanese text-processing system, and a neural machine translation model for reconstructing kanji from hiragana or katakana. In a manual evaluation, raters rated 95.4% of the proposed spelling variants as plausible. ASR results show that our method, which does not penalize the system for choosing a valid alternate spelling of a word, affords a 2.4%–3.1% absolute reduction in CER depending on the task.

## introduction: "word" error rate for decades, a standard measure of performance in automatic speech recognition (asr) has been word error rate (wer), which gives a measure of how poorly a transcription hypothesized by the asr system matches a reference transcription and which, while often criticized-e.g. @xcite -is still widely used. while the expression wer uses the term word, it is important to note that what is matched is not really words, but rather spelled forms. to take a simple example from english, the reference transcription might have the token advisor, whereas the corresponding token in the hypothesis is adviser. although these are variant spellings of the same word, the system would be assessed as getting the word wrong, since the spellings do not match. if one used instead character error rate (cer), the effect of the spelling discrepancy would be of course be less, but there would still be an error. arguably this should really not count as an error, since the spelling alternates are both valid. orthographic variation @xcite , section 4.6), is common in the world's writing systems, but for many systems the effect is a minor one. in english, for example, orthographic variation is of two main types: regional variation, in particular british versus american spelling (e.g. neighbour vs. neighbor); and more or less free variation within a regional variety such as the advisor/adviser example above, or issues such as whether to write a space in noun compounds (e.g. doghouse vs. dog house). in the former case, one can argue that a spelling discrepancy should count as an error since in contexts where, say, flavour would be an appropriate spelling, neighbor would not be, and vice versa. in the latter case, the variants should probably not be counted as errors, but a naive wer or cer computation would so count them. still, since the amount of such spelling variation is relatively small, one can usually ignore this effect, or use cleanup scripts to handle the few cases that occur. wer is a fiction, but it is a fiction that can largely be ignored. ## japanese spelling inconsistency in japanese, unlike in english, spelling variation is rampant, and the fiction becomes too great to be ignored. japanese spelling is very inconsistent, with many words that have kanji (chinese character) spellings also appearing in text in hiragana (one of the two syllabaries used in japanese), or even, for emphasis or other reasons, in katakana (the other syllabary). thus common words like だめ (hiragana) dame 'not allowed' also frequently appear as ダメ (katakana) for emphasis, but there is also a somewhat infrequent but nonetheless occurring form in kanji, 駄目. there are many reasons for the variation. some of them have to do with style-on which see section 6. katakana is frequently used to mark emphasis so that in japanese, orthographic variation is used to mark what in english would involve either, to adopt the terminology of @xcite , graphetic variation such as italics, or graphemic variation such as capitalization. @xcite give the example of mechamecha 'absurdly', normally written in katakana メチャメチャ, being written in kanji as 目茶目茶 in a sentence with foreign words or emphasized words, both written in katakana. they suggest the reason for the kanji spelling in this case is to provide visual distinctiveness. spelling variants may also be used for artistic reasons @xcite . one would like to have a measure of error rate that takes these sorts of variation into account. one could of course propose developing reference transcriptions that are highly standardized so that, e.g., dame is always written だめ, thus eliminating the problem. indeed corpora such as the corpus of spontaneous japanese foot_0 exist that have highly standardized orthographic transcriptions. but this is not a practical solution in general for a couple of reasons. first, a large amount of potential training data that comes with transcriptions-for example youtube videos-will not have been subjected to rigorous transcription guidelines, and the cost of retranscribing such data would be prohibitive. second, downstream applications cannot be expected to adhere to whatever guidelines have been adopted, and one would like the flexibility to provide transcriptions that can match what downstream applications expect. over and above this, normalizing everything to a standard spelling misses the fact that variation is a normal part of japanese spelling, and one can ignore this only by adopting an artificial standard. we propose therefore to try to model what every native speaker/reader of japanese knows, namely that だめ, ダメ and 駄目 are all legal ways to write dame 'not allowed'. at the same time, one cannot allow the system to be too loose. to return to an example cited above, for うまい umai 'good', one can also have 上手い, as above, but in addition another possible written form is 美味い. the second spelling, 上手い just means 'good' (i.e, good at something), whereas 美味い means 'good tasting'. these two senses are both available for うまい, but they are not interchangeable, and this issue comes up if the ground truth has a kana spelling, whereas the hypothesized form uses kanji. if ground truth has うまい, and the system transcribes 美味い 'delicious', whereas a native speaker could tell from context that what was intended was 上手 い 'good', this should count as an error. in reconstructing spelling variants in kanji from reference forms in kana, the system therefore needs to do sense disambiguation. ## proposed method our method starts with the creation of a lattice of possible respellings, given a reference transcription for an utterance. in order to illustrate the method, we consider the hypothetical reference transcription この 拉麺 は うまい 。 kono rāmen ha umai . for each hiragana word, we also want a katakana equivalent-cf., the example of だめ/ダメ above. this is a fairly straightforward conversion and in the example at hand would produce ラーメン for rāmen, which also happens to be the way this word is normally written. this completes the conversion of kanji tokens into kana, and the next step is to convert in the other direction. for example, the last non-punctuation token in the utterance うまい umai 'delicious', also has a common kanji spelling 旨い. however as noted above, in this as in many other cases, one needs to be careful, since another possible spelling for うまい is 上手い, which would not be appropriate in this instance since it means 'skillful'. for this conversion we train a transformer-based neural machine translation model (nmt)-e.g. @xcite -on japanese web text where we first converted successive kanji spellings into hiragana using the text normalization system previously described. for example, consider the input sentence: 再び、mtサミットが日本で futatabi, mt samitto-ga nihon de 'again, the mt summit is in japan' which contains two words containing kanji 再び futatabi 'again' and 日本 nihon 'japan'. consider the second of these, which has the hiragana transcription にほん. we replace this into the sentence above and tag it with a special @xmath0...</@xmath1, given this context. we also need to train the model to predict cases where kana spellings should not be replaced by kanji: for example the final で does not have a kanji variant, and so in this case we would produce a variant of the input sentence with that token tagged @xmath2...</to_kanji>, and the system would be trained to replace it with itself. the nmt transformer is configured with 6 layers, 8 attention heads and a hidden-layer dimension of 2048, and trained on a web corpus of 7.3 billion tokens. at runtime the model is applied to each hiragana word in the sentence in turn to predict whether that token should be replaced with a word spelled in kanji, and if so which word. the token error rate for kanji restoration on a held out corpus is approximately 3.8%. in the case at hand, the system would correctly predict 旨い as an appropriate spelling of うまい. finally, 旨い has another possible kanji spelling, 美味い, which we have already seen in section 2. to allow for these variants we use lexical resources licensed from the cjk institute ( www.cjki.org ), in particular the japanese orthographic dictionary, which lists spelling equivalence classes for several tens of thousands of japanese words. foot_1 these equivalence classes are 'safe' in the sense that one can substitute any spelling in the class for any other without considering the context of the word. thus 美味い and 旨い both mean 'delicious' and can be safely substituted for each other. to the cjki institute data we have added additional equivalence classes mined from various data sources such as wikipedia, for a total of about 54,200 equivalence classes. figure 1 shows the complete lattice that is reconstructed for the input sentence given above. lattices are implemented in openfst @xcite , with weights represented in the tropical semiring. during evaluation, the levenshtein edit distance @xcite between the reference lattice and the hypothesized transcription is computed using the algorithm reported in @xcite , pp. 93-96. as with standard cer, we define our lenient cer as the lattice edit distance-the sum of the substitution, insertion and deletion errors-divided by the number of characters in the best matching path in the reference lattice. in future work (section 6) we also wish to incorporate style/register language models to rank different transcriptions, and we will thus want to preserve language model weights for the various spelling alternatives. to that end, we first convert the tropical weights into @xmath3, @xmath4, where the first dimension is reserved for the edit distance weights, and the second dimension preserves the language model weights. this will guarantee that the path in the lattice closest to the hypothesized string is selected, with the language model score of that path preserved in the second dimension. after the shortest path has been computed, the result can be converted back to the tropical semiring with just the (seconddimension) language model weights. in the experiments reported in section 5, we compare the results with multiple lattice variants, which are indicated with the terms boldfaced below: ## related work while the contribution of spelling variation to error rate computation for japanese asr has been noted-see @xcite , page 72-as far as we can tell, there has been no prior work that specifically addresses solutions to this problem. however, the problem of spelling variation in japanese is similar to cases in other languages where no standardized spelling exists. for example, @xcite -and see also @xcite -present an approach for asr for arabic dialects. unlike modern standard arabic, which has an official and standardized orthography, arabic regional varieties such as levantine, gulf arabic, or maghrebi are spoken languages that have no generally agreed standard written form. nonetheless, particularly with the advent of social media, people increasingly communicate in arabic dialects in written form. but since there is no prescribed standard there is a substantial amount of variation in how words are spelled. @xcite propose the werd ("word error rate for dialects") metric, which depends on a spelling variants table, which they construct from social media. variants are collected by mining tokens that share the same context, occur a sufficient number of times, and are within a levenshtein edit-distance bound of each other. this kind of approach for finding potentially intersubstitutable terms has been used in other applications: for example, @xcite propose a similar approach for finding potential pairs of words and novel abbreviations of those words. once the spelling variants table is constructed, @xcite use it to match asr candidates against the reference transcription similar to the way in which our lattice-based matching works. related work includes @xcite , who report on an asr system for swiss german, which like dialectal arabic, has no standard orthography, but where spellings are loosely based on pronunciation. another case of spelling variation can be found with transliteration, say when someone whose native language is hindi using the devanagari script, transliterates a hindi word into english. as @xcite discuss, this problem has a practical application, since while keyboards for devanagari and other south asian scripts exist, they tend to be difficult to use, whereas many users are used to typing in english. therefore many users prefer to type in latin script transliteration, and have the system automatically convert to the native script. but this introduces a problem since, while there are standards for transliteration of south asian languages into latin script, few people adhere to them. the result is that one can find quite a large amount of variation in how to spell words in latin script, whereas there is generally one way to correctly write a given word in the native script. @xcite investigated a variety of methods including both neural and pair n-gram methods, and found that they got the best performance with a pair 6-gram model using a katz-smoothed trigram language model for the output. while the above cases are similar to the problem with japanese spelling variation, there is also an important difference. for dialectal arabic, and transliterated south asian languages, there is no standard, and so long as the message can be communicated, users are more or less unconstrained in how they will spell words. in the case of japanese, spelling variation is not completely unconstrained: there are definitely wrong spellings for words, even if there is in any given case no single right spelling. while this does not dictate a particular approach to the problem, it does mean that the variation needs to be constrained by lexical knowledge implemented in some fashion. our use of neural mt models for kanji restoration is related to the similar use of nmt models for transliteration: see, e.g., grundkiewicz and heafield (2018) and @xcite . finally, we note that the problem of lenient evaluation comes up in other domains, for example in evaluation of mt systems. for example, @xcite argue that the rich morphology of arabic has a negative impact on bleu scores in that a naive application of bleu can rank correct translations lower than incorrect ones. they propose a lenient metric they term "al-bleu", which takes morphological variation into account. they argue that this metric provides a more defensible evaluation metric. ## experiments we investigated our proposed evaluation metrics on several japanese asr tasks. using large-scale multiple domain datasets, we calculated error reductions from conventional naive cer, using lattices that incorporate the additional resources discussed in the last section. we also conducted human evaluations to validate the generated spelling alternatives. ## conclusions and future work in this paper we have proposed a lattice-based lenient evaluation method applied to computing character error rate in japanese asr. the method combines lexical resources, a japanese text-processing system, and a neural mt system to reconstruct kanji from kana spellings in context. we evaluated on three different commercial japanese asr domains, and demonstrated a 2.4%-3.1% absolute reduction of cer-translating into an over 25% relative error reduction for the voice search domain. obviously these reductions in cer are not due to any improvement in the asr method itself, but rather reflect a more defensible measure than naive comparison to a single reference transcription. this in turn points to the importance of taking spelling variation into account when evaluating systems on languages where such variation is simply a fact of life. as noted in section 3, we plan in future work to address another issue, namely style and register. while it is true that one often sees spelling variation for words even within the same text, it is also the case that style and register are important factors in deciding which spellings are felicitous in any given context. thus while the word kawaii 'cute', has a kanji spelling 可愛い, that spelling would not usually be found in social media where the hiragana かわいい or katakana カワイイ variants would be more expected, especially if the goal is to communicate a more "friendly" message. we are currently experimenting with using style/register language models that are trained on different genres of text ranging from (informal) social media texts scraped from the web to (formal) official japanese government documents. in the context of the system described in this paper, the language models will be used to rank alternative spellings. thus, a hypothesized spelling for a sentence may be a technically valid variant for a given reference transcription, but may also not be the most consistent in terms of style, and thus should be evaluated somewhat worse than a transcription that is more consistent. this would involve not only considering the edit distance measure-first dimension of the lexicographic semiring described in section 3, but also the language-model cost in the second dimension. we are also investigating using the spellingvariant-augmented reference lattices during training of the asr system rather than just evaluation. currently the asr systems are trained with a single ground truth, which means that the asr systems themselves are not sensitive to spelling variation. to this end, we are developing lattice-based loss functions that can be used during asr training. in addition to the above, we will continue to improve the system in various ways. we plan to include more lexical resources, such as publicly available resources like jmdict (see footnote 2), as well as improve the nmt-based kanji restoration model, with a goal to reducing human-judged unacceptable substitutions below the current 4.6%. an open question is whether large language models such as gpt-4 or bard can be induced to provide judgments on whether two japanese spellings are intersubstitable in a given context, and we also plan to investigate this in future work. finally, while japanese provides a particularly rich example of spelling variation compared to other modern writing systems, as discussed in section 4, there are many languages that are primarily oral, and for which there no accepted written standard. in such languages, one can expect a fair amount of variation in spelling when people attempt to write them, and the methods proposed in this paper could be applicable to such cases.