Creating Data in I celandic for Text Normalization (2021)

Abstract: There is no natural way to acquire normalized data so we try to create good enough data to attempt more advanced methods for text normalization. We manually annotated the first normalized corpus in Icelandic, 40,000 sentences, and developed Regína, a rule-based system for text normalization. Regína gets 90.83% accuracy compared to the manually annotated corpus on non-standard words. Regína showed a significant improvement in accuracy when compared to an older normalization system for Icelandic. The normalized corpus and Regína will be released as open source.

## introduction text normalization is an integral part of a tts system. unrestricted input texts can contain so-called non-standard words (nsws), which are impossible for a computer to read without being formatted into regular strings of alphabetical letters and punctuation marks. these nsws are divided into semiotic classes and include abbreviations, numbers, and special characters. the degree of importance of text normalization in tts is not obvious even though its utility is known. most words do not need to be normalized, and therefore normalized datasets and their unnormalized counterparts are almost identical. however, without expanding nsws, a tts system skips those words, making the text inaccurate and incomplete. to clarify, let us look at an example of a sentence before and after normalization. haesti tindur esjunnar er 914 m. (esjan's highest peak is 914m.) ↓ haesti tindur esjunnar er níu hundruð og fjórtán metrar. (esjan's highest peak is nine hundred and fourteen meters.) text normalization systems are customarily rule-based but are moving in the direction of neural networks (nns). models made with nns require less human effort @xcite but need a vast amount of correctly annotated data to learn from, and these do not naturally exist for text normalization. people can generally read nsws without requiring an explanation, so there is no motivation to create data with normalized text, such as in translation. to acquire data in icelandic for the training of more sophisticated systems, we start by making a system that can make data good enough for further training. we compare the results of this system with manually annotated data to better assess the quality. ## methodology icelandic is an inflected language, where each word can have various forms of words depending on the context. for example, the number 2 (two) can be expanded as tveir, tvo, tveimur, tveggja, tvaer, or tvö, depending on the next word's case. the ordinal number 2. (second) can then be annar, annan, öðrum, annars, önnur, aðra, annarri, annarrar, annað, öðru, annars, aðrir, annarra, or aðrar. only the first four numbers (one, two, three, and four) have this inflected nature. the most significant ambiguity in the data was whether to write hyphens and dashes as til (to) or silence when it was used to describe sports results. in icelandic, a sentence like leiknum lauk með 2-1 sigri (the game ended with a 2-1 victory), is read as leiknum lauk með tvö (2) eitt (1) sigri and the hyphen is silent. in a tts system, the idea is that the user can either mark the topic herself or run the text through data-driven topic classification. the system built in this research uses regular expressions and grammar rules to determine how a word should be expanded. it has been given the name regína. the first step of regína is to run rules for expansions of abbreviations, measurements, money, weblinks, and roman numerals through the unnormalized text. the rules for measurements take prepositions into account. for example, this could help when the base version of km is kílómetrar. if we say til 2 km, regína uses the preposition til to expand the word to the genitive case, kílómetra. the next step is to run this expanded text through a part-of-speech (pos) tagger. @xcite instead of reading km as an abbreviation (and giving it a tag as such), the tagger now recognizes the word kílómetra and knows from context it is in genitive case. now regína is preserving part-of-speech tags for each word. next, the semiotic class of remaining nsws is determined. rules for numbers are applied to cardinal and ordinal numbers, decimals and fractions. in this step, the words tagged as numbers consider the next word's tag. the numbers that are not followed by an adjective or a noun are assigned a default case. the final step of the system is to run the text through rules for other semiotic classes: time, sports results, digits, letters, dates, and symbols. for comparison, the normalized text was re-aligned with the manually annotated text, with each sentence and word indexed to keep the structure clear. in appendix a, the pipeline for regína is shown. ## results the dataset with general news had 729,763 words, of which 701,088 did not need normalization. the baseline of the system without any work was thus 96.08%. the remaining 28,675 words were split into cardinal, ordinal, and decimal numbers, digits, fractions, letter sequences, abbreviations, weblinks, measurements, clock times, dates, and symbols. the accuracy and size of each class are shown in