Scaling Neural ITN for Numbers and Temporal Expressions in T amil: Findings for an Agglutinative Low-resource Language (2023)

Abstract: ITN involves rewriting the verbalised form of text from spoken transcripts to its corresponding written form. The task inherently expects challenges in identifying ITN entries due to spelling variations in words arising out of dialects, transcription errors etc. Additionally, in Tamil, word boundaries between adjacent words in a sentence often get obscured due to Punarchi, i.e. phonetic transformation of these boundaries. Being morphologically rich, the words in Tamil show a high degree of agglutination due to inflection and clitics. The combination of such factors leads to a high degree of surface-form variations, making scalability with pure rule-based approaches difficult. Instead, we experiment with fine-tuning three pre-trained neural LMs, consisting of a seq2seq model (s2s), a non-autoregressive text editor (NAR) and a sequence tagger + rules combination (tagger). While the tagger approach works best in a fully-supervised setting, s2s performs the best (98.05 F-Score) when augmented with additional data, via bootstrapping and data augmentation (DA&B). S2S reports a cumulative percentage improvement of 20.1 %, and statistically significant gains for all our models with DA&B. Compared to a fully supervised setup, bootstrapping alone reports a percentage improvement as high as 14.12 %, even with a small seed set of 324 ITN entries.

## introduction inverse text normalisation (itn) is a textrewriting approach that converts the verbalized form of text in spoken conversational systems to its written form. 1 the written and verbalised forms often diverge in their surface-forms @xcite .such words, henceforth to be referred to as itn entities, typically include numbers, dates, money, etc. at large, such categories are referred to as semiotic classes @xcite . itn is generally perceived as a task for improving text-readability for any language @xcite . however, recent research suggests that identifying itn entities may additionally improve the performance of systems designed for downstream nlu tasks @xcite @xcite . in this work, we identify and address the challenges in developing itn systems for a low-resource and morphologically rich agglutinative language, tamil. we primarily consider four different categories of itn entities, in our task. three of them are numerals belonging to semiotic classes @xcite and the fourth one is linguistic phrases denoting temporal expressions. the three numerical categories are money, date and time, and other numerical values. temporal expressions, though typically do not require a rewrite, are also considered as an additional category in our task. for instance, consider the statement 'i will pay by the end of this month'. while the temporal expression 'end of this month' may not require a rewrite for readability, the information it conveys is similar to that one would expect from itn entities, such as '30th may'. hence, such expressions are also identified, for further downstream processing. tamil is a morphologically-rich agglutinative classical dravidian language, widely spoken in india, sri lanka, etc. @xcite . itn systems typically are developed using rulebased systems @xcite , neural text rewriting methods @xcite , or a combination of neural taggers followed by rulebased methods @xcite . a purely rule-based approach may be challenging for tamil, due to the aforementioned characteristics of the language. hence, we explore three different neural approaches, a sequence-to-sequence model @xcite , a non-auoregressive text-editor model @xcite and a combination of neural tagger @xcite with rules. we leverage pretrained large language models for fine-tuning these models for our task. however, tamil is a low-resource language. hence, we additionally explore data augmentation and bootstrapping to obtain additional data to train our models. specifically, we perform data augmentation by obtaining a substitution matrix of common spelling variations, generating verbalised forms of numerals, and identifying temporal expressions from publicly available corpora. for bootstrapping, our default setting involves a human-in-the-loop (hitl) approach for candidate verification at each iteration. we compare the default setting with two other experimental setups, a) a fully automated setup replacing hitl with a number classifier, and b) a warm start scenario with a seed set many times larger than the default setup. our major observations from this work are: input inta mātattavan . ai muppattorāyiratti ȇt . nūtti muppattañciruvā final output from system {inta māta}# [this month] ttavan . ai {muppattorāyiratti ȇt . nūtti muppattañciruvā} [|31,835]# final sentence with improved readability inta mātattavan . ai |31,835 ## itn models itn is a monotone sequence transduction task where the input and output sequences typically have considerable lexical overlap and generally follow monotonicity in their alignments @xcite . here, we formulate the task in three different setups. a) a sequence tagger @xcite coupled with a rulebased system; b) a seq2seq model @xcite ); c) a non-autoregressive text-editor @xcite ## dataset generation tamil being a low resource language, we employ both bootstrapping @xcite and data augmentation @xcite for obtaining the training data for the task. ## conclusion our work focuses on developing a neural itn system for a morphologically-rich agglutinative language, tamil. tamil is a morphologically productive language with rich agglutination, which along with punarchi leads to high degree of surface-form variation in the utterances generated. we observe that both bootstrapping and data-augmentation for data generation help improve the performance of all the three systems we experimented with. s2s reports the highest gain. it surpasses tagger and reports the best score, when using data from data generation. without data augmentation, tagger reports the best scores in all the other settings. even in a cold start setting, we observe that a fully automated candidate verification can lead to performance improvements in these models. however, our hitl cold start setting or alternatively the fully automated solution in the warm start setting has shown to further improve the performance of these models. overall, we find that both seq2seq and tagger models perform satisfactorily for our use cases and helps in downstream applications.