Zero-shot Disfluency Detection for I ndian Languages (2022)

Abstract: Disfluencies that appear in the transcriptions from automatic speech recognition systems tend to impair the performance of downstream NLP tasks. Disfluency correction models can help alleviate this problem. However, the unavailability of labeled data in low-resource languages impairs progress. We propose using a pretrained multilingual model, finetuned only on English disfluencies, for zero-shot disfluency detection in Indian languages. We present a detailed pipeline to synthetically generate disfluent text and create evaluation datasets for four Indian languages: Bengali, Hindi, Malayalam, and Marathi. Even in the zero-shot setting, we obtain F1 scores of 75 and higher on five disfluency types across all four languages. We also show the utility of synthetically generated disfluencies by evaluating on real disfluent text in Bengali, Hindi, and Marathi. Finetuning the multilingual model on additional synthetic Hindi disfluent text nearly doubles the number of exact matches and yields a 20-point boost in F1 scores when evaluated on real Hindi disfluent text, compared to training with only English disfluent text.

## introduction disfluencies (e.g., filled pauses, repetitions, discourse markers) are artefacts that are inherent to spontaneous or conversational speech. disfluencies typically obey the following surface structure comprising: a reparandum, an interruption point (+) that marks the end of the reparandum, an interregnum, and finally the repair @xcite . the reparandum consists of one or more words that are not intended by the speaker and will be replaced or ignored. the interregnum consists of an editing term indicating that the reparandum will be edited, or it may be empty, or it can contain fillers, discourse markers, etc. the repair section reflects the fluent part of the utterance. words from the reparandum are repeated or corrected in the repair section, or a new chain of thought is started in case of a false start. consider the following example that illustrates two disfluency types: {well} [i think + {you know} i think] the idea will work the words highlighted in red and green refer to discourse marker and repetition disfluency types, respectively. the part in blue is the fluent version of the original sentence. the example also follows the standard annotation scheme: disfluencies in automatically transcribed text pose a major challenge for downstream nlp tasks such as machine translation, summarization, etc. @xcite . disfluency detection/correction is often used as a preprocessing step for nlp, where the goal is to identify/remove the disfluent words @xcite . while disfluency correction has been extensively studied for english @xcite , it has received far less attention in other languages. this is largely due to the lack of labeled data for other languages. in this work, our main objective is to build disfluency detection models for four indian languages -bengali, hindi, malayalam, and marathi -in the zero-shot setting with no access to labeled disfluent data in these languages. to the best of our knowledge, this is the very first study of disfluency detection across multiple indian languages and also the very first to investigate the ability of large pretrained models to do zero-shot disfluency detection. we specify a rule-based procedure to generate disfluencies starting from fluent sentences. it is worth noting that the synthetically generated disfluent data might not completely reflect real world disfluencies. nevertheless, we find the synthetic data to be useful in improving disfluency detection for low-resource indian languages. also, we can create near-real disfluent data by manually editing the synthetic data, which will take significantly less time than annotating from scratch. our rulebased pipeline is targeted at indian languages, and the same set of rules is applied to sentences in four indian languages -bengali, hindi, malayalam, and marathi. from within these synthetic datasets, native speakers of the respective languages manually identified disfluent sentences that seemed natural. this resulted in manually-verified evaluation datasets for all the four languages. we also constructed evaluation datasets for bengali, hindi, and marathi with real foot_0 disfluent sentences, transcribed and extracted from conversational speech in interviews. using these datasets as evaluation benchmarks and inspired by prior work on crosslingual zero-shot transfer using large pretrained multilingual models @xcite , we investigated the effectiveness of a large pretrained multilingual model muril @xcite on the task of zero-shot disfluency detection. muril is a multilingual transformer-based model that is pretrained on large amounts of text in a number of different indian languages. we finetuned muril using labeled disfluent sentences in english @xcite (and synthetic disfluent text) and evaluated disfluency correction for all four indian languages in the zero-shot setting. 2 four muril-based disfluency detection models were trained, viz. those using (1) only real english disfluent data, (2) both real english and synthetically generated bengali disfluent data, (3) both real english and synthetically generated hindi disfluent data, and (4) both real english and synthetically generated marathi disfluent data. model (2) significantly improves disfluency detection on the real bengali disfluent data compared to model (1). we also observe similar results for the other two languages. this validates our claim that our synthetically generated data is effective in capturing (some subset of) the kinds of disfluencies that are encountered in real conversational data. the main ideas in this work can be summarized as follows: there are three main categories of approaches for disfluency detection. they are based on (1) sequence tagging, (2) parsing, and (3) a noisy channel model @xcite . sequence tagging based approaches use classification techniques to label individual words @xcite . parsing-based approaches detect disfluencies along with identifying the syntactic structure of the sentence (rasooli and tetreault, @xcite . the main idea behind a noisy channel model of disfluency is that we assume there is a fluent source sentence x to which some noise has been added, resulting in a disfluent sentence y . the goal is to find the most likely fluent sentence given y @xcite . prior works @xcite have used pretrained multilingual models for many zero-shot nlp tasks such as named entity recognition (ner), part of speech (pos) tagging, question answering (qa), etc. however, this is the first work to attempt disfluency detection in a zero-shot setting and the very first work to study disfluency detection for multiple indian languages. our work on synthetic disfluency data generation has parallels to the recent work of @xcite where they focus on an artificial disfluency generation algorithm. they focus broadly on repetitions, replacements, and restarts and only focus on english. @xcite is another prior work that has looked into inducing disfluencies in english fluent text. our disfluencies are much more fine-grained in construction (e.g., pronoun corrections, missing syllables, etc.) compared to prior work and apply to indian languages. ## generating synthetic disfluencies we focus on four major disfluency types as listed in @xcite , i.e., fillers, repetitions foot_2 , corrections, and false starts. we specify a total of nine rules across the four disfluency types to introduce disfluencies in fluent sentences. we show examples of bengali disfluent sentences in appendix a for all the disfluency types. apart from what we describe in this section, there are some more fine-grained details governing how and where various disfluencies are introduced in a fluent sentence; these details are specified in our released codebase. ## experimental setup in this work, we use muril @xcite which is a bert model @xcite pretrained on 16 indian languages (including the four we consider) and english. muril is pretrained using two language modeling objectives: masked language modeling and translation language modeling. ## results & analysis this section presents the evaluation results and analyses the quantitative and qualitative performance of our models. ## conclusion we propose the use of a pretrained multilingual model muril for zero-shot disfluency detection in indian languages. we evaluate our model on bengali, english, hindi, malayalam, and marathi disfluency detection tasks. we also show that synthetically generated bengali/hindi/marathi disfluency detection data using simple rules, when combined with real english disfluency data during finetuning, helps improve f1 scores on real bengali/hindi/marathi disfluencies. our overall results support the claim that it is possible to do cross-lingual transfer of disfluency detection with-out any labeled data in the target languages. for future work, we intend to evaluate the model on more diverse disfluencies. feng wang, wei chen, zhen yang, qianqian dong, shuang xu, and bo xu. 2018. semi-supervised disfluency detection. in proceedings of the 27th international conference on computational linguistics, pages 3529-3538, santa fe, new mexico, usa. association for computational linguistics. shaolei wang, zhongyuan wang, wanxiang che, sendong zhao, and ting liu. 2021. combining self-supervised learning and active learning for disfluency detection. acm trans. asian low-resour. lang. inf. process., 21(3). wen wang, gokhan tur, jing zheng, and necip fazil ayan. 2010. automatic disfluency removal for improving spoken language translation. in 2010 ieee international conference on acoustics, speech and signal processing, pages 5214-5217. ieee. shuangzhi wu, dongdong zhang, ming zhou, and tiejun zhao. 2015. efficient disfluency detection with transition-based parsing. in proceedings of the 53rd annual meeting of the association for computational linguistics and the 7th international joint conference on natural language processing (volume 1: long papers), pages 495-503, beijing, china. association for computational linguistics. masashi yoshikawa, hiroyuki shindo, and yuji matsumoto. 2016. joint transition-based dependency parsing and disfluency detection for automatic speech recognition texts. in proceedings of the 2016 conference on empirical methods in natural language processing, pages 1036-1041, austin, texas. association for computational linguistics. vicky zayats, mari ostendorf, and hannaneh hajishirzi. 2016. disfluency detection using a bidirectional lstm. in proc. interspeech 2016, pages 2523-2527. victoria zayats, mari ostendorf, and hannaneh hajishirzi. 2014. multi-domain disfluency and repair detection. in fifteenth annual conference of the international speech communication association. simon zwarts and mark johnson. 2011. the impact of language models and loss functions on repair disfluency detection. in proceedings of the 49th annual meeting of the association for computational linguistics: human language technologies, pages 703-711, portland, oregon, usa. association for computational linguistics.