Speech Recognition and Meaning Interpretation: Towards Disambiguation of Structurally Ambiguous Spoken Utterances in I ndonesian (2023)

Abstract: Despite being the world’s fourth-most populous country, the development of spoken language technologies in Indonesia still needs improvement. Most automatic speech recognition (ASR) systems that have been developed are still limited to transcribing the exact word-by-word, which, in many cases, consists of ambiguous sentences. In fact, speakers use prosodic characteristics of speech to convey different interpretations, which, unfortunately, these systems often ignore. In this study, we attempt to resolve structurally ambiguous utterances into unambiguous texts in Indonesian using prosodic information. To the best of our knowledge, this might be the first study to address this problem in the ASR context. Our contributions include (1) collecting the Indonesian speech corpus on structurally ambiguous sentences; (2) conducting a survey on how people disambiguate structurally ambiguous sentences presented in both text and speech forms; and (3) constructing an Indonesian ASR and meaning interpretation system by utilizing both cascade and direct approaches to map speech to text, along with two additional prosodic information signals (pause and pitch). The experimental results reveal that it is possible to disambiguate these utterances. In this study, the proposed cascade system, utilizing Mel-spectrograms concatenated with F0 and energy as input, achieved a disambiguation accuracy of 79.6%, while the proposed direct system with the same input yielded an even more impressive disambiguation accuracy of 82.2%.

## introduction ambiguity is one of the challenges in natural language processing. it has been observed that nearly * *this work was conducted while the first author was doing internship at ha3ci laboratory, jaist, japan under jst sakura science program. 1 our corpus is available at https://github.com/ ha3ci-lab/struct_amb_ind figure 1 : an example of a structurally ambiguous sentence in indonesian and english. efforts have been made to address the issue of ambiguity in natural language. however, the resolution of lexical ambiguity is more commonly studied than structural ambiguity. there is even a specific task dedicated to its resolution, namely word-sense disambiguation. furthermore, most studies on the resolution of structural ambiguity are focused on parsing tasks @xcite @xcite . to the best of our knowledge, there have been no studies directly yielding disambiguated sentences as output in the field of structural ambiguity resolution. speech is the most natural form of human-tohuman communication. it is also considered the richest form of human communication in terms of bandwidth, conveying valuable supplementary information not found in text, particularly the prosodic structure @xcite , which listeners can use to resolve structural ambiguities @xcite . therefore, understanding spoken language is one of the earliest goals of natural language processing @xcite . today, automatic speech recognition (asr) technology has advanced and is widely used. unfortunately, most asr systems do not typically address issues of ambiguity. consequently, in this work, we have made the first attempt to create a system capable of disambiguating structurally ambiguous utterances into unambiguous text by using the provided prosodic information. we propose to build the system by adapting both the cascade and direct approaches to speech-to-text mapping. the proposed cascade system combines two components: asr and a novel text disambiguation (td) model, while the proposed direct system utilizes another novel speech disambiguation (sd) model. the td model is specifically designed to transform the asr output, which is the transcription of structurally ambiguous utterances, into the intended unambiguous text. an illustration of the disambiguation system is shown in figure 2 . ## related work indonesia has one of the largest populations in the world @xcite , yet the availability of resources and the progress of nlp research in indonesian lag behind @xcite . nevertheless, research on indonesian, both speech and text, is gradually progressing despite the challenges mentioned earlier. even though the majority of research and existing data in indonesian comes from the text modality, several works have attempted some speech processing tasks in indonesian. to date, some speech corpora have also been developed @xcite @xcite , as well as the development of asr @xcite @xcite @xcite and tts @xcite @xcite . most indonesian asrs and ttss have achieved good results. however, specifically for asr tasks, most asr that has been developed in indonesian and also in other languages is still limited to transcribing the exact word-by-word, ignoring ambiguities in the transcription. a number of attempts have also been made to address the issue of ambiguity in indonesian, particularly in the text domain. with regard to wordsense disambiguation, several studies have been conducted @xcite @xcite . nevertheless, research on structural ambiguity resolution, especially in speech, is far less common, and there is a notable gap in this area. on the other hand, some studies have attempted to utilize prosodic information in speech to resolve ambiguity across various tasks. for example, @xcite developed a dependency parser that type ambiguous sentences disambiguation sentences 4 the book on the chair i just bought is good. ## conclusions this paper introduces the first indonesian structural ambiguity corpus, comprising 400 structurally ambiguous sentences, each paired with two unambiguous texts, totaling 4800 speech utterances. we also conducted an evaluation of human performance in disambiguating written and spoken structurally ambiguous sentences, highlighting the advantages of the spoken format. additionally, we developed systems for disambiguating structurally ambiguous sentences in indonesian by adapting both cascade and direct approaches to speech-to-text mapping, using two additional prosodic information signals (pauses and pitch). the experiment demonstrates that by utilizing prosodic information, structurally ambiguous sentences can be disambiguated into unambiguous interpretations. among the systems tested, the proposed cascade system achieved average bleu, rouge-1, rouge-2, and rouge-l scores of 63.8 and a disambiguation accuracy of 79.6%. in contrast, the proposed direct system, which uses input mel-spectrograms concatenated with f0 and energy (rms energy), obtained slightly lower average bleu, rouge-1, rouge-2, and rouge-l scores of 62.8, but it achieved the highest disambiguation accuracy of 82.2%. this represents a significant improvement compared to the proposed cascade system.