Semantic Parsing of Disfluent Speech (2021)

Abstract: Speech disfluencies are prevalent in spontaneous speech. The rising popularity of voice assistants presents a growing need to handle naturally occurring disfluencies. Semantic parsing is a key component for understanding user utterances in voice assistants, yet most semantic parsing research to date focuses on written text. In this paper, we investigate semantic parsing of disfluent speech with the ATIS dataset. We find that a state-of-the-art semantic parser does not seamlessly handle disfluencies. We experiment with adding real and synthetic disfluencies at training time and find that adding synthetic disfluencies not only improves model performance by up to 39% but can also outperform adding real disfluencies in the ATIS dataset.

## introduction spoken language differs from written language. unlike written texts, spontaneous speech frequently contains disfluencies such as filled pauses, repetitions, repairs, and false starts. these affect around 6% of words @xcite @xcite and occur in both human-human and human-computer interactions @xcite . while considerable research has been done on speech disfluencies in syntactic parsing @xcite @xcite , semantic parsing of disfluent speech has received less attention. semantic parsing is the task of mapping written text to a representation of its meaning. in voice assistants, such as amazon alexa, google assistant, or siri, an automated speech recognition (asr) component is often followed by a semantic parsing task that identifies the intent and slots of the transcribed utterance. in voice-based contexts, speech disfluencies are to be expected, and so a voice assistant must accurately support disfluent speech to support more natural user interaction. despite the growing popularity of voice assistants, semantic parsing research to date has focused on written language. most popular semantic parsing datasets only include written forms that were generated from grammars @xcite or crowdsourced @xcite and do not contain speech phenomena such as disfluencies. one approach to parsing disfluent speech is to detect and remove transcribed disfluencies in a post-processing step after asr with a disfluency detector @xcite . alternatively, some asr models attempt to directly produce fluent transcriptions from disfluent inputs @xcite but have lower performance. we investigate semantic parsing of disfluent utterances directly, without asr post-processing, by training a single semantic parsing model that handles both fluent and disfluent utterances. our approach has the advantage of being simple and easier to implement than approaches that require multiple models in a pipeline. we investigate semantic parsing of disfluent speech using the atis semantic parsing dataset @xcite , which was originally collected as spontaneous speech. atis is a small dataset but currently the only semantic parsing dataset that contains speech disfluencies. as such, our findings are limited by dataset size and results may differ given more data, however, we still find interesting preliminary results. first, we compare performance of a state-of-the-art semantic parser on disfluent and fluent examples in atis and identify a performance gap of 79%. to address this gap, we experiment with augmenting the training data with real disfluencies, synthetically-generated disfluencies, and a combination of both. we find that adding synthetic disfluencies adds up to 39% improvement in exact-match accuracy for disfluent utterances. we also find that real disfluencies degrade perfor-fluent please find a flight from detroit michigan to st. petersburg arriving before 10 pm speech please find a @xmath0. petersburg excuse me from detroit michigan to st. petersburg arriving before 10 pm ## atis the airline travel information system (atis) corpus @xcite unlike most semantic parsing datasets, the atis corpus contains recorded speech transcribed in detail, which has been used to research characteristics of disfluencies @xcite . however, most semantic parsing work on atis uses a cleaned version of the utterances where disfluencies are removed @xcite . we use the original atis transcriptions to restore disfluencies to the dataset. we reintroduce deleted words in repairs, partial words longer than 1 character, and filled pauses @xcite . utterances identified as disfluent are manually verified. an example is shown in ## experiments we run four experiments to evaluate how our semantic parsing model handles speech disfluencies. each model is evaluated against three test sets: the original atis test set of 893 fluent utterances, the disfluent subset of 56 disfluent test set utterances identified in §2, and the fluent subset containing the fluent versions of these 56 utterances. the fluent and disfluent subsets are small, however we train multiple models and report results as the average exact-match accuracy over 5 runs with standard deviation. we also calculate slot-match accuracy for one model per setting on the disfluent subset, which measures whether the slots in the representation are correct. finally, we provide a breakdown of performance based on disfluency type for a selection of models in ## conclusion in this work, we evaluate semantic parsing of disfluent utterances in the atis dataset. we highlight the importance of considering spoken language phenomena when building semantic parsers by showing that a state-of-the-art semantic parsing model trained on fluent data is not robust to naturally occurring speech disfluencies. although the atis dataset is small, we present preliminary results showing that adding real or synthetic disfluencies at training time can significantly improve performance by up to 39%. while adding real disfluencies comes with degradation in performance on fluent utterances, adding synthetic disfluencies improves performance on disfluent utterances with almost no degradation on fluent utterances and requires no additional costs for annotations or labeling. these findings show that adding simpler, targeted synthetic disfluencies can be a practical and effective way to improve a semantic parser's performance on disfluent utterances. with the release of more and larger spoken language semantic parsing datasets in the future, we hope to replicate and strengthen our findings in future work.