Using Pause Information for More Accurate Entity Recognition (2021)

Abstract: Entity tags in human-machine dialog are integral to natural language understanding (NLU) tasks in conversational assistants. However, current systems struggle to accurately parse spoken queries with the typical use of text input alone, and often fail to understand the user intent. Previous work in linguistics has identified a cross-language tendency for longer speech pauses surrounding nouns as compared to verbs. We demonstrate that the linguistic observation on pauses can be used to improve accuracy in machine-learnt language understanding tasks. Analysis of pauses in French and English utterances from a commercial voice assistant shows the statistically significant difference in pause duration around multi-token entity span boundaries compared to within entity spans. Additionally, in contrast to text-based NLU, we apply pause duration to enrich contextual embeddings to improve shallow parsing of entities. Results show that our proposed novel embeddings improve the relative error rate by up to 8% consistently across three domains for French, without any added annotation or alignment costs to the parser.

## introduction and related work voice assistants are increasingly becoming a part of our daily lives as they empower us to complete diverse and complex tasks. today, voice assistants are able to make phone calls, take notes, set alarms, narrate bed time stories, or even rap hip-hop. given a user's request, the assistant transcribes the input audio to text, and a natural language understanding (nlu) system classifies and routes the transcribed text to components that handle the request (e.g., phone call, notes, alarm). typical conversational nlu systems rely on text, the output of an automatic speech recognition (asr) system @xcite . these systems have proven successful, but relying wholly on text means that some analyses may be inaccurate @xcite , since acoustic signals may be needed to arrive at the right intent. for example, in the utterance "play thank you next", the pause duration between "you" and "next" can help determine if the request is to play the specific song "thank you next" or to play the song "thank you" but next in the queue. this disambiguating function of pause information is well known, but the potential role of pause information in signalling different types of constituents, as described in @xcite , has not, to our knowledge, been investigated in this setting. our goal is two-fold. our first objective is to analyze the findings of @xcite in the context of spoken conversations typical of those processed by voice assistants. specifically, we show that differences in pause length around named entity spans is a statistically reliable processing cue. our second goal is to demonstrate that the intuitive role of pause duration in aiding language understanding can be used in practice. specifically, we present novel pause-grounded contextual embeddings that consistently outperform text-based representations in a language understanding task across multiple domains. a standard approach to a tighter integration between acoustic information and nlu has been to use end-to-end models that can consume acoustic waveforms directly and learn nlu labels @xcite . however, this approach can be difficult to adapt to traditional text-based tasks since audio inputs need to be labeled with task-specific information, introducing additional annotation costs. an alternative approach is suggested by the recent and growing body of research which encodes text along with other modalities, such as audio ## analysis of pause duration in @xcite , time-aligned spoken language corpora in multiple languages were compared in terms of speech rate and pause duration. the variable of interest was the difference between nouns and verbs, with the subtlety that nominalisations of verbs still counted as verbs, and (less frequent) verbalisings of nouns counted as nouns. this means the the distinction was close to the traditional semantic distinction of 'object' and 'action' words. their main findings were that speech rate slows relatively before nouns, and for most languages, pauses are more likely before nouns than before verbs. we are most interested in the observation about pauses before nouns, as a potential signal for an upcoming named entity sequence. named entities may not be limited to single nouns, but may be a span of tokens with at least a head noun. we analyze if pauses are more likely and/or longer before (or after) the boundary of an entity span than within the span itself. we investigate this as follows. we consider a set of utterances from a large internal, anonymous dataset, collected over several months and domains (referred to throughout as usagedata). given the sampling methodology, it can be concluded that there are no speaker-dependent effects in our dataset. the utterances have been taken from an upstream asr system which provides each utterance hypothesis with pause durations following each token, in milliseconds. in order to test the findings of @xcite in the voice assistant setting, we randomly sample usagedata utterances from the sports, movies, and music domains in english and french (referred as engfrpausedata); see we manually annotate engfrpausedata with bio-style tags @xcite ; o for non-entity, b for beginning of an entity, i for inside an entity. furthermore, if a token is tagged with a b or i, it contains an additional entity tag (e.g., artist). we combine the pause information from asr with the named entity tags to derive pauses before, during, and after an entity. we aggregate pause and token duration statistics across all utterances for all domains in a language and fit distributions for the following pairs of settings: o-b, b-i, i-i, b-o, i-o. for example, an o-b pair marks the pause duration between a token tagged o and the token following it tagged b, while an i-i pair denotes the pause between a token tagged i and the token following it note that the deep purple-grey color is indicative of the red and blue histograms overlapping. to analyze whether the study in @xcite can be extended to spoken entity sequences, we ask whether the pause duration before, within, and after an entity is different in a statistically significant way. to do this, we first log normalize the frequency of pauses before, inside, and after an entity to account for differences in their raw counts. in fig. 1 and fig. 2 we plot and compare histograms showing the normalized frequency distributions of pauses before an entity (o-b) and inside an entity (b-i and i-i), and inside an entity and after an entity (b-o and i-o) for french and english respectively. we observe that pauses before uttering and after uttering an entity are longer than pauses within the entity itself. for french, the mean duration for pause within an entity is 18.17ms, for pause before an entity is 55.04ms, and for pause after an entity is 63.86ms. the differences in mean pause duration for both within and before an entity, and for within and after an entity are found statistically significant using a two sample statistical significance t-test; p-values are 4.59e-34 and 3.97e-20, respectively. for english, the mean duration for pause within an entity is 15.21ms, for pause before an entity is 33.99ms, and for pause after an entity is 38.08ms. the differences in pause duration are again found to be statistically significant for both before and within an entity (p-value: 7.31e-33), and within and after an entity (p-value: 6.81e-36). these findings are in line with those from @xcite and suggests that acoustic pauses are indeed a useful and complementary signal to text-based features. likewise, in figure 3 , we show the percentage frequency of pauses greater than or equal to 60ms for different types of entity labels. here, too, we observe that such pauses are more frequent after or before entities (label types o-b, i-o, b-o) as compared to within entities (label types b-i, i-i), for both french (figure 3a ) and english (figure 3b ). note that the pause length difference within and outside an entity span tends to be greater in french than in english. similarly, the difference in frequency of longer pauses at entity boundaries versus within entity spans is greater in french than in english, as can be visualized in figure 3 . we therefore carried out our further experiments on the french data (hereafter, frpausedata), both for reasons of time, and on the grounds that any effects were more likely to show up there. foot_0 3 shallow parsing our primary downstream setting that consumes contextual embeddings (described in section 4) comprises of shallow parsing, as a sequence labelling task. it takes text output from an asr system as the input, and produces an output label for each input token. the labels aim to recognize the intent of the spoken utterance, typically picking note that the deep purple-grey color is indicative of the red and blue histograms overlapping. ## pre-training embeddings with pause information the baseline version of the shallow parser uses a text-only embedding. this embedding is a bertstyle text-based language model @xcite trained without the next sentence prediction auxiliary task of the original bert architecture. to investigate the effect of pauses, we compare 2 "thank you next" is the asr recognition result for the song "thank u, next". the baseline architecture with an identical bertbased model extended with an additional spoken pause prediction task, as shown in fig. 5 . we are given, for each utterance, the pauses following each token in milliseconds, and we learn a model jointly optimizing a text-based language modeling task and a token-level pause duration prediction task. we add additional linear projections over intermediary representations from the encoder to predict pause durations following a token. we refer to the encoder as the "joint utterance encoder", and to the linear projection as the "pause prediction layer". we use negative log-likelihood (nll) for categorical loss and mean squared error (mse) for regression loss optimizations. we backpropagate the sum of the losses from both the text-based and the speech-based pause prediction tasks into the joint utterance encoder. we consider two variants of the prediction task: hierarchical bin classification [hbc]: the task is to perform hierarchical classification where the first linear projection predicts the boolean presence/absence of a pause and the second linear projection maps non-zero pauses (in the case of presence of a pause) into one of three labels: short, medium, and long, indicative of the pause durations. ## experimental design in our experiments, we compare the parser with the baseline version of text-only embeddings against a version using embeddings enhanced with pause information (hbc and nlr). this is the only difference between the two systems: the speech pause [mask] thank [mask] next . . . . . . . . . . . . you play (a) pause-sensitive contextual embedding (b) hbc no pause pause present short medium long (c) nlr [0 1] ⋯ σ key joint utterance encoder pause prediction layer linear layer figure 5: pause-sensitive contextual embedding. (a) joint utterance encoder, in which the prediction layer can be (b) hbc, (c) nlr, or absent (baseline). for pre-training the bert-style language models, we train over a random sample from the french section of usagedata without entity annotations @xcite . frtraindata contains about 50 million utterances with token-level asr feature tuples consisting of recognized text and post-token pause duration (in milliseconds) for each input point. to generate label boundaries for the hbc task, we sort pause durations in frpausedata in ascending order and divide the resulting list of pauses into three parts with an equal number of data points. these serve as the label boundaries for the s, m, and l length labels for hbc. empirically, pause durations less than 60ms are labeled as s, those between 60ms and 310ms are labeled as m, and those greater than 310ms are labeled as l. during training, we discard any pause greater than 10,000ms as noise from upstream asr. for nlr, we normalize the pauses between 0 and 1 by dividing each duration by 10,000ms (the longest pause duration in the pruned dataset). for the classification task, we opt for hbc to address the class imbalance problem in our data @xcite (see figure 1 , which shows a large proportion of tokens with zero pause detected). this coarse-to-fine approach helps with learning the presence/absence of pauses (coarse label) over the skewed dataset while also allowing the auxiliary s/m/l prediction (fine label) to be suf- ## results we compare the performance of our pause-sensitive model with that of our baseline text-based bert architecture in looking at performance across domains, we observe that improvement in eer and ter for the movies domain is consistently higher than that for music and sports. we look at frpausedata to find that entity names in the movies domain typically contain longer sequences of tokens than entities for the music and sports domains, where bynames and abbreviations are often used. on average, entity spans in the movies domain are relatively longer than entity spans in music by 49.64%, and in sports by 39.71% in our dataset. for example, the majority entity span type in the movies domain is 'movie title' which on average consists of 2.54 tokens and makes 34% of the domain. this token length is more than double that of the majority label type 'team name' in the sports domain which makes 60% of the domain and contains 1.25 tokens on average. we note a similar pattern on observing the top-3 most frequent entity types per domain. for example, the mean token length for the top-3 most frequent labels in the movies domain is 2.43 tokens compared with only 1.59 tokens in the music domain. this relates with our earlier experiment with frpausedata-we noted a longer mean pause duration of 120.23ms for movies, compared with 39.06ms for music, and 29.38ms for sports. we hypothesize that longer sequences of entity names cause speakers to make longer pauses around these entity spans. finally, since the nlr model improves performance across all domains under study, we emphasize that it can be used in settings where a pretrained set of embeddings are shared across multiple applications, which is particularly relevant in open-topic voice assistant nlu systems where the domain of a user request is ascertained at run time. ## conclusion in this paper, we analyzed the statistically significant correlation of pause lengths with syntactic properties in real-world usage data for three entityrich application domains. to exploit this property, we presented a novel pause-sensitive contextual embedding model. the model outperforms textbased representations in a language processing task across multiple domains. our approach does not impose additional task-specific annotation requirements and can be extended to more text-based target tasks. to the best of our knowledge, this is the first study on grounding textual representations in pause signals for improved understanding in voice assistants.