On The Diversity of ASR Hypotheses In Spoken Language Understanding
Abstract
In Conversational AI, an Automatic Speech Recognition (ASR) system is used to transcribe the user’s speech, and the output of the ASR is passed as an input to a
Spoken Language Understanding (SLU) system, which outputs semantic objects (such as intent, slot-act pairs, etc.).
Recent work, including the state-of-the-art methods in SLU utilize either Word lattices or N-Best Hypotheses from the ASR.
The intuition given for using N-Best instead of 1-Best is that the hypotheses provide extra information due to errors in the transcriptions of the ASR system, i.e., the performance gain is attributed to the word-error-rate (WER) of the ASR.
We empirically show that the gain in using N-Best hypotheses is related to not WER but to the diversity of hypotheses.
Code and datasets are available at this URL.
Introduction
In Conversational AI [1], an Automatic Speech Recognition (ASR) system first transcribes a user’s speech.
A Spoken Language Understanding (SLU) system parses the speech into semantic objects such as intents and slots.
These intents or slots are used by dialog policy to produce the bot’s response.
As the ASR is not perfect the transcripts are noisy and contain errors, so the SLU system needs to be aware of and robust to the errors caused by ASR.
Prior work enables this "awareness of ASR errors" by providing extra information such as Word Lattices or Word Confusion Networks [2] or N-Best lists [3] in case WCNs are not available.
Prior work [4, 5] argues that using WCNs is better when compared to using 1-Best hypothesis as the oracle path has the lowest word error with respect to the ground truth transcript; similar arguments are made in the case of N-Best hypotheses [3, 6].
In this work, we argue that this assumption that the benefit of using WCNs or N-Best hypotheses is only due to the errors made by the ASR system might not be the entire truth.
We empirically show that the diversity of the hypotheses plays a crucial role in determining the gain in performance when using 1-Best v/s. N-Best.
This work investigates the conditions under which N-Best ASR Hypotheses-based methods are more beneficial and show that the diversity in ASR Hypotheses is an important factor.
The contributions of this work are summarized below:
• Propose novel metrics to measure the diversity of ASR hypotheses.
• Show that WER of the 1-Best ASR hypothesis is not indicative of how useful the non-1-Best hypotheses are.
• Show that the ASR hypotheses’ diversity might indicate the gain in performance.
[Figure] [Figure]
Related Work 2.1 Notation
Let D and | D | denote the dataset, and the number of examples in the dataset, respectively.
N is the number of ASR hypotheses considered, x ji is the j -th-best hypothesis in the i -th instance in D .
y i is the corresponding intent/slot label.
x ∗ i is the "gold" or ground-truth transcript obtained from the ASR dataset.
| x i | denotes the number of tokens in x i .
We denote L ce as the Cross-Entropy loss function.
f ( ., θ ) is a neural network parameterized by θ , a set of parameters.
Throughout the work we discuss the word-error-rate (WER), which is defined as below:
[Equation]
Here, S, D , and I are the numbers of substitution, deletion, and insertion errors.
N is the total number of tokens in x ∗ i .
WER is computed between x 1 i and x ∗ i .
WER is averaged across the dataset D , and we denote the averaged WER as simply WER .
2.2 Word Confusion Networks and Word Lattices
Word lattices [7] and Word Confusion Networks (WCNs) [2, 8] as shown in Figure 1a provides a graphical representation of hypotheses, where each path from the start to the end node representshypothesis.
Lattices are a normalised and a more compact topology of WCNs; while WCNs retain the topology of the original search space.
Liu et al.
[8] show that using WCN as input yields better results for intent performance ( 2% to 4%) compared to the N-Best hypotheses and Word Lattices.
There are similar improvements over 1-Best and N-Best on using word lattices as discussed in [7].
2.3 N-Best ASR Transformer
Ganesan et al.
[3] propose a method that concatenates N-Best ASR hypotheses, illustrated in Fig.
1b, that feeds to a pretrained language model (LM).
This simple method outperformed state-of-the-art methods by leveraging information contained in non-1-Best alternatives while having several benefits over prior baselines, such as leveraging pretrained LMs, and plug-and-play support for third-party
ASR services.
The input is constructed as follows:
[Equation]
where (cid:76) is the string concatenation operation, and [ CLS ] and [ SEP ] are special tokens denoting and separator tokens.
The loss used is L ce (ˆ x, y, θ ) .
Experiments
Following [3], the proposed methods in this section utilize a pretrained XLM-RoBERTa (base).
The set of parameters is denoted by θ .
We train models in cases when a) top-N ASR hypotheses are available, b) only 1-Best ASR hypothesis is available and for completeness, c) when only ground-truth transcripts are available.
We report
Macro F1 scores on ASR hypotheses as inputs.
The code to reproduce results on the public datasets will be provided as supplementary material.
3.1 Datasets
N depends on the ASR used to generate the hypotheses.
For example, suppose the beam-width is 2 .
In that case, the number of hypotheses available is N = 2 , i.e., N is the property of the dataset and is not treated as a hyperparameter for the sake of this study.
2 and SNIPS-TTS
[Table] [Table]
DSTC-2: We build an act-slot-pair multi-label classification dataset using the original act-slot-value
DSTC-2 dataset [9].
The resulting dataset consists of 10.8K and 9.1K training and testing examples with 21 labels (act-slot pairs).
We discard the "value" labels as this requires a change in the model architecture.
We use N = 5 .
SNIPS-TTS: We use the 7-class classification dataset introduced by [10].
However, few train and test set examples do not have N-Best alternatives available.
We filter such examples.
The train and test sets have 11K and 0.6K examples, respectively.
We use N = 9 .
In-house : This multi-class classification dataset consists of utterances collected using a deployed voice assistant (VA) in the banking domain.
The VA handles queries about blocking debit/credit cards, checking bank balances, reporting fraud, etc.
Human annotators label the speech audio with transcriptions, and ASR hypotheses are tagged with one of 42 intents.
We perform an inner-join of these datasets, resulting in a dataset of 3.4K utterances, which we split into training and testing sets of 2.7K and 0.68K examples.
We use N = 5 .
In Figure 2, we illustrate some examples of each of the datasets used.
We observe that DSTC-2 and
SNIPS-TTS have highly similar ASR alternatives.
In the next section, we define metrics that quantify this observation.
Diversity of ASR Hypotheses
In Table 1, we note that the gain in F1 scores due to using N -best hypotheses is much lower when compared with the gains in our In-house dataset.
We believe this is because the hypotheses generated are less diverse in DSTC-2 and SNIPS-TTS, and { 2 , 3 , . . .
, N } -best hypotheses do not provide much extra information to the SLU model.
We compute various statistics based on token-level and sequence-level string-similarity functions to support the above claim.
For a string-similarity function s ( u, v ) computes a scalar value that denotes the similarity between two strings u and v .
[Equation]
S is the averaged statistic across D .
S denotes how much, on average, the non-1-Best hypotheses differ from the 1-Best hypothesis.
It is considered a measure of diversity in ASR hypotheses.
The string-similarity functions s considered are listed below:
• Gestalt-PM : Gestalt Pattern Matching [11], also known as the Ratcliff-Obershelp algorithm for sequence matching.
A higher score means higher similarity and vice-versa.
• Jaccard-Index : A token-level set-similarity measure [12].
A higher score means higher similarity and vice-versa.
Table 2 contains pair-wise string-similarity statistics between 1-Best ASR Hypothesis and others, averaged across all examples in the train and test sets.
SNIPS-TTS has a higher similarity of hypotheses, followed by DSTC-2 and In-house, across both string-similarity functions.
SNIPSTTS has the highest WER of 0 . 443 while having the least diversity in alternatives.
The In-house dataset on the other hand, has a similar WER; however, the hypotheses are much more diverse.
We emphasize that the WER of ASR seems unrelated to ∆ 1 BestNBest and the diversity of hypotheses.
∆ ab is the percentage difference in ASR F1 when using methods a and b .
In hindsight, it would seem obvious that if the hypotheses contained very similar tokens, it would not be useful in helping the SLU "guess" the correct transcription and classify the utterance correctly.
However, to our knowledge, this intuition was never empirically proven to be valid.
Limitations
The number of datasets on which the experiments were carried out may be too few (3) to make a concrete claim on the importance of diversity.
However, we argue that this is enough evidence to show that the intuition used by prior work, which relies on the WER of the 1-best hypothesis, is false.
Conclusion
In this work, we first benchmark N-best ASR Transformer on three different datasets with varying word-error-rates.
We show that for the datasets we tried, the gain in performance when using N-best hypotheses is unrelated to WER.
Instead, we show that the diversity of alternatives is an important factor.
To support this, we introduce novel metrics to quantify the ASR hypotheses’ diversity and show a positive correlation between the gain in performance when using N-best hypotheses and the diversity of alternatives.