Zero-Shot End-to-End Spoken Language Understanding via Cross-Modal Selective Self-Training (2024)

Abstract: End-to-end (E2E) spoken language understanding (SLU) is constrained by the cost of collecting speech-semantics pairs, especially when label domains change. Hence, we explore zero-shot E2E SLU, which learns E2E SLU without speech-semantics pairs, instead using only speech-text and text-semantics pairs. Previous work achieved zero-shot by pseudolabeling all speech-text transcripts with a natural language understanding (NLU) model learned on text-semantics corpora. However, this method requires the domains of speech-text and text-semantics to match, which often mismatch due to separate collections. Furthermore, using the entire collected speech-text corpus from any domains leads to imbalance and noise issues. To address these, we propose cross-modal selective self-training (CMSST). CMSST tackles imbalance by clustering in a joint space of the three modalities (speech, text, and semantics) and handles label noise with a selection network. We also introduce two benchmarks for zero-shot E2E SLU, covering matched and found speech (mismatched) settings. Experiments show that CMSST improves performance in both two settings, with significantly reduced sample sizes and training time. Our code and data are released in https://github.com/amazon-science/zero-shot-E2E-slu.

## introduction end-to-end (e2e) spoken language understanding (slu) models train on speech-semantics pairs, inferring semantics directly from acoustic features @xcite and leveraging nonlexical information like stress and intonation. in contrast, pipelined slu models @xcite operate on speech-transcribed text, omitting the acoustic information. in all, e2e slu has gained significant research attention. however, training e2e slu models faces a significant * the work was done during an aws ai labs internship. two works have explored zero-shot e2e slu. @xcite trained an nlu model by textsemantics pairs and used it to predict pseudolabels for the text of all speech-text pairs, similar to figure 1 @xcite . they then trained an e2e slu model using the speech audio from the speech-text pairs, paired with the predicted pseudolabels. in another way, @xcite mapped the text of all text-semantics pairs to speech embeddings, creating "pseudospeech"-semantics pairs. however, both works assume matched domains for text-semantics and speech-text pairs, with data collected from the same scenario. in practice, however, these pairs are often separately collected, leading to potential domain mismatches. in such cases, directly using all speech-text and text-semantics pairs for zero-shot e2e slu leads to two types of issues as below. noise. sample noise comes from speech-text pairs whose transcripts (texts) are out-of-domain (ood) for the nlu task. passing all transcripts through nlu inference leads to inaccurate pseudolabels on the ood data, impacting slu learning. this exacerbates label noise, which refers to incorrect nlu model predictions that are then (wrongly) treated as pseudolabels; this issue is inherent to self-training and also impacts performance @xcite . imbalance. since the text-semantics and speechtext pairs are separately collected, even after removing ood speech-text pairs, the remaining text in speech-text pairs may be heavily imbalanced within the nlu domain, e.g., one semantics dominates all others. besides, imbalanced speech, e.g., having only female voices, can bias e2e slu learning. though a model may succeed despite the imbalance, this can waste training resources that could have been used on representative speech-text pairs. for these issues, @xcite and @xcite ignore sample noise and imbalance by selecting speech-text pairs that are matched and balanced; however, in practice, it is hard to gain such well-matched and well-balanced speech-text corpus. furthermore, neither work is selective with pseudodata, which in @xcite led to degradation when more external speech-text was added, due to label noise. instead, with selection as a unifying perspective, we make the following contributions: (i). zero-shot e2e slu benchmarks for both matched and found speech. for the matched domain setting, we define voxpopuli2slue, combining text-semantics pairs of slue's ner-annotated subset @xcite of voxpopuli @xcite with speech-text pairs from voxpopuli, similar to @xcite . then, for the found (mismatched) speech setting, we define minips2slurp, combining the home-assistant text-semantics pairs of slurp @xcite with speech-text pairs from the generaldomain people's speech corpus @xcite . ## benchmarks for zero-shot e2e slu we define a traditional slu model as θ a l,t , that is trained on data d a l,t with pairs of speech audio a and semantic labels l. these samples are in a target domain t. besides, we will use superscript t l to denote text t to semantic labels, and a t to denote speech audio to text. in our zero-shot setting, instead of having a speech-to-semantics dataset d a l,t , we have a text-to-semantics pair set d t l,t in the target domain, and an external speech-to-text pair set d a t . unlike @xcite or @xcite , the provided speech-to-text data d a t may be independently collected and have sample pairs from an external domain. we divide d a t into two disjoint subsets, with samples either in the target domain t or being external domain ϵ: a domain denotes data collection scenarios. the ϵ can be matched or mismatched to the t domain. given d t l,t and d a t , we aim to learn an e2e slu model θa l that performs close to θ a l,t . this is zero-shot, as training our θa l uses no speech-semantics pairs d a l,t . we created the below two datasets to study this problem: matched speech: voxpopuli2slue. we use slue-voxpopuli @xcite as the target domain text-to-semantics data d t l,t . the external speech-to-text data d a t is from voxpopuli @xcite . we denote this dataset as voxpopuli2slue. its domain is matched, because slue-voxpopuli and voxpopuli are both from european parliamentary proceeding scenario. found speech: minips2slurp. we use slurp @xcite as the target domain text-to-semantics data d t l,t . mini-ps @xcite provides the externaldomain speech-to-text pairs d a t,ϵ . slurp is in the voice command domain for controlling family robots. but mini-ps is a subset of people's speech corpus, with 32,255 speech-to-text pairs in diverse domains, such as tv, news, and sermons. we then mix d a t,ϵ from mini-ps and d a t,t from slurp for d a t . minips2slurp is mismatched with slurp's target domain, as it is dominated by found data from other domains. for fair comparison, in the above two datasets, we provide d a l,t that has the same size and speech as d a t,t . the d a l,t is only used to learn θ a l,t and not applied to learn our θa l . we use the full slurp test set as the test set in minips2slurp, and half of the dev set in slue-voxpopuli as the test set in voxpopuli2slue. the dataset statistics, data annotations, and data usages are in ## experiments we now compare our proposed framework to baselines on the two datasets introduced in sec. 3. ## conclusion to advance zero-shot e2e slu research, we create two datasets: voxpopuli2slue and minips2slurp, catering to matched and found speech, respectively. in addition, our framework cmsst tackles the noise and imbalance issues that have been disregarded in previous works. cmsst incorporates mcss, a method that selects speechtext pairs to simultaneously enhance the diversity of acoustic, text, and semantics, thus addressing the imbalance. furthermore, cmsn is proposed to mitigate the impact of low-confidence pseudolabels, thereby alleviating the effects of label noise. extensive experiments on both datasets demonstrated the effectiveness and efficacy of our framework.