Introducing Semantics into Speech Encoders (2023)

Abstract: Recent studies find existing self-supervised speech encoders contain primarily acoustic rather than semantic information. As a result, pipelined supervised automatic speech recognition (ASR) to large language model (LLM) systems achieve state-of-the-art results on semantic spoken language tasks by utilizing rich semantic representations from the LLM. These systems come at the cost of labeled audio transcriptions, which is expensive and time-consuming to obtain. We propose a task-agnostic unsupervised way of incorporating semantic information from LLMs into self-supervised speech encoders without labeled audio transcriptions. By introducing semantics, we improve existing speech encoder spoken language understanding (SLU) performance by over 5% on intent classification (IC), with modest gains in named entity resolution (NER) and slot filling (SF), and spoken question answering (SQA) FF1 score by over 2%. Our approach, which uses no ASR data, achieves similar performance as methods trained on over 100 hours of labeled audio transcripts, demonstrating the feasibility of unsupervised semantic augmentations to existing speech encoders.

## introduction realizing artificial intelligence (ai) that can understand and respond to spoken language is a north star for many speech and natural language processing (nlp) researchers. a particularly effective framework for this is the encoder-decoder architecture, where an encoder represents input audio signals as high-dimensional embeddings and a decoder converts said embeddings to outputs for different downstream tasks. benchmarks for such systems include spoken language understanding, where intent, named entities, or slot values are predicted from input utterances @xcite @xcite , and spoken question answering, where the start and end frames of * equal contribution an input audio passage answering an input audio question are predicted @xcite . a particularly notable setup of the encoderdecoder framework is the universal representation setup @xcite , where a shared selfsupervised speech encoder is pretrained upstream once and frozen for all downstream tasks, then a different lightweight decoder is fine-tuned on each downstream task. this setup is appealing for building speech systems as maintaining a separate large specialized model for every task is not computationally efficient. the universal representation setup has been widely adopted in other areas of research, such as computer vision @xcite and nlp @xcite , and production when there are many downstream tasks or domains @xcite . the current state-of-the-art speech encoders under this setup are w2v2 and hubert @xcite @xcite , which are transformer-based models trained with self-supervised learning (ssl) on raw audio and have achieved impressive performance on various tasks. recently, analytical works found ssl speech encoders capture primarily acoustic, not semantic, information @xcite . thus, researchers proposed end-to-end systems @xcite @xcite @xcite that introduce semantic information through large language models (llms), such as roberta @xcite or bart @xcite , which are pretrained to capture language semantics @xcite . this is typically accomplished by the pipeline approach @xcite , which passes audio input through the ssl speech encoder, then bridge module, then llm. the bridge module converts speech encoder embedding outputs into llm token inputs @xcite @xcite . unsupervised asr models (asr-u) @xcite @xcite have also seen recent success. the state-of-the-art asr-u model uses generative adversarial networks (gans) @xcite to generate text transcription from input audio @xcite . current works combining ssl speech encoders and llms do not satisfy the universal representation framework, as they either (1) rely on asr data on the downstream task, which is expensive to collect and maintain, (2) are not lightweight, requiring training the whole system end-to-end, or (3) are not general, as they do not consider a wide variety of downstream tasks @xcite @xcite . similarly, asr-u was proposed for speech recognition and the focus is not improving ssl speech encoders @xcite . we propose introducing semantics into speech encoders, sse, a task-agnostic unsupervised way of incorporating semantic information from llms into self-supervised speech encoders without labeled audio transcriptions. concretely, sse adopts the pipeline approach to obtain semantic embeddings, with an asr-u bridge connector to extract information from llms. as asr-u is inherently noisy, sse introduces attention residual connection @xcite between the speech encoder and llm. sse also efficiently aligns the llm with the speech encoder through adapter modules @xcite . sse improves w2v2 @xcite and hubert @xcite on 3 slu tasks across 3 datasets, all under the universal representation setup. sse also outperforms state-of-the art no-asr method, dual @xcite , in sqa. while recent works use asr-u to augment existing speech encoders with phoneme-level llms @xcite @xcite , subword-level llms contain much more pertinent and measurable semantic information @xcite . other works in sqa rely on clustering to assign audio frames to frequent subword tokens, but this requires heavy finetuning on the downstream task @xcite . to the best of our knowledge, we are the first to propose a task-agnostic ssl speech encoder which directly interfaces with subword-based llms, unblocking many other applications and future work in this domain. to this end, attention residual con-nections and adapters are essential to successfully extracting semantic information from noisy intermediary transcriptions. we summarize our contributions below: w2v @xcite trains a convolutional neural network model to reconstruct the quantized cluster representations. w2v2 @xcite uses transformers and a discrete codebook quantization module. hubert @xcite improves w2v2 by disentangling the clustering and ssl objectives and using a bert-style encoder @xcite . the speech processing universal performance benchmark (su-perb) @xcite @xcite shows ssl speech encoders are the most effective method for solving multiple downstream tasks with minimal fine-tuning. a recent analytical work finds ssl speech encoders successfully encode acoustic information, but lack semantic information @xcite . in response, contentvec @xcite propose disentangling the speaker and semantic content of audio via an ssl objective. speechlm @xcite propose training a multi-modal speech and text encoder. ## proposed method we propose to introduce semantics into ssl speech encoders by using asr-u to interface with llms. section 3.2 describes how to use asr-u to link a speech encoder with a llm. section 3.3 describes how to combine both acoustic and semantic information and deal with asr transcriptions errors. finally, section 3.4 describes how to align llms with the speech encoder for downstream tasks. ## conclusion we presented a compelling case for introducing semantics into ssl speech encoders and an effective method of doing so. our approach boosts the performance of existing speech encoders on multiple slu and sqa tasks and datasets. we provide reasoning for what tasks may benefit more or less from incorporating semantics. furthermore, our approach is task agnostic and can augment any existing ssl speech encoder. with sse-tune, we show merging acoustic and semantic information and effectively aligning llms to the speech encoder on downstream tasks can further boost performance with minimal parameter overhead. as it can generalize to many downstream tasks, sse provides an important step towards ai that can understand and respond to spoken language.