R - BASS : Relevance-aided Block-wise Adaptation for Speech Summarization (2024)

Abstract: End-to-end speech summarization on long recordings is challenging because of the high computational cost. Block-wise Adaptation for Speech Summarization (BASS) summarizes arbitrarily long sequences by sequentially processing abutting chunks of audio. Despite the benefits of BASS, it has higher compute time due to sequential processing of all blocks, regardless of whether they are relevant to the final summary. In this paper, we propose R-BASS, a new relevance-aware block-wise adaptation method. First, we introduce two approaches to automatically estimate block relevance based on lexical and semantic similarity between the block-level transcript and the summary. Experiments on the How2 dataset show that using ground truth relevance during inference improves efficiency by 63.9 % by dropping irrelevant blocks. Finally, we incorporate relevance scores into training using a novel relevance loss and relevance predictor, and the proposed R-BASS model makes it possible to drop 86.3 % of the blocks while retaining comparable performance, resulting in a 2.2x speedup over BASS.

## introduction generative models @xcite have revolutionized the field of artificial intelligence. speech summarization @xcite @xcite @xcite is the task of taking in long input recordings, identifying parts of the speech with essential information, and generating a short textual summary that concisely conveys the important information. end-to-end speech summarization @xcite @xcite has been shown to improve performance over cascade models that first transcribe long recordings, and then summarize tran-* author is now at google scripts @xcite . however, such models are difficult to train on very large inputs owing to compute restrictions @xcite . to address the challenge of long inputs, block-wise adaptation for speech summarization (bass) @xcite chunks the long input speech into blocks. these blocks are then processed independently, with the semantic context being passed across blocks to facilitate remembering information from past blocks. though bass has better performance and lower computational cost over training directly on long sequences and can process arbitrarily long sequences by updating summaries based on new acoustic information, processing of all relevant and irrelevant blocks is computationally inefficient. in this paper, we introduce r-bass, a relevance-aware block-wise model that first predicts whether the new block of acoustic information is relevant to the summary before integrating new information only from relevant blocks into the semantic context. to decide whether a given block is relevant or not, we analyze the acoustics of the block and the generated summary thus far. if the acoustic information within a new block possesses higher semantic similarity with the previously produced summary, we deem such blocks to be relevant. then, we examine automatic methods to label the relevance of blocks based on lexical and semantic similarity with the block transcript. lexical similarity involves looking at the number of words in the transcript of a given block that are present in the final summary. semantic similarity is assessed by calculating the similarity between bert (kenton and toutanova, 2019) embeddings of the given block's transcript and the summary. finally, we devise a relevance loss that can be used to guide model predictions of relevance to be similar to the ones obtained by automatic annotations. from experiments on how2, r-bass improves efficiency while retaining comparable performance. ## block-wise adaptation for speech summarization (bass) bass is implemented as an attention-based encoder-decoder model and comprises three main blocks : (1) encoder, (2) updater, and (3) decoder. the input sequence x is represented as a sequence of t abutting blocks of length b, i.e., given the i-th block of input x i , the encoder computes a high-level latent representation h i of the input. a semantic embedding s i is used to represent semantic context from all blocks until the i-th block. an updater considers the previous semantic embedding s i-1 and the new acoustic information represented by h i and produces the updated semantic embedding s i for the current block. the decoder finally uses the updated semantic embedding s i to obtain a hypothesis for the summary ŷ i . from a probabilistic perspective, at each block, the bass model estimates the probability of the summary at the i-th block y i given all prior speech features x 1:i p(y i |x 1:i , y 1:i-1 ), which can be decomposed as shown in equation 1. the terms in the decomposition p (s i |x i ) and p (y i |s i ) are modeled by the encoder and decoder respectively. to make training tractable over arbitrarily large inputs, backpropagation is performed block-wise rather than utterance-wise. in the former, only tensors that pertain to the current block of acoustic input remain in the computational graph and gpu memory while the latter is infeasible for long inputs because all the tensors corresponding to all blocks in the recording would have to be stored in the computational graph and gpu memory. to perform block-level optimization, block-level targets are required to compute a loss. the i-th block x i produces a block-level output y i which is compared to the reference summary for the entire recording y to obtain the loss, and backpropagation follows. to combine information from the prior block and the current encoded output, we first use concatenation, which is a simple approach. in this the previous semantic embedding is concatenated along the sequence (time) dimension with the current acoustic embedding to produce the current semantic embedding. this mechanism preserves more information but can be less efficient than using a fixed size of semantic embedding for all blocks. bass not only allows one to use standard selfattentions and avoid approximation errors, but it can also be efficient for streaming speech summarization, where summaries are expected to be updated given new acoustic information. however, it can result in longer training and inference times than methods that use efficient self-attention for offline training and inference. this can be mitigated by using the fact that all blocks are not equally useful to the summary. since processing all blocks to generate the final output summary at the last block is computationally expensive, we propose r-bass, a method to predict and use block-level relevance. ## conclusion in this paper, we address the challenge of efficiency within blockwise models for speech summarization. first, we introduce a novel model r-bass that only processes relevant blocks rather than all blocks to produce a summary more efficiently. to realize r-bass, a relevance estimator is used to predict whether an acoustic block is relevant based on its similarity with the previous block summary. to obtain labels to train the relevance estimator, we propose to obtain binary relevance labels using lexical and semantic similarity between the block transcript and reference summary. experiments demonstrate that there exist multiple irrelevant blocks, which can be ignored to improve efficiency while retaining performance. finally, we introduce a relevance loss to teach bass models to predict and use relevance during inference time. experiments show that training with the proposed semantic similarity loss enables faster processing by dropping around 86% of blocks as irrelevant, resulting in a 2.2x faster inference than bass while obtaining relatively small performance degradations.