Searchable Hidden Intermediates for End-to-End Models of Decomposable Sequence Tasks (2021)

Abstract: End-to-end approaches for sequence tasks are becoming increasingly popular. Yet for complex sequence tasks, like speech translation, systems that cascade several models trained on sub-tasks have shown to be superior, suggesting that the compositionality of cascaded systems simplifies learning and enables sophisticated search capabilities. In this work, we present an end-to-end framework that exploits compositionality to learn searchable hidden representations at intermediate stages of a sequence model using decomposed sub-tasks. These hidden intermediates can be improved using beam search to enhance the overall performance and can also incorporate external models at intermediate stages of the network to re-score or adapt towards out-of-domain data. One instance of the proposed framework is a Multi-Decoder model for speech translation that extracts the searchable hidden intermediates from a speech recognition sub-task. The model demonstrates the aforementioned benefits and outperforms the previous state-of-the-art by around +6 and +3 BLEU on the two test sets of Fisher-CallHome and by around +3 and +4 BLEU on the English-German and English-French test sets of MuST-C.

## introduction the principle of compositionality loosely states that a complex whole is composed of its parts and the rules by which those parts are combined (lake and @xcite . this principle is present in engineering, where task decomposition of a complex system is required to assess and optimize task allocations @xcite , and in natural language, where paragraph coherence and discourse analysis rely on decomposition into sentences @xcite and sentence level semantics relies on decomposition into lexical units @xcite ). 1 all code and models are released as part of the espnet toolkit: https://github.com/espnet/espnet . similarly, many sequence-to-sequence tasks that convert one sequence into another @xcite can be decomposed to simpler sequence subtasks in order to reduce the overall complexity. for example, speech translation systems, which seek to process speech in one language and output text in another language, can be naturally decomposed into the transcription of source language audio through automatic speech recognition (asr) and translation into the target language through machine translation (mt). such cascaded approaches have been widely used to build practical systems for a variety of sequence tasks like hybrid asr @xcite , phrase-based mt @xcite , and cascaded asr-mt systems for speech translation (st) @xcite . end-to-end sequence models like encoderdecoder models @xcite , are attractive in part due to their simplistic design and the reduced need for hand-crafted features. however, studies have shown mixed results compared to cascaded models particularly for complex sequence tasks like speech translation @xcite and spoken language understanding @xcite . although direct target sequence prediction avoids the issue of error propagation from one system to another in cascaded approaches @xcite , there are many attractive properties of cascaded systems, missing in end-to-end approaches, that are useful in complex sequence tasks. in particular, we are interested in (1) the strong search capabilities of the cascaded systems that compose the final task output from individual system predictions @xcite @xcite , (2) the ability to incorporate external models to re-score each individual system @xcite , (3) the ability to easily adapt individual components towards out-of-domain data @xcite , and finally (4) the ability to monitor performance of the individual systems towards the decomposed sub-task @xcite . in this paper, we seek to incorporate these properties of cascaded systems into end-to-end sequence models. we first propose a generic framework to learn searchable hidden intermediates using an auto-regressive encoder-decoder model for any decomposable sequence task ( §3). we then apply this approach to speech translation, where the intermediate stage is the output of asr, by passing continuous hidden representations of discrete transcript sequences from the asr sub-net decoder to the mt sub-net encoder. by doing so, we gain the ability to use beam search with optional external model re-scoring on the hidden intermediates, while maintaining end-to-end differentiability. next, we suggest mitigation strategies for the error propagation issues inherited from decomposition. we show the efficacy of searchable intermediate representations in our proposed model, called the multi-decoder, on speech translation with a 5.4 and 2.8 bleu score improvement over the previous state-of-the-arts for fisher and callhome test sets respectively ( §6). we extend these improvements by an average of 0.5 bleu score through the aforementioned benefit of re-scoring the intermediate search with external models trained on the same dataset. we also show a method for monitoring sub-net performance using oracle intermediates that are void of search errors ( §6.1). finally, we show how these models can adapt to out-of-domain speech translation datasets, how our approach can be generalized to other sequence tasks like speech recognition, and how the benefits of decomposition persist even for larger corpora like must-c ( §6.2). ## proposed framework in this section, we present a general framework to exploit natural decompositions in sequence tasks which seek to predict some output c from an input sequence a. if there is an intermediate sequence b for which @xmath0, then the original @xmath1. in other words, if we can learn p (b | a) then we can learn the overall task of p (c | a) through max b (p (c | a, b)p (b | a)), approximated using viterbi search. we define a first encoderdecoder sub @xmath2, h d b . then we define a subsequent sub @xmath3. therefore, we call h d b hidden intermediates. the following equations shows the two sub-networks of our framework, sub @xmath4, which can be trained end-to-end while also exploiting compositionality in sequence tasks. foot_0 sub @xmath5: note that the final prediction, given by equation 6, does not need to be a sequence and can be a categorical class like in spoken language understanding tasks. next we will show how the hidden intermediates become searchable during inference. ## results our model improves by +2.9/+0.3 (fisher/callhome) over the best cascaded baseline and by +5.6/+1.5 bleu over the best published end-to-end baselines. with speech-attention, our model improves by +3.4/+1.6 bleu over the cascaded baselines and +7.1/+2.8 bleu over encoder-decoder baselines. both the multi-decoder and multi-decoder w/ speech-attention on average are further improved by +0.9/+0.4 bleu through asr re-scoring. 4 ## discussion and relation to prior work compositionality: a number of recent works have constructed composable neural network modules for tasks such as visual question answering @xcite , neural @xcite , and synthetic sequence-to-sequence tasks (lake, 2019). modules that are first trained separately can subsequently be tightly integrated into a single end-to-end trainable model by passing differentiable soft decisions instead of discrete decisions in the intermediate stage @xcite . further, even a single encoder-decoder model can be decomposed into modular components where the encoder and decoder modules have explicit functions @xcite . joint training with sub-tasks: end-to-end sequence models been shown to benefit from introducing joint training with sub-tasks as auxiliary loss functions for a variety of tasks like asr @xcite , st @xcite @xcite , slu @xcite . they have been shown to induce structure @xcite and improve the model performance @xcite , but this joint training may reduce data efficiency if some sub-nets are not included in the final endto-end model @xcite . our framework avoids this sub-net waste at the cost of computational load during inference. speech translation decoders: prior works have used asr/mt decoding to improve the overall st decoding through synchronous decoding @xcite , dual decoding @xcite , and successive decoding @xcite . these works partially or fully decode asr transcripts and use discrete intermediates to assist mt decoding. @xcite and @xcite are closest to our multi-decoder st model, however the benefits of our proposed framework are not entirely explored in these works. two-pass decoding: two-pass decoding involves first predicting with one decoder and then re-evaluating with another decoder @xcite @xcite . the two decoders iterate on the same sequence, so there is no decomposition into sub-tasks in this method. on the other hand, our approach provides the subsequent decoder with a more structured representation than the input by decomposing the complexity of the overall task. like two-pass decoding, our approach provides a sense of the future to the second decoder which allows it to correct mistakes from the previous first decoder. auto-regressive decoding: as auto-regressive decoders inherently learn a language model along with the task at hand, they tend to be domain specific @xcite . this can cause generalizability issues during inference @xcite , impacting the performance of both the task at hand and any downstream tasks. our approach alleviates these problems through intermediate search, external models for intermediate re-scoring, and multi-sequence attention. ## conclusion and future work we present searchable hidden intermediates for endto-end models of decomposable sequence tasks. we show the efficacy of our multi-decoder model on the fisher-callhome @xmath6, achieving state-of-the-art results. we present various benefits in our framework, including sub-net performance monitoring, beam search for better hidden intermediates, external models for better search, and error propagation avoidance. further, we demonstrate the flexibility of our framework towards out-of-domain tasks with the ability to adapt our sequence model at intermediate stages of decomposition. finally, we show generalizability by training multi-decoder models for the speech recognition task at various levels of decomposition. we hope insights derived from our study stimulate research on tighter integrations between the benefits of cascaded and end-to-end sequence models. exploiting searchable intermediates through beam search is just the tip of the iceberg for search algorithms, as numerous approximate search techniques like diverse beam search @xcite and best-first beam search @xcite have been recently proposed to improve diversity and approximation of the most-likely sequence. incorporating differentiable lattice based search @xcite can also allow the subsequent sub-net to digest n-best representations.