  Voice Assistants such as Alexa, Siri, and Google Assistant typically use a
two-stage Spoken Language Understanding pipeline; first, an Automatic Speech
Recognition (ASR) component to process customer speech and generate text
transcriptions, followed by a Natural Language Understanding (NLU) component to
map transcriptions to an actionable hypothesis. An end-to-end (E2E) system that
goes directly from speech to a hypothesis is a more attractive option. These
systems were shown to be smaller, faster, and better optimized. However, they
require massive amounts of end-to-end training data and in addition, don't take
advantage of the already available ASR and NLU training data.
  In this work, we propose an E2E system that is designed to jointly train on
multiple speech-to-text tasks, such as ASR (speech-transcription) and SLU
(speech-hypothesis), and text-to-text tasks, such as NLU (text-hypothesis). We
call this the Audio-Text All-Task (AT-AT) Model and we show that it beats the
performance of E2E models trained on individual tasks, especially ones trained
on limited data. We show this result on an internal music dataset and two
public datasets, FluentSpeech and SNIPS Audio, where we achieve
state-of-the-art results. Since our model can process both speech and text
input sequences and learn to predict a target sequence, it also allows us to do
zero-shot E2E SLU by training on only text-hypothesis data (without any speech)
from a new domain. We evaluate this ability of our model on the Facebook TOP
dataset and set a new benchmark for zeroshot E2E performance. We will soon
release the audio data collected for the TOP dataset for future research.
