An end-to-end speech-to-text translation (ST) takes audio in a source language and outputs the text in a target language. Inspired by neuroscience, humans have perception systems and cognitive systems to process different information, we propose TED,T ransducer-E ncoder-D ecoder , a unified framework with triple supervision to decouple the end-to-end speech-to-text translation task. In addition to the target  sentence translation loss, %DIFDELCMD < \method %%%  includes two auxiliary supervising signals to guide the acoustic transducer that extracts acoustic features from the input, and the semantic encoder to extract semantic features relevant to the source transcription text. Our method achieves state-of-the-art performance on both English-French and English-German  speech translation benchmarks .
An end-to-end speech-to-text translation (ST) takes audio in a source language and outputs the text in a target language. Inspired by neuroscience, humans have perception systems and cognitive systems to process different information, we propose LUT, Listen-Understand-Translate , a unified framework with triple supervision to decouple the end-to-end speech-to-text translation task. In addition to the target language sentence translation loss, %DIFDELCMD < \method %%% LUT includes two auxiliary supervising signals to guide the acoustic encoder to extracts acoustic features from the input, and the semantic encoder to extract semantic features relevant to the source transcription text. We do experiments on English-French , English-German and English-Chinese speech translation benchmarks and the results demonstrate the reasonability of LUT. Our code is available at URL
D
TED,
D
T
D
ransducer-
D
E
D
ncoder-
D
D
R
ecoder
LUT, Listen-Understand-Translate
A
language
A
LUT
R
transducer that
encoder to
R
Our method achieves state-of-the-art performance on both
We do experiments on
R
and
,
A
and English-Chinese
R
.
and the results demonstrate the reasonability of LUT. Our code is available at URL