Spoken language understanding (SLU) requires a model to analyze input acoustic signals to understand its linguistic content and make predictions. To boost the models' performance, various pre-training methods have been proposed to utilize large-scale unlabeled text and speech data . However, the inherent disparities between the two modalities necessitate a mutual analysis. In this paper, we propose a novel semi-supervised learning method, AlignNet , to jointly pre-train the speech and language modules. Besides  a self-supervised masked language modeling of the two individual modules , AlignNet aligns representations from paired speech and transcripts in a shared latent semantic space . Thus, during fine-tuning, the speech module alone can produce representations carrying both acoustic information and contextual semantic knowledge  . Experimental results verify the effectiveness of our approach on various SLU tasks. For example, AlignNet improves the previous state-of-the-art accuracy on the Spoken SQuAD dataset by 6.2 \%.
Spoken language understanding (SLU) requires a model to analyze input acoustic signal to understand its linguistic content and make predictions. To boost the models' performance, various pre-training methods have been proposed to learn rich representations from large-scale unannotated speech and text . However, the inherent disparities between the two modalities necessitate a mutual analysis. In this paper, we propose a novel semi-supervised learning framework, SPLAT , to jointly pre-train the speech and language modules. Besides conducting a self-supervised masked language modeling task on the two individual modules using unpaired speech and text, SPLAT aligns representations from the two modules in a shared latent space using a small amount of paired speech and text . Thus, during fine-tuning, the speech module alone can produce representations carrying both acoustic information and contextual semantic knowledge of an input acoustic signal . Experimental results verify the effectiveness of our approach on various SLU tasks. For example, SPLAT improves the previous state-of-the-art performance on the Spoken SQuAD dataset by more than 10 \%.
R
signals
signal
R
utilize
learn rich representations from
R
unlabeled text and speech data
unannotated speech and text
R
method, AlignNet
framework, SPLAT
A
conducting
R
of
task on
R
, AlignNet
using unpaired speech and text, SPLAT
R
paired speech and transcripts
the two modules
R
semantic space
space using a small amount of paired speech and text
A
of an input acoustic signal
R
AlignNet
SPLAT
R
accuracy
performance
R
6.2
more than 10