Cross-modal Contrastive Learning for Speech Translation
Abstract
How can we learn uniﬁed representations for spoken utterances and their written text?
Learning similar representations for semantically similar speech and text is important for speech translation.
To this end, we propose ConST , a cross-modal contrastive learning method for end-to-end speech-to-text translation.
We evaluate ConST and a variety of previous baselines on a popular benchmark
MuST-C. Experiments show that the proposed
ConST consistently outperforms the previous methods on, and achieves an average BLEU of 29.4.
The analysis further veriﬁes that ConST indeed closes the representation gap of different modalities — its learned representation improves the accuracy of cross-modal speechtext retrieval from 4% to 88%.
Code and models are available at https://github. com/ReneeYe/ConST .
Introduction
End-to-end speech-to-text translation (E2E ST) becomes important in many internet products and real applications.
An E2E ST system accepts audio signals as the input and generates the target translation using a single model.
Compared with the conventional cascade ST models, E2E ST models have achieved almost comparable (Bentivogli et al., 2021) or even superior (Ansari et al., 2020;
Potapczyk and Przybysz, 2020; Xu et al., 2021) performance.
The performance of an E2E ST model is still restricted for languages with relatively small parallel data, compared to text machine translation (MT).
Existing approaches for ST focus on using additional data from MT and automatic speech recognition (ASR).
This can be realized through pretraining approaches (Zheng et al., 2021; Dong et al., 2021b,a) or multi-task training frameworks (Tang et al., 2021b;
Ye et al., 2021; Han et al., 2021).
“Speech”
Transcript “It is a nice day!”
“What are you going to do today?”
It is a nice day!
What are you going to do today?
“It's a new day full of energy.”
It's a new day full of energy.
“if you take chances” if you take chances “It is a nice day!”
“What are you going to do today?”
It is a nice day!
What are you going to do today?
“It's a new day full of energy.”
It's a new day full of energy.
“if you take chances” if you take chances “Speech”
[Figure]
Different from the data perspective, this paper investigates the bottleneck of E2E ST from the neural representation perspective.
We believe that when the representation of audio input is similar to its corresponding textual representation, it is easier for information to transfer from MT to ST, thus improving speech translation performance.
We analyze Transformer models for speech translation and observe a noticeable modality gap between encoder representations of speech and text (Sec. 6 has more details) from existing ST models.
An ideal representation should satisfy: if the content of the speech and transcription are similar, their encoded representations should likewise be close to each other.
Nevertheless, how to learn uniﬁed and aligned speech-text representations?
Inspired by the recent progress of contrastive learning approaches in cross-lingual (Lample and
Conneau, 2019; Pan et al., 2021) and cross-modal vision-and-language domains (Li et al., 2021;
Zhou et al., 2020; Dong et al., 2019), we designed a simple con trastive learning method for ST ( ConST ) to learn the representations that meet the aforementioned conditions explicitly.
On the one hand, our model inherits the advantages of the previous multi-task learning methods.
On the other hand, it reduces the gap between the representations of speech and its corresponding transcription.
Our contributions are as follows.
• We develop ConST for speech translation, a cross-modal contrastive learning method, on top of the multi-task training framework.
• Our experiments on the MuST-C benchmark to show ConST achieves an average BLEU score of 29.4, outperforming the best previous baseline.
• We show that ConST indeed learns similar representations for two modalities and better retrieves text with speech input.
Related Work
End-to-end ST
To alleviate the error propagation in the cascaded ST systems and to make the deployment simpler, Bérard et al.
Weiss et al.
(2017) proposed to use an end-to-end architecture to directly translate speech into text in another language, without the intermediate transcription.
Kano et al.
; Berard et al.
; Inaguma et al.
; Wang et al.
(2020a)
; Zhao et al.
(2021a) implemented several off-the-shelf encoderdecoder E2E-ST models, such as BiLSTM (Greff et al., 2016) and Speech-Transformer (Dong et al., 2018).
However, training an end-to-end speech translation model is difﬁcult because we need to design a cross-modal cross-language model, meanwhile, the speech-transcription-translation supervised data for speech translation is signiﬁcantly less than that of MT and ASR.
Methods, like data augmentation (Park et al., 2019; Pino et al., 2020;
Chen et al., 2021), pre-training (Weiss et al., 2017;
Berard et al., 2018; Bansal et al., 2019; Wang et al., 2020b; Alinejad and Sarkar, 2020; Dong et al., 2021a; Zheng et al., 2021), self-training (Pino et al., 2020; Wang et al., 2021a), utilizing selfsupervised pre-trained audio representation (Wu et al., 2020; Han et al., 2021;
Ye et al., 2021; Wang et al., 2021a), are proved to be effective.
Meanwhile, some work has shown that the encoderdecoder model with a single encoder cannot encode speech information well.
For example, Dong et al.
(2021b)
ﬁrst proposed a second encoder to further extract semantic information of the speech sequence.
Xu et al.
(2021) proposed a stacked acoustic-and-textual encoder and introduced largescale out-of-domain data.
Also, multi-task frameworks (Le et al., 2020; Tang et al., 2021b;
Ye et al., 2021) are widely applied to further enhance the robustness for ST.
As a cross-modal task, some work has noted the problem of the modality gap.
(Han et al., 2021) designed a ﬁx-size semantic memory module to bridge such a gap, from the neuroscience perspective.
However, we ﬁnd that this approach actually sacriﬁces the effect of MT.
So in this paper, we propose a simple yet effective contrastive learning method to bridge the gap and to improve
ST performance.
Cross-modal grounding learning This paper attempts to address the problem in speech translation from the perspective of cross-speech-text representation learning.
We are also inspired by crossmodal representation learning in the acoustic word embedding (AWE) (Palaskar et al., 2019; Kamper et al., 2020;
Hu et al., 2020) and the visuallanguage pre-training (VLP) (Wu et al., 2019; Lu et al., 2019; Chen et al., 2020b; Li et al., 2021) tasks.
These works usually focus on enhancing textual representations with acoustic or visual information, in other words, grounding learning.
In this work, we consider the its dual form, i.e., grounding speech representations using text.
Contrastive learning
Our method is motivated by the recent success in contrastive representation learning.
The contrastive learning method was ﬁrst proposed to learn representations from unlabeled datasets (hence the term, self-supervised learning) by telling which data points are similar or distinct, especially in the ﬁeld of computer vision (Chopra et al., 2005; Gutmann and Hyvärinen, 2010;
Schroff et al., 2015; Sohn, 2016;
Oord et al., 2018; Chen et al., 2020a; Grill et al., 2020).
Khosla et al.
(2020) extended the self-supervised batch contrastive approach to the fully-supervised setting and proposed a supervised contrastive learning method.
In speech processing, representative methods focused on speaker identiﬁcation (Ravanelli and Bengio, 2018), speech recognition (Schneider et al., 2019), and audio representation learning (van den
Oord et al., 2018; Baevski et al., 2020).
In the
NLP area, the contrastive framework is used for sentence representation learning (Fang et al., 2020;
Shen et al., 2020; Gao et al., 2021; Wu et al., 2021;
Yan et al., 2021), machine translation (Pan et al., 2021), and summarization (Wang et al., 2021b;
Cao and Wang, 2021).
Very recently, contrastive learning is also applied to learning a uniﬁed representation of image and text (Dong et al., 2019;
Zhou et al., 2020; Li et al., 2021).
Motivated by the contrastive learning frameworks in cross-lingual and cross-modal topics, we introduce a similar idea
Wav2vec2.
CNN
S - E n c
Span-Masked Aug. (SMA)
Word Repetition(Rep)
Cut-off “Thank you” …
Text Emb
Average pooling
Transformer Encoder Transformer Decoder
Crossmodal
Contrastive
Loss
[Figure]
<en> Thank you .
<fr> Merci.
<en> Thank you.
𝐿 !"
𝐿 #" 𝐿 $!
% 𝐿 &"% “It is a nice day!”
“What do you like to eat?”
“It's a new day full of energy.”
“I love vanilla ice cream.”
S - E n c
A v er ag e p oo li n g
It is a nice day!
What do you like to eat?
It's a new day full of energy.
I love vanilla ice cream.
T e x t
E m b
A v er ag e p oo li n g
Speech
Transcription
[Figure]
Learning
Positive example Negative example
Hard Examples Mining Operation (optional) in speech translation.
The ConST Approach
An end-to-end speech translation model directly translates audio sequence s =
( s 1 , ..., s | s | ) to the text y
=
( y 1 , ..., y | y | ) in the target language.
Speech translation corpus D
= { ( s , x , y ) } provides transcript x = ( x 1 , ..., x | x | ) in the source language, as well.
In this section, we present the overall speech translation model and cross-modal contrastive learning.
We also provide several feasible strategies to construct more positive and negative pairs to enhance the contrastive learning.
3. Model Framework
Our model consists four sub-modules: a speech encoder, a word embedding layer, a Transformer
Encoder and a Transformer decoder (Figure 2).
It is designed to take either speech or a sentence as input, and to output either source transcript or target translation text.
Such architecture enables a universal framework for multiple tasks, including
ST, MT and ASR.
The speech encoder module (S-Enc) is designed to extract low-level features for speech signals.
It contains Wav2vec2.0 (Baevski et al., 2020) and two additional convolutional layers.
The input is raw waveform signal sampled at 16kHz.
Each convolutional layer has a stride of 4 and d channels.
In total, it shrinks the time dimension by a factor of 4 .
Denote a = S-Enc
( s ) as the audio representation of the speech, | a | (cid:28) | s | .
Parallel to the speech encoder is the word embeeding layer .
It is the same as word embedding for text translation.
Both the speech encoder and word embedding layer are connect to Transformer encoder and then passed to the Transformer decoder .
The Transformer encoder and decoder are using the same conﬁguration as the original (Vaswani et al., 2017).
To explain, the Transformer encoder further extracts the high-level semantic hidden representation of two modalities.
The Transformer decoder generates the word sequences (transcription and translation) for ST, MT and ASR tasks.
Since our model has a complete Transformer encoder-decoder as a sub-module, this makes it possible to pre-train using large-scale extra MT parallel data.
Previous work has shown that multi-task learning on ST, MT and ASR improves translation performance (Indurthi et al., 2020;
Tang et al., 2021b;
Ye et al., 2021).
Our training loss consists of the following elements.
[Equation]
where
[Equation]
The ﬁrst three elements are cross-entropy losses on <speech, target text> , <speech, source text> and <source text, target text> pairs.
These pairs are built from the triplet ST data.
We also introduce a cross-modal contrastive loss term L CTR (see
Section 3.2 for details).
It aims to bring the representation between the speech and textual transcription modalities closer (its effect will be analyzed in detail in Section 6).
λ is a tuned hyper-parameter of the weighted contrastive loss term.
3. Cross-modal Contrastive Learning
As mentioned in the beginning, since we need to produce similar representations for the speech and transcript sharing the same semantic meanings, we propose cross-modal contrastive learning method to bring their representations closer together.
The main idea of cross-modal contrastive learning is to introduce a loss that brings speech and its corresponding transcript (positive example) near together while pushing irrelevant ones (negative examples) far apart.
Given a positive example of such a speechtranscript pair ( s , x ) , we randomly pick a set of
N − 1 transcripts { x − i } N − 1 i =1 from the same batch as negative examples.
For speech s and its transcript x , we ﬁrst average them in terms of the time dimension,
[Equation]
and apply the multi-class N-pair contrastive loss (Sohn, 2016):
[Equation]
where A = { x } ∪ { x − i } N − 1 i =1 , τ is the temperature hyper-parameter, and sim is the cosine similarity function sim ( a, b )
= a (cid:62) b/ (cid:107) a (cid:107)(cid:107) b (cid:107) .
In the implementation, negative examples { x − i } N − 1 i =1 are from the same training batch of data (Figure 2(b)).
3. Mining Hard Examples for Contrastive Learning
To further enhance the contrastive learning, we introduce three strategies to mine additional hard examples.
These strategies are at input and representation (gray shaded modules in Figure 2(a)).
Speciﬁc schematic illustrations of each operations are shown in Figure 3.
Span-Masked Augmentation
We mask consecutive segments of an original audio waveform sequence s to obtain a new modiﬁed speech s (cid:48) .
We
[Figure]
take s (cid:48) as an input to the model, and compute the contrastive loss on its original corresponding transcript.
We randomly sample without replacement all time steps in the original waveform of the speech to be the starting indices with a probability p , and then we set the sub-sequence M successive time steps to be blank.
In the experiment, we tried multiple conﬁgurations, and found p = 0 .
25 , M = 3600 the best, resulting in a masked span of 0 .
225 second.
Since the masked speech fragment is very short, we consider the masked speech and the original transcript to be positive pairs, and the remaining transcripts in the same batch to be negative pairs.
Word Repetition
The word repetition strategy randomly replicates some words (or sub-words) in the original sentences, with two advantages for improving representation robustness.
First, as the length of the sentence is shorter than that of its audio representation, randomly repeating the words in the sentence is a simple yet useful technique to increase the length.
Second, repeating words does not change the semantics and is suitable as an extra positive example of the corresponding speech.
Speciﬁcally, given sentence x , each sub-word token x i can be duplicated k more times, resulting in the duplicated sentence x (cid:48) , where k = 0 , 1 , 2 , ... and k ∼ Poisson (1) .
We regard x (cid:48)
as the additional positive example for the speech s and the samples with the same operation in the same batch as the negative examples.
Cut-off strategy Recent studies on natural language understanding and generation have proved cut-off strategy to be successful (Shen et al., 2020;
Yan et al., 2021).
We analogize a similar idea to the cut-off approach for speech representation.
We entirely erase a slice of the T × d representation matrix along each dimension and set the erased terms to 0.
Here, we present two variants: sequence cut-off , which erases some sequence dimension, and feature cut-off , which erases some feature dimension.
Note that there is a difference between cut-off and dropout.
Dropout randomly sets some elements to 0, while cut-off is a dimensional “block" dropout.
Similarly, we treat the cut-off audio representation and the original transcribed sentence as positive pairs, and the rest sentences in the same batch as negative pairs.
Experiments 4. Experimental Setups
ST datasets
We conduct experiments on all the translation directions in
MuST-C dataset 1 (Di Gangi et al., 2019): English (En) to
German (De), Spanish (Es), French (Fr), Italian (It), Dutch (Nl), Portuguese (Pt), Romanian (Ro) and Russian (Ru).
As one of the largest ST benchmarks, MuST-C contains more thanhours of TED talks for each direction.
MT datasets We also introduce external WMT datasets (Bojar et al., 2016) for En-De/Es/Fr/Ro/Ru and OPUS100 datasets (Zhang et al., 2020) for EnIt/Nl/Pt directions, as the expanded setup.
Table 8 (in Appendix.
A) lists the statistics of all the datasets included.
Model Conﬁgurations The Wav2vec2.0 in the SEnc is only pre-trained on Librispeech (Panayotov et al., 2015) speech without any downstream ﬁnetuning 2 .
Two layers of CNNs after the Wav2vec2.
are set to kernel size 5, stride size 2 and hidden size 512.
The Transformer follows the base conﬁguration, with 6 layers of encoder and decoder, hidden size d = 512 , 8 attention heads, and
FFN hidden states.
We use pre-layer normalization for stable training.
The model with the above conﬁgurations has a total of about 150M parameters.
Experiment Details We evaluate case-sensitive detokenized BLEU using sacreBLEU 3 (Post, 2018) on MuST-C tst-COMMON set.
In the analysis, we also report the ChrF++ score 4 (Popovi´c, 2017) and the learning-based BLEURT score 5 .
We use the raw 16-bit 16kHz mono-channel speech input.
We jointly tokenize the bilingual text using SentencePiece (Kudo and Richardson, 2018), with a vocabulary size of 10k, which is the same as Ye et al.
(2021)’s setup.
For the training loss, we set contrastive temperature τ = 0 .
02 and weight of contrastive term λ
= 1 . 5 for German and Dutch, and λ = 1 .
for the other languages.
Appendix B contains more detailed settings and explanations for the baseline models in Table 1.
Appendix C shows the experiments on the choice of the hyper-parameters.
4. Main Results
Comparison with end-to-end ST models Tableshows the main results.
Since many existing works regard “leveraging external data” to be one of their model’s features, their strong performances are largely predicated on the utilization of auxiliary data, especially large-scale MT data.
For a relatively fair comparison, we investigate two cases: (1) without external MT data and (2) with external MT data.
Without the external MT data, our method already gains an average improvement of 0.5 BLEU over the previous best models.
Also when speech data is introduced for pre-training, our method works better than others (Self-training,
W-Transf. and XSTNet).
When extra MT data are introduced, our method also outperforms SOTA by an average of 0.6 BLEU.
Among the benchmark models, with the same goal of closing two modality gaps, Chimera (Han et al., 2021) constructed an extra ﬁxed-length shared semantic space.
However, the shared memory with ﬁxed size actually compromises the MT performance, while our contrastive learning approach is more straightforward and effective.
Comparison with cascaded ST systems We compare our method with several cascade baselines, where Ye et al.
(2021) and Xu et al.
(2021) provided two strong cascade systems trained using MuSTC and external ASR and MT data (LibriSpeech,
WMT, and Opensubtitles).
From Table 2, we ﬁnd that as an end-to-end model, ConST can outperform these strong cascade models.
In Appendix 7, we provide a case study to show such improvement.
Analysis 5. Is contrastive loss effective?
With the same model architecture and the same pretraining + ﬁne-tuning procedure, the main differ-
Models
[Table] [Table]
Xu et al.
(2021) are two strong cascaded models.
ence between ConST and XSTNet (Ye et al., 2021) is whether we use the contrastive loss term during the ﬁne-tuning or not.
Comparing the BLEU results of w/o and w/ external MT data situations in Table 1, we ﬁnd that ConST further improves 0 .
5 and 0 .
6 BLEU scores in terms of eight translation directions on average, which proves the effectiveness of the cross-modal contrastive learning.
By gradually removing each losses in Eq.
( 1), Table 3 shows the improvements bringing by the multi-task learning and the contrastive learning.
For En-De translation direction, contrastive learning can bring an average improvement of 0.9 BLEU over the baseline models by only optimizing L ST (corresponding to the last row of the Table 3), and multi-task learning can lead to a further improvement of about 1.2 BLEU on top of that.
External MT
Conﬁg.
without with
[Table]
We test the results under settings with and without the introduction of external MT data.
5. Which layer to contrast on?
An intriguing question is which representations should be considered in the contrastive loss function.
In the method part (Section 3.2), we use averaged audio representation u for speech s (Eq.(2)) and averaged lexical embedding v for the transcript x (Eq.(3)), denoted as low-level repr. .
Whereas inspired by a recent study in multilingual MT (Pan et al., 2021), we also provide an alternative contrastive loss as a comparison, whose speech and text features are average-pooled semantic representations derived from the Transformer encoder, denoted as high-level repr. .
Table 4 shows that contrastive learning using the low-level representations (Line 1) is better than using the high-level ones (Line 2 ) .
On the
Representations
BLEU
ChrF
[Table]
+
BLEURT low
[Table]
[Table] [Table]
[Table] [Table]
w/
[Table]
contrative loss 27.
[Table]
other hand, although the performance of Line 2 is relatively inferior, it still outperforms the multi-task model without the contrastive loss (Line 3).
The detailed analysis of possible explanations will be shown in Section 6.2.
5. Is contrastive loss better than other losses?
Our goal for introducing the contrastive loss term (denoted as CTR Loss) is to close the distance be
- tween speech and text representations.
Whereas, there are other options to achieve this goal, such as L2 loss and CTC loss.
• L2 Loss :
Without introducing any negative samples, L2 loss directly reduces the Euclidean distance between the representations of two modalities by minimizing L = (cid:107) u − v (cid:107) 2 .
L2 loss can be viewed as an implementation based on the idea of knowledge distillation (Heo et al., 2019; Dong et al., 2021b).
• CTC Loss : The connectionist temporal classiﬁcation (CTC) loss (Graves et al., 2006) is commonly used in speech-related tasks (Xu et al., 2021;
Dong et al., 2021b).
Unlike contrastive loss that cares about the representation, CTC loss connects the two modalities by establishing speech-text alignment and maximizing p ( x | a ) =
(cid:80)
[Equation]
) , where Π s , a is the set of all valid alignments.
Compared to the other two ways of bridging the modality gap, L2 and CTC loss, is the contrastive loss term better?
The answer is yes according to the results in Table 5.
Our explanation is that information on the negative samples beneﬁts the contrastive loss, bringing the the distance between the speech and its corresponding transcription closer while pushing the distance to the irrelevant text farther.
5. Analysis on the hard example mining strategies
In Section 3.
we proposed four methods to mine the hard examples for contrastive learning, namely
Extra Loss
BLEU
ChrF
[Table]
+
BLEURT
[Table]
[Figure]
: p < 0 . 01 ).
span-masked augmentation (SMA), word repetition (Rep), sequence cut-off (SCut), and feature cut-off (FCut).
In this section, we study how effective these methods are, and to do so, we consider the BLEU performances of their 15 combinations (Figure 4).
Note that “Original” means the original contrastive loss in Eq.(4) without any additional hard examples mining operation, and the diagonal in the heat map represents only one strategy used.
For an easy and fair comparison, we set the weight of the contrastive term to 1.0 uniformly.
We have the following observations.
All the hard examples mining methods are effective.
All the BLEU scores in Figure 4 exceed the strong multi-task model trained without contrastive learning ( 27 . 1 ).
Among all the strategies, the combination of the original and SCut reaches the best result ( 28 . 3 ), and is better than the model without any expanded operations ( p < 0 . 01 ).
Generally, to ﬁnd the best model, we suggest adopting multiple strategies and choosing the best check-
[Figure]
point
The combinations of the hard examples mining methods and the “original” have relatively better performances.
We argue that we need the original positive and negative examples to give more accurate representations (without any dropout) for contrastive learning.
On the contrary, without the help of “original” loss, the performance with both sequence cut-off and feature cut-off is the worst in Figure 4, probably because too much information is lost by superimposing the two.
Why does cross-modal contrastive learning work? — Analysis on the Modality Gap
As mentioned earlier, the existing multi-task training models cannot address the speech-text modality gap .
Does ConST reduce the representation gap between speech and text?
6. Visualization of Representation
Does the speech-text modality gap exist without explicitly bridging the two?
Speech-text modality gap means the discrepancy between the audio representations and transcription sentence embeddings.
To visualize it, we plot the bivariate kernel density estimation (Parzen, 1962) (KDE) contour of their dim-reduced features, where T-SNE (Van der
Maaten and Hinton, 2008) is used to reduce the dimension into two (Figure 5).
Ideally, if the representations of speech and its corresponding transcript are similar, their KDEs will be similar, and thus the contour lines will overlap as much as possible.
However, Figure 5(a) is the KDE contour of the multi-task framework without any explicit modeling to bring two modalities together (Ye et al., 2021).
It shows that the representations are so dissimilar that they are organically divided into two clusters, i.e. speech-text modality gap exists.
Does ConST reduce the modality gap?
As shown in Figure 5(b), compared to the baseline model without contrastive learning, ConST with cross-modal contrastive learning is able to bring representations of different modalities much closer.
This means that the audio representation contains more linguistic information similar to that of the textual transcription, which is more advantageous for the downstream ST generation through the shared Transformer encoder and Transformer decoder .
6. Cross-modal Retrieval
How good is the cross-modal representation space learned from ConST?
To answer this question, we conduct a retrieval experiment, i.e. ﬁnding the nearest (smallest cosine similarity) transcript based on the speech representation.
We compare ConST model with the baseline without crossmodal contrastive learning and report the top-1 retrieval accuracy using (1) the low-level representations and (2) the high-level semantic representations, in Table 7.
When retrieving the text using low-level representations, our method gains a substantial 79% increase over the baseline.
In addition, we ﬁnd that without explicit contrastive modeling, the baseline can achieve retrieval accuracy of more than 94% according to the semantic representations outputted from the Transformer encoder .
We believe that such high accuracy is automatically learned from the triple-supervised data itself under the multi-task learning framework.
With such a degree of crossmodal alignment, if we construct the contrastive loss with semantic representations, its gain to the
ST performance turns out to be limited, which exactly corroborates the ﬁndings in Section 5.2 – low-level representations are preferred in the crossmodal contrastive learning.
Case Analysis
In this section, we use several cases that ConST generates.
We compare our model with the cascaded model and the previous end-to-end model,
XSTNet (Ye et al., 2021).
For this ﬁrst case, the cascaded system fails to give a right translation due to the mis-punctuation
Models
CASE
[Table]
CASE
[Table] [Table]
issue ( klingt is a verb), while the end-to-end model,
XSTNet and ConST translate correctly.
For the second case, the previous end-to-end XSTNet model cannot accurately translate the phrase “started exploring this idea of”, which performs worse than the cascaded one.
Whereas ConST successfully conveys the meaning of “this idea” , and translates more accurately than XSTNet.
We believe this improvement comes from the cross-modal contrastive learning.
Conclusion
In this paper, we propose ConST, a simple yet effective contrastive learning framework bridging the speech-text representation gap and facilitating the
ST with limited data.
We also provide feasible hard example mining methods to learn robust representations.
The results on the MuST-C ST dataset prove the effectiveness of the method.
Broader Impact
This work improves the performance of ST tasks on public datasets by learning speech representations that are more similar to text representations, but the model is far from being achieved for industrialgrade implementations.
In real scenarios, for example, the original voice is noisier and the distribution of speech lengths is more complex than in the public dataset, which cannot be handled by an end-toend model alone.
The shortcoming of this model is that it still needs a certain amount of labeled data for training, especially <speech,transcription> to learn better speech representation, and for the more than 7 , 000 languages and dialects in the world, most of them do not have corresponding translations or even transcriptions, our method does not work in untranscribed scenarios.
In this paper, we focus on the improvement brought by the better speech representation on the ST task, and obtained good results with hundreds of hours of speech data.
We hope that our work achieves better results using more data ( e.g. raw speech, raw text, ASR, MT data) in the future.
B Experimental Details Training and Implementation Details
We use
Adam optimizer ( β 1 = 0 .
9 , β 2 = 0 . 98 ) with learning rate = 1 e − 4 and warmup 25k steps during the
ST training.
We also implement the expanded setting with the introduction of external WMT to train the Transformer module.
In the pre-training stage, we set the learning rate = 7 e − 4 and warmupsteps.
For robust training, we set label smoothing to 0 .
1 , and dropout rate to 0 .
The hyper-parameters for different data augmentation methods are as follows: for masked audio span strategy, we set masking probability p = 0 .
25 and masking span length
M = 3600 frames; for both sequence and feature cut-off, we set the cut-off dropout rate as 0 . 1 .
We save the checkpoint with the best BLEU on dev-set and average the last 10 checkpoints.
For decoding, we use a beam size of 10 and length penalty 0 .
7 for
German, 1 . 0 for French, and 0 . 4 for Russian.
We train the models in 8 Nvidia Tesla V100 GPUs for each experiment.
We use Fairseq (Ott et al., 2019) as the code-base for our implementation.
Baseline Models
In Table 1, we compared our method with end-to-end baseline models whose audio inputs are 80-channel log Mel-ﬁlter bank, including: FairseqST (Wang et al., 2020a),
NeurST
(Zhao et al., 2021a), Espnet ST (Inaguma et al., 2020), Dual-decoder Transformer (Le et al., 2020), SATE (Xu et al., 2021), Speechformer (Papi et al., 2021), self training (Pino et al., 2020) and mutual learning (Zhao et al., 2021b) method, STAST (Liu et al., 2020b), bi-KD (Inaguma et al., 2021), MLT method (Tang et al., 2021b), Lightweight Adaptor (Le et al., 2021),
JT-S-MT (Tang et al., 2021a),
FAT-ST (Zheng et al., 2021), TaskAware (Indurthi et al., 2021), and STPT (Tang et al., 2022). We also compare our method to baseline models that have pretrained Wav2vec2.0 as a module, including:
• W-Transf.
(Ye et al., 2021): the model has the same structure as ours, but is only trained on <speech, translation> parallel data.
• Chimera-ST
(Han et al., 2021): the model that builds a shared semantic memory for both audio and text modalities.
XSTNet (Ye et al., 2021): the model has the same structure as ours, and adopted a multi-task ﬁne-tuning strategy.
• STEMM (Fang et al., 2022): the model that bridges the modality representation gap by minimizing the Jensen–Shannon divergence between the original speech representation and the manifold mix-up representation.
C The Choice for Hyper-parameters
Inﬂuence of Temperature In the contrastive loss, the temperature hyper-parameter is provided to control the smoothness of the distribution normalized by softmax operation.
A high temperature helps to smooth the distribution, making it more difﬁcult for the model to distinguish between positive and negative samples (corresponding to correct transcriptions and other transcriptions in this work), while the low temperature behaves just the opposite.
We choose several temperature hyper-parameters ranging from 0 .
01 to 0 .
5 , and Figure 6 shows their
BLEUs on the test and dev sets .
We ﬁnd that (1) the choice of the temperature does not drastically affect the ﬁnal BLEU score, and (2) we recommend that the temperature τ be set between 0.02 and 0.
to ensure a relatively good ST performance.
In the experiment, we use τ = 0 .
[Figure]
Inﬂuence of Contrastive Loss Weight
The total loss we optimize, Eq.(1), is a linear combination of the multi-task cross-entropy losses L MLT and the contrastive term L CTR .
To investigate how much the contrastive terms affect BLEU, we ﬁx its temperature τ = 0 .
02 , adjust the values of its loss weight λ from 0.1 to 2.0, performed three experiments for each value, and test the average BLEU on
En-De tst-COMMON set.
Figure 7 depicts the performances.
First, all objective functions containing
L CTR , even if their weights λ take different values, are apparently better than the baseline model with
L MLT only L CTR .
Then, the best BLEU score is achieved at loss weight λ = 1 . 5 , corresponding to the results in Table 1.
And when analyzing the effect of data augmentation strategies (Section 5.4), since we need to consider the combination between them, which is more complicated.
Therefore, we set the loss weight to 1 . 0 uniformly for simplicity.
In general, we recommend that the weight hyperparameter takes a value between 0 . 8 and 1 . 5 .
[Figure]
D Data Scale for Fine-tuning
The experiments in the main paper show that our model can perform well by introducing external
MT data pre
training.
Here, we simulate the scenario with plenty of MT and speech data and limited ST triple-labeled data, and does ConST have the ability of low-resource learning?
In the experiment, we reduce the labeled ST data to 1, 10, and 100 hours, corresponding to sentence counts of about 500, 5k, and 50k sentences.
For a fair comparison, we use the same MT pre-trained Transformer module as in the main paper.
We ﬁnd the contrastive loss particularly helpful when the amount of speech data is extremely small, like only 1 hour of speech.
Second, the multi-task training strategy is also very effective in improving the robustness of the model performance.
We also ﬁnd that by using easily accessible MT and speech pretraining, our model could reach the previous baseline results without pre-training using only 1 / 4 of the original data, i [Figure] e. 100 hours of labeled ST data.
[Figure]