Boosting Monolingual Sentence Representation with Large-scale Parallel Translation Datasets
Abstract
Although contrastive learning greatly improves sentence representation, its performance is still limited by the size of existing monolingual datasets.
So can semantically highly correlated massively parallel translation pairs be used for pre-training of monolingual models?
This paper proposes an exploration of this.
We leverage parallel translated sentence pairs to learn singlesentence sentence embeddings and demonstrate superior performance in balancing alignment and consistency.
We achieve new state-of-the-art performance on the mean score of Standard Semantic
Text Similarity (STS), outperforming both SimCSE and Sentence-T5.
1. Introduction
Gao et al.
2021 demonstrates that a contrastive objective can be extremely effective when coupled with pre-trained language models and sentence-pair datasets.
However, the generality and capability of the language model are strictly limited by the size of existing sentence-pair datasets (Bowman et al., 2015; Williams et al., 2017).
Meanwhile, there have accumulated large-scale parallel translation datasets ( 100 x larger than existing monolingual sentence-pair datasets) in multilingual learning community (Yang et al., 2019a;
Feng et al., 2020; Pan et al., 2021), which have not been utilized for learning sentence representations.
Furthermore, given parallel translation pairs, previous contrastive learning frameworks (Radford et al., 2021; Gao et al., 2021) cannot well balance the alignment and uniformity (Wang &
Isola, 2020) of monolingual sentence embeddings, where alignment calculates the expected distance between positive embeddings and uniformity measures how well the embeddings are uniformly distributed.
In this paper, we build on the top of dual encoder (Radford et al., 2021;
Yang et al., 2019b), and adopt a similar strategy as Frozen (Tsimpoukelli et al., 2021), where we freeze the source language encoder and only train the target language encoder for better monolingual sentence embeddings.
The source language encoder is fixed and provides consistent embeddings to supervise the target language encoder via contrastive learning.
The corresponding source-target translation pairs are regarded as positives.
We conduct a comprehensive evaluation protocol following
SimCSE (Gao et al., 2021) on seven standard semantic textual similarity (STS) tasks (Agirre et al., 2012; 2013;
Marelli et al., 2014; Agirre et al., 2014; 2015; 2016; Cer et al., 2017).
We outperforme SimCSE (Gao et al., 2021) and Sentence-T5 (Ni et al., 2021) by a large margin.
On the average score of STS tasks, our pre-trained BERT base with or without fine-tuning surpasses SimCSE-BERT base by 4.39% and 3.25% respectively, and RoBERTa large achieves 85.58 on average.
Surprisingly, BERT base with fine-tuning achieves better results than Sentence-T5 (11B) with only 1% parameters in comparison.
In summary, we provide an exploration of utilizing existing large-scale parallel translation pairs for learning monolingual sentence representation, based on cross-lingual contrastive learning framework that well balances alignment and uniformity.
Our approach achieves a new state-of-theart on standard semantic textual similarity (STS), and the best performance in corresponding tracks on transfer tasks evaluated by SentEval 1 .
2. Proposed Approach 2.1. Background
Scaling up the size of training dataset (Radford et al., 2021;
Jia et al., 2021) has proved to be effective to improve robustness and generalization of representations in contrastive learning framework.
However, previous works (Reimers & Gurevych, 2019; Gao et al., 2021) only utilize limited size of monolingual sentence pairs to learn sentence embeddings, such as MNLI datasets (Williams et al., 2017)
[Figure] [Figure]
and SNLI (Bowman et al., 2015).
In contrast, there have existed large-scale well-annotated parallel translation pairs ( 100 x larger than monolingual paired datasets) in the community of multilingual learning.
Instead of training on limited monolingual sentence pairs, utilizing existing parallel translation datasets shows better flexibility and a potential to further improve the performance of sentence embeddings, where a parallel translation pair that is highly correlated in semantic can be treated as a positive sample.
Preliminaries.
To utilize paired inputs, single multilingual encoder (Ma et al., 2020; Pan et al., 2021) and dual encoder (He et al., 2020; Radford et al., 2021; Ni et al., 2021) are the most commonly adopted strategies for learning multilingual representations.
Multilingual encoder embeds sentences from different languages into a single semantic space using a unified encoder, based on the hypothesis that multilingual learning leads to better multilingual sentence representation.
Its architecture is illustrated in A, Figure 1.
Dual encoder, also known as two-tower, models the paired data with two independent encoders, and projects the embeddings of paired inputs into the same semantic space through joint training.
Its architecture is illustrated in B, Figure 1.
2.2. Method
Although multilingual encoder and dual encoder can use parallel translation pairs straightforwardly, they both suffer from the imbalance between alignment and uniformity, as source language encoder and target language encoder keep updating in the training process.
In other words, while they pull the positive samples (source-target translation pairs) closer and the negative samples (source-non target translation pairs) farther away through an explicit contrastive learning objective, the alignment and uniformity of embeddings from monolingual sentence pairs cannot be guaranteed.
Specifically, let ( s i , t i ) denote the representation of a parallel translation pair generated by the source language encoder and target language encoder, respectively.
We simplify the explicit contrastive objective as Eq 1.
[Equation]
Where L p and L n represent the distance for positives and negatives of parallel translation pairs as defined in Eq 2 and
Eq 3, α denote the linear weights, D is a distance function, and i ̸
= j .
The explicit contrastive objective is to minimize the distance between positives and maximize the distance between negatives.
[Equation]
Given parallel translation pairs, we also define the implicit
[Figure]
(Ours) (Preliminaries)
[Figure]
or actual objective that has not been considered into contrastive learning framework in Eq 4, which measures the alignment and uniformity of monolingual sentence embeddings. Although L implicit is not considered in the explicit contrastive objective, we expect to retain good alignment and uniformity of monolingual sentence embeddings from the target encoder, as the actual objective is to learn monolingual sentence embeddings from parallel translation pairs.
[Equation]
Where L ′ p and L ′ n represent the distance for positives and negatives of monolingual pairs as defined in Eq 5 and Eq 6.
s
+ i and t
+ i represent the monolingual positive samples for s i and t i , respectively.
β denote linear weights.
[Equation]
In preliminaries, as shown in (A) and (B), Figure 1, the source language encoder keeps updating in training and can not provide consistent supervision for the target language encoder.
The implicit objective for preliminaries is Eq 4, where the alignment and uniformity of source embeddings and target embeddings are both required to be implicitly optimized.
However, given two independent implicit objectives, it becomes hard to find a local optimum through Eqwithout any constraints.
To effectively improve the uniformity and retain the alignment simultaneously, and optimize the implicit objective (4) through an explicit objective (1), we propose to soften the implicit objective for better optimization with our modified architecture, built on the top of regular dual encoder.
To be clear, we freeze the side of the source language encoder, so that the alignment and uniformity of source embeddings are frozen in the training.
In this case, the implicit objective degrades to Eq 7.
[Equation]
As the optimization space shrinks and the implicit objective relaxed, finding the local optimal solution becomes easier and more efficient.
We show the differences between our approach (C) and preliminaries (A, B) in Figure 1.
To validate the effectiveness of our approach, we take the checkpoint of our model and preliminaries every 100 steps during training and visualize their alignment and uniformity (Wang & Isola, 2020) on a monolingual sentence-pair dataset and parallel translation dataset in Figure 4, training details can be found in 3.4.2. Figure 4, we show the promising results of implicit objective (the alignment and uniformity of target encoder), given monolingual sentence pairs as input, where we greatly improve uniformity and retain a steady alignment, while others dramatically degrade alignment. [Figure]
3. Experiments 3.1. Training Datasets
We adopt WMT and source-mixed datasets that have parallel translation pairs for cross-lingual contrastive learning, while the Chinese NLI dataset 2 that has monolingual Chinese sentence pairs is only utilized for fine-tuning.
WMT Dataset 3 is a common-used machine translation dataset composed of various sources.
We perform an elaborate cleaning process following (Meng et al., 2020) to filter out low-quality pairs.
We get 19,442,200 Chinese-English translation parallel pairs after cleaning.
Source-mixed Dataset collects from more open-sourced translation datasets built on the top of WMT dataset, including AIC (Wu et al., 2017), translation2019zh (Xu, 2019),
UN Corpus (Ziemski et al., 2016), etc.
Finally, we establish a larger-scale dataset including 56,741,808 Chinese-English translation pairs.
This dataset is used to show that further scaling up the size of the training set helps improve overall performance.
3.2. Training Details
We elaborate the training details of our pipeline that is shown in Figure 2.
We maintain a consistent memory queue (He et al., 2020) of negative embeddings, where the current mini-batch of the source language encoder’s embeddings are enqueued and the oldest are dequeued.
The pooling method used in the training is [CLS] with an MLP layer following SimCSE.
All experiments are conducted on
V100 GPUs.
The batch size in experiments represents the batch size on each GPU.
3.2.1. T RAINING A C HINESE E NCODER
As shown in (A), Figure 2, the first step is to train a target language (Chinese) encoder.
Specifically, we adopt the pre-trained SimCSE-RoBERTa large model as the source language (English) encoder, and initialize a Chinese
RoBERTa large model 4 with pre-trained weights as the target language (Chinese) encoder.
We adopt a series of hyperparameters from 3.2.2: learning rate is 5e-5, batch size is 200, dropout is 0.1, and the input sentence length is 50.
We freeze the source language (English) encoder and only update the target language (Chinese) model.
We evaluate every 250 training steps on the development set of Chinese
STS-B
and save the best checkpoint.
3.2.2. T RAINING AN E NGLISH E NCODER
As shown in B, Figure 2, we train a target language (English) encoder that generates sentence embeddings.
Specifically, we reuse the pre-trained Chinese encoder from 3.2.1 as the source language (Chinese) encoder and freeze its parameters.
We evaluate every 250 training steps on the development set of STS-B and save the best checkpoint.
For BERT base (or RoBERTa base ), the learning rate is 1e-4, batch size is 400, and the dropout is defaulted set as 0.1.
In the term of RoBERTa large (or BERT large ), we set the learning rate to 5e-5, batch size to 200, all other hyperparameters keep the same as BERT base .
3.3. Evaluation Results
Following Gao et al.
, we evaluate our models on seven transfer and seven STS tasks by SentEval tools.
As the main goal of learning sentence embeddings is to cluster semantically similar sentences, we also take STS result as the main metric.
Semantic textual similarity tasks.
We evaluate our approach under zero-shot and fine-tuned settings, respectively.
To fairly compare with previous works (Gao et al., 2021; Ni et al., 2021), we adopt seven STS tasks including STS 2012–2016 (Agirre et al., 2012; 2013; 2014; 2015; 2016),
STS Benchmark (Cer et al., 2017) and SICKRelatedness (Marelli et al., 2014).
STS tasks are widely used in measuring the discriminative power of sentence embeddings.
We compare several well-known benchmarks, include:
Sentence-BERT (Reimers & Gurevych, 2019), CTBERT (Carlsson et al., 2020), SimCSE and Sentence-T5 (Ni et al., 2021) 11B model, which contains 11 billion parameters.
Table 1 reports the evaluation results on seven
STS tasks.
Our approach can substantially improve results on all the datasets with or without extra NLI supervision, greatly outperforming the previous state-of-the-art models.
Specifically, our approach outperforms the averaged scores of SimCSE by 1.27-2.65 under a zero-shot setting in all tracks.
When using NLI datasets, Ours-BERT base further pushes the state-of-the-art results from 84.94 to 85.15.
The gains are more pronounced on RoBERTa encoders, and our method achieves 85.58 with RoBERT large .
3.4. Ablation Studies
We investigate the impact of source language encoder and contrastive objectives.
We use BERT base (WMT) without fine-tuning as our benchmark.
3.4.1. T HE EFFECT OF SOURCE LANGUAGE ENCODER
To analyze the role of source language encoder, we train a SimCSE-RoBERTa large model on the Chinese NLI dataset directly and use it as the source language (Chinese) encoder.
For comparison, we train two RoBERTa large models on the
WMT dataset following the steps in 3.2.1 with and without fine-tuning.
Then, we train three target language (English) encoders as 3.2.2 given different source language models and evaluate them on the SST-B development set.
We report the results in table 2.
[Table]
[Table]
[Table]
3.4.2. T HE EFFECT OF CONTRASTIVE OBJECTIVES
To show the effectiveness of our cross-lingual contrastive learning scheme, we train models with multilingual encoder, dual encoder and our modified dual architecture, respectively, and evaluate their performance on STS-B development set.
For dual encoder, we adopt the pre-trained source language (Chinese) encoder from 3.2.1 and a pre-trained RoBERTa base , then train it via contrastive learning.
For multilingual encoder, we adopt a RoBERTa base -xlm
(Lample & Conneau, 2019) model that accepts multilingual input.
For our modified dual architecture, we use the same source and target encoder as dual encoder, while keeping the source encoder frozen.
All models are trained on WMT dataset.
For a fair comparison, we unify the hyperparameters of different objectives: batch size is 128, learning rate is 2e4.
The only difference between dual encoder and ours is whether the source language encoder is frozen in the training.
Table 3 shows the effectiveness of our approach.
4. Conclusion
In this work, we provide the first exploration of utilizing existing large-scale parallel translation pairs for learning sentence representation, propose a modified dual architecture that well balances the alignment and uniformity of embeddings.
We demonstrated that our method achieves a new state-of-the-art on standard semantic textual similarity (STS), and the best performance on corresponding tracks on transfer tasks, outperforming both SimCSE and