Transcormer: Transformer for Sentence Scoring with Sliding Language Modeling
Abstract
Sentence scoring aims at measuring the likelihood score of a sentence and is widely used in natural language processing scenarios, like reranking, which is to select the best sentence from multiple candidates.
Previous works on sentence scoring mainly adopted either causal language modeling (CLM) like GPT or masked language modeling (MLM) like BERT, which have some limitations: 1)
CLM only utilizes unidirectional information for the probability estimation of a sentence without considering bidirectional context, which affects the scoring quality; 2) MLM can only estimate the probability of partial tokens at a time and thus requires multiple forward passes to estimate the probability of the whole sentence, which incurs large computation and time cost.
In this paper, we propose
Transcormer – a Transformer model with a novel sliding language modeling (SLM) for sentence scoring.
Specifically, our SLM adopts a triple-stream self-attention mechanism to estimate the probability of all tokens in a sentence with bidirectional context and only requires a single forward pass.
SLM can avoid the limitations of CLM (only unidirectional context) and MLM (multiple forward passes) and inherit their advantages, and thus achieve high effectiveness and efficiency in scoring.
Experimental results on multiple tasks demonstrate that our method achieves better performance than other language models.
Our code and pre-trained models will be released at: https://github.com/microsoft/CyBERTron-LM/
Transcormer .
Introduction
Sentence scoring is to measure the log-likelihood score of a sentence via language model, so that it can be used to represent the relative likeliness of a sentence.
Specifically, a good sentence should have a relatively lower log-likelihood score, which means more linguistically acceptable for the sentence [1].
Due to such nature, sentence scoring has been widely used in many natural language processing (NLP) scenarios.
For instance, it can be used to rerank candidates in neural machine translation (NMT) or automatic speech recognition (ASR) tasks [2, 3] or evaluate sentences in linguistic acceptability [4].
Therefore, how to design effective language models to calculate sentence scores efficiently is very important.
Recently, neural network based language models (LM) [5, 6, 7, 8] have been considered as the most widely used technique for sentence scoring, since they can produce density estimation of the whole sentence by computing the probability of each token and summing up their values as the sentence score.
Specifically, causal language modeling (CLM) [9] and masked language modeling (MLM) [10] are the most representative LMs.
For a given sentence x = { x 1 , · · · , x | x | } , where x i is the i -th token of x .
CLM can predict next token conditioned on unidirectional context and its objective is to optimize (cid:80) | x | i =1 log P ( x i | x <i ) .
Hence, CLM is usually used for solving natural language generation (NLG) tasks [9, 11, 12].
While for MLM, it replaces a subset of tokens x S in x as special symbol [MASK] and then predicts the masked tokens based on the corrupted sequence x \S .
The objective of MLM is to optimize (cid:80) |S| i =1 log P ( x S i | x \S ) , so it is able to learn bidirectional context and can be used for solving natural language understanding (NLU) tasks [10, 13, 14, 15].
Since CLM and MLM can be learned in an unsupervised fashion, many works [10, 13, 14, 9, 11, 12, 16, 17, 18, 15, 19, 20] have pre-trained these LMs on large-scale corpus to extract powerful linguistic representations, and these models can be directly used out of the box to predict the probabilities of tokens.
Inspired by the successes of LMs, some works [21, 22, 23, 24, 25] have tried to use pre-trained CLM or MLM to generate sentence scores on NMT or ASR reranking tasks and achieved some promising results.
However, we notice that both CLM and MLM still suffer from some deficiencies in calculating sentence scores.
For instance, some works [25] utilized GPT-style model for ASR reranking within single inference, yet GPT model can only extract unidirectional information due to the limitation of CLM, without considering the whole sentence semantics, and thus affect the sentence score.
To utilize bidirectional context, some works [22, 23, 24] applied BERT model for rescoring.
However, the nature of MLM is to mask some tokens in the sentence for prediction, which means it requires the
BERT model to forward multiple times and each forward pass only masks one token for prediction.
As a result, it is time-consuming to directly adopt MLM for scoring sentence.
In summary, we find that for calculating sentence scores, CLM needs one-pass inference but only uses unidirectional information and MLM is costly in computing sentence score although it can use bidirectional context.
Therefore, a natural question arises: is it possible to design a language model to use bidirectional context for sentence scoring with only one inference pass.
In this paper, we introduce Transcormer , a Transformer model designed for sentence scoring.
More specifically, our Transcormer leverages a novel language modeling scheme, named as sliding language modeling (SLM), that produces the probability of all tokens within single inference pass and simultaneously utilizes bidirectional context.
To fulfill this target, we innovatively design a triple-stream self-attention mechanism, which consists of two content streams (a forward stream and a backward stream) and one query stream.
By employing specifically-designed mask strategies on the attention matrix, our method allows each token in the query stream to leverage all token information except itself (i.e., the tokens before and after it) for estimating its probability to avoid any information leakage.
To the best of our knowledge, SLM is the first language modeling tailored for sentence scoring.
We pre-train our SLM on large-scale corpus, and then evaluate it on multiple datasets.
Experimental results demonstrate that Transcormer outperforms the baselines by up to + 0.8/0.6 BLEU score on small/large-scale NMT tasks, ∼ 20% relative improvements on ASR tasks.
The main contributions of this work are summarized as follows:
• We analyze the pros and cons of CLM and MLM when using them for scoring sentences, and propose Transcormer with a new sliding language modeling, which uses bidirectional context for probability estimation within a single pass.
• We introduce a novel triple-stream self-attention mechanism in SLM, which has two content streams to collect forward/backward semantics, and a query stream to estimate the probability of each token in a sentence.
• Experimental results on multiple datasets demonstrate the effectiveness and efficiency of our SLM for sentence scoring.
Background 2.1 Sentence Scoring
Sentence scoring has a long history in NLP applications, especially in reranking tasks ( e.g. , reranking for machine translation [26, 27, 28, 29] or speech recognition [30]).
Generally, given n -best candidates generated by text generation models, reranking aims at scoring each candidate to select the best answer.
Early works [26, 27, 28, 29, 31, 32] mainly used statistical LM or combined it with
RNN-based LM to calculate the sentence scores.
Recently, end-to-end neural network based LM has became the de facto approach for scoring and has been applied in many NLP tasks [6, 5, 33].
Specifically, causal language modeling (CLM) and masked language modeling (MLM) are the most representative language modeling methods, among which GPT [9, 11, 12] and BERT [10] are the most famous examples, respectively.
As aforementioned, CLM conditions on the previous states to predict next token so that it can obtain the probability of each token of the sentence in a single pass, but can only capture unidirectional information.
For MLM, it is able to use bidirectional context for prediction but the masked-prediction mechanism limits MLM to producing the probability of all tokens within one forward pass (since MLM can only provide the probability of masked tokens).
To calculate the sentence score, a kind of solutions [22, 23, 24] is to forward multiple times and only mask one token each time.
Therefore, the MLM for sentence scoring is formulated as: (cid:80) | x | i =1 log P ( x i | x \ x i ) .
We can find that the cost of MLM for scoring needs | x | inference passes, which is too time-consuming.
Some recent works [22, 21, 34, 35] have been proposed to alleviate these issues in MLM.
Wang et al.
[22] and Salazar et al.
[21] attempted to use stochastic estimation or distillation to avoid N-passes problem to approximately estimate the probability of each toke produced by MLM, with a sacrifice of performance.
Clark et al.
[34] adopted a two-cloze tower [36] with noise-contrastive estimation to provide sentence probability, and Shin et al.
[35] only considered word embedding as the inputs of key and value in transformer without any interaction.
Besides, some works [37, 38] adopted discriminative language modeling to approximately estimate sentence scores based on the paired data, but this paradigm must require labeled data from the downstream tasks and cannot utilize unlabled data for pre-training.
More discussions about related works can be found in Appendix A.2.
Therefore, how to calculate sentence scores efficiently with bidirectional context is the main challenge in MLM.
Overall, CLM only needs a single forward pass to estimate the probability of all tokens but cannot extract bidirectional context, while MLM leverages bidirectional information but needs multiple inference passes.
Consequently, we raise a natural question: is it possible to design a pre-trained language model to support all token prediction in a single pass and simultaneously leverage bidirectional information?
This is exactly the motivation of our method.
2.2 Multiple-Stream Self-Attention
The pioneer of multiple-stream self-attention is XLNet [16], which introduces two-streams selfattention to incorporate autoregressive pre-training for language understanding, which pre-trains
Transformer [39] via using a content stream and a query stream.
In details, for the t -th step, the content stream is able to capture the dependency from the tokens before the t -th step and itself ( i.e. , x ≤ t ), while the query stream is only allowed to view the tokens before the t -th step ( i.e. , x <t ) to avoid information leakage.
Besides, there are some other variants of two-stream self-attention [17, 40, 41], which are designed for solving NLU tasks.
For example, MPNet [17] used two-stream self-attention to build masked and permuted pre-training.
ProphetNet [40] designed multiple query streams to predict N-gram future steps for sequence-to-sequence tasks, and ERNIE-GAN [41] proposed a multiflow generation model, which includes two query streams for span and word prediction.
We observe that these works mainly used a single content stream and then used one or many query streams to predict more information.
Different from these works, we introduce a triple-stream self-attention mechanism, which enables query stream to leverage two content streams for prediction, and thus enjoys the benefits of additional bidirectional context for estimating token probability.
Transcormer
To inherent the advantages of CLM and MLM for sentence scoring and avoid their limitations, we propose Transcormer – a Transformer model with a novel sliding language modeling for sentence scoring.
First, we summarize that an ideal language modeling for sentence scoring should satisfy two requirements: 1) model should be able to use bidirectional context for effective probability estimation of each token; 2) model should produce the probability of all tokens in a sentence within a single inference pass for efficiency.
Therefore, to fulfill these two requirements, we formulate a new language modeling, named as sliding language modeling (SLM), and describe it in Section 3.1.
In SLM, we propose a Triple-Stream Self-Attention mechanism based on Transformer (please see
Section 3.2 for details) to use bidirectional context for each token prediction and avoid information leakage.
We also discuss the differences between SLM and other LMs in Section 3.3.
Figurepresents the pipeline of our Transcormer for sentence scoring with SLM.
[Figure]
xxx
P P
Pxx
P
P_ _ _
P
P
P_ _
P
Pxxx
P P
Pxx
P P
Transcormer
[Figure]
3.1 Sliding Language Modeling
Considering the pros and cons of CLM and MLM for scoring, we notice: 1) CLM can produce probability of all tokens within one forward pass, and thus obtain the unidirectional information of the whole sentence; 2) MLM for sentence scoring needs multiple inference passes, so that many context has been repeatedly calculated and cause a waste of computation.
So, is it possible to reuse token information to build bidirectional context for token prediction?
Therefore, we propose sliding language modeling (SLM) to address the inherent flaws in previous
LMs ( i.e. , CLM and MLM) for sentence scoring.
Specifically, we first maintain two individual streams to collect forward (left-to-right) context and backward (right-to-left) context.
And for each token prediction, we decompose the sentence information as the past tokens (the tokens before it) and future tokens (the tokens after it) respectively.
As a result, our SLM enforces each token to only capture the dependency from its past tokens and its future tokens concurrently, so that each token can utilize the whole sentence information (except itself) to estimate token probability.
The objective function of SLM is formulated as:
[Equation]
where x <i and x >i respectively correspond to the tokens before the i -th token and after the i -th token, and θ represents the parameters of SLM.
Thanks to such design, our SLM can utilize bidirectional context to produce the probability of each token within one forward pass, and thus satisfy the above requirements for sentence scoring.
However, previous experiences [16, 42] pointed out that the states with bidirectional information will cause information leakage when propagating to the next layer 2 .
Therefore, how to implement SLM to avoid information leakage and maintain different states together is still a troublesome problem.
3.2 Triple-Stream Self-Attention
Based on the Eqn 1 of our proposed SLM, model needs to maintain two states to collect forward and backward contexts for prediction, and we call these two states as the forward stream and backward xxx
P
P
Pxx
P
Pxxx
P P
Pxx
P
P_ _ _
P
P
P_ _
P P
Q
KV forward stream
Q
KV backward stream
Q
KV,KV
Query stream
Input Sequence
Attention (a) x 1 x 2 x 3 x 4 xxxxxxx 1 x 2 x 3 x 4 xxxxxx
P
P P
P
Pquery stream forward stream backward stream (
[Figure]
query the hidden both
The is the attention the forward and the for the query
The cell in means this cannot be stream respectively.
To avoid information leakage, we additionally maintain an individual state for prediction and control it to only capture the dependency from the forward and backward streams.
We name this state as the query stream.
Therefore, we propose a novel Triple-Stream Self-Attention to update each stream, and the detailed design is described as following.
To fulfill our target, we choose Transformer [39] as our basic model, due to its flexibility in capturing global dependency.
Assume the input sequence as { w 1 , w 2 , · · · , w n } and its positions as { p 1 , p 2 , · · · , p n } , where w i and p i represents the embedding of i -th token and its position in the sentence, and n is the token number.
For the query stream, we only use the positional embeddings { p 1 , p 2 , · · · , p n } as the input.
For the forward and backward streams, we also maintain two individual states and both of them use the tokens plus its position ( i.e. , { w 1 + p 1 , w 2 + p 2 , · · · , w n + p n } ) as the input.
For the l -th layer calculation, we denote the forward and backward streams of the position i as ⃗h li and ⃗ h li , and they are updated as:
[Equation]
where Attention( · , · ) refers to the self-attention [39] in Transformer and Q , K , V denote the query, key and value in self-attention.
Hence, ⃗h li collects the information before the position i and itself, and ⃗ h li collects the information after the position i and itself.
For the query stream, we denote q li as its hidden states, and concatenate forward stream ⃗h li and backward stream ⃗ h li of the current layer to be the key/value of query stream.
So q li is updated as:
[Equation]
Here we find that q li is required to only capture the dependency from the forward stream before the position i and the backward stream after the position i , rather than itself.
Due to such design, the query stream is able to capture bidirectional context for estimating token probability and avoid information leakage, which is more effective than CLM in using context.
More importantly, our triple-stream self-attention enables model to predict the probability of all tokens in a sentence within a single forward pass, which demonstrates more efficiency than MLM.
Figure 2 presents the detailed design of our triple-stream self-attention.
We can find that in the query stream, the masked matrix is like a sliding window to control each token to view its previous states in forward stream and its future states in backward stream.
And that is why we name our model as sliding language modeling.
[Table]
3.3 Discussion
To better understand our SLM, we analyze the advantages of our SLM over other LMs.
The comparisons are listed in Table 1.
We select three representative LMs for comparisons, which are
CLM (BERT), MLM (GPT) and bidirectional LM (Bi-LM, used in ELMO [43] 3 ) respectively.
From
Table 1, we have the following observations:
When compared with CLM, our SLM requires 3 × computations.
However, our SLM can fully use the whole sentence information for prediction while CLM can only use unidirectional information.
Even scaling CLM as 3 × parameters, it still can not use bidirectional context for prediction.
This also demonstrates the effectiveness of our SLM in using context.
MLM is powerful at extracting bidirectional context but it needs n × inferences to calculate the whole sentence information limited to its masked prediction.
Our SLM just needs a single inference and uses bidirectional information for prediction with only 3 × computations.
Especially, our SLM shows higher efficiency compared with MLM when n is large.
Bi-LM can also extract forward and backward contextual information, but it just simply concatenates the forward and backward features for the final prediction, without any interactions.
Instead, our SLM can iteratively fuse the bidirectional information thanks to our triple-stream self-attention mechanism.
Overall, the design of SLM is dedicated for sentence scoring, while CLM prefers NLG tasks and
MLM/Bi-LM prefer NLU tasks.
Experiments
In this section, we describe our experimental setup, and the results on NMT and ASR datasets.
4.1 Experimental Setup
We adopt Transformer [39] as the backbone network.
Following previous works [10], we adopt a base setting and a small setting for our model as Transcormer base with 110M parameters and
Transcormer small with 34M parameters, that consists of 12/6 transformer layers and each layer has 768/512 hidden size and 12/8 attention heads.
During the pre-training, we use wikipedia plus bookcorpus (16GB) as the training corpus, to be consistent with previous works [10].
Our model is trained at the sentence-level ( i.e. , one sentence per sample).
We choose Adam [45] as the default optimizer with learning rate of 5 e − 4 , β 1 = 0 .
9 , β 2 = 0 .
98 and ϵ = 1 e − 6 , and weight decay is set as 0.01.
The learning rate warms up over the first 10,000 steps and then linearly decays.
We set the batch size as 8192 tokens per batch, and the training step is 125,000 steps.
We use 32 NVIDIA Tesla 32GB GPUs, with FP16 speedup.
The total training needs 5.5 days for Transcormer base .
For more experimental settings ( e.g. , dataset and its split sizes), please refer to the Appendix.
The code and pre-trained models will be released at: https://github.com/microsoft/CyBERTron-LM/ Transcormer .
[Table]
4.2 Experiments on Neural Machine Translation
We choose IWSLT14 dataset [46], which includes multiple small-scale translation tasks from different languages to English, and WMT14 English-German dataset 4 for evaluation.
We adopt
Transformer [39] as the machine translation model with 6-6 transformer layers to generate multiple candidates for reranking, with a beam size of 10.
The hidden size and attention head are set as 512/1024 and 8/16 for IWSLT and WMT tasks respectively.
During the reranking, we combine the original score produced by the machine translation model and the LM score with a hyper-parameter λ , by following previous experiences [7, 21].
The hyper-parameter λ is tuned on dev set with a range of [0 . 0 , 2 . 0]
[Table]
and then select the best λ to evaluate the test set.
The results are reported in Table 2, in terms of BLEU [47].
Each task will be evaluated by five times based on different pre-trained checkpoints, and report the mean value, with a variance of 0.05.
From Table
[Table]
we have the following observations:
• Our Transcormer base can obtain better performance than CLM and Bi-LM 5 in both small-scale IWSLT and large-scale WMT tasks, which indicates the importance of bidirectional context for sentence scoring, and further validate the ability of our SLM in utilizing bidirectional information.
• When compared with MLM, our Transcormer base also achieves comparable results.
Considering that the computation of MLM for scoring is linear to the input length and needs n inference passes, our SLM show higher efficiency with only 3 × computations in a single pass to maintain three streams.
Overall
[Table]
all of the results reveal that our model is more effective in using contextual information for probability estimation and more efficient with only a single forward pass, especially for long sentences.
[Table] [Table]
O v e r a ll A N A .
A G R A R G .
S T R B
I N D
I N G C T R L .
R A
I
S .
D - N A G R E
L L
I P S
I S F I L L E R G
A P
I R R E G U L
A R
I S L A N D N P
I Q U
A N T
I F
I E R S
S - V A G R
[Table]
4.3 Experiments on Automatic Speech Recognition
We choose LibrSpeech [48] to evaluate the performance of our model for reranking on ASR task.
We train a Conformer model [49] on LibrSpeech, which has 12 encoder layers and 6 decoder layers with 512 hidden size and 8 attention heads, and the beam size is set as 10.
In addition, we use SpecAug [50] as a data augmentation technology to further improve the accuracy of ASR system.
We use word error rate (WER) to evaluate the performance of ASR tasks.
We follow the same tuning technique used in NMT tasks for hyper-parameter λ , but with a larger range as [0 . 0 , 5 . 0] .
The results are reported in Table 3.
From Table 3, we find that our model can give nearly 20% relative improvements over the baseline and also outperform other LMs, including CLM, MLM and Bi-LM.
The results on
ASR task further demonstrates the generalization and effectiveness of our SLM in sentence scoring.
4.4 Experiments on Linguistic Acceptability
Following previous experiences [21], we also conduct experiments on Benchmark of Linguistic
Minimal Pairs (BLiMP) [4], which includes 67K minimal pairs that contrast in grammatical acceptability and isolate specific phenomena in syntax, morphology or semantics.
BLiMP provides an unsupervised setting that uses language models to evaluate sentences and the acceptable sentence can be assigned by a lower log-likelihood score.
We compare our model with GPT-2, BERT and
RoBERTa, and the results are reported in Table 4.
We can find that our Transcormer can easily beat
GPT model in BLiMP, which further validates the effectiveness of our SLM in using bidirectional context for sentence scoring.
And when compared with BERT model, our Transcormer can also match or outperform BERT performance slightly.
Considering that our model only needs a single pass to produce the probability of all tokens, which also manifests the efficiency of our SLM in sentence scoring.
Due to the limitation of resources, our model currently cannot beat RoBERTa since
RoBERTa costs more computations than BERT and thus extract better semantics for prediction.
But we believe our Transcormer can achieve the better performance by training our model with the same computations.
Analyses
In this section, we conduct some method analyses on our proposed SLM and CLM/MLM.
Besides, we also provide more analyses in Appendix.
5.1 Latency Comparisons between MLM and SLM
To better manifest the efficiency of our SLM in rescoring when compared with MLM, we further test the inference latency between SLM and MLM at both GPU and CPU under different lengths of the input sequence.
Specifically, we measure the inference latency of each model at a batch size of 1 to validate the efficiency, and all results are reported in Table 5.
From Table 5, we can find that MLM (BERT) produces very high latency when compared with SLM (Transcormer), especially in CPU devices.
Even using a smaller BERT model, it still cannot avoid the inherent issues ( i.e. , requires n × computations) in MLM for scoring, which needs 20 × and 166 × additional computations over
Transcormer (SLM) in GPU and CPU.
These results also demonstrate the efficiency of our model in using bidirectional context for calculating sentence scores
[Table]
5.2 Comparable Computation between CLM and SLM
As aforementioned, our SLM needs 3 × computations when compared with CLM.
To make a fair comparison, we also pre-train a Transcormer small with 34M parameters in total, which consists 6 transformer layers and each layer has 512 hidden size and 8 attention heads.
Hence, our Transcormer small has approximately 13 parameters of Transcormer base , and has the similar computations as GPT base .
We conduct experiments on three NMT tasks and an ASR task (LibriSpeech dataset) for comparisons and the results are listed in Table 2 and 3 ( i.e. , “Transcormer small row”).
We can find that even under the same computation, our model still outperforms CLM and this result further validates the necessity of using bidirectional context for sentence scoring.
Besides, considering that Transcormer small has fewer parameters, our model is also friendly to the device deployments ( e
[Table]
g. , CPU).
[Table]
5.3 Varying Numbers of Forward Passes in MLM
As mentioned above, MLM needs to forward n passes as each time only mask one token.
So what will happen if we allow each pass to mask more tokens?
Therefore, we design experiments that enforces MLM to forward k tokens for prediction so that it only needs ⌈ n/k ⌉ passes, and investigate the effect of different k .
For a certain k , we randomly split the sentence as ⌈ n/k ⌉ sets, and each time masks one subset for prediction.
The comparisons are listed in Table 6.
We find that using larger k will severely harm the performance, even if it can reduce the number of inference passes.
When k is set as n 3 , which is equal to the cost of our SLM, it can hardly give any improvements over the baseline.
We think that masking more tokens in the sentence will make it more difficult to estimate the token probability at a time.
These comparisons also highlight the efficiency and effectiveness of our SLM for sentence scoring.
5.4 Sentence Scoring Quality at Each Position
passes to predict each
Following previous experiences [21], we count the crossentropy loss of each position in sentences for each LM, to better analyze the effectiveness of bidirectional context to estimate token probability.
Specifically, we sample a subset of sentences as S , and each sentence has the same token number n (here n is 20).
For each position i , we count the average cross-entropy over the sampled subset
S based on the output probability of each LM.
The results are displayed in Figure 3.
We can find that: 1) For CLM, the cross-entropy loss is higher at the first several positions and gradually decreases for the subsequent positions but is still higher than MLM and SLM, which indicates that only using undirectional information is not enough to measure the sentence score precisely.
2) SLM can almost obtain similar loss as MLM at each position.
Considering SLM just needs a single pass while MLM needs n passes, this phenomenon further validates the superiority and efficiency of SLM in scoring sentences.
Conclusion
In this paper, we propose Transcormer, a Transformer with a novel sliding language modeling for sentence scoring.
Specifically, our SLM is able to produce the probability of each token over the whole sentence within a single forward pass, and utilizes bidirectional context for prediction, and thus inherents the advantages of CLM and MLM and avoids their deficiencies.
To the best of our knowledge, the proposed Transcormer is the first pre-trained language model tailored for sentence scoring.
Experimental results on multiple datasets demonstrate the effectiveness of our Transcormer in computing sentence score for reranking tasks.
Besides, we summarize some potential directions of our Transcormer and SLM as the future works:
• Currently our Transcormer is only conducted on the English domain under the base setting, due to limited computation.
We expect to develop large-scale Transcormer and use different language domains or multilingual data for training in the future.
• We design sliding language modeling for sentence scoring, and our experiments are mainly on reranking task.
However, based on the characteristics of our SLM, we believe our model can also be used for other scenarios ( e.g. , error correction [51, 52, 53], data selection), and we will explore the specific fine-tuning techniques when applying our SLM on different downstream tasks.
• Besides, our Transcormer mainly pre-trains SLM on an encoder framework.
However, our SLM is not limited to model structure.
For example, SLM can be easily extended to encoder-decoder framework [18] based on paired data.
Therefore, we also expect to explore the possibility of using SLM on different frameworks.
• Although our paper mainly focuses on text data, we want to highlight that SLM can also be extended to other different modalities with sequential characteristic ( e.g. , image, speech [54] and time series data).
Consequently, how to apply SLM to other modalities is also a valuable topic in the future. Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] (c) Did you discuss any potential negative societal impacts of your work?
[N/A] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2. If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results? [N/A] (b) Did you include complete proofs of all theoretical results? [N/A] 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] (b) Did you specify all the training details (e
g., data splits, hyperparameters, how they were chosen)?
[Yes] (c) Did you report error bars (e
g., with respect to the random seed after running experiments multiple times)?
[Yes] Please see Section refsubsec:nmt and Appendix.
(d) Did you include the total amount of compute and the type of resources used (e
g., type of GPUs, internal cluster, or cloud provider)?
[Yes] Please see Section 4. 4.
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] (b) Did you mention the license of the assets?
[Yes] (c) Did you include any new assets either in the supplemental material or as a URL?
[N/A] (d) Did you discuss whether
and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
A Appendix A.1 Experimental Setup A.1.1 Datasets
IWSLT 2014 is the evaluation campaign of the 11th International Workshop on Spoken Language
Translation.
It consist of a lot of small-scale translation tasks collected from TED talks, including
German (De), Spanish (Es), Italian (It), Dutch (NL), Polish (PL), Romanian (Ro), Russian (Ru),
Turkish (Tr) to English.
We randomly split each dataset as the training set and dev set with a ratio of [Table] It
Nl [Table] And each task concatenates TED.tst2010, TED.tst2011, TED.dev2010 and TED.tst2012 as the test set.
The statistics of each sub-task is described as:
WMT14 English-German comprises 4.5M bilingual data collected from Europarl v7, Common
Crawl corpus and News Commentary.
We concatenate newstest2012 and newstest2013 as the valid set, and choose newstest2014 as the test set for WMT14 English-German.
Our experiments mainly focuses on German → English.
LibriSpeech [48] includes 1000hr speech data, sampled at 16k Hz.
LibriSpeech includes four subsets for evaluation, which are dev-clean, dev-other, test-clean and test-other.
[Table]
A.1.2 Hyper-parameter Setup
The pre-training hyper-parameters of Transcormer are described in Table 8.
A.2 More Details about Related Works
As mentioned in Section 2.1, some works [22, 21, 34, 35] tried to alleviate the efficient problem in
MLM model caused by N-passes.
Specifically, [22] proposed to calculate pseudo log-likelihood score via stochastic estimation, that is randomly sampling K tokens and computing the probability of these
K tokens via masked prediction as the final sentence probability.
It can reduce the time complexity from O ( | x | ) to O ( K ) but will harm model performance.
[21] also suggested a distillation strategy to cover this problem, that requires model to compute sentence score via N-passes first ( i.e. , teacher model) and then distills it to the output vector of the [ CLS ] token ( i.e. , student model) during the pre-training.
However, this paradigm also will under-perform regular LMs in their experiments [21].
[35] designed a model named DLM to produce token-wise probability via a single inference pass.
To fulfill this target, DLM only feeds word embeddings as the key/value for each Transformer layer, rather than the previous layer.
Such design allows the query stream to capture the whole sentence information but without any contextualized semantics.
Electric [34] is a model built upon a Two-Cloze Tower [36], based on noise contrastive estimation.
More in details, Electric trained a left-to-right Transformer and a right-to-left Transformer and then concatenated together at the final layer to predict each token.
Just as discussed in Section 3.3, this model learns forward and backward context individually and only fuse semantics at the final layer while the query stream of our SLM is able to fuse bidirectional context iteratively.
Overall, our SLM can make full use of bidirectional information over all Transformer layers and predict token-wise probabilities simultaneously.
Besides, there also remain some works [37, 38] which use discriminative language modeling to approximately estimate sentence scores.
[37] borrowed the idea of the energy-based model into
Dev
Test
Model
Domain clean other clean other
[Table]
[Table]
Electric [34]
[Table] [Table] [Table] [Table]
sentence reranking.
[38] proposed a discriminative language model that minimizes the KL-divergence between the target distribution and the output distribution.
These methods can be considered as the discriminative language modeling which directly predicts a single value to be the sentence score. Discriminative language model usually needs target datasets for fine-tuning, while our proposed language model is independent to downstream tasks. We think discriminative language models are complementary to our works and we leave this combination as the future works. A.3 Results A.3.1 Comparison with other works
As aforementioned, previous works [35, 34] have tried some strategies to calculate the probabilities of all tokens simultaneously to avoid the limitations in N-passes.
To validate the advantages of our model in using bidirectional context, we also conduct experiments to make a comparison with these methods.
For the sake of fairness, all of our experiments are deployed on the same datasets used in [23] and the results are shown in Table 9.
From Table 9, we find that our Transcormer can outperform these models.
These improvements also demonstrate the superiority of our model in utilizing bidirectional context to predict sentence probability.
A.4 Accelerating SLM Training
Although SLM is more advantageous than CLM in using bidirectional context and demonstrates higher efficiency than MLM for sentence scoring, directly training SLM from scratch is still timeconsuming since it needs 3 × computations to maintain query and forward/backward streams.
So is it possible to accelerate the training of SLM?
We note that both SLM and MLM adopt the masked token plus its position to predict its target based on the context while the main difference in context is that
MLM adopts one bidirectional context and SLM adopts forward and backward contexts.
We think that MLM-based model with bidirectional context should also be able to produce good unidirectional context and SLM does not modify model structure, so that we can use the MLM-based model (e.g.,
BERT) for initialization to accelerate SLM training.
Specifically, we continue to train BERT model with addition 5 epochs (nearly 20K steps) and the results can be found in Table 4 and Table 9.
We can find that only needs 5 epochs, our Transcormer with BERT model initialization can match the performance that is trained from scratch, which means using BERT initialization can accelerate our
SLM training.
A.5 Analyses A.5.1 Pre-training Strategy
In our experiment setup, we use sentence-level data as the input for pre-training.
To better analyze the effect of using different data processing for pre-training, we conduct experiments by using stream-level data (concatenate multiple sentences as a fixed-length, e.g. , 512) to make a comparison.
We apply two strategies on NMT & ASR tasks, and then evaluate the average (top-1) accuracy of each token in a sentences with different lengths based on the output probability for SLM.
The results are reported in Table 10.
We can find that using stream-level data for pre-training can not achieve good accuracy when sentence length is too short.
We guess that is because using stream-level data causes model can not fit short sentence since it always pre-trains under the longer sentences ( i.e. , 512).
Considering that our downstream scenarios mainly consist of single sentence, which is usually too short, directly using stream-level data for pre-training can not achieve promising performance.
As a result, we recommend to use sentence-level data for pre-training, and we also expect to explore more effective pre-training strategies in the future.
[Table]
A.5.2 Domain Adaption
Following previous experiences [21], we also study the effect of using in-domain data for pre-training.
For NMT tasks, we randomly sample 20GB monolingual data from NewsCrawl data to build the pre-training corpus for pre-training.
And for ASR tasks, as LibriSpeech includes 4GB in-domain data, we direct use this data as our pre-training corpus to handle ASR tasks.
The results of NMT and ASR tasks are reported in Table 11 and Table 12.
We can find that using in-domain data for pre-training is useful to improve the downstream tasks
[Table]
[Table]
[Table]
The sentiment of [ Review ] is [ GT Label ] . [Table]
[ Review ] [Table] [Table]
Label ]
[Table]
[ Review ] is a [ GT Label ] sentiment.
[Table]
As aforementioned in Section 3.3, both SLM and MLM can learn bidirectional context to predict token-wise probability, so why we cannot directly use SLM for language understanding tasks?
First, we want to highlight that although the query stream is able to enjoy completely bidirectional context like MLM due to the design of the triple-stream self-attention mechanism, there still remain some differences between MLM and SLM.
First, the query and content streams in the MLM-based model are shared together to enjoy the benefits of the bidirectional context, while our SLM maintains query and forward/backward streams individually.
In other words, the content in MLM can learn bidirectional context while SLM is used to collect forward and backward context.
Considering previous state-of-the-art works [10, 13, 16, 14] have proven that BERT-style models can be fine-tuned to obtain superior performance in NLU tasks due to the benefits of bidirectional context.
We think that MLM prefers NLU tasks while SLM is more suitable for scoring.
In our internal experiments, we have tried to directly fine-tune SLM on SST-2 [55] like BERT ( i.e. , using bidirectional context).
Due to the mismatch ( i.e. , forward/backward context v.s. bidirectional context) between pre-training and fine-tuning, our SLM only achieves 91.9% accuracy while the standard BERT can obtain 92.8%.
This phenomenon may also validate our hypothesis that SLM is not the optimal method for solving NLU tasks in a BERT-style fine-tuning paradigm.
However, there still remain some alternative possibilities for SLM to adapt NLU scenarios.
A possible solution is that converts language understanding tasks into ranking classification tasks.
For example, assuming the task is sentiment classification and we have a review sentence, so we can construct two sentences: “the sentiment of [review] is positive" and “the sentiment of [review] is negative".
Based on the constructed sentences, we can use our model on each sentence to calculate their sentence scores and the sentence with ground truth should have a lower log-probability score.
It can be considered as a variant of prompt-based learning [12, 56].
We have simply built two different patterns to formulate sentiment classification, and conduct experiments on SST-2 in a zero-shot setting ( i.e. , without fine-tuning).
The results are shown in Table 13.
We find that using patterns like “ [ Review ] is a [ GT Label ] sentiment.”
can obtain an accuracy of 71.0% on SST-2 in a zero-shot setting (BERT fine-tuning cannot be used in a zero-shot scenario).
These experiments also demonstrate that we can transform NLU tasks into ranking tasks and then apply our model for ranking, which also indicates the potential of our model in solving NLU or other NLP tasks.
We will also continue to explore more techniques to refine this paradigm to improve performance in the future.