Transcormer: Transformer for Sentence Scoring with Sliding Language Modeling
Anonymous Author
AffiliationAddress
Abstract
Sentence scoring aims at measuring the likelihood score of a sentence and is widelyused in natural language processing scenarios, like reranking, which is to select thebest sentence from multiple candidates.
Previous works on sentence scoring mainlyadopted either causal language modeling (CLM) like GPT or masked languagemodeling (MLM) like BERT, which have some limitations: 1) CLM only utilizesunidirectional information for the probability estimation of a sentence withoutconsidering bidirectional context, which affects the scoring quality; 2) MLM canonly estimate the probability of partial tokens at a time and thus requires multipleforward passes to estimate the probability of the whole sentence, which incurslarge computation and time cost.
In this paper, we propose Transcormer – a
Transformer model with a novel sliding language modeling (SLM) for sentencescoring.
Specifically, our SLM adopts a triple-stream self-attention mechanismto estimate the probability of all tokens in a sentence with bidirectional contextand only requires a single forward pass.
SLM can avoid the limitations of CLM(only unidirectional context) and MLM (multiple forward passes) and inherittheir advantages, and thus achieve high effectiveness and efficiency in scoring.
Experimental results on multiple tasks demonstrate that our method achieves betterperformance than other language models.
Introduction
Sentence scoring is to measure the log-likelihood score of a sentence via language model, so thatit can be used to represent the goodness of a sentence.
Specifically, a good sentence should have arelatively lower log-likelihood score while a bad sentence should have a higher one.
Due to suchnature, sentence scoring has been widely used in many natural language processing (NLP) scenarios.
For instance, it can be used to rerank candidates in neural machine translation (NMT) or automaticspeech recognition (ASR) tasks [1, 2] or evaluate sentences in linguistic acceptability [3].
Therefore,how to design effective language models to calculate sentence scores efficiently is very important.
Recently, neural network based language models (LM) [4, 5, 6, 7] have been considered as the most widely used technique for sentence scoring, since they can produce density estimation of the whole sentence by computing the probability of each token and summing up their values as the sentence score.
Specifically, causal language modeling (CLM) [8] and masked language modeling (MLM) [9] are the most representative LMs.
For a given sentence x = { x 1 , · · · , x | x | } , where x i is the i -thtoken of x .
CLM can predict next token conditioned on unidirectional context and its objective is tooptimize (cid:80) | x | i =1 log P ( x i | x <i ) .
Hence, CLM is usually used for solving natural language generation
(NLG) tasks [8, 10, 11].
While for MLM, it replaces a subset of tokens x S in x as special symbol[MASK] and then predicts the masked tokens based on the corrupted sequence x \S .
The objectiveof MLM is to optimize (cid:80) |S| i =1 log P ( y S i | y \S ) , so it is able to learn bidirectional context and can beused for solving natural language understanding (NLU) tasks [9, 12, 13, 14].
Since CLM and MLMcan be learned in an unsupervised fashion, many works [9, 12, 13, 8, 10, 11, 15, 16, 17, 14, 18, 19]have pre-trained these LMs on large-scale corpus to extract powerful linguistic representations, andthese models can be directly used out of the box to predict the probabilities of tokens.
Inspired by thesuccesses of LMs, some works [20, 21, 22, 23, 24] have tried to use pre-trained CLM or MLM togenerate sentence scores on NMT or ASR reranking tasks and achieved some promising results.
However, we notice that both CLM and MLM still suffer from some deficiencies in calculatingsentence scores.
For instance, some works [24] utilized GPT-style model for ASR reranking withinsingle inference, yet GPT model can only extract unidirectional information due to the limitationof CLM, without considering the whole sentence semantics, and thus affect the sentence score.
Toutilize bidirectional context, some works [21, 22, 23] applied BERT model for rescoring.
However,the nature of MLM is to mask some tokens in the sentence for prediction, which means it requires the
BERT model to forward multiple times and each forward pass only masks one token for prediction.
As a result, it is time-consuming to directly adopt MLM for scoring sentence.
In summary, we findthat for calculating sentence scores, CLM needs one-pass inference but only uses unidirectionalinformation and MLM is costly in computing sentence score although it can use bidirectional context.
Therefore, a natural question arises: is it possible to design a language model to use bidirectionalcontext for sentence scoring with only one inference pass.
In this paper, we introduce Transcormer , a Transformer model designed for sentence scoring.
Morespecifically, our Transcormer leverages a novel language modeling scheme, named as sliding language modeling (SLM), that produces the probability of all tokens within single inference passand simultaneously utilizes bidirectional context.
To fulfill this target, we innovatively design atriple-stream self-attention mechanism, which consists of two content streams (a forward streamand a backward stream) and one query stream.
By employing specifically-designed mask strategieson the attention matrix, our method allows each token in the query stream to leverage all tokeninformation except itself (i.e., the tokens before and after it) for estimating its probability to avoidany information leakage.
To the best of our knowledge, SLM is the first language modeling tailoredfor sentence scoring.
We pre-train our SLM on large-scale corpus, and then evaluate it on multipledatasets.
Experimental results demonstrate that Transcormer outperforms the baselines by up to +0.8/0.6 BLEU score on small/large-scale NMT tasks, ∼ 20% relative improvements on ASR tasks.
The main contributions of this work are summarized as follows:
• We analyze the pros and cons of CLM and MLM when using them for scoring sentences, andpropose Transcormer with a new sliding language modeling, which uses bidirectional context for
probability estimation within a single pass.
• We introduce a novel triple-stream self-attention mechanism in SLM, which has two contentstreams to collect forward/backward semantics, and a query stream to estimate the probability ofeach token in a sentence.• Experimental results on multiple datasets demonstrate the effectiveness and efficiency of our SLMfor sentence scoring.
Background2.1 Sentence Scoring
Sentence scoring has a long history in NLP applications, especially in reranking tasks ( e.g. , rerankingfor machine translation [25, 26, 27, 28] or speech recognition [29]).
Generally, given n -best candidates generated by text generation models, reranking aims at scoring each candidate to select thebest answer.
Early works [25, 26, 27, 28, 30, 31] mainly used statistical LM or combined it with
RNN-based LM to calculate the sentence scores.
Recently, end-to-end neural network based LMhas became the de facto approach for scoring and has been applied in many NLP tasks [5, 4, 32].
Specifically, causal language modeling (CLM) and masked language modeling (MLM) are the mostrepresentative language modeling methods, among which GPT [8, 10, 11] and BERT [9] are the mostfamous examples, respectively.
As aforementioned, CLM conditions on the previous states to predictnext token so that it can obtain the probability of each token of the sentence in a single pass, but canonly capture unidirectional information.
For MLM, it is able to use bidirectional context for predictionbut the masked-prediction mechanism limits MLM to producing the probability of all tokens withinone forward pass (since MLM can only provide the probability of masked tokens).
To calculate thesentence score, a kind of solutions [21, 22, 23] is to forward multiple times and only mask one tokeneach time.
Therefore, the MLM for sentence scoring is formulated as: (cid:80) | x | i =1 log P ( x i | x \ x i ) .
We canfind that the cost of MLM for scoring needs | x | inference passes, which is too time-consuming.
Some recent works [21, 20, 33, 34] have been proposed to alleviate these issues in MLM.
Wang et al.[21] and Salazar et al.
[20] attempted to use stochastic estimation or distillation to avoid N-passesproblem to approximately estimate the probability of each toke produced by MLM, with a sacrificeof performance.
Clark et al.
[33] adopted a two-cloze tower [35] with noise-contrastive estimationto provide sentence probability, and Shin et al.
[34] only considered word embedding as the inputsof key and value in transformer without any interaction.
Besides, some works [36, 37] adopteddiscriminative language modeling to approximately estimate sentence scores based on the paired data.
More discussions about related works can be found in Appendix A.2.
Therefore, how to calculatesentence scores efficiently with bidirectional context is the main challenge in MLM.
Overall, CLM only needs a single forward pass to estimate the probability of all tokens but cannotextract bidirectional context, while MLM leverages bidirectional information but needs multipleinference passes.
Consequently, we raise a natural question: is it possible to design a pre-trained language model to support all token prediction in a single pass and simultaneously leverage bidirectionalinformation?
This is exactly the motivation of our method.
2.2 Multiple-Stream Self-Attention
The pioneer of multiple-stream self-attention is XLNet [15], which introduces two-streams selfattention to incorporate autoregressive pre-training for language understanding, which pre-trains
Transformer [38] via using a content stream and a query stream.
In details, for the t -th step, thecontent stream is able to capture the dependency from the tokens before the t -th step and itself ( i.e. ,x ≤ t ), while the query stream is only allowed to view the tokens before the t -th step ( i.e. , x <t ) to avoidinformation leakage.
Besides, there are some other variants of two-stream self-attention [16, 39, 40],which are designed for solving NLU tasks.
We observe that these works mainly used a single contentstream and then used one or many query streams to predict more information.
Different from theseworks, we introduce a triple-stream self-attention mechanism, which enables query stream to leveragetwo content streams for prediction, and thus enjoys the benefits of additional bidirectional context forestimating token probability.
Transcormer
To inherent the advantages of CLM and MLM for sentence scoring and avoid their limitations, wepropose Transcormer – a Transformer model with a novel sliding language modeling for sentencescoring.
First, we summarize that an ideal language modeling for sentence scoring should satisfytwo requirements: 1) model should be able to use bidirectional context for effective probabilityestimation of each token; 2) model should produce the probability of all tokens in a sentence withina single inference pass for efficiency.
Therefore, to fulfill these two requirements, we formulate anew language modeling, named as sliding language modeling (SLM), and describe it in Section 3.1.
In SLM, we propose a Triple-Stream Self-Attention mechanism based on Transformer (please see
Section 3.2 for details) to use bidirectional context for each token prediction and avoid information x
[Figure]
xxx
P P
Pxx
P
P_ _ _
P
P
P_ _
P
Pxxx
P P
Pxx
P P
Transcormer
[Figure]
leakage.
We also discuss the differences between SLM and other LMs in Section 3.3.
Figure 1presents the pipeline of our Transcormer for sentence scoring with SLM.
3.1 Sliding Language Modeling
Considering the pros and cons of CLM and MLM for scoring, we notice: 1) CLM can produceprobability of all tokens within one forward pass, and thus obtain the unidirectional information ofthe whole sentence; 2) MLM for sentence scoring needs multiple inference passes, so that manycontext has been repeatedly calculated and cause a waste of computation.
So, is it possible to reusetoken information to build bidirectional context for token prediction?
Therefore, we propose sliding language modeling (SLM) to address the inherent flaws in previous
LMs ( i.e. , CLM and MLM) for sentence scoring.
Specifically, we first maintain two individualstreams to collect forward (left-to-right) context and backward (right-to-left) context.
And for eachtoken prediction, we decompose the sentence information as the past tokens (the tokens before it)and future tokens (the tokens after it) respectively.
As a result, our SLM enforces each token to onlycapture the dependency from its past tokens and its future tokens concurrently, so that each token canutilize the whole sentence information (except itself) to estimate token probability.
The objectivefunction of SLM is formulated as:
[Equation]
where x <i and x >i respectively correspond to the tokens before the i -th token and after the i -th token,and θ represents the parameters of SLM.
Thanks to such design, our SLM can utilize bidirectionalcontext to produce the probability of each token within one forward pass, and thus satisfy the aboverequirements for sentence scoring.
However, previous experiences [15, 41] pointed out that the stateswith bidirectional information will cause information leakage when propagating to the next layer 1 .
Therefore, how to implement SLM to avoid information leakage and maintain different states togetheris still a troublesome problem.
xxx
P P
Pxx
P
Pxxx
P P
Pxx
P
P_ _ _
P
P
P_ _
P P
Q
KV forward stream
Q
KV backward stream
Q
KV,KV
Query stream
Input Sequence
Attention (a) x 1 x 2 x 3 x 4 xxxxxxx 1 x 2 x 3 x 4 xxxxxx
P
P P
P
Pquery stream forward stream backward stream (
[Figure]
3.2 Triple-Stream Self-Attention
Based on the Eqn 1 of our proposed SLM, model needs to maintain two states to collect forward andbackward contexts for prediction, and we call these two states as the forward stream and backwardstream respectively.
To avoid information leakage, we additionally maintain an individual state forprediction and control it to only capture the dependency from the forward and backward streams.
Wename this state as the query stream.
Therefore, we propose a novel Triple-Stream Self-Attention toupdate each stream, and the detailed design is described as following.
To fulfill our target, we choose Transformer [38] as our basic model, due to its flexibility in capturing global dependency.
Assume the input sequence as { w 1 , w 2 , · · · , w n } and its positions as{ p 1 , p 2 , · · · , p n } , where w i and p i represents the embedding of i -th token and its position in thesentence, and n is the token number.
For the query stream, we only use the positional embeddings{ p 1 , p 2 , · · · , p n } as the input.
For the forward and backward streams, we also maintain two individualstates and both of them use the tokens plus its position ( i.e. , { w 1 + p 1 , w 2 + p 2 , · · · , w n + p n } ) asthe input.
For the l -th layer calculation, we denote the forward and backward streams of the positioni as ⃗h li and ⃗ h li , and they are updated as:
[Equation]
where Attention( · , · ) refers to the self-attention [38] in Transformer and Q , K , V denote the query,key and value in self-attention.
Hence, ⃗h li collects the information before the position i and itself, and⃗ h li collects the information after the position i and itself.
For the query stream, we denote q li as itshidden states, and concatenate forward stream ⃗h li and backward stream ⃗ h li of the current layer to bethe key/value of query stream.
So q li is updated as:
[Equation]
Here we find that q li is required to only capture the dependency from the forward stream beforethe position i and the backward stream after the position i , rather than itself.
Due to such design,the query stream is able to capture bidirectional context for estimating token probability and avoidinformation leakage, which is more effective than CLM in using context.
More importantly, our
[Table]
triple-stream self-attention enables model to predict the probability of all tokens in a sentence withina single forward pass, which demonstrates more efficiency than MLM.
Figure 2 presents the detaileddesign of our triple-stream self-attention.
We can find that in the query stream, the masked matrixis like a sliding window to control each token to view its previous states in forward stream and itsfuture states in backward stream.
And that is why we name our model as sliding language modeling.
3.3 Discussion
To better understand our SLM, we analyze the advantages of our SLM over other LMs.
Thecomparisons are listed in Table 1.
We select three representative LMs for comparisons, which are
CLM (BERT), MLM (GPT) and bidirectional LM (Bi-LM, used in ELMO [42] 2 ) respectively.
From
Table 1, we have the following observations:
When compared with CLM, our SLM requires 3 × computations.
However, our SLM can fully use
the whole sentence information for prediction while CLM can only use unidirectional information.
Even scaling CLM as 3 × parameters, it still can not use bidirectional context for prediction.
This
also demonstrates the effectiveness of our SLM in using context.
MLM is powerful at extracting bidirectional context but it needs n × inferences to calculate
the whole sentence information limited to its masked prediction.
Our SLM just needs a singleinference and uses bidirectional information for prediction with only 3 × computations.
Especially,our SLM shows higher efficiency compared with MLM when n is large.
Bi-LM can also extract forward and backward contextual information, but it just simply concatenates the forward and backward features for the final prediction, without any interactions.
Instead,our SLM can iteratively fuse the bidirectional information thanks to our triple-stream self-attentionmechanism.
Overall, the design of SLM is dedicated for sentence scoring, while CLM prefers NLG tasks and
MLM/Bi-LM prefer NLU tasks.
Experiments
In this section, we describe our experimental setup, and the results on NMT and ASR datasets.
4.1 Experimental Setup
We adopt Transformer [38] as the backbone network.
Following previous works [9], we adopta base setting and a small setting for our model as Transcormer base with 110M parameters and
Transcormer small with 34M parameters, that consists of 12/6 transformer layers and each layerhas 768/512 hidden size and 12/8 attention heads.
During the pre-training, we use wikipedia plusbookcorpus (16GB) as the training corpus, to be consistent with previous works [9].
Our model is
[Table]
trained at the sentence-level ( i.e. , one sentence per sample).
We choose Adam [44] as the defaultoptimizer with learning rate of 5 e − 4 , β 1 = 0 .
9 , β 2 = 0 .
98 and ϵ = 1 e − 6 , and weight decay is setas 0.01.
The learning rate warms up over the first 10,000 steps and then linearly decays.
We set thebatch size as 8192 tokens per batch, and the training step is 125,000 steps.
We use 32 NVIDIA
Tesla32GB GPUs, with FP16 speedup.
The total training needs 5.5 days for Transcormer base .
For moreexperimental settings ( e.g. , dataset and its split sizes), please refer to the Appendix.
The code andpre-trained models will be released later.
4.2 Experiments on Neural Machine Translation
We choose IWSLT14 dataset [45], which includes multiple small-scale translation tasks from different languages to English, and WMT14 English-German dataset 3 for evaluation.
We adopt
Transformer [38] as the machine translation model with 6-6 transformer layers to generate multiplecandidates for reranking, with a beam size of 10.
The hidden size and attention head are set as512/1024 and 8/16 for IWSLT and WMT tasks respectively.
During the reranking, we combine theoriginal score produced by the machine translation model and the LM score with a hyper-parameterλ , by following previous experiences [6, 20].
The hyper-parameter λ is tuned on dev set with a rangeof [0 . 0 , 2 . 0] , and then select the best λ to evaluate the test set.
The results are reported in Table 2,in terms of BLEU [46].
Each task will be evaluated by five times based on different pre-trainedcheckpoints, and report the mean value, with a variance of 0.05.
From Table 2, we have the followingobservations
• Our Transcormer base can obtain better performance than CLM and Bi-LM 4 in both small-scaleIWSLT and large-scale WMT tasks, which indicates the importance of bidirectional context forsentence scoring, and further validate the ability of our SLM in utilizing bidirectional information.• When compared with MLM, our Transcormer base also achieves comparable results.
Consideringthat the computation of MLM for scoring is linear to the input length and needs n inference passes,our SLM show higher efficiency with only 3 × computations in a single pass to maintain threestreams.
Overall, all of the results reveal that our model is more effective in using contextual informationfor probability estimation and more efficient with only a single forward pass, especially for longsentences.
4.3 Experiments on Automatic Speech Recognition
We choose LibrSpeech [47] to evaluate the performance of our model for reranking on ASR task
We
[Table]
512 hidden size and 8 attention heads, and the beam size is set as 10.
In addition, we use SpecAug[49] as a data augmentation technology to further improve the accuracy of ASR system.
We use worderror rate (WER) to evaluate the performance of ASR tasks.
We follow the same tuning techniqueused in NMT tasks for hyper-parameter λ , but with a larger range as [0 . 0 , 5 . 0] .
The results arereported in Table 3.
From Table 3, we find that our model can give nearly 20% relative improvementsover the baseline and also outperform other LMs, including CLM, MLM and Bi-LM.
The results on
ASR task further demonstrates the generalization and effectiveness of our SLM in sentence scoring.
Analyses
In this section, we conduct some method analyses on our proposed SLM and CLM/MLM.
Besides,we also provide more analyses in Appendix.
5.1 Comparable Computation between CLM and SLM
As aforementioned, our SLM needs 3 × computations when compared with CLM.
To make a fair comparison, we also pre-train a Transcormer small with 34M parameters in total, which consists 6 transformer layers and each layer has 512 hidden size and 8 attention heads.
Hence, our Transcormer smallhas approximately 13 parameters of Transcormer base , and has the similar computations as GPT base .
We conduct experiments on three NMT tasks and an ASR task (LibriSpeech dataset) for comparisonsand the results are listed in Table 2 and
( i
[Table]
e. , “Transcormer small row”).
We can find that even underthe same computation, our model still outperforms CLM and this result further validates the necessityof using bidirectional context for sentence scoring.
Besides, considering that Transcormer small hasfewer parameters, our model is also friendly to the device deployments ( e
[Table]
g. , CPU).
[Table]
5.2 Varying Numbers of Forward Passes in MLM
As mentioned above, MLM needs to forward n passes as each time only mask one token.
So whatwill happen if we allow each pass to mask more tokens?
Therefore, we design experiments thatenforces MLM to forward k tokens for prediction so that it only needs ⌈ n/k ⌉ passes, and investigate the effect of different k .
For a certain k , we randomly split the sentence as ⌈ n/k ⌉ sets, and each timemasks one subset for prediction.
The comparisons are listed in Table 4.
We find that using largerk will severely harm the performance, even if it can reduce the number of inference passes.
Whenk is set as n 3 , which is equal to the cost of our SLM, it can hardly give any improvements over thebaseline.
We think that masking more tokens in the sentence will make it more difficult to estimatethe token probability at a time.
These comparisons also highlight the efficiency and effectiveness ofour SLM for sentence scoring.
5.3 Sentence Scoring Quality at Each Position
Following previous experiences [20], we count the crossentropy loss of each position in sentences for each LM,to better analyze the effectiveness of bidirectional contextto estimate token probability.
Specifically, we sample asubset of sentences as S , and each sentence has the sametoken number n (here n is 20).
For each position i , wecount the average cross-entropy over the sampled subset
S based on the output probability of each LM.
The resultsare displayed in Figure 3.
We can find that: 1) For CLM,the cross-entropy loss is higher at the first several positionsand gradually decreases for the subsequent positions but isstill higher than MLM and SLM, which indicates that onlyusing undirectional information is not enough to measurethe sentence score precisely.
2) SLM can almost obtainsimilar loss as MLM at each position.
Considering SLMjust needs a single pass while MLM needs n passes, thisphenomenon further validates the superiority and efficiency of SLM in scoring sentences.
Conclusion
In this paper, we propose Transcormer, a Transformer with a novel sliding language modeling forsentence scoring.
Specifically, our SLM is able to produce the probability of each token over thewhole sentence within a single forward pass, and utilizes bidirectional context for prediction, andthus inherents the advantages of CLM
and MLM and
avoids
their deficiencies.
To the best of ourknowledge, the
proposed Transcormer
is the first pre-
trained language
model tailored for sentencescoring.
Experimental results on multiple datasets demonstrate the effectiveness of our Transcormerin computing
sentence score for reranking tasks
Besides we summarize some potential directions of our Transcormer and SLM as the future works:
• Currently our Transcormer is only conducted on the English domain under the base setting, due tolimited computation.
We expect to develop large-scale Transcormer and use different languagedomains or multilingual data for training in the future.• We design sliding language modeling for sentence scoring, and our experiments are mainly onreranking task.
However, based on the characteristics of our SLM, we believe our model can alsobe used for other scenarios ( e.g. , error correction [50, 51, 52], data selection), and we will explorethe specific fine-tuning techniques when applying our SLM on different downstream tasks.• Besides, our Transcormer mainly pre-trains SLM on an encoder framework.
However, our SLMis not limited to model structure.
For example, SLM can be easily extended to encoder-decoderframework [17] based on paired data.
Therefore, we also expect to explore the possibility of usingSLM on different frameworks.• Although our paper mainly focuses on text data, we want to highlight that SLM can also be extendedto other different modalities with sequential characteristic ( e.g. , image, speech and time series data).Consequently, how to apply SLM to other modalities is also a valuable topic in the future.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?
[Yes](b) Did you describe the limitations of your
work?
[Yes]
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A](b)
Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters,
how they
were chosen)?
[Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experi-
ments multiple times)?
[Yes] Please see Section refsubsec:nmt and Appendix.
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] Please see Section 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[Yes]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
d) Did you discuss whether and how consent was obtained from
people
whose data you’re
using/curating?
[N
/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]
A Appendix
A.1 Experimental Setup
A.1.1 Datasets
IWSLT 2014 is the evaluation campaign of the 11th International Workshop on Spoken Language
Translation.
It consist of a lot of small-scale translation tasks collected from TED talks, including
German (De), Spanish (Es), Italian (It), Dutch (NL), Polish (PL), Romanian (Ro), Russian (Ru),
Turkish (Tr) to English.
We randomly split each dataset as the training set and dev set with a ratio of25:1.
And each task concatenates TED.tst2010, TED.tst2011, TED.dev2010 and TED.tst2012 as thetest set.
The statistics of each sub-task is described as:
[Table]
datasets.
WMT14 English-German comprises 4.5M bilingual data collected from Europarl v7, Common
Crawl corpus and News Commentary.
We concatenate newstest2012 and newstest2013 as the validset, and choose newstest2014 as the test set for WMT14 English-German.
Our experiments mainlyfocuses on German → English.
LibriSpeech [47] includes 1000hr speech data, sampled at 16k Hz.
LibriSpeech includes four subsetsfor evaluation, which are dev-clean, dev-other, test-clean and test-other.
[Table]
[Table]
O v e r a ll A N A .
A G R A R G .
S T R B
I N D
I N G C T R L .
R A
I
S .
D - N A G R E
L L
I P S
I S F I L L E R G
A P
I R R E G U L
A R
I S L A N D N P
I Q U
A N T
I F
I E R S
S - V A G R
[Table]
A.1.2 Hyper-parameter Setup
The
pre-training hyper-parameters of Transcormer are described in Table 6.
A.2 More Details about Related Works
As mentioned in Section 2.1, some works [21, 20, 33, 34] tried to alleviate the efficient problem in
MLM model caused by N-passes.
Specifically, [21] proposed to calculate pseudo log-likelihood scorevia stochastic estimation, that is randomly sampling K tokens and computing the probability of these
K tokens via masked prediction as the final sentence probability.
It can reduce the time complexityfrom O ( | x | ) to O ( K ) but will harm model performance.
[20] also suggested a distillation strategyto cover this problem, that requires model to compute sentence score via N-passes first ( i.e. , teachermodel) and then distills it to the output vector of the [ CLS ] token ( i.e. , student model) during thepre-training.
However, this paradigm also will under-perform regular LMs in their experiments [20].[34] designed a model named DLM to produce token-wise probability via a single inference pass.
To fulfill this target, DLM only feeds word embeddings as the key/value for each Transformerlayer, rather than the previous layer.
Such design allows the query stream to capture the wholesentence information but without any contextualized semantics.
Electric [33] is a model built upona Two-Cloze Tower [35], based on noise contrastive estimation.
More in details, Electric trained aleft-to-right Transformer and a right-to-left Transformer and then concatenated together at the finallayer to predict each token.
Just as discussed in Section 3.3, this model learns forward and backwardcontext individually and only fuse semantics at the final layer while the query stream of our SLM isable to fuse bidirectional context iteratively.
Overall, our SLM can make full use of bidirectionalinformation over all Transformer layers and predict token-wise probabilities simultaneously.
Besides, there also remain some works [36, 37] which use discriminative language modeling toapproximately estimate sentence scores.
[36] borrowed the idea of the energy-based model intosentence reranking.
[37] proposed a discriminative language model that minimizes the KL-divergencebetween the target distribution and the output distribution.
These methods can be considered as thediscriminative language modeling which directly predicts a single value to be the sentence score.
Discriminative language model usually needs target datasets for fine-tuning, while our proposedlanguage model is independent to downstream tasks.
We think discriminative language models arecomplementary to our works and we leave this combination as the future works.
A.3 Results
A.3.1 Results on BLiMP
Following previous experiences [20], we also conduct experiments on Benchmark of Linguistic
Minimal Pairs (BLiMP) [3], which includes 67K minimal pairs that contrast in grammatical acceptability and isolate specific phenomena in syntax, morphology or semantics.
BLiMP provides an
Dev
Test
Model
Domain clean other clean other
[Table]
unsupervised setting that uses language models to evaluate sentences and the acceptable sentencecan be assigned by a lower log-likelihood score.
We compare our model with GPT-2, BERT and
RoBERTa, and the results are reported in Table 7.
We can find that our Transcormer can easily beat
GPT model in BLiMP, which further validates the effectiveness of our SLM in using bidirectionalcontext for sentence scoring.
And when compared with BERT model, our Transcormer can alsomatch or outperform BERT performance slightly.
Considering that our model only needs a singlepass to produce the probability of all tokens, which also manifests the efficiency of our SLM insentence scoring.
Besides, we also report RoBERTa results for reference.
Due to the limitation of resources, our modelcurrently cannot match RoBERTa since RoBERTa costs more computations than BERT so it canextract better semantics for prediction.
But we believe our Transcormer can also achieve the betterperformance by training our model with the same computations.
A.3.2 Comparison with other works
As aforementioned, previous works [34, 33] have tried some strategies to calculate the probabilitiesof all tokens simultaneously to avoid the limitations in N-passes.
To validate the advantages ofour model in using bidirectional context, we also conduct experiments to make a comparison withthese methods.
For the sake of fairness, all of our experiments are deployed on the same datasetsused in [22] and the results are shown in Table 8.
From Table 8, we find that our Transcormer canoutperform these models.
These improvements also demonstrate the superiority of our model inutilizing bidirectional context to predict sentence probability.
A.4 Accelerating SLM Training
Although SLM is more advantageous than CLM in using bidirectional context and demonstrateshigher efficiency than MLM for sentence scoring, directly training SLM from scratch is still timeconsuming since it needs 3 × computations to maintain query and forward/backward streams.
So is itpossible to accelerate the training of SLM?
We note that both SLM and MLM adopt the masked tokenplus its position to predict its target based on the context while the main difference in context is that
MLM adopts one bidirectional context and SLM adopts forward and backward contexts.
We thinkthat MLM-based model with bidirectional context should also be able to produce good unidirectionalcontext and SLM does not modify model structure, so that we can use the MLM-based model (e.g.,
BERT) for initialization to accelerate SLM training.
Specifically, we continue to train BERT modelwith addition 5 epochs (nearly 20K steps) and the results can be found in Table 7 and Table 8.
Wecan find that only needs 5 epochs, our Transcormer with BERT model initialization can match the performance that is trained from scratch, which means using BERT initialization can accelerate our
SLM training.
A.5 Latency
To better manifest the efficiency of our SLM in rescoring when compared with MLM, we further testthe inference latency between SLM and MLM at both GPU and CPU under different lengths of theinput sequence.
Specifically, we measure the inference latency of each model at a batch size of 1 tovalidate the efficiency, and all results are reported in Table 9.
From Table 9, we can find that MLM(BERT) produces very high latency when compared with SLM (Transcormer), especially in CPUdevices.
Even using a smaller BERT model, it still cannot avoid the inherent issues ( i.e. , requiresn × computations) in MLM for scoring, which needs 20 × and 166 × additional computations over
Transcormer (SLM) in GPU and CPU.
These results also demonstrate the efficiency of our model inusing bidirectional context for calculating sentence scores.
[Table]
CPU is at Intel (R) Xeon (R) Platinum 8168 CPU @
2.70GHz.
The units of numbers in “GPU” and “CPU” columns are at millisecond (ms) and second (s).
A.6 AnalysesA.6.1 Pre-training Strategy
In our experiment setup, we use sentence-level data as the input for pre-training.
To better analyzethe effect of using different data processing for pre-training, we conduct experiments by usingstream-level data (concatenate multiple sentences as a fixed-length, e.g. , 512) to make a comparison.
We apply two strategies on NMT & ASR tasks, and then evaluate the average (top-1) accuracy ofeach token in a sentences with different lengths based on the output probability for SLM.
The resultsare reported in Table 10.
We can find that using stream-level data for pre-training can not achievegood accuracy when sentence length is too short.
We guess that is because using stream-level datacauses model can not fit short sentence since it always pre-trains under the longer sentences ( i.e. ,512).
Considering that our downstream scenarios mainly consist of single sentence, which is usuallytoo short, directly using stream-level data for pre-training can not achieve promising performance.
As a result, we recommend to use sentence-level data for pre-training, and we also expect to exploremore effective pre-training strategies in the future.
[Table] [Table]
length ( e.g. , 20, 250, 500) to evaluate the token accuracy of SLM in sentences ( i.e. , obtain the top-1 accuracy of each token and calculate the sentence accuracy by averaging the accuracy of all tokens).
A.6.2 Domain Adaption
Following previous experiences [20], we also study the effect of using in-domain data for pre-training.
For NMT tasks, we randomly sample 20GB monolingual data from NewsCrawl data to build thepre-training corpus for pre-training.
And for ASR tasks, as LibriSpeech includes 4GB in-domaindata, we direct use this data as our pre-training corpus to handle ASR tasks.
The results of NMTand ASR tasks are reported in Table 11 and Table 12.
We can find that using in-domain data forpre-training is useful to improve the downstream tasks.
[Table] [Table]
A
.7 Can SLM be used for Language Understanding Tasks?
As aforementioned in Section 3.3, both SLM and MLM can learn bidirectional context to predicttoken-wise probability, so why we cannot directly use SLM for language understanding tasks?
First,a difference between MLM and SLM is that the query and content streams in the MLM-based modelare shared together to enjoy the benefits of the bidirectional context, while our SLM maintains queryand forward/backward contents.
Previous state-of-the-art works [9, 12, 15, 13] have proven that
BERT-style models can achieve state-of-the-art performance in NLU tasks due to the benefits ofbidirectional context.
Although the query stream is able to enjoy completely bidirectional contextlike MLM due to the design of the triple-stream self-attention mechanism, the content of SLM isdecomposed as the forward and backward streams respectively to collect undirectional semantics,which may be weaker than MLM-based model in NLU tasks if using bidirectional context for finetuning.
In our internal experiments, we have tried to directly fine-tune SLM on SST-2 [53] like
BERT ( i.e. , using bidirectional context).
Due to the mismatch ( i.e. , forward/backward context v.s.bidirection context) between pre-training and fine-tuning, our SLM only achieves 91.9% accuracywhile the standard BERT can obtain 92.8%.
This phenomenon may also validate our hypothesis that SLM is not the optimal method for solving NLU tasks in a BERT-style fine-tuning paradigm.
However, there still remain some alternative possibilities for SLM to adapt NLU scenarios.
A possiblesolution is that converts language understanding tasks into ranking classification tasks.
For example,assuming the task is sentiment classification and we have a review sentence, so we can construct twosentences: “the sentiment of [review] is positive" and “the sentiment of [review] is negative".
Basedon the constructed sentences, we can use our model on each sentence to calculate their sentencescores and the sentence with ground truth should have a lower log-probability score.
It can beconsidered as a variant of prompt-based learning [11, 54].
We have simply built two different patternsto formulate sentiment classification, and conduct experiments on SST-2 in a zero-shot setting ( i.e. ,
[Table] The sentiment of [ Review ] is [ GT Label ] . The sentiment of [ Review ] is [ Wrong Label ] . [Table]
[ Review ] is a [ GT Label ] sentiment. [Table] [Table] [ Wrong Label ]
[Table]
without fine-tuning).
The results are shown in Table 13.
We find that using patterns like “ [ Review ] is
[Table] a [ GT Label ] sentiment.” can obtain an accuracy of 71.0% on SST-2 in a zero-shot setting (BERT [Table] fine-tuning cannot be used in a zero-shot scenario).
These experiments also demonstrate that we can [Table] transform NLU tasks into ranking tasks and then apply our model for ranking, which also indicatesthe potential of our model in solving NLU or other NLP tasks.
We will also continue to explore moretechniques to refine this strategy to improve performance in the future.