Transcormer: Transformer for Sentence Scoring with Sliding Language Modeling
Abstract
Sentence scoring aims at measuring the likelihood score of a sentence and is widely used in natural language processing scenarios, like reranking, which is to select the best sentence from multiple candidates.
Previous works on sentence scoring mainly adopted either causal language modeling (CLM) like GPT or masked language modeling (MLM) like BERT, which have some limitations: 1)
CLM only utilizes unidirectional information for the probability estimation of a sentence without considering bidirectional context, which affects the scoring quality; 2) MLM can only estimate the probability of partial tokens at a time and thus requires multiple forward passes to estimate the probability of the whole sentence, which incurs large computation and time cost.
In this paper, we propose
Transcormer – a Transformer model with a novel sliding language modeling (SLM) for sentence scoring.
Specifically, our SLM adopts a triple-stream self-attention mechanism to estimate the probability of all tokens in a sentence with bidirectional context and only requires a single forward pass.
SLM can avoid the limitations of CLM (only unidirectional context) and MLM (multiple forward passes) and inherit their advantages, and thus achieve high effectiveness and efficiency in scoring.
Experimental results on multiple tasks demonstrate that our method achieves better performance than other language models.
Our code and pre-trained models will be released at: https://github.com/microsoft/CyBERTron-LM/
Transcormer .
Introduction
Sentence scoring is to measure the log-likelihood score of a sentence via language model, so that it can be used to represent the relative likeliness of a sentence.
Specifically, a good sentence should have a relatively lower log-likelihood score, which means more linguistically acceptable for the sentence [1].
Due to such nature, sentence scoring has been widely used in many natural language processing (NLP) scenarios.
For instance, it can be used to rerank candidates in neural machine translation (NMT) or automatic speech recognition (ASR) tasks [2, 3] or evaluate sentences in linguistic acceptability [4].
Therefore, how to design effective language models to calculate sentence scores efficiently is very important.
Recently, neural network based language models (LM) [5, 6, 7, 8] have been considered as the most widely used technique for sentence scoring, since they can produce density estimation of the whole sentence by computing the probability of each token and summing up their values as the sentence score.
Specifically, causal language modeling (CLM) [9] and masked language modeling (MLM) [10] are the most representative LMs.
For a given sentence x = { x 1 , · · · , x | x | } , where x i is the i -th token of x .
CLM can predict next token conditioned on unidirectional context and its objective is to optimize (cid:80) | x | i =1 log P ( x i | x <i ) .
Hence, CLM is usually used for solving natural language generation (NLG) tasks [9, 11, 12].
While for MLM, it replaces a subset of tokens x S in x as special symbol [MASK] and then predicts the masked tokens based on the corrupted sequence x \S .
The objective of MLM is to optimize (cid:80) |S| i =1 log P ( x S i | x \S ) , so it is able to learn bidirectional context and can be used for solving natural language understanding (NLU) tasks [10, 13, 14, 15].
Since CLM and MLM can be learned in an unsupervised fashion, many works [10, 13, 14, 9, 11, 12, 16, 17, 18, 15, 19, 20] have pre-trained these LMs on large-scale corpus to extract powerful linguistic representations, and these models can be directly used out of the box to predict the probabilities of tokens.
Inspired by the successes of LMs, some works [21, 22, 23, 24, 25] have tried to use pre-trained CLM or MLM to generate sentence scores on NMT or ASR reranking tasks and achieved some promising results.
However, we notice that both CLM and MLM still suffer from some deficiencies in calculating sentence scores.
For instance, some works [25] utilized GPT-style model for ASR reranking within single inference, yet GPT model can only extract unidirectional information due to the limitation of CLM, without considering the whole sentence semantics, and thus affect the sentence score.
To utilize bidirectional context, some works [22, 23, 24] applied BERT model for rescoring.
However, the nature of MLM is to mask some tokens in the sentence for prediction, which means it requires the
BERT model to forward multiple times and each forward pass only masks one token for prediction.
As a result, it is time-consuming to directly adopt MLM for scoring sentence.
In summary, we find that for calculating sentence scores, CLM needs one-pass inference but only uses unidirectional information and MLM is costly in computing sentence score although it can use bidirectional context.
Therefore, a natural question arises: is it possible to design a language model to use bidirectional context for sentence scoring with only one inference pass.
In this paper, we introduce Transcormer , a Transformer model designed for sentence scoring.
More specifically, our Transcormer leverages a novel language modeling scheme, named as sliding language modeling (SLM), that produces the probability of all tokens within single inference pass and simultaneously utilizes bidirectional context.
To fulfill this target, we innovatively design a triple-stream self-attention mechanism, which consists of two content streams (a forward stream and a backward stream) and one query stream.
By employing specifically-designed mask strategies on the attention matrix, our method allows each token in the query stream to leverage all token information except itself (i.e., the tokens before and after it) for estimating its probability to avoid any information leakage.
To the best of our knowledge, SLM is the first language modeling tailored for sentence scoring.
We pre-train our SLM on large-scale corpus, and then evaluate it on multiple datasets.
Experimental results demonstrate that Transcormer outperforms the baselines by up to + 0.8/0.6 BLEU score on small/large-scale NMT tasks, ∼ 20% relative improvements on ASR tasks.
The main contributions of this work are summarized as follows:
• We analyze the pros and cons of CLM and MLM when using them for scoring sentences, and propose Transcormer with a new sliding language modeling, which uses bidirectional context for probability estimation within a single pass.
• We introduce a novel triple-stream self-attention mechanism in SLM, which has two content streams to collect forward/backward semantics, and a query stream to estimate the probability of each token in a sentence.
• Experimental results on multiple datasets demonstrate the effectiveness and efficiency of our SLM for sentence scoring.
Background 2.1 Sentence Scoring
Sentence scoring has a long history in NLP applications, especially in reranking tasks ( e.g. , reranking for machine translation [26, 27, 28, 29] or speech recognition [30]).
Generally, given n -best candidates generated by text generation models, reranking aims at scoring each candidate to select the best answer.
Early works [26, 27, 28, 29, 31, 32] mainly used statistical LM or combined it with
RNN-based LM to calculate the sentence scores.
Recently, end-to-end neural network based LM has became the de facto approach for scoring and has been applied in many NLP tasks [6, 5, 33].
Specifically, causal language modeling (CLM) and masked language modeling (MLM) are the most representative language modeling methods, among which GPT [9, 11, 12] and BERT [10] are the most famous examples, respectively.
As aforementioned, CLM conditions on the previous states to predict next token so that it can obtain the probability of each token of the sentence in a single pass, but can only capture unidirectional information.
For MLM, it is able to use bidirectional context for prediction but the masked-prediction mechanism limits MLM to producing the probability of all tokens within one forward pass (since MLM can only provide the probability of masked tokens).
To calculate the sentence score, a kind of solutions [22, 23, 24] is to forward multiple times and only mask one token each time.
Therefore, the MLM for sentence scoring is formulated as: (cid:80) | x | i =1 log P ( x i | x \ x i ) .
We can find that the cost of MLM for scoring needs | x | inference passes, which is too time-consuming.
Some recent works [22, 21, 34, 35] have been proposed to alleviate these issues in MLM.
Wang et al.
[22] and Salazar et al.
[21] attempted to use stochastic estimation or distillation to avoid N-passes problem to approximately estimate the probability of each toke produced by MLM, with a sacrifice of performance.
Clark et al.
[34] adopted a two-cloze tower [36] with noise-contrastive estimation to provide sentence probability, and Shin et al.
[35] only considered word embedding as the inputs of key and value in transformer without any interaction.
Besides, some works [37, 38] adopted discriminative language modeling to approximately estimate sentence scores based on the paired data, but this paradigm must require labeled data from the downstream tasks and cannot utilize unlabled data for pre-training.
More discussions about related works can be found in Appendix A.2.
Therefore, how to calculate sentence scores efficiently with bidirectional context is the main challenge in MLM.
Overall, CLM only needs a single forward pass to estimate the probability of all tokens but cannot extract bidirectional context, while MLM leverages bidirectional information but needs multiple inference passes.
Consequently, we raise a natural question: is it possible to design a pre-trained language model to support all token prediction in a single pass and simultaneously leverage bidirectional information?
This is exactly the motivation of our method.
2.2 Multiple-Stream Self-Attention
The pioneer of multiple-stream self-attention is XLNet [16], which introduces two-streams selfattention to incorporate autoregressive pre-training for language understanding, which pre-trains
Transformer [39] via using a content stream and a query stream.
In details, for the t -th step, the content stream is able to capture the dependency from the tokens before the t -th step and itself ( i.e. , x ≤ t ), while the query stream is only allowed to view the tokens before the t -th step ( i.e. , x <t ) to avoid information leakage.
Besides, there are some other variants of two-stream self-attention [17, 40, 41], which are designed for solving NLU tasks.
For example, MPNet [17] used two-stream self-attention to build masked and permuted pre-training.
ProphetNet [40] designed multiple query streams to predict N-gram future steps for sequence-to-sequence tasks, and ERNIE-GAN [41] proposed a multiflow generation model, which includes two query streams for span and word prediction.
We observe that these works mainly used a single content stream and then used one or many query streams to predict more information.
Different from these works, we introduce a triple-stream self-attention mechanism, which enables query stream to leverage two content streams for prediction, and thus enjoys the benefits of additional bidirectional context for estimating token probability.
Transcormer
To inherent the advantages of CLM and MLM for sentence scoring and avoid their limitations, we propose Transcormer – a Transformer model with a novel sliding language modeling for sentence scoring.
First, we summarize that an ideal language modeling for sentence scoring should satisfy two requirements: 1) model should be able to use bidirectional context for effective probability estimation of each token; 2) model should produce the probability of all tokens in a sentence within a single inference pass for efficiency.
Therefore, to fulfill these two requirements, we formulate a new language modeling, named as sliding language modeling (SLM), and describe it in Section 3.1.
In SLM, we propose a Triple-Stream Self-Attention mechanism based on Transformer (please see
Section 3.2 for details) to use bidirectional context for each token prediction and avoid information leakage.
We also discuss the differences between SLM and other LMs in Section 3.3.
Figurepresents the pipeline of our Transcormer for sentence scoring with SLM.
[Figure]
xxx
P P
Pxx
P
P_ _ _
P
P
P_ _
P
Pxxx
P P
Pxx
P P
Transcormer
[Figure]
3.1 Sliding Language Modeling
Considering the pros and cons of CLM and MLM for scoring, we notice: 1) CLM can produce probability of all tokens within one forward pass, and thus obtain the unidirectional information of the whole sentence; 2) MLM for sentence scoring needs multiple inference passes, so that many context has been repeatedly calculated and cause a waste of computation.
So, is it possible to reuse token information to build bidirectional context for token prediction?
Therefore, we propose sliding language modeling (SLM) to address the inherent flaws in previous
LMs ( i.e. , CLM and MLM) for sentence scoring.
Specifically, we first maintain two individual streams to collect forward (left-to-right) context and backward (right-to-left) context.
And for each token prediction, we decompose the sentence information as the past tokens (the tokens before it) and future tokens (the tokens after it) respectively.
As a result, our SLM enforces each token to only capture the dependency from its past tokens and its future tokens concurrently, so that each token can utilize the whole sentence information (except itself) to estimate token probability.
The objective function of SLM is formulated as:
[Equation]
where x <i and x >i respectively correspond to the tokens before the i -th token and after the i -th token, and θ represents the parameters of SLM.
Thanks to such design, our SLM can utilize bidirectional context to produce the probability of each token within one forward pass, and thus satisfy the above requirements for sentence scoring.
However, previous experiences [16, 42] pointed out that the states with bidirectional information will cause information leakage when propagating to the next layer 2 .
Therefore, how to implement SLM to avoid information leakage and maintain different states together is still a troublesome problem.
3.2 Triple-Stream Self-Attention
Based on the Eqn 1 of our proposed SLM, model needs to maintain two states to collect forward and backward contexts for prediction, and we call these two states as the forward stream and backward xxx
P
P
Pxx
P
Pxxx
P P
Pxx
P
P_ _ _
P
P
P_ _
P P
Q
KV forward stream
Q
KV backward stream
Q
KV,KV
Query stream
Input Sequence
Attention (a) x 1 x 2 x 3 x 4 xxxxxxx 1 x 2 x 3 x 4 xxxxxx
P
P P
P
Pquery stream forward stream backward stream (
[Figure]
query the hidden both
The is the attention the forward and the for the query
The cell in means this cannot be stream respectively.
To avoid information leakage, we additionally maintain an individual state for prediction and control it to only capture the dependency from the forward and backward streams.
We name this state as the query stream.
Therefore, we propose a novel Triple-Stream Self-Attention to update each stream, and the detailed design is described as following.
To fulfill our target, we choose Transformer [39] as our basic model, due to its flexibility in capturing global dependency.
Assume the input sequence as { w 1 , w 2 , · · · , w n } and its positions as { p 1 , p 2 , · · · , p n } , where w i and p i represents the embedding of i -th token and its position in the sentence, and n is the token number.
For the query stream, we only use the positional embeddings { p 1 , p 2 , · · · , p n } as the input.
For the forward and backward streams, we also maintain two individual states and both of them use the tokens plus its position ( i.e. , { w 1 + p 1 , w 2 + p 2 , · · · , w n + p n } ) as the input.
For the l -th layer calculation, we denote the forward and backward streams of the position i as ⃗h li and ⃗ h li , and they are updated as:
[Equation]
where Attention( · , · ) refers to the self-attention [39] in Transformer and Q , K , V denote the query, key and value in self-attention.
Hence, ⃗h li collects the information before the position i and itself, and ⃗ h li collects the information after the position i and itself.
For the query stream, we denote q li as its hidden states, and concatenate forward stream ⃗h li and backward stream ⃗ h li of the current layer to be the key/value of query stream.
So q li is updated as:
[Equation]
Here we find that q li is required to only capture the dependency from the forward stream before the position i and the backward stream after the position i , rather than itself.
Due to such design, the query stream is able to capture bidirectional context for estimating token probability and avoid information leakage, which is more effective than CLM in using context.
More importantly, our triple-stream self-attention enables model to predict the probability of all tokens in a sentence within a single forward pass, which demonstrates more efficiency than MLM.
Figure 2 presents the detailed design of our triple-stream self-attention.
We can find that in the query stream, the masked matrix is like a sliding window to control each token to view its previous states in forward stream and its future states in backward stream.
And that is why we name our model as sliding language modeling.
[Table]
3.3 Discussion
To better understand our SLM, we analyze the advantages of our SLM over other LMs.
The comparisons are listed in Table 1.
We select three representative LMs for comparisons, which are
CLM (BERT), MLM (GPT) and bidirectional LM (Bi-LM, used in ELMO [43] 3 ) respectively.
From
Table 1, we have the following observations:
When compared with CLM, our SLM requires 3 × computations.
However, our SLM can fully use the whole sentence information for prediction while CLM can only use unidirectional information.
Even scaling CLM as 3 × parameters, it still can not use bidirectional context for prediction.
This also demonstrates the effectiveness of our SLM in using context.
MLM is powerful at extracting bidirectional context but it needs n × inferences to calculate the whole sentence information limited to its masked prediction.
Our SLM just needs a single inference and uses bidirectional information for prediction with only 3 × computations.
Especially, our SLM shows higher efficiency compared with MLM when n is large.
Bi-LM can also extract forward and backward contextual information, but it just simply concatenates the forward and backward features for the final prediction, without any interactions.
Instead, our SLM can iteratively fuse the bidirectional information thanks to our triple-stream self-attention mechanism.
Overall, the design of SLM is dedicated for sentence scoring, while CLM prefers NLG tasks and
MLM/Bi-LM prefer NLU tasks.
Experiments
In this section, we describe our experimental setup, and the results on NMT and ASR datasets.
4.1 Experimental Setup
We adopt Transformer [39] as the backbone network.
Following previous works [10], we adopt a base setting and a small setting for our model as Transcormer base with 110M parameters and
Transcormer small with 34M parameters, that consists of 12/6 transformer layers and each layer has 768/512 hidden size and 12/8 attention heads.
During the pre-training, we use wikipedia plus bookcorpus (16GB) as the training corpus, to be consistent with previous works [10].
Our model is trained at the sentence-level ( i.e. , one sentence per sample).
We choose Adam [45] as the default optimizer with learning rate of 5 e − 4 , β 1 = 0 .
9 , β 2 = 0 .
98 and ϵ = 1 e − 6 , and weight decay is set as 0.01.
The learning rate warms up over the first 10,000 steps and then linearly decays.
We set the batch size as 8192 tokens per batch, and the training step is 125,000 steps.
We use 32 NVIDIA
Tesla 32GB GPUs, with FP16 speedup.
The total training needs 5.5 days for Transcormer base .
For more experimental settings ( e.g. , dataset and its split sizes), please refer to the Appendix.
The code and pre-trained models will be released at: https://github.com/microsoft/CyBERTron-LM/
Transcormer .
[Table]
4.2 Experiments on Neural Machine Translation
We choose IWSLT14 dataset [46], which includes multiple small-scale translation tasks from different languages to English, and WMT14 English-German dataset 4 for evaluation.
We adopt
Transformer [39] as the machine translation model with 6-6 transformer layers to generate multiple candidates for reranking, with a beam size of 10.
The hidden size and attention head are set as 512/1024 and 8/16 for IWSLT and WMT tasks respectively.
During the reranking, we combine the original score produced by the machine translation model and the LM score with a hyper-parameter λ , by following previous experiences [7, 21].
The hyper-parameter λ is tuned on dev set with a range of [0 . 0 , 2 . 0]
[Table]
and then select the best λ to evaluate the test set.
The results are reported in Table 2, in terms of BLEU [47].
Each task will be evaluated by five times based on different pre-trained checkpoints, and report the mean value, with a variance of 0.05.
From Table
[Table]
we have the following observations:
• Our Transcormer base can obtain better performance than CLM and Bi-LM 5 in both small-scale IWSLT and large-scale WMT tasks, which indicates the importance of bidirectional context for sentence scoring, and further validate the ability of our SLM in utilizing bidirectional information.
• When compared with MLM, our Transcormer base also achieves comparable results.
Considering that the computation of MLM for scoring is linear to the input length and needs n inference passes, our SLM show higher efficiency with only 3 × computations in a single pass to maintain three streams.
Overall
[Table]
all of the results reveal that our model is more effective in using contextual information for probability estimation and more efficient with only a single forward pass, especially for long sentences.
[Table] [Table]
O v e r a ll A N A .
A G R A R G .
S T R B
I N D
I N G C T R L .
R A
I
S .
D - N A G R E
L L
I P S
I S F I L L E R G
A P
I R R E G U L
A R
I S L A N D N P
I Q U
A N T
I F
I E R S
S - V A G R
[Table]
4.3 Experiments on Automatic Speech Recognition
We choose LibrSpeech [48] to evaluate the performance of our model for reranking on ASR task.
We train a Conformer model [49] on LibrSpeech, which has 12 encoder layers and 6 decoder layers with 512 hidden size and 8 attention heads, and the beam size is set as 10.
In addition, we use SpecAug [50] as a data augmentation technology to further improve the accuracy of ASR system.
We use word error rate (WER) to evaluate the performance of ASR tasks.
We follow the same tuning technique used in NMT tasks for hyper-parameter λ , but with a larger range as [0 . 0 , 5 . 0] .
The results are reported in Table 3.
From Table 3, we find that our model can give nearly 20% relative improvements over the baseline and also outperform other LMs, including CLM, MLM and Bi-LM.
The results on
ASR task further demonstrates the generalization and effectiveness of our SLM in sentence scoring.
4.4 Experiments on Linguistic Acceptability
Following previous experiences [21], we also conduct experiments on Benchmark of Linguistic
Minimal Pairs (BLiMP) [4], which includes 67K minimal pairs that contrast in grammatical acceptability and isolate specific phenomena in syntax, morphology or semantics.
BLiMP provides an unsupervised setting that uses language models to evaluate sentences and the acceptable sentence can be assigned by a lower log-likelihood score.
We compare our model with GPT-2, BERT and
RoBERTa, and the results are reported in Table 4.
We can find that our Transcormer can easily beat
GPT model in BLiMP, which further validates the effectiveness of our SLM in using bidirectional context for sentence scoring.
And when compared with BERT model, our Transcormer can also match or outperform BERT performance slightly.
Considering that our model only needs a single pass to produce the probability of all tokens, which also manifests the efficiency of our SLM in sentence scoring.
Due to the limitation of resources, our model currently cannot beat RoBERTa since
RoBERTa costs more computations than BERT and thus extract better semantics for prediction.
But we believe our Transcormer can achieve the better performance by training our model with the same computations.
Analyses
In this section, we conduct some method analyses on our proposed SLM and CLM/MLM.
Besides, we also provide more analyses in Appendix.
5.1 Latency Comparisons between MLM and SLM
To better manifest the efficiency of our SLM in rescoring when compared with MLM, we further test the inference latency between SLM and MLM at both GPU and CPU under different lengths of the input sequence.
Specifically, we measure the inference latency of each model at a batch size of 1 to validate the efficiency, and all results are reported in Table 5.
From Table 5, we can find that MLM (BERT) produces very high latency when compared with SLM (Transcormer), especially in CPU devices.
Even using a smaller BERT model, it still cannot avoid the inherent issues ( i.e. , requires n × computations) in MLM for scoring, which needs 20 × and 166 × additional computations over
Transcormer (SLM) in GPU and CPU.
These results also demonstrate the efficiency of our model in using bidirectional context for calculating sentence scores
[Table]
5.2 Comparable Computation between CLM and SLM
As aforementioned, our SLM needs 3 × computations when compared with CLM.
To make a fair comparison, we also pre-train a Transcormer small with 34M parameters in total, which consists 6 transformer layers and each layer has 512 hidden size and 8 attention heads.
Hence, our Transcormer small has approximately 13 parameters of Transcormer base , and has the similar computations as GPT base .
We conduct experiments on three NMT tasks and an ASR task (LibriSpeech dataset) for comparisons and the results are listed in Table 2 and 3 ( i.e. , “Transcormer small row”).
We can find that even under the same computation, our model still outperforms CLM and this result further validates the necessity of using bidirectional context for sentence scoring.
Besides, considering that Transcormer small has fewer parameters, our model is also friendly to the device deployments ( e
[Table]
g. , CPU).
[Table]
5.3 Varying Numbers of Forward Passes in MLM
As mentioned above, MLM needs to forward n passes as each time only mask one token.
So what will happen if we allow each pass to mask more tokens?
Therefore, we design experiments that enforces MLM to forward k tokens for prediction so that it only needs ⌈ n/k ⌉ passes, and investigate the effect of different k .
For a certain k , we randomly split the sentence as ⌈ n/k ⌉ sets, and each time masks one subset for prediction.
The comparisons are listed in Table 6.
We find that using larger k will severely harm the performance, even if it can reduce the number of inference passes.
When k is set as n 3 , which is equal to the cost of our SLM, it can hardly give any improvements over the baseline.
We think that masking more tokens in the sentence will make it more difficult to estimate the token probability at a time.
These comparisons also highlight the efficiency and effectiveness of our SLM for sentence scoring.
5.4 Sentence Scoring Quality at Each Position
passes to predict each
Following previous experiences [21], we count the crossentropy loss of each position in sentences for each LM, to better analyze the effectiveness of bidirectional context to estimate token probability.
Specifically, we sample a subset of sentences as S , and each sentence has the same token number n (here n is 20).
For each position i , we count the average cross-entropy over the sampled subset
S based on the output probability of each LM.
The results are displayed in Figure 3.
We can find that: 1) For CLM, the cross-entropy loss is higher at the first several positions and gradually decreases for the subsequent positions but is still higher than MLM and SLM, which indicates that only using undirectional information is not enough to measure the sentence score precisely.
2) SLM can almost obtain similar loss as MLM at each position.
Considering SLM just needs a single pass while MLM needs n passes, this phenomenon further validates the superiority and efficiency of SLM in scoring sentences.
Conclusion
In this paper, we propose Transcormer, a Transformer with a novel sliding language modeling for sentence scoring.
Specifically, our SLM is able to produce the probability of each token over the whole sentence within a single forward pass, and utilizes bidirectional context for prediction, and thus inherents the advantages of CLM and MLM and avoids their deficiencies.
To the best of our knowledge, the proposed Transcormer is the first pre-trained language model tailored for sentence scoring.
Experimental results on multiple datasets demonstrate the effectiveness of our Transcormer in computing sentence score for reranking tasks.
Besides, we summarize some potential directions of our Transcormer and SLM as the future works:
• Currently our Transcormer is only conducted on the English domain under the base setting, due to limited computation.
We expect to develop large-scale Transcormer and use different language domains or multilingual data for training in the future.
• We design sliding language modeling for sentence scoring, and our experiments are mainly on reranking task.
However, based on the characteristics of our SLM, we believe our model can also be used for other scenarios ( e.g. , error correction [51, 52, 53], data selection), and we will explore the specific fine-tuning techniques when applying our SLM on different downstream tasks.
• Besides, our Transcormer mainly pre-trains SLM on an encoder framework.
However, our SLM is not limited to model structure.
For example, SLM can be easily extended to encoder-decoder framework [18] based on paired data.
Therefore, we also expect to explore the possibility of using SLM on different frameworks.
• Although our paper mainly focuses on text data, we want to highlight that SLM can also be extended to other different modalities with sequential characteristic ( e.g. , image, speech [54] and time series data).
Consequently, how to apply SLM to other modalities is also a valuable topic in the future.