Addressing Token Uniformity in Transformers via Singular Value Transformation
Abstract
Token uniformity is commonly observed in transformer-based models, in which different tokens share a large proportion of similar information after going through stacked multiple self-attention layers in a transformer.
In this paper, we propose to use the distribution of singular values of outputs of each transformer layer to characterise the phenomenon of token uniformity and empirically illustrate that a less skewed singular value distribution can alleviate the ‘token uniformity’ problem.
Base on our observations, we define several desirable properties of singular value distributions and propose a novel transformation function for updating the singular values.
We show that apart from alleviating token uniformity, the transformation function should preserve the local neighbourhood structure in the original embedding space.
Our proposed singular value transformation function is applied to a range of transformer-based language models such as BERT, ALBERT, RoBERTa and DistilBERT, and improved performance is observed in semantic textual similarity evaluation and a range of GLUE tasks.
Our source code is available at https:// github.com/hanqi-qi/tokenUni.git .
INTRODUCTION
In Natural Language Processing (NLP), approaches built on the transformer architecture have achieved the state-ofthe-art performance in many tasks [Veitch et al., 2020].
However, recent work identified an anisotropy problem in language representations generated by transformer-based deep models [Ethayarajh, 2019, Gao et al., 2019, Li et al., 2020], i.e., the learned embeddings occupy a narrow cone in the representation space.
Such anisotropic shape is very different from what would be expected in an expressive embedding space [Arora et al., 2016, Mu and Viswanath, 2018].
This problem is called token uniformity or information diffusion , i.e., different tokens share a large proportion of similar information after going through stacked multiple self-attention layers in a transformer.
Goyal et al.
[2020] showed that using different transformer-encoded tokens in an input sample as a classification unit can achieve almost the same result.
Recently, Dong et al.
[2021] found that pure self-attention networks, i.e., transformers without skip-connections and
MLPs, have their outputs converging to a rank one matrix, and such rank deficiency can lead to token uniformity.
They therefore concluded that skip-connection and MLP help alleviate the token uniformity problem.
However, in our experiments, we still observe the token uniformity problem in the full transformer model with self-attention layers, skipconnections and MLPs, even when its output hidden state matrices are full-rank.
In this paper, we instead investigate the token uniformity problem via exploring the distribution of singular values of the transformer-encoded hidden states of input samples.
Our analysis reveals that the learned embedding space is a high-dimensional cone-like hypersphere which is bounded by the singular values.
Also, skewed probability distribution of singular values is indicative of token uniformity (See in §3.1).
Therefore, making the distribution less skewed towards small singular values can help alleviate the token uniformity issue.
Unlike existing methods [Gao et al., 2019,
Wang et al., 2020] that implicitly or explicitly guide the spectra training of the output embedding matrix by adding a regularisation term to control the singular value decay, we propose a novel approach to address the token uniformity via smoothing the singular value distribution (See in §4.2).
In order to verify the effectiveness of our proposed singular value transformation function in transformer-based structures, we apply it to four commonly used large-scale pretrained language models (PTLMs).
In particular, the singular value distribution of the final layer output from a
PTLM is modified using our proposed transformation function.
Then, the transformed singular values are used to re- construct the hidden states in the last layer of the PTLM, which are subsequently used for prediction in downstream tasks.
Our extensive experiments on a variety of NLP tasks including semantic textual similarity evaluation and a range of General Language Understanding Evaluation (GLUE) tasks [Wang et al., 2019] across thirteen datasets show that our proposed transformation function can effectively reduce the skewness of singular value distributions in qualitative analysis and achieve noticeable performance gains.
Our contribution can be summarised as follows:
• We have presented both geometric interpretation and empirical study of the token uniformity problem.
Based on our observations, we have designed a set of desirable properties of singular value distributions and proposed a singular value transformation function to alleviate the token uniformity issue.
• We have proposed a range of methods to evaluate the transformed features in terms of uniformity and the preservation of the local neighbourhood structure.
• Our proposed transformation function has been applied to four widely-used PTLMs and evaluated on both unsupervised and supervised tasks.
The results demonstrate the effectiveness of our proposed method on addressing the token uniformity problem while preserving the local neighbourhood structure in the original embedding space.
RELATED WORK
Transformer-based mask language models, such as
BERT [Devlin et al., 2018], ALBERT [Lan et al., 2019],
RoBERTa
[Liu et al., 2019] and DistilBERT [Sanh et al., 2019], have achieved significant success in NLP.
However, token uniformity, i.e., different tokens share similar representations, is commonly observed with the increasing network depth.
Many studies [Ganea et al., 2019, Yang et al., 2019] claimed that token uniformity is caused by rank collapse of the layer-wise outputs because the transformer architecture learns the token representation based on the normalised weighted sum of the context representations.
Another line of work, which observed token uniformity in empirical studies, argued that the desirable word representations should be isotropic and focused on studying the distribution of the learned embeddings [Mu and Viswanath, 2018].
Gao et al.
[2019] and Bis et al.
[2021] defined the problem as ‘representation degeneration’ and gave a theoretical analysis, which asserts that this phenomenon is caused by frequencies of rare words.
Wang et al.
[2020] proposed to add an exponential decay term in training objective so as to control the singular value distribution.
All the aforementioned work focused on token-level features and tasks.
More recent work argued that the sentence-level features can also be anisotropic due to the anisotropy in word features.
Contrasting learning can also alleviate the anisotropy problem both theoretically and empirically [Carlsson et al., 2021,
Gao et al., 2021, Gu and Yeung, 2021].
Li et al.
[2020] proposed BERT-flow to transform the representations learned by PTLMs into a standard Gaussian distribution in order to make the token/sentence representations isotropic.
Other researchers turned to the whitening-based post-processing strategy to normalise sentence embeddings to derive almost perfectly isotropic representation distribution [Su et al., 2021, Huang et al., 2021].
We argue that on the one hand addressing rank collapse does not necessarily solve the token uniformity problem, as it is still observed even with the network components such as skip-connections which can guarantee the full rank feature space [Dong et al., 2021].
On the other hand, while whitening methods can effectively solving the token uniformity problem, they failed to preserve the local neighbourhood structure of the original embedding space, which is important for downstream tasks.
We propose a novel singular value transformation function which can alleviate the token uniformity and at the same time preserve the local neighbourhood structure.
SINGULAR VALUE DISTRIBUTION OF TRANSFORMER BLOCK OUTPUTS
In a typical transformer block ℓ , assuming the input token is x l − 1 , the information propagation process is given by:
[Equation]
where φ is an element-wise nonlinear function applied to a feed-forward layer, whose weight matrix, W l ∈
R n l × n l − 1 , transforms the feature dimension from n l − 1 to n l , Self-Attention ( x l − 1 ) returns the weighted value vector of all input representations where weights are derived by multiplying the query vector of the current input x l −with the key vectors from other inputs.
Between every two transformer blocks, there is a skip-connection and a layer normalisation.
The former mechanism bypasses the transformer block ℓ and adds the input x l − 1 directly to the output v l of this block, while the latter normalises the input across the feature dimension.
Taking BERT as an example, we assume that the input of the model is X
= x 1 ⊕ x 2 ⊕ ... ⊕ x m , X ∈ R n × m , where x i ∈
R n , m is the number of tokens in an input sentence (we do not distinguish special tokens such as [ CLS ])] , and ⊕ is the concatenation operator.
The output of a transformer block ℓ is denoted as X l ∈ R n l × m , where n l is the dimension of output feature in layer ℓ .
Without loss of generality, we assume n l > m for all layers since the embedding size for tokens is larger than the maximum length of sentences in most BERT models.
Existing work mainly focused on the discussion of the rank of features learned by a transformer-based language model.
For example, it was stated in [Dong et al., 2021] that with the growth of depth in a pure transformer model, the rank of the output representation matrix will converge to 1 exponentially.
However, in practice, a position embedding matrix, which is usually full rank, is added to the output representation of each layer. In addition, skip connections are used.
Therefore, rank collapse rarely occurs. It can be observed from the empirical cumulative density function of singular values from different layers of BERT, derived from real-world data and shown in Figure 2, that there is no zero singular value.
In this section, we instead study the token uniformity problem by the singular value density of the representation matrix X for a deep network with ℓ transformer blocks rather than the rank of X .
Since the distribution of the singular values of X l determines the extent to which the input signals become distorted or stretched as they propagate through the network, we aim to understand the entire singular value spectrum of X l in transformers.
In particular, we want to study the degree of skewness of the singular value distribution, because highly skewed distributions indicate strong anisotropy in the embedded feature space, the radical reason for token uniformity.
3.1 SINGULAR VALUE VANISHING IN TRANSFORMER
In this subsection, we give a geometric interpretation of the problem of vanishing singular values in transformers.
We assume that X l ∈ R n l × m is a full rank matrix.
We can perform SVD on ( X l ) ⊺ = UΣV , where U and V are orthogonal matrices and Σ is the diagonal singular value matrix.
Without loss of generality, we sort the singular values in a descending order, λ 1 ≥ λ 2 ≥ ... ≥ λ m ≥ 0 , where λ k , k ∈ { 1 , · · · , m } , is a diagonal element in Σ .
We can choose a positive value C such that λ 1 ≥ ... ≥ λ k ≥ C ≥ λ k +1 ≥ ... ≥ 0 , which defines two subspaces, denoted as
S l [1 ,k ] , and S l [ k +1 ,m ] , respectively. For any token embedding, x ∈ X l , we can find a point x ′ in the subspace S l [1 ,k ] such that their difference is no larger than C .
That is, we are able to establish the following bound:
Theorem: ∀ x ∈ X l , ∃ x ′ ∈ S l [1 ,k ] , where the subspace S l [1 ,k ] is defined based on λ k ≥ C ≥ λ k +1 , then ∥ x − x ′ ∥ 2 ≤ C .
According to this result, the embedding space is bounded by two components, the largest singular value in the subspace
X l [1 ,k ] , and the upper bound C of the small singular values, as the radius to span the k -dimensional X l [1 ,k ] subspace into the m -dimensional space.
Furthermore, the weights of the m components in the ℓ -th layer is constrained by selfattention: Σ mi =1 α li = 1 , which indicates that the embedding space is a spherical cone-like k -dimension hypershere.
This phenomenon has been observed in many studies [Gao et al., 2019, Zhang et al., 2020, Wang et al., 2020, Li et al., 2020].
We assume the Probability Density Function (PDF) of the distribution of singular values is f p ( λ ) , where λ is the singular values in the learned embedding space, then we can obtain the value of k based on the Cumulative Distribution
Function (CDF) of singular values larger than C , and the size of input tokens m , k =
⌜ (cid:82) + ∞ C m · f p ( λ ) dλ ⌝ .
Therefore, the shape of the embedding space is now decided by two parameters: C , the radius of the hypercone, and m − k , the size of almost vanished dimensions when C is small.
However, due to the complex network operations in transformer blocks, it is difficult to derive the exact form of f p ( λ ) .
Hence, we resort to empirical study to understand the singular value distribution which appears to be an exponential long-tail distribution, and use the skewness to measure the risk of dimension vanishing in the rest of this paper.
3.2 EMPIRICAL STUDY OF TOKEN UNIFORMITY IN BERT
Existing studies have observed the token uniformity issue in
PTLMs and skewed singular value distributions of outputs from the intermediate network layers [Goyal et al., 2020].
Few of them though has explored the impact of different shapes of singular value distributions on the downstream task performance.
Here, taking BERT as an example, we empirically illustrate that the singular value distribution of outputs from different intermediate transformer blocks on the Corpus of Linguistic Acceptability (CoLA) dataset.
The empirical CDF of singular values of the hidden states (i.e., intermediate token representations) from BERT in layer 2, 4, 6, 8, 10 and 12 is shown in Figure 1.
It clearly reveals that the CDF is steeper when closer to the origin, which indicates that the probability of singular values λ less than a small x is high ( F ( x ) =
Pr ( λ < x ) ).
With the increase of network depth, the CDF curve tends to be steeper, indicating that the shape of the spherical conelike embedding space tends to be long and narrow, leading to token uniformity.
In the last layer for the prediction (i.e., Layer 12), the representation is projected to an embedding space guided by supervised label information, therefore showing a lower degree of token uniformity.
Nevertheless, simply leveraging the supervision from labels is not enough to address the problem of vanished dimensions during deep network training.
We calculate the average cosine similarity between every token pair, and [ CLS ] tokens from different BERT layers as a proxy measure of token uniformity.
Figure 2 shows the skewness of singular values and token uniformity measurement increase as the network layer goes deeper.
We also observe the gradual vanishing of smaller singular values as the median of the singular value decreases drastically (from
[Figure]
More flattened curve indicates a more balanced distribution of singular values.
For a given x , the larger F ( x ) can cover a higher percentage of singular values which are less than x .
Hence, the top curve in this figure indicates more vanished dimensions in the embedding space.
0.12 to 0.0397).
Our results empirically show that the degree of skewness of singular value distributions is indicative of the degree of token uniformity.
TRANSFORMATION FUNCTION
Having empirically illustrating the changes of the singular value distributions of the transformer layer outputs and the measures of token uniformity across the transformer blocks, we now provide insights of designing a desirable singular value transformation function.
4.1 MOTIVATION
As in the geometric interpretation presented in Section 3, the highly skewed singular value distribution in the embedded feature space would mean that the axis of the ellipsoid is significantly longer than the corresponding axis of the sphere, leading to the token uniformity problem.
A variety of techniques have been developed to alleviate this problem, and the most popular ones are a series of normalisation methods that can be explained in a unified framework of constraining the contribution of every feature onto a sphere [Ioffe and
Szegedy, 2015, Wu and He, 2018, Ba et al., 2016].
A notable example is layer normalisation, an essential module in the Transformer architecture, which scales the variance of an individual activation across the layers to one.
One common property of these normalisation methods is that they preserve the trace of the covariance matrix (i.e., the first moment of its singular value distribution), but they do not control higher moments of the distribution, such as the skewness.
A consequence is that there may still be a large imbalance in singular values.
Here, we propose a transformation function that can adjust the skewness of singular value distributions by modifying small singular values to avoid dimension vanishing.
4.2 PROPERTIES OF DESIRABLE SINGULAR VALUE TRANSFORMATION
We want to alleviate the token uniformity problem in PTLMs by adjusting the singular value distributions of outputs of transformer layers (see Section 3.2).
Since SVD is computationally expensive 1 and a common practice is to fine-tune a
PTLM on downstream tasks, rather than applying the transformation in each transformer block 2 , we propose to only apply it in the last transformer block to modify the final output token distribution.
On one hand, we don’t want the singular value distribution to be too skewed towards very few large singular values.
On the other hand, we don’t want it to be too flat as we want to keep the relative difference between singular values learned from powerful pre-trained models.
To this end, we propose the following three desirable properties of an singular value transformation function f ( λ ) :
1) f ′ ( λ ) ≥
The large PTLMs have achieved promising performance in a broad spectrum of NLP tasks, showing their capabilities in learning good feature representations.
In order to preserve the original data manifold that is mainly defined by the feature singular values, we would like to keep the original order of the singular values.
As the input to f is a monotonically increased singular value sequence, f should be monotonically increasing: f ( λ i ) >
f ( λ j ) , [Equation]
2) f ′′ ( λ ) ≤ 0 To make the transformed singular value distribution more balanced while keeping the largest singular value unchanged, intuitively, we should increase the smaller singular values.
The increment ∆ i for each singular value is defined as f ( λ i ) − λ i and ∆
=
(i.e., the largest singular value is kept unchanged).
To reduce that gap between larger and smaller singular values, we propose a simple solution that a smaller singular values should have an equal or larger increment than those larger ones while preserving the original order of the singular values.
That is: ∆ i ≥ ∆ j , if λ i < λ j , where ∆ i =
f
( λ i ) − λ
i i.e., f ′ ( λ i ) − f ′ ( λ j ) λ
i − λ j ≤ 0 , the second-order derivative of f should be monotonically decreasing.
3) f ( λ max ) ≈ λ max
To guarantee the bounded embedding space, we require the transformation keep the largest singular value unchanged as much as possible.
Existing studies have also shown that the largest singular value of the data covariance matrix is significant to model training [Pennington and Worah, 2017].
4.3 SOFTDECAY FUNCTION
We develop a non-linear and trainable function built on the soft-exponential function [Godfrey and Gashler, 2015].
[Equation]
Specially, when α < 0 , these curves are monotonically increasing with smaller slop.
This is consistent with our properties (1)(2).
For the property (3), it can be proved that for any λ , there is λ ≥ f ( λ ) when α < 0 .
Hence, we have λ max ≥ f ( λ max ) ≥ λ 2 , where λ 2 is the second largest singular value.
Combing the desirable properties of singular value distribution and the non-linear transformation function, we describe our proposed transformation in Algorithm 1:
4.4 TRANSFORMED FEATURE EVALUATION
Existing research in text representation learning showed that the features should be roughly isotropic (i.e., directionally uniform)
[Mu and Viswanath, 2018, Zhang et al., 2020,
Wang et al., 2020, Li et al., 2020, Su et al., 2021] to prevent the feature space squeezing into a narrow cone and preserve as much information of the data as possible.
We argue that the evaluation of transformed features should consider both the uniformity and the preservation of local neighbourhood structure in the original embedding space. dimension
Uniformity.
We propose to measure the distribution uniformity in three different ways.
First, we examine the features similarity ( TokenUni ): TokenUni
[Equation]
where f ( · ) transforms an input feature by the SoftDecay .
Second, we use the Radial Basis Function (RBF) kernel, RBF dis , to measure feature similarity, as it has been shown a great potential in evaluating representation uniformity [Wang and Isola, 2020].
[Equation]
where t is a constant.
We use the logarithmic RBF dis in experiment.
Finally, as few predominant singular values will result in an anisotropic embedding space, we can check the difference of variances in different directions or singular values and use the E xplained V ariance ( EV k )
[Zhou et al., 2021]:
[Equation]
where λ i is the i -th singular value sorted in a descending order, m is the number of all the singular values.
In the extreme case when EV 1 approximates to 1, most of the variations concentrate on one direction, and the feature space squeezes to a narrow cone.
Preservation of Local Neighbourhood.
Ideally, the transformed embeddings should preserve the local neighbourbood structure in the original embedding space.
Inspired by the Locally Linear
Embedding [Roweis and Saul, 2000], we propose the L ocal S tructure D iscrepancy Score ( LSDS ) to measure the degree of preserving the original local neighbourhood.
First, for a data point x i in the original embedding space, we choose its k -nearest-neighbours, then define the weight connecting x
i and its neighbour x j as the distance measured by the RBF kernel, w ij = exp( −∥ x i − x j ∥ 2 /t ) .
In the transformed space, the new feature ˜ x i = f
( x i ) is supposed to be close to the linear combination of its original neighbours in the transformed space weighted by the distance computed in the original space:
[Equation]
where N ( x i ) denotes the k -nearest-neighbours of x i .
EXPERIMENTS
We implement our proposed transformation functions on four transformer-based Pre-Trained Language Models (PTLMs), BERT, ALBERT, RoBERTa and DistilBERT, and evaluate on semantic textual similarity (STS) datasets and
General Language Understanding Evaluation (GLUE) tasks [Wang et al., 2019], including unsupervised and supervised comparison.
5.1 UNSUPERVISED EVALUATION ON STS
Setup The STS task is a widely-used benchmark of evaluating sentence representation learning.
We conduct experiments on seven STS datasets, namely, the SICK-R
[Marelli et al., 2014], and the STS tasks (Agirre et al.
We compare our approach with unsupervised methods on adjusting anisotropy in STS tasks, including
BERT-flow [Li et al., 2020], SBERT-WK
[Wang and
Kuo, 2020], BERT-whitening [Su et al., 2021] and
WhiteBERT [Huang et al., 2021].
Our SoftDecay is directly applied to the last layer of the original PTLMs to derive the transformed sentence representation without any fine-tuning 4 .
Results It can be observed from Table 1 that: (1) whitening-based methods ( BERT-Whitening and
WhiteBERT ), which transform the derived representations to be perfectly isotropic, perform better than the other baselines such as BERT-flow , which applies a flow-based approach to generate sentence-embedding from a Gaussian distribution.
Our proposed SoftDecay gives superior results across all seven datasets significantly, 23.5% of improvement over the base PLTMs and 5% over the best baseline on BERT-based methods.
(3) When comparing the results from different PLTMs, we observe more significant improvements on the ALBERT-based models (23%), and modest improvements on the DistilBERT-based models (8%).
This is somewhat expected as the token uniformity issue is more likely to occur in deeper models.
Therefore, less obvious improvements are found on DistilBERT with only 6 layers, compared to others with 12 layers.
The crosslayer parameter sharing in ALBERT could potentially lead to more serious token uniformity, and thus benefits more from the mitigation strategies.
To further understand how
SoftDecay alleviates token uniformity, we show the CDF of singular values from DistilBERT and ALBERT before and after applying SoftDecay in Figure 4.
We can observe that before applying SoftDecay , the outputs of ALBERT across various layers are very similar while the outputs of
DistilBERT across different layers are more different.
After applying SoftDecay , the singular value distribution of the last layer output (red curve) of ALBERT is less skewed compared to DistilBERT (the brown curve).
Feature Evaluation To gain insights into the characteristics of desirable features for the STS task, we visualise the sentence representations in STS-15 via tSNE and present the results using our proposed metrics in
Figure 3.
BERT-Whitening transforms vanilla features from BERT into perfectly isotropic distribution, which is evidenced in results of the uniformity measures that nearly all the features are orthogonal to each other as TokenUni is zero and they have the smallest RBF dis .
It also has the lowest EV k score of its top singular value.
However, BERT-Whitening fails to preserve the local neighbourhood of BERT embeddings in its transformed space as shown by its larger Local Structure Discrepancy Score ( LSDS ) compared to SoftDecay .
By contrast,
SoftDecay not only significantly improves the uniformity compared to the vanilla BERT feature distribution, but also maintains a similar distribution shape.
Our results show that transforming learned text representations into isotropic distributions does not necessarily lead to better performance.
Our proposed SoftDecay is better in preserving the local neighbourhood structure in the transformed embedding space, leading to superior results compared to others. In Appendix C.3, we also discuss the connections and differences between SoftDecay and a representative contrastive learning method SimCSE
[Gao et al., 2021], which also considers the uniform distribution as a desirable property of representation learning.
5.2 SUPERVISED EVALUATION ON GLUE DATASETS
Setup We evaluate our method on five sentence-level classification datasets in GLUE [Wang et al., 2019], including grammar acceptability assessment on the Corpus of Linguistic Acceptability (CoLA)
[Warstadt et al., 2019], sentiment classification on the Stanford Sentiment Treebank (SST2)
[Socher et al., 2013], paraphrase detection on the
Microsoft Research Paraphrase Corpus (MRPC)
[Dolan and Brockett, 2005], natural language inference on the QuestionAnswering NLI (QNLI) data and the Recognizing Textual
Entailment (RTE) data.
We apply our proposed SoftDecay on top of the last
[Table] [Table]
TokenUni( [Table] see in §4.4) are listed on the right. We can see our method preserves the local neighbourhood structure better than Whitening with lower LSDS and address token uniformity in BERT well with lower scores in the first three metrics.
encoder layer in BERT, ALBERT and DistilBERT, and then fine-tune the PTLM weights, along with α on different tasks.
In addition to the PTLMs, we include two more baselines, i.e., sentence-level embedding learning models, SentenceBERT ( S-BERT for short)
[Reimers and Gurevych, 2019] and BERT-CT [Carlsson et al., 2021] .
• S-BERT adds a pooling operation to the output of BERT to derive a sentence embedding and fine-tunes a siamese BERT network structure on sentence pairs. • BERT-CT improves the PTLMs by incorporating contrastive loss in the training objective to retain a semantically distinguishable sentence representation.
The two methods aim at making the sentence-level embeddings more discriminative, which in turn alleviate the token uniformity problem.
Since GLUE did not release the test set, the test results can only be obtained by submitting the trained models to the GLUE leaderboard 8 .
We show the test results returned by the GLUE leaderboard in Table 3.
Results It can be observed from Table 2 that SoftDecay is more effective on BERT-based model, while gives less noticeable improvement on DistilBERT, similar to what we observed for the STS tasks since DistillBERT has fewer layers.
For the vanilla PLTMs, BERT has the better results over all the single-sentence tasks (except for MRPC, sentencepair paraphrase detection).
All the three models achieve better results on inference task (QNLI and RTE), especially [Table] [Table] on the smaller dataset RTE.
The CDF of singular value distributions on RTE before and after applying SoftDecay shown in Figure 5 further verifies the effectiveness of our proposed transformation function.
We also observe that models trained on a larger training set tend to generate more similar representations 9 .
On MRPC, using SoftDecay is effective on BERT, but gives slight performance drop on
ALBERT and DistilBERT.
One possible reason is the much smaller training set size.
On the GLUE test results shown in Table 3, we observe that SoftDecay outperforms both
S-BERT and BERT-CT across all tasks.
rte
CONCLUSION AND FUTURE WORK
In this paper, we have empirically shown that the degree of skewness of singular value distributions correlates with the degree of token uniformity.
To address the token uniformity problem, we have proposed a singular value transformation function by alleviating the skewness of the singular values.
We have also shown that a perfect isotropic feature space fails to capture the local neighborhood information and leads to inferior performance in downstream tasks.
Our proposed transformation function has been evaluated on unsupervised and supervised tasks.
Experimental results show that our methods can more effectively address token uniformity compared to existing approaches.
Our paper explores the token uniformity issue in the information propagation in the transformer encoder, where self-attention is used.
It would be promising to extend our approach to the encoder-decoder structure and explore its performance in language generation tasks.
One promising future direction is to improve the generation diversity via addressing the token uniformity, as 1) many existing works show that anisotropy is related to the word frequency 2)
In the decoding phase, sampling words from more isotropic word embedding distributions could potentially lead to more diverse results.
Acknowledgements
This work was funded by the the UK Engineering and Physical Sciences Research Council (grant no.
EP/T017112/1,
EP/V048597/1).
YH is supported by a Turing AI Fellowship funded by the UK Research and Innovation (grant no.
EP/V020579/1).