Speeding Up Entmax
Abstract
Softmax is the de facto standard for normalizing logits in modern neural networks for language processing.
However, by producing a dense probability distribution each token in the vocabulary has a nonzero chance of being selected at each generation step, leading to a variety of reported problems in text generation.
αentmax of Peters et al.
(2019) solves this problem, but is unfortunately slower than softmax.
In this paper, we propose an alternative to αentmax, which keeps its virtuous characteristics, but is as fast as optimized softmax and achieves on par or better performance in machine translation task.
Introduction
Sparseness of vector representations is a desirable trait in neural network models for natural language processing (NLP): words (subwords) are discrete objects by their nature, and, accordingly, are encoded by one-hot embeddings at the input and output of neural networks.
However, to predict a categorical response in neural models, softmax is most often used, which produces a dense probability distribution, i e. every category (word/subword) receives a non-zero probability.
Recent studies suggest that it is this output density that poses problems when the trained NLP model is used for inference.
For example, in the case of text generation, unconstrained sampling from a trained language model results in poor quality of the resulting text (Holtzman et al., 2020).
In neural machine translation (NMT), exact decoding from a trained model often results in empty text (Stahlberg and Byrne, 2019).
1 To get around these problems, constrained decoding techniques have been proposed, most of which artiﬁcially impose sparsity on softmax prediction.
For example, Fan et al.
(2018) propose to sample from the top- k probable words, and Holtzman et al.
(2020) propose to sample from the most probable words, which comprise the cumulative probability p .
While these methods are effective, they are ad-hoc solutions that lead to a mismatch between how the model is trained and how it is used at inference.
In this regard, the works on sparse alternatives to softmax stand apart since they allow us to make inference from the model in the same way than it was trained.
Some of the most successful and elegant solutions are sparsemax (Martins and Astudillo, 2016) and its generalization α -entmax (Peters et al., 2019).
When coupled with suitable losses, these transformations are not inferior to softmax, and sometimes even surpass it as measured with ﬁnal performance metrics on a number of tasks.
A problem with these transformations however is that they are signiﬁcantly slower than softmax when the number of categories (vocabulary size) is tens of thousands, as in the case of text generation.
This is because α -entmax transformation—in its original formulation—requires sorting over the logits.
In this work, we ask the question: is it possible to obtain a sparse output like that of α -entmax, but without its degradation in computational speed?
Our answer is afﬁrmative—we propose a sparse output transformation that
• is on par or superior to softmax and α -entmax in the NMT tasks, • works as fast as softmax during training and at inference, • gives the same training dynamics as α -entmax (in training steps).
The most surprising thing is that such a transformation is simply a shifted ReLU raised to power
α − 1 , which we call α -ReLU .
The rest of the paper is organised as follows.
In
Sect.
2 we motivate the choice of α -ReLU as the output transformation, and also select an appropriate loss function.
In Sect.
3 we experimentally conﬁrm our claims about performance and output speed of α -ReLU in the NMT task.
Sect.
4 is devoted to a comparative analysis of α -ReLU and α -entmax in terms of sparsity, ability to solve the empty translation problem, and training dynamics.
α -ReLU at Output
Our departure point is the α -entmax transformation of Peters et al.
(2019) which can be deﬁned for z ∈ R d as
[Equation]
where [ x ] + := max { x, 0 } , and τ : R d → R is the (unique) function that satisﬁes (cid:80) j
[( α − [Equation]
z j − τ ( z )] 1 α [Equation] 1 + = 1 for any z .
It is this threshold τ that makes the computation of α -entmax slow, because one needs to sort the components of z to ﬁnd τ (Peters et al., 2019, Alg. 2).
As we can see, the threshold τ is only needed to ensure that α -entmax( z ) is a probability distribution.
We loosen this constraint, and only require non-negative weights, which is sufﬁcient for most uses.
Consider then a transformation
[Equation]
where τ is a constant that does not depend on z .
In order to force α -ReLU( z ) —applied to the logits z —to converge to the one-hot vector e y of the gold label y we need to adjust the corresponding loss.
This can easily be done by feeding the logits z and the output α -ReLU( z ) into the following loss, which we call α -ReLU loss .
(cid:96)
[Equation]
where H α [ p ] :=
[Equation]
[Equation]
1 , is the Tsallis α -entropy (Tsallis, 1988), and 1 := . , 1) ∈ R
d is a vector of ones.
The rationale for coupling α -ReLU with the loss (3) is the following
Lemma 1.
For any τ ∈ R , the gradient of the α -ReLU loss (3) is given by
[Equation]
Proof.
The proof is in Appendix B.1.
By Lemma 1, gradient-based minimization of (cid:96) indeed forces α -ReLU
( z ) → e y .
Notice that this is similar to what happens when the softmax normalization is coupled with the cross-entropy loss or when α -entmax is coupled with the entmax loss.
In both cases differentiating the loss with respect to logits gives p − e y , where p is either softmax( z ) or α -entmax( z ) (Martins and Astudillo, 2016; Peters et al., 2019).
Remark.
Recall that α -entmax is a generalization of sparsemax.
For example, 2-entmax is essentially sparsemax, and for α ∈ (1 , 2) we get a smoothed version of sparsemax.
Similarly, αReLU is a kind of generalization of ReLU.
So, the standard ReLU is 2-ReLU (with τ = 0 ), and for α ∈ (1 , 2) we get a smoothed ReLU (see Fig. 1). , 2] , with τ = 0 . 2-ReLU is a standard ReLU( x ) := [ x ] + .
Experiments
In theory, nothing prevents α -ReLU from learning what α -entmax is learning.
However, in practice we can have a different picture, because training is conditioned by many factors—the size of the dataset, the architecture of the neural network, the optimization algorithm, etc.
In this section, we compare α -ReLU empirically with α -entmax (as well as with sparsemax and softmax), assuming all other factors are ﬁxed.
The goal of these experiments is to evaluate the consequences of using α -ReLU as drop-in replacement for α -entmax.
We test α -ReLU at output in a neural machine translation task (Sutskever et al., 2014), which is essentially a conditional text generation task.
Compared to open-ended text generation, there is a
[Table] Transform
Loss
IWSLT De
→ En
WMT En → De
WMT
[Table]
→ Ru
[Table]
clearer metric of the quality of the generated text— the BLEU score (Papineni et al., 2002).
As in open-ended text generation, at each prediction step, the NMT system needs to make a choice from all words (subwords) of the vocabulary, the size of which can reach several tens of thousands.
Therefore, the sparsity of the output distribution becomes critical in such setups, since it can explicitly prevent the occurrence of most of the words that are inappropriate in the context.
3. Setup Data.
We conduct experiments on three datasets of varied sizes:
• IWSLT’14 De
→ En (Cettolo et al.), 172K training examples, • WMT’
En → De (Bojar et al., 2014), 4.5M training examples, • WMT’
En → Ru (Bojar et al., 2013),
1.3M
tranining examples.
We preprocess all datasets using the byte pair encoding algorithm (Sennrich et al., 2016) with 10K merge operations on IWSLT, 40K merge operations on WMT
En → De, and 60K merge operations on WMT
En → Ru.
We report detokenized casesensitive BLEU with SacreBLEU (Post, 2018).
Hyperparameters α and τ .
In all experiments we set α = 1 . 5 , because this value was recommended by Peters et al.
; Peters and Martins (2021) as the middle ground between α = 1 (softmax) and α = 2 (sparsemax).
The value for τ is chosen as follows: we run the ﬁrst batch through a non-trained neural network, which has 1.5-entmax at the output, in the forward direction and determine the average τ value across the batch.
This value is then used to train the 1.5ReLU network.
Our preliminary experiments have shown that 1.5-ReLU convergence is sensitive to the τ value, and that having output close to the probability distribution early in the learning phase works well with the rest of hyperparameters which are set to their default values.
Training.
We trained the Transformer Base (Vaswani et al., 2017) using the OpenNMT-py 2.
toolkit (Klein et al., 2017).
Optimization details are in Appendix A.
3. Results
The results are given in Table 1.
Reported are test BLEU scores for best checkpoints which are selected based on validation BLEU.
We observe that the 1.5-ReLU performs on par with 1.5-entmax or better, while sparsemax is inferior to all others.
Training Time.
Fig.
2&3 show the training dynamics in training steps and in wall time on
WMT’
En → De.
Despite the closeness of performance in intermediate steps and at the end of training, we see that on the larger datasets 1.5-entmax is slower in wall time than softmax and 1.5-ReLU.
To speed up the learning process, Peters et al.
(2019) recommended limiting the number of sorted logits in the α -entmax to the k largest logits.
We tried this[-on the WMT’
 En → De dataset-] using k = 100 , which is the default value in the author’s implementation of α -entmax.
The resulting training dynamics are shown as dashed curves in Fig.
As we can see, partial sorting indeed speeds up the learning process, and at the same time does not harm the quality of the translation compared to α -entmax with full sorting.
But in the end, learning is still slower than in the case of 1.5-ReLU.
Of course, one can try to select such k that the speed of calculating the 1.5-entmax will be as close as possible to the speed of 1.5-ReLU without losing quality, but this requires additional efforts on the part of the user, and this must be done for each case separately.
Also note that both 1.5-entmaxes (with full and partial sorting) cannot learn the English-Russian data set as well as 1.5-ReLU.
In this regard, 1.5-ReLU does not require additional ﬁne-tuning, converges as fast as softmax in absolute time and performs on par or better.
Thus 1.5-ReLU combines all three desired properties: computation speed, task performance, and sparsity of output.
Inference Time.
We measured inference time of translating the WMT
En → Ru test data with the different strategies and with different beam sizes.
The results—normalized by the smallest value— are shown in Fig.
As can be seen the relative difference seems independent of the beam size: softmax is almost twice faster than 1.5-entmax (with full sorting over the logits).
Even though the softmax version is optimized through the softmax CUDA kernel, it performs equivalent to the 1.5-ReLU model in terms of computation speed.
Analysis 4. Empty Translations
We remind the reader that the cat got your tongue problem (Stahlberg and Byrne, 2019) is one of the main motivations for using sparse transformations when generating text.
As Peters and Martins (2021) have shown, 1.5-entmax successfully tackles this problem by signiﬁcantly lowering the proportion of cases where an empty string is more likely than the beam search hypothesis.
For 1.5-ReLU, we also calculated this proportion, and compared it with the proportions for softmax and sparsemax (Table 2).
As we see, 1.5-ReLU also successfully tackles the cat got your tongue problem.
Output
IWSLT
WMT WMT
Transform
De
→ En
En → De
En → Ru softmax 7.5% 29.8% 31.7% sparsemax 0% 0.03% 0% 1.5-entmax 0% 0.2% 0% 1.5-ReLU 0% 0.3% 0.1%
4. Sparsity
To compare the sparsity of 1.5-ReLU and 1.5entmax we depict in Fig.
5 the distributions of the number of zero components after applying these transformations (recall that for softmax all components are always nonzero).
Since we constructed the α -ReLU in such way that it mimics the αentmax (at least in the early stages of training), we expected that these two transformations would have similar properties, including sparsity.
However, this is not the case: as we can see, the 1.5-ReLU is signiﬁcantly less sparse than the 1.5-entmax.
It is noteworthy that lower sparsity in this case correlates with a better performance in the translation task (see Table 1).
A possible explanation for the difference in sparsity levels could be that α -ReLU, in contrast to α -entmax, behaves signiﬁcantly differently on the test set than on the training set.
However, this is not the case: for example, comparing the sparsity on the IWSLT training set (Fig. 6), we see that the distributions of non-zero components are almost the same as on the test set for 1.5-ReLU and 1.5-entmax.
Note that the sparsity of α -ReLU and α -entmax is approximately the same at the beginning of training due to how we initialize τ in 1.5-ReLU (making it as close as possible to 1.5-entmax’s τ in the untrained model, Sec. 3.1).
However, during training, α -ReLU ’s τ remains ﬁxed , and the model can only adapt the logits themselves so that α -ReLU( z ) converges to the corresponding one-hot vector. At the same time, in α -entmax , τ ( z ) adapts together with logits z . We hypothesize that during training, the entmax’s τ ( z ) gradually increases which entails greater sparsity by the end of the training. However, the logits themselves also change during training, so the increase in τ may not be the cause of greater sparsity.
To ﬁnd out, we track the dynamics of mean logit norm (cid:107) z (cid:107) and mean τ during training for both 1.5-entmax and 1.5-ReLU (Fig. 7).
As we can see, the logit sizes grow in both cases.
At the same time, the 1.5-entmax’s τ ( z ) increases following the logit size, while the 1.5-ReLU’s τ remains constant.
From this we conclude that the sparsity of 1.5-entmax is inevitably less than the sparsity of 1.5-ReLU.
4. Impact of τ
The selection of τ was described in Section 3.1.
However, the question arises: does the described approach lead to the choice of the optimal τ ?
To ﬁnd out, we trained the α -ReLU models for τ ∈ { 0 , 0 . 1 , 0 .
9 , 1 , 2 , 5 , 10 } on the IWSLT data.
Note that all of these τ ’s have led to almost the same result at the end of the training (as predicted by Lemma 1).
In Fig.
8, we present the dynamics of early training only for τ ∈ { 0 , 0 . 1 , 0 . 2 , 0 . 3 , 5 , 10 } , since the curves for τ ∈ { 0 . 4 , ..., 0 . 9 , 1 , 2 } practically coincided with the optimal curve corresponding to τ = 0 .
Note that our τ selection method gave a value of 0 .
33 , thus we have no evidence against the adequacy of our method.
4. Estimation of τ without data
On closer inspection, we noticed that the preentmax logits in the untrained Transformer model are distributed according to the normal law, regardless of what data is supplied to the input, ShapiroWilk test, p-value > 0 . 15 .
This allows us, using asymptotic theory, to estimate τ as
[Equation]
where d model is the size of hidden representations, d vocab is the vocabulary size for a target language, Φ − 1 ( · ) is the probit function and p ∗ is the solution of a non-linear equation that involves functions related to the standard normal distribution (see Appendix B.2 for details).
Table 3 compares the ˜ τ calculated by running data through an untrained model with the estimate ˆ τ obtained from (4).
As we can see, ˆ τ practically coincides with ˜ τ with an
IWSLT’
WMT’ WMT’
De
→ En
En → De
En → Ru
[Table]
[Table]
accuracy of two decimal places.
Unfortunately, the formula (4) is not universal: it is only true for the
Transformer architecture.
4. Self-normalization
The attentive reader may have noticed that the output of α -ReLU is not normalized, i.e. the components of α -ReLU( z ) do not have to sum up to 1.
Accordingly, the question arises: how correct is it to compare translation scores at different steps of the beam-search decoding if the conditional probabilities are not normalized?
However, the comparison is possible if the α -ReLU( z ) components add up to approximately the same number, i.e. if the model is self-normalizing.
To check this, we ran the trained α -ReLU model on the IWSLT and
WMT’14 test sets, and looked at the distribution of (cid:80) i
α -ReLU
i ( z ) at each decoding step.
The results are shown in Fig.
As we can see, the sum of the α -ReLU( z ) components concentrates well around its mean ≈ 1 .
24 (IWSLT) and 1 .
(WMT’14), which might indicate that the model indeed has a self-normalization property.
4. Training Dynamics
As we noted in Sect.
3.2, the training dynamics are similar in all three cases (softmax, 1.5- entmax, 1.5-ReLU) when time is measured in training steps.
Here we attempt to explain this phenomenon through the recently proposed Neural
Tangent Kernel (NTK) approach of Jacot et al.
Roughly speaking, the NTK theory suggests that a sufﬁciently wide neural network trains like a kernel regression.
We use this theory to show (in Appendix B.3) that in all three cases the logits z ( x, t ) for a training instance x at a training step t evolve (approximately) according to the same differential equation
[Equation]
where expectation is over training examples ( x (cid:48) , y (cid:48) ) , σ ( · ) is one of the transformations considered (softmax, α -entmax, or α -ReLU), and
K σ ( x, x (cid:48) )
∈ R
d × d is a positive semi-deﬁnite matrix that depends on σ .
The Equation (5) is a non-linear matrix differential equation which in general cannot be solved analytically.
However, it has an equilibrium point z ( x, t ) such that
E ( x (cid:48) ,y (cid:48) )
[ K σ ( x, x (cid:48) ) · ( σ ( z (cid:48) )
− e y (cid:48) )]
= 0 , thus its solution converges to this point as t → ∞ .
This similarity in the evolution of σ ( z ) implies the similarity in the evolution of the perfomance metric—such as
BLEU—across all three transformations.
4. Human Evaluation
Although the BLEU metric (Papineni et al., 2002) has stood the test of time, it is still an automated assessment of translation quality.
To double-check the reliability of the results from Table 1, we decided to manually evaluate the translations from the WMT’
En → Ru test split.
To do this, we followed the human evaluation setup from (Berard et al., 2019).
We formed two random samples of 135 instances each and gave them to two annotators.
45 instances were shared across two samples in order to calculate inter-annotator agreement.
Each instance consists of an original sentence in English and 4 candidate translations into Russian (reference, softmax, entmax, α -ReLU).
The annotators were to rate each translation on a 4-point scale.
For annotation instructions, see Appendix C.
The order of candidate translations was shufﬂed for each instance, so the annotators did not know which sentence is from which model.
Nevertheless, the annotator always had a good chance of guessing which translation was the reference one, due to the large difference in quality between human and machine translation. [Table]
The results of human evaluation are shown in
Table 4.
Cohen’s κ = 0 .
56 , indicating moderate agreement between annotators.
As we can see, all three models give approximately the same translation quality, and all three are signiﬁcantly inferior to the reference translation.
This is generally consistent with the results of 1.5-ReLU and 1.5-entmax in Table 1, but at the same time casts doubt on the softmax lag behind 1.5-ReLU and 1.5-entmax as the BLEU metric suggests.
In Appendix D we give a few examples where 1.5-ReLU translates better than 1.5-entmax and vice versa.
Related Work
Sparse seq2seq models.
Our proposed α -ReLU transformation is based on the α -entmax transformation of Peters et al.
(2019), which in turn is a generalization of the sparsemax transformation (Martins and Astudillo, 2016).
In our work, we study sparseness at the output of a neural network.
Nevertheless, there are a number of works aimed at sparsiﬁcation within a neural network.
For example, Malaviya et al.
; Peters et al.
Correia et al.
(2019) show that sparsemax and αentmax can replace softmax in the attention mechanism with some success.
A recent work of Zhang et al.
(2021) attempted to replace softmax with a component-wise ReLU in the attention mechanism.
Unfortunately, in its pure form, this replacement leads to the inability of the model to learn at all, since its loss function does not decrease during optimization.
The authors solve this problem by adding a normalizing layer on top of the attention layer.
These and other works (Zhang et al., 2019) state that sparsity in the weights of attention produces more interpretable patterns.
However, Meister et al. (2021) questioned this claim and were unable to ﬁnd clear evidence to support it.
Therefore, in this work, we focused on the application of α -ReLU to the output of the transformer model, and not to the mechanism of attention, but at the same time we do not deny the possibility of studying the latter.
Self-normalization.
Self-normalizing training aims to bypass the need of normalization during inference time.
This is done by tweaking the learning mechanism so that the sum of all predictions sums (approximately) to a constant value.
Theoretical work on why this works is poorly understood (Andreas et al., 2015) but early work in neural machine translation has shown its empirical value.
Vaswani et al.
(2013) achieves that by using noisecontrastive estimation (the neural model is used to re-rank the output of a hierarchical phrase-based machine translation system).
Noise-contrastive estimation is also the standard training mechanism for word2vec (more popular than the alternative hierarchical softmax), which also eschews any expensive normalization.
Differently, Devlin et al.
(2014) changes the training loss to include a factor that encourages the normalizing factor to be 1.
At inference time, this is just assumed and decoding time is reported to achieve a 15x speed-up.
Limitations and Risks
We believe that the main limitations of our work are as follows:
• α -ReLU’s output is still not a probability distribution, as required by the classical formulation of a probabilistic classiﬁcation model.
• τ evaluation requires either running the data through an untrained model with α -entmax at the output, or deriving a formula similar to (4) for each individual architecture.
• Our approach only works for the case when αReLU is used at the output of the model, but it is not clear how to use it as an alternative to softmax/ α -entmax in the attention layer.
The last mentioned limitation leads to the potential risk of inability to learn if α -ReLU is misused in the intermediate layers of the neural network such as attention layers.
The experiments of Zhang et al.
(2021) using vanilla ReLU (2-ReLU with τ =in our notation) instead of softmax to produce attention weights lead to a divergence of the loss function of the Transformer model.
This translates into a waste of energy, especially when training large models on large datasets.
Therefore, we believe that in the future, a preliminary mathematical analysis and/or experiments with small models on small datasets should be carried out as to why the unnormalized distribution of attention weights leads to the inability of the model to learn.
Conclusion
It seems that the sparsity of the output is natural for (sub)word prediction models.
Nevertheless, sparsity does not have to come with slowdown of computations, as our work shows.
The proposed transformation, α -ReLU, gives a sparse output, shows competitive performance, and is as fast as softmax.
The reduced dependency on the vocabulary size seems particularly important in translation, where neural models are moving more and more towards multi-lingual ones, which in general have a much higher vocabulary size in order to accommodate enough tokens for all languages.
A natural extension of this work will be the evaluation of α -ReLU in the problem of open-ended text generation, as well as a replacement for softmax in the attention layers of Transformer models. Our standalone implementation of α -ReLU in PyTorch is available at https://github. com/MaxatTezekbayev/alpha-relu . Acknowledgements
The authors would like to thank Christopher Dance, André Martins, Bill Byrne, Vlad Niculae and anonymous reviewers for their feedback. The work of Maxat Tezekbayev and Zhenisbek Assylbekov is supported by the Nazarbayev University Collaborative Research Program 091019CRP2109. A. GPU Power Consumption
Dataset
IWSLT’
En → De
WMT’14 De
→ En
WMT’
En → Ru
GPU(s) 1 × RTX 2080 Ti 2 × RTX4 × Tesla V100 SXM
Power consumption, W 2 ×4 ×
Training time, hours softmax 15.
sparsemax 24.
1.5-entmax 26.
1.5-ReLU 16.
TOTAL hours 82.
TOTAL kW-hours 20.
GRAND TOTAL kW
[Table]
hours 365. We do not report CO 2 consumption, as experiments were run in different countries, making aggregate statistics difﬁcult to compute.
The largest experiment (on WMT’13), were run in France, which beneﬁts from a very low CO 2 emission intensity in its electrical mix.
B Proofs
Notation.
We let R denote the real numbers.
Bold-faced lowercase letters ( x ) denote vectors in
Euclidean space, bold-faced uppercase letters ( A ) denote matrices, plain-faced lowercase letters ( x ) denote scalars, (cid:107) · (cid:107) denotes the Euclidean norm: (cid:107) x (cid:107) :=
√ x (cid:62) x .
The gradient of f : R d → R is denoted by ∇ f .
The Jacobian of z (cid:55)→ g ( z ) is denoted by J g ( z ) .
Also, we denote ReLU( x ) :=
[ x ] + := max { x, 0 } , [ d ] := { 1 , . . .
, d } , ∆ d − 1 := { p ∈ R d | (cid:80) i p i = 1 , p i ≥ 0 } , e y :
= (0 , . .
. , 0) where 1 is at y th position.
B.
Proof of Lemma 1.
First, let us calculate the Jacobian of the mapping z (cid:55)→ α -ReLU
( z ) .
Recall that
[Equation]
Therefore, the partial derivatives are given by
[Equation]
Thus, the Jacobian can be written concisely as
[Equation]
where raising to power is done component-wise (i
[Equation]
e. x β =
[ x β 1 , . .
, x βd ] ), and diag[ x ] is a diagonal matrix with x on its diagonal.
Recall the deﬁnition of the Tsallis α -entropy:
[Equation]
Its gradient w.r.t.
p is
[Equation]
Combining this with (6)
[Equation]
and using the chain rule
[Equation]
we have
[Equation]
where (cid:12) is the Hadamard product (element-wise multiplication), and we used diag[ x ] · y = x (cid:12) y .
Taking into account (7), the gradient of the α -ReLU loss (3) w.r.t.
z is ∇ z (cid:96) ( z , y ) =
∇ z
[Equation]
[Equation]
[Equation]
= ( α -ReLU( z ) − e y
[Equation]
+
[Equation]
= α -ReLU( z ) − e y , where in the fourth line we used [ x ] β + (cid:12) x
= [ x ] β + (cid:12) [ x ] + =
{+[ x ] β +
+}
[-[ x ] β +
-]
+ .
This concludes the proof.
B. Approximation of τ for 1.5-entmax
We derive the formula (4) in two steps: ﬁrst in Lemma 2, we approximate τ ( z ) of 1.5-entmax when z is an arbitrary random sample from the normal distribution with zero mean and variance σ 2 ; next in Lemma 3, we compute σ 2 for the case when z is the pre-softmax vector of logits in the Transformer model.
Lemma 2.
Let z 1 , . .
z d be independent and identically distributed random variables from the normal distribution N (0 , σ 2 ) .
Then the thresholding function of 1 . 5 - entmax( z ) can be approximated as
[Equation]
where Φ − 1 ( · ) is a probit function, and p ∗ is the solution of
[Equation]
with
[Equation]
Proof.
Let z (1) ≥ . .
≥ z ( d ) be a sorting of z 1 , . .
z d in descending order.
Peters et al.
(2019) showed that
[Equation]
where k ∈ [ d ] is any index that satisﬁes
[Equation]
with
[Equation]
Approximating z ( i ) by its asymptotic mean σ Φ − 1 (cid:0) 1 − id (cid:1) (Arnold et al., 2008), and denoting (cid:15) :
= 1 d , p :=
kd , we have
[Equation]
where we approximated the average of ﬁnitely many numbers { Φ − 1 (1 − i/d ) } ki =1 by the mean value of the function Φ − 1 (1 − x ) , and then we used the fact that − φ (Φ − 1 ( x )) is an antiderivative for the probit function Φ − 1 ( x ) ; and m ( p ) is deﬁned by (8).
Similarly, for the second empirical moment, we have
[Equation]
and thus
S
[Equation]
where s ( p ) is deﬁned by (9).
Hence, ﬁnding k ∈ [ d ] that satisﬁes (11) is (approximately) equivalent to ﬁnding p ∈ (0 , 1) that satisﬁes
[Equation]
Let p ∗ be the solution of (12)
[Equation]
Then, taking into account (10), we have
[Equation]
which concludes the proof
[Equation]
Lemma 3.
Let z
=
Wx be a pre-softmax vector of logits in the OpenNMT-py (Klein et al., 2017) implementation of the Transformer model (Vaswani et al., 2017).
Then for any input, in a non-trained model the logits z 1 , . .
z d are distributed according to the normal distribution N (cid:16) 0 ,
[Equation]
d vocab (cid:17) , where d model is the size of hidden representations, and d vocab is the vocabulary size for a target language.
Proof.
The default Transformer conﬁguration in OpenNMT-py implies that the elements w ij of W are initialized from a uniform distribution U [ − a, a ] , where a = (cid:113)
[Equation]
, thus
[Equation]
Since x is the result of a layer normalization (Ba et al. [Equation] [Equation] we have
[Equation]
Therefore, from (13) and (14)
[Equation]
we have
[Equation]
Being a sum of independent random variables, by the Central Limit Theorem, each z i tends to normal distribution with the mean and variance above.
B. Derivation of the Equation (5)
We provide derivation for the case of α -ReLU.
Extension to α -entmax and softmax is done analogously.
Let x ∈ R n 0 be the input vector.
We deﬁne a feedforward neural network with L − 1 hidden layers recursively:
[Equation]
where W ( k − 1) ∈ R n k × n k − 1 is the weight matrix in the k th hidden layer, and σ ( · ) is a nonlinear activation function applied element-wise.
We consider the case of a multi-label classiﬁcation, i.e. the output layer is a vector
[Equation]
which is fed into the α -ReLU loss:
[Equation]
where H α [ p ] := [Equation] [Equation]
j ( p
[Equation]
− p αj ) , α (cid:54) = 1 , is the Tsallis
α -entropy (Tsallis, 1988).
Given a training sample S :
= { ( x, y ) } learning is performed by minimizing the training error
[Equation]
with respect to the network parameters θ :=
vec (cid:0) { W ( k − 1) } k ∈
[ L − 1] (cid:1) .
Lemma 4.
Let the training error (16) be minimized by gradient descent with inﬁnitesimally small learning rate.
Let z ( x, t ) ∈
R
d be the network output on any training instance x at time t , and y be the desired output.
Then, as the widths of hidden layers n k → ∞ , ∀ k ∈
[ L − 1] , the output z ( x, t ) follows the following evolution [Equation]
[Equation]
where K ( x, x (cid:48) ) ∈ R
d × d is a positive semideﬁnite matrix, and z (cid:48) [Equation] = z
[Equation] x [Equation] t ) .
Proof.
From (16) and Lemma 1 we have
[Equation]
where we denoted z :
= z ( x, t ) and z (cid:48) :
= z ( x (cid:48) , t ) for shorthand.
Now, consider the gradient descent update
[Equation]
[Equation]
where η is the learning rate.
Taking the limit in (19) as η → 0 , we have:
[Equation]
where the last equality is due to the chain rule.
Combining this with (18), we get
[Equation]
Applying the chain rule again, and using (20), we have
[Equation]
The quantity K ( x, x (cid:48) ; θ ) was named the Neural Tangent Kernel by Jacot et al.
They also showed (see their Theorem 1) that
[Equation]
where K ( x, x (cid:48) ) ∈ R
d × d is the deterministric kernel that does not depend on θ .
This concludes the proof.
C Instructions for Human Annotators
You are shown a reference sentence and several candidate translations.
Please indicate, for each, on a 4-point scale, how much of the meaning is represented in the translation, ignoring the language quality.
Imagine you are a forgiving reader, ignoring any error that does not prevent you from getting the meaning of the text.
So please ignore language oddities, typographic errors and the like.
(This is difﬁcult but key to us!)
The scale of meaning preservation is: 4 =
Everything / 3 =
Most / 2 = Little / 1 =
None
As we are interested in comparing system’s output, you can reﬁne your judgement using + or − , e.g. 3 + .
When you do not know, simply leave empty.
For instance, given the reference sentence “This restaurant is beautiful and the staff is very friendly” , valid judgements for different translations are provided in Table 6.
[Table]
We insist that evaluating by meaning differs from a natural intuitive evaluation.
Provided the meaning is not impacted, we want to ignore the language quality, the punctuation, the casing.
D Translation Examples
[Table]