R EGULARLY VARYING REPRESENTATION FOR SENTENCE EMBEDDING
Anonymous authors
Paper under double-blind review
A BSTRACT
The dominant approaches to sentence representation in natural language rely on learning embeddings on massive corpuses.
The obtained embeddings have desirable properties such as compositionality and distance preservation (sentences with similar meanings have similar representations).
In this paper, we develop a novel method for learning an embedding enjoying a dilation invariance property.
We propose two algorithms: Orthrus , a classiﬁcation algorithm, constrains the distribution of the embedded variable to be regularly varying, i.e. multivariate heavy-tail.
and uses Extreme Value Theory (EVT) to tackle the classiﬁcation task on two separate regions: the tail and the bulk.
Hydra , a text generation algorithm for dataset augmentation, leverages the invariance property of the embedding learnt by Orthrus to generate coherent sentences with controllable attribute, e.g. positive or negative sentiment.
Numerical experiments on synthetic and real text data demonstrate the relevance of the proposed framework.
I NTRODUCTION
Representing the meaning of natural language in a mathematically grounded way is a scientiﬁc challenge that has received increasing attention with the explosion of digital content and text data in the last decade.
Relying on the richness of contents, several sentence embeddings have been proposed (Peters et al.
; Radford et al.
; Devlin et al.
(2018)) with demonstrated efﬁciency for the considered tasks.
These embeddings, learnt on massive datasets, are commonly used to perform downstream tasks such as classiﬁcation or data generation.
In a classiﬁcation context, e.g. when the goal is to distinguish between positive and negative sentiments in user reviews, most classiﬁers are based on Empirical Risk Minimization (ERM) strategies and variants of it.
However, nothing guarantees that such classiﬁers perform satisfactorily on the tails of the explanatory variables, i.e. in regions attached to rare events or even outside the domain of observed data.
The present paper builds upon the methodological framework proposed by (Jalalzai et al.
performing classiﬁcation in extreme regions with guarantees concerning the excess risk of the ERM classiﬁer.
Their approach relies on multivariate extreme value theory (EVT) and is valid under regularity assumptions concerning the multivariate distributional tail which appear to be an homogeneity property above large thresholds: the tail behaviour at inﬁnity is similar to the behaviour of polynomial function near inﬁnity (see equation 1 below).
To wit, it is necessary for the multivariate survival function of the considered random vector to be approximately homogeneous of degree − 1 , up to appropriate marginal standardization.
Although there is experimental evidence that their framework improves the performance of baseline classiﬁers in extreme regions on low dimensional data, there is no reason to assume that the previously mentioned text embeddings satisfy the required regularity assumptions.
The aim of the present work is to apply Jalalzai et al.
(2018)’s methodology to text data represented by state of the art embeddings.
To achieve this, we train the algorithm to perform a transformation that allows it to map any text embedding X onto a random vector Z which satisﬁes the aforementioned regularity assumptions.
The transformation is learnt by an adversarial strategy (Goodfellow et al.
As a by-product, we obtain a novel data augmentation mechanism which takes advantage of the scale invariance properties of Z to generate synthetic sentences that keep invariant the attribute of the original sentence.
The work detailed in this paper can be connected to other articles (Socher et al.
; Guu et al.
(2018); Siffer et al.
(2017)), though our framework is different.
Our contribution is twofold.
We introduce ( i ) a novel classiﬁcation algorithm ( Orthrus ), taking as input an embedded sentence to map it onto a heavy-tailed vector in the latent space satisfying Equation 1.
( ii ) a semi-supervised data augmentation method ( Hydra ), which leverages the homothetic invariance property of the embedding learnt by Orthrus to produce new sentences with prescribed label.
Label preserving data augmentation, which increases the number of training samples in machine learning systems, is an effective solution to data insufﬁciency and is an efﬁcient pre-processing step in computer vision (Wang & Perez (2017)).
Due to the discrete nature of text, data augmentation is a much harder problem in natural language processing (NLP) than in computer vision.
Current work in natural language (NL) dataset augmentation with label preservation mainly relies either on back-translation (Shleifer (2019)), on synonym replacements (Kobayashi (2018)), on slot ﬁlling (Hou et al.
(2018)) or on the use of handcrafted heuristics (Wei & Zou (2019))
( e.g swap, deletion, synonym replacements).
Unlike other state of the art solutions, Hydra can generate multiple sentences with a considered sentiment using a unique input sentence.
Hydra does not require any additional resources (e.g. synonym dictionary), nor does it rely on hand-crafted heuristics, or use style transfer methods (Hu et al.
(2017); Colombo et al.
In this paper, we work with output vectors issued by BERT sentence embedding as inputs (Devlin et al.
BERT is currently one of the state-of-the art algorithms for sentence embedding.
We demonstrate on both synthetic and real datasets that the embedding learnt by Orthrus on top of BERT indeed follows a heavy-tail distribution, and that Orthrus is able to outperform a state-of-the art classiﬁer built on BERT.
On the dataset augmentation task, quantitative and qualitative experiments demonstrate the ability of Hydra to generate new sentences while preserving labels.
The rest of this paper is organized as follows.
Section 2 introduces the necessary background in multivariate extremes and adversarial learning.
The methodology we propose is detailed at length in section 3.
Illustrative numerical experiments on both synthetic and real data are gathered in section 4.
Additional experimental results are available in the supplementary material.
B ACKGROUND 2. H EAVY TAILS AND REGULAR VARIATION
By deﬁnition, heavy-tail phenomena are those which are ruled by very large values, occurring with a far from negligible probability and with signiﬁcant impact on the system under study.
When the phenomenon of interest is described by the distribution of a univariate random variable, the theory of regularly varying functions provides the appropriate mathematical framework for the study of heavy-tailed distributions.
For the sake of clarity and in order to introduce notations we recall some related theoretical background.
One may refer to (Resnick (2013)) for an excellent account of the theory of regularly varying functions and its application to the study of heavy-tailed distributions.
A random variable X with cumulative distribution function ( c.d.f. )
F is heavy-tailed of index
α >if and only if for any ﬁxed x > 0 :
[Equation]
where F − 1 ( u ) = inf { t : F ( t ) ≥ u } denotes F ’s generalized inverse.
Based on this characterization, the heavy-tail model is classically extended to the multivariate setup as follows.
Consider now a d -dimensional random vector X =
( X (1) , . . .
, X ( d ) ) taking its values in R d + .
Then X is said to be heavy-tailed with tail index α >
if there exists a positive Radon measure µ on the punctured set [0 , ∞ ] d \{ 0 } and a function b ( t ) → ∞ such that:
[Equation]
For any Borelian set A ⊂ [0 , ∞ ] d which is bounded away from 0 and such that the measure µ of the boundary ∂A is zero.
In such a case, µ fulﬁlls the homogeneity property µ ( tA )
=
t − α µ ( A ) for all t > 0 and any set A in [0 , ∞ ]
d \{ 0 } .
Using the homogeneity property, one may show that µ can be decomposed into a radial component and an angular component Φ , which are independent from each other.
Indeed, for all x = ( x 1 , . . .
, x d ) ∈ R d , set
[Equation]
τ τ λτ λτ where S is the positive orthant of the unit sphere in R d for the chosen norm (cid:107) .
(cid:107) .
The choice of the norm is unimportant as all norms are equivalent in R d .
Then, for every B ⊂ S , we have:
[Equation]
where Φ is referred to as the spectral or angular measure.
2. C LASSIFICATION IN E XTREME R EGION
In the standard setup of binary classiﬁcation, ( X, Y ) ∈ R d + × {− 1 , 1 } is a random pair deﬁned on a certain probability space with unknown joint probability distribution P .
Now, equipped with the concepts introduced in the previous section we shall assume regular variation for both classes with the same tail index.
Stated otherwise (as formulated in (Jalalzai et al.
we shall make the assumption that:
[Equation] where σ ∈ {− 1 , +1 } , A ⊂ R d + , 0 (cid:54)∈ ∂A .
If one considers ( X ∞ , Y ∞ ) a random pair deﬁned by P { X ∞ ∈
A, Y ∞ =
+1 } =
lim t →∞ , P { X ∈ tA, Y = +
| ||
X || > t } , P { Y ∞ =
+1 } =
lim t →∞ P { Y = +
| ||
X || > t } , then one can prove (see Theorem 1 in Jalalzai et al. that the Bayes regression function η ∞ ( x ) =
P { Y ∞ = 1 | X ∞ = x } for the limiting pair is constant along rays i.e. is a function of Θ( x ) ∈ S only.
The same is true of the Bayes classiﬁer for the asymptotic pair.
As a consequence, the optimal classiﬁers on extreme regions are based on indicator functions of truncated cones on the kind {|| x || > r, Θ( x ) B } , where B ⊂ S .
As a consequence, the optimal classiﬁers on extreme regions are based on indicator functions of truncated cones on the kind {(cid:107) x (cid:107) > r, Θ( x ) ∈ B } , where B ⊂ S (see Figure 1).
This observation justiﬁes an ERM strategy using the extreme points of the dataset to train an angular classiﬁer.
Let { ( X i , Y i ) } ni
=1 be n i.i.d copies of ( X, Y ) .
By sorting the training observations by decreasing order of magnitude, we introduce X ( i ) (with corresponding sorted label Y ( i ) ) the i -th order statistics, i.e. || X (1) || ≥ . . . ≥ || X ( n ) || . Let G S be a class of classiﬁers deﬁned on the sphere S with ﬁnite VC dimension V G S < ∞ .
Thus for any g ∈ G S and any θ ∈ S , g ( θ ) ∈ {− 1 , 1 } .
By extension, for any x ∈ R d + we deﬁne g ( x ) =
g (cid:0) Θ( x ) (cid:1) ∈ {− 1 , 1 } .
Let t τ be the quantile at level (1 − τ ) of the r.v || X
|| where τ > 0 corresponds to a small fraction of extreme observations: P {|| X || > t τ } = τ .
Set k
= (cid:98) nτ (cid:99) and consider the empirical risk dedicated to the extreme samples:
[Equation]
which corresponds to the empirical version of L t τ , the loss at level t τ .
Now consider solutions to the minimization problem:
[Equation]
Thence the authors of Jalalzai et al.
(2018) provide guarantees concerning the classiﬁcation risk in out-of-sample regions.
Theorem 1 (Jalalzai et al.
Let (cid:98)
g k be any solution of equation 3.
Recall k = (cid:98) nτ (cid:99) .
Then, for δ ∈ (0 , 1) , ∀ n ≥ 1 , we have with probability larger than 1 − δ :
[Equation]
where C is a constant independent from n , τ and δ .
2. A DVERSARIAL LEARNING
Adversarial networks, introduced in (Goodfellow et al.
(2014)), form in a system where two neural networks are competing.
A ﬁrst model called the generator generates samples as close as possible to the input dataset.
A second model called the discriminator aims at distinguishing samples produced by the generator from the input dataset.
The goal of the generator is to maximize the probability of the discriminator making a mistake.
Hence, if P input is the distribution of the input dataset then the adversarial network intends to minimize the distance (as measured by the Jensen-Shannon divergence) between the distribution of the generated data P G and P input .
Auto-encoders and derivations (Goodfellow et al.
; Laforgue et al.
; Fard et al.
(2018)) shape a subclass of neural networks whose purpose is to learn a suitable representation by learning encoding and decoding functions which capture the core properties of the input data.
An adversarial auto-encoders (see Makhzani et al. (2015)) is a speciﬁc kind of auto-encoder where the code plays the role of the generator of an adversarial network.
Thus the latent code is forced to follow a given distribution while containing information relevant to reconstructing the input.
D ILATION I NVARIANT R EPRESENTATION 3. L EARNING A DILATION INVARIANT REPRESENTATION
We now introduce Orthrus , a novel algorithm for classiﬁcation of text data represented by high dimensional vectors as issued by pre-trained embeddings such as BERT which is assumed to not satisfy the regular variation assumption.
The idea behind Orthrus is to modify the output X of BERT in order to increase the information carried by the resulting representation Z = ϕ
( X ) regarding the label Y , including in the upper tail regions of Z corresponding to low probability sentences, in order to improve the performance of a downstream classiﬁer.
This is achieved by training an encoding function ϕ in such a way that (i) the marginal distribution q ( z ) of the code Z be close to a user-speciﬁed heavy tailed distribution p satisfying the regularity property (1) ; and (ii) the classiﬁcation loss of a multilayer perceptron trained on the code Z be small.
A major difference distinguishing Orthrus from existing auto-encoding schemes is that the target distribution on the latent space is not chosen as a Gaussian distribution but as a heavy-tailed one.
As the Bayes classiﬁer in the extreme region has a potentially different structure from the Bayes classiﬁer on the bulk (recall from Section 2 that the regression function at inﬁnity depends on the angle Θ( x ) only), Orthrus trains two different classiﬁers, C ext on the extreme region of the latent space on the one hand, and C bulk on its complementary set on the other hand.
Given a high threshold t , the extreme region of the latent space is deﬁned as the set { z : (cid:107) z (cid:107) > t } .
In practice, the threshold t is chosen as an empirical quantile of order ( 1 − κ ) (for some small, ﬁxed κ ) of the norm of encoded data (cid:107) Z i (cid:107)
=
(cid:107) ϕ
( X i ) (cid:107) .
In the end, the classiﬁer issued by Orthrus is of the kind
C ( z ) =
C ext ( z )1 {(cid:107) z (cid:107) > t } + C
[Equation]
z )1 {(cid:107) z
[Equation]
≤ t } .
Our goal is to minimize the weighted risk
[Equation]
where Z = ϕ ( X ) , d is the Jensen-Shannon distance between the heavy tailed target distribution p and the code distribution q , and λ 1 , λ 2 , λ 3 are positive weights.
Following common practice in the adversarial literature, the Jensen-Shannon distance is approached (up to a constant term) by the empirical proxy (cid:98) L ( q, p ) =
sup D ∈ Γ (cid:98) L ( q, p, D )) , with (cid:98) L
[Equation]
where Γ is a wide class of discriminator functions, and where independent samples Z i , ˜
Z i are respectively sampled from the target distribution and the code distribution q .
In the end, Orthrus solves the following min-max problem inf C ext ,C bulk ,ϕ sup D (cid:98) R ( ϕ, C ext , C bulk , D ) where
[Equation]
λ
[Equation]
where { Z ( i ) =
ϕ ( X ( i ) ) , i = 1 , . .
. , n } are the encoded observations with associated labels Y ( i ) sorted by decreasing magnitude of (cid:107) Z (cid:107)
( i.e. (cid:107) Z (1) (cid:107) ≥ · · · ≥ (cid:107) Z ( n ) (cid:107) ), and k = (cid:98) κn (cid:99) is the number of extreme samples among the n encoded observations.
This minimization problem results in Algorithm 1 detailed below, where m is the batch size, ρ is the proportion of the points given to train the extreme classiﬁer and the cost function (cid:96) is the negative log-likelihood.
3. A DILATION INVARIANT REPRESENTATION FOR DATASET AUGMENTATION
We now introduce Hydra , a data augmentation algorithm, which relies on the dilation invariance of labels on the extreme embeddings learnt by Orthrus .
Let U =
( u 1 , . . .
, u T ) be a sequence of inputs of length T .
The dataset augmentation problem consists in generating multiple sentences { U (cid:48) i } i ∈ N such that each sequence U (cid:48)
i is label coherent with U .
Hydra follows the seq2seq approach (Sutskever et al.
and learns two decoders with attention (Bahdanau et al.
(2014)), one for each region of the representation previously learnt by Orthrus .
On the whole, a decoder takes an input Z (the latent code) and generates an output sequence U (cid:48)
i =
( u (cid:48)
i 1 , . . .
, u (cid:48)
i T (cid:48) ) , where each word u (cid:48)
i k is in the vocabulary.
To generate an output word u (cid:48) k , the decoder iteratively takes as input the previously generated word u (cid:48)
k − 1 ( u (cid:48)
being a start symbol), updates its internal state, and returns the next word with highest probability.
This process is repeated until the decoder generates either a stop symbol or the length of the generated sequence reaches the maximum sequence length.
Hydra is a semi-supervised algorithm in that it requires a ﬁrst labeled dataset D n to learn embeddings through Orthrus and a second dataset D g n (not necessarily labeled) to train a decoder G ext ( resp
G bulk ) for the extremes ( resp non extremes) on D g n .
The learning is carried out by optimising the classical negative log likelihood of individual tokens (cid:96) gen .A detailed description of Hydra is provided in Algorithm 2 below.
After Hydra has been trained, the data augmentation proceeds as follows:
For each input X with extreme embedding, multiple sentences with constant label are obtained by applying a λ dilation (see Figure 2b).
In other words, if G ext is a decoder learnt by Hydra and applicable to the extreme region of the representation, synthetic sentences of the form G ext ( λz ) can be obtained, with the same attribute as z .
[Equation]
[Figure]
INPUT: Coef.
λ 1 ,λ 2 ,λ 3 >
, Training dataset D n = { ( U 1 ,Y 1 ) , ..., ( U n ,Y n ) } , proportion κ ≤ 1 of extreme observations, batch size m ≤ n .
Initialization: parameters of the encoder ϕ τ , classiﬁers, C extθ , C bulkθ (cid:48) , decoders and discriminator D γ while ( τ,θ,θ (cid:48) ) not converged do
Sample { ( U 1 ,Y 1 ) ..., ( U m ,Y m ) } from D n .
Sample { Z 1 ,...,Z m } from the prior P Z .
Sample ˜ Z i from ϕ τ ( Z | X i ) for i ∈ { 1 ,...,m } .
Update D γ by ascending: λ 3 m m (cid:88) i =
log D γ ( Z i )
+ log(1 − D γ
( ˜ Z i )) .
Sort { ˜ Z i } i ∈{ 1 ,...,m } by decreasing order of magnitude ||
˜ Z (1) || ≥ ... ≥ ||
˜ Z ( m ) || .
Update C ext θ by descending:
L ext def = λ 1 (cid:98) κm (cid:99) (cid:98) κm (cid:99) (cid:88)
i = (cid:96) (cid:0) Y ( i ) ,
C ext θ ( ˜ Z ( i ) ) (cid:1) .
Update C bulk θ (cid:48) by descending:
L bulk def =
λ 2 m − (cid:98) κm (cid:99) m (cid:88) i = (cid:98) κm (cid:99) + (cid:96) (cid:0) Y ( i ) ,C bulk θ (cid:48)
( ˜ Z ( i ) ) (cid:1) .
Update ϕ τ by descending: m m (cid:88) i = − λ 3 · log D γ ( ˜ Z i )
+ L ext + L bulk .
end while Compute { ˜ Z i } i ∈{ 1 ,...,n } = ϕ
( X i ) i ∈{ 1 ,...,n }
Sort { ˜ Z i } i ∈{ 1 ,...,n } by decreasing order of magnitude || ˜ Z (1) || ≥ ... ||
˜ Z ( (cid:98) κn (cid:99) )
|| ≥ ... ≥
||
˜ Z ( n ) || .
OUTPUT: encoder ϕ , classiﬁers C ext θ applicable on the region { x : || ϕ
( x ) || ≥ ||
˜ Z ( (cid:98) κn (cid:99) )
||}
and
C bulk θ applicable on the region { x : || ϕ ( x ) (cid:1)
|| <
||
˜ Z ( (cid:98) κn (cid:99) )
||} .
Algorithm
INPUT: Coef.
λ 1 ,λ 2 ,λ 3 > 0 , Training datasets D n = { ( U 1 ,Y 1 ) , ..., ( U n ,Y n ) } , D g n = { U g 1 , ..., U g n } batch size m ≤ n ,
Initialization: parameters of the encoder ϕ , classi- ﬁers, C extθ , C bulkθ (cid:48) , decoders, G extψ , G bulkψ (cid:48) and discriminator D γ
Optimization: ϕ , C ext , C bulk =
Orthrus ( λ 3 , D n ,) while ( ψ,ψ (cid:48) ) not converged do Sample { ( X g 1 ) ..., ( X g m ) } from the training set D g n . Sample ˜
Z i from ϕ
( Z | X g i ) for i ∈ { 1 ,...,m } .
Sort { ˜ Z i } i ∈{ 1 ,...,m } by decreasing order of magni- tude || ˜ Z (1) || ≥ ... ≥ || ˜ Z ( m ) || .
Update G ext ψ by descending:
L ext g def = λ 1 (cid:98) ρm (cid:99) (cid:98) ρm
(cid:99) (cid:88)
i = (cid:96) gen.
(cid:0) Y ( i ) ,G ext ψ ( ˜ Z ( i ) ) (cid:1) .
Update G bulk ψ (cid:48) by descending:
L bulk def =
λ 2 m − (cid:98) ρm (cid:99) m (cid:88) i = (cid:98) ρm
(cid:99)
+ (cid:96) gen.
(cid:0) Y ( i ) ,G bulk ψ (cid:48)
( ˜ Z ( i ) ) (cid:1) .
end while
Compute { ˜ Z i } i ∈{ 1 ,...,n } = ϕ
( X i ) i ∈{ 1 ,...,n }
Sort { ˜ Z i } i ∈{ 1 ,...,n } by decreasing order of magnitude || ˜ Z (1) || ≥ ... ||
˜ Z ( k ) || ≥ ... ≥ ||
˜ Z ( n ) || .
OUTPUT: encoder ϕ , decoder G ext applicable on the region { x : || ϕ
( x ) || ≥ ||
˜ Z ( (cid:98) κn (cid:99) )
||} and G bulk applicable on the region { x : || ϕ ( x )
|| <
||
˜ Z ( (cid:98) κn (cid:99) )
||} .
Algorithm: Orthrus and Hydra .
For both algorithm X is the embedding provided by BERT to an input sentence U .
N UMERICAL E XPERIMENTS
Heavy-tail distribution:
In the remaining of this paper, the regularly varying target distribution is a logistic distribution with parameter δ = 0 .
9 (see Appendix B.2).
The logistic distribution is widely used in the context of extreme values analysis.
This distribution is simulated according to Stephenson (2003) which is deﬁned in R d with parameter δ ∈ (0 , 1] by its parametric c.d.f.
F ( x ) = exp (cid:8) − ( (cid:80) dj
=1 x ( j ) 1 δ ) δ (cid:9) .
Figure 4 in Appendix illustrates this distribution with various values of δ .
Experimental Settings: Classiﬁers C bulk , C ext are Multi Layer Perceptrons (MLP).
For each dataset, one-fourth of data is retained as a test set while the remaining data points are used as the train set.
We denote by T the extreme test set region as selected with Algorithm 1.
The considered norm is the (cid:96) ∞ norm.
For the generation task, the dilation coefﬁcient varies between 1 and 1 . 5 ( λ ∈ [1 , 1 . 5] ).
4. A PPLICATION TO S ENTIMENT A NALYSIS
We assess Othrus ability to perform sentiment classiﬁcation tasks using two balanced binary labeled datasets: Yelp, Amazon introduced by (Kotzias et al.
Further results on the datasets are reported in the Appendix.
Each dataset is balanced and is composed of 1000 sentences and each sentence expresses a strong positive or negative sentiment.
Baseline: Orthrus is compared with a state of the art classiﬁer: a MLP trained on the initial BERT embedding ( NN ) taking as input the same training data.
Classiﬁcation performance: Table 1 illustrates the performance of considered models in terms of
Hamming loss on the test set.
Performance of Orthrus is better than NN on both dataset on the whole test set.
The difference in terms of performance is even greater when considering the extreme samples.
[Table]
4. A PPLICATION TO DATASET AUGMENTATION FOR LABELLED SENTENCES
Comparison to related work: We compare Hydra with two state of the art methods for dataset augmentation: EDA (Wei & Zou (2019)) and round-trip translation (Shleifer (2019)).
Compared to EDA which uses heuristics and a synonym dictionary, Hydra generates grammatically correct sentences and conserve the labels.
Compared to round-trip translation Hydra can generate multiple sentences for a single input and conserves sentence labels.
We also compare Hydra to a Vanilla Sequence to Sequence to demonstrate the validity of our approach.
In this section, we validate Orthrus using the Yelp dataset.
To train Orthrus the dataset D n is the Yelp dataset introduced in (Kotzias et al.
D g n is a subset of the full Yelp Corpus composed of 5 · 10 5 sentences.
Evaluation: Automatic evaluation of generative models for text is still an open research problem.
We evaluate our models through three criteria ( C1, C2, C3 ).
C1 measures Cohesion (Crossley & McNamara (2010))
(Are the generated sentences grammatically and semantically consistent?).
C2 (named Sent.
in Table 1a.
) evaluates label conservation (Does the expressed sentiment in the generated sentence match the sentiment of the input sentence?).
C3 measures the diversity of the sentences (Is the dataset augmented with diverse sentences?)
We carry out two types of evaluations: a qualitative one based on a user study – that provides evaluation through C1 and C2 – and a quantitative one – that provides evaluation through C measures.
We measure diversity ( C3 ) with two automatic methods: (1) Distinct n , the diversity measure introduced by (Li et al.
(2015)) which relies on a n-grams count; (2) a F1 score that reports the improvement induced by the data augmentation while training a fastText (Joulin et al.
(2016)) classiﬁer.
This improvement reﬂects the information added by newly generated sentences.
[Table] [Table] Qualitative Evaluation : We compare the different methods on the extreme set of sentences according to Hydra .
As previously, the extreme test set T consists of 92 sentences.
For each sentence, we randomly sample λ ∈ [1 , 1 . 5] and generate a new sentence with Hydra .
To compare our method, we use Eda, round-trip translation and a Vanilla Seq2seq to generate new sentences.
Results of the qualitative evaluation are reported in Table 2a.
As expected Hydra exhibits comparable performance with all other models except Eda in terms of cohesion.
Such a result is expected since Eda’s use of random swaps or permutations may result in ungrammatical sentences.
In terms of expressed sentiments, Hydra outperforms Seq2seq and round-trip translation, demonstrating the relevance of the proposed approach.
Quantitative Evaluation : We report in (Table 2b) the results of evaluation for C3 .
We observe that Hydra outperforms all other methods in term of distinct 1 and distinct 2.
Table 1b shows that improvement in F1 score induced by dataset augmentation by Hydra beats all other methods and is only equalled by EDA.
Example of generated sentences : We report in Table 3 sentences generated by Hydra .
While we make no claim that these generated sentences are better than sentences generated by existing methods, we believe that these samples are at least competitive with the generative models in the literature and highlight the potential of the introduced framework.
D ISCUSSION & P ERSPECTIVES
The approach promoted in this paper relies on learning a regularly varying representation by minimizing an empirical proxy of an objective function.
The latter writes as a weighted sum of a classiﬁcation risk and a regularization term penalizing the distance between the representation distribution and a heavy-tailed target.
Our experiments show that the obtained representation allows to diminish the classiﬁcation error compared to a MLP with comparable complexity.
The obtained representation used for a text data augmentation task, is competitive with existing data augmentation methods.
The attribute invariance under code dilation is the key to generate meaningful sentences with prescribed attribute.
Future work will investigate the possibility to train an auto-encoder satisfying this dilation invariance property without relying on a pretrained classiﬁer such as Orthrus .
A PPENDIX A R EMARKS
Remark 1 ( Selection of k )
To the best of our knowledge, selection of k in Algorithm 1 and Algorithm 2 is still a vivid problem in EVT which is not solved yet.
As k gets large the number of extreme points increases including samples which are not large enough and deviates from the asymptotic distribution of extremes.
Smaller values of k increase the variance of the classiﬁer/generator.
This bias-variance trade-off is beyond the scope of this paper.
Remark
In Figure 3a selecting the extreme samples on the input space is not a straightforward step since the tails of the input clusters are not heavy-tail.
It was decided to standardise the input data ( X i ) i ∈{ 1 , ··· ,n } by applying the rank-transformation:
[Equation]
for all x = ( x 1 , · · · x d ) ∈ R d where (cid:98) F j def = 1 n +1 (cid:80) ni =1 1 { X ji ≤ x } is the j th empirical marginal distribution.
It comes ∀ i ∈ { 1 , · · · , n } , V i
= ˆ T ( | X i | ) .
V i margins follow a standard Pareto distribution.
Extreme samples are thus { V i ,
||
V
i || ≥ n/k } with k = √ n by convention.
One may notice that extremes in Figure 3c are more label balanced compared to Figure 3a. B E XPERIMENTS
Further synthetic experiments were conducted to show the improvement of the classiﬁer.
B. S YNTHETIC DATA
Dataset description : Synthetic data was generated using Scikit-Learn (Pedregosa et al. Focus is made on the classiﬁcation dataset where clusters of points are normally distributed for each class (see Figure 3a). Convergence to a logistic : In Figure 3b we visualise the latent space of the output of the encoder ϕ
(see Figure 2).
In Orthrus , the adversarial approach to minimize the discrepancy between the distribution of the output of the encoder and a logistic distribution. Notice the similarity between Figure 3b and Figure 4a. Extremes selection: Extreme samples from Figure 3a are assessed directly on the input data while extremes from Figure 3c are obtained from the latent space.
Extremes from the latent space (output of ϕ after the training phase) are located on the border of the distribution which comforts the correct behaviour of the training. We also note similarities while comparing in the input space: extreme samples selected in the input space and extreme ones selected in the learnt latent space. In Figure 3a no extreme observation appear on the negative class whereas labels of extremes of Figure 3c are balanced.
B. L OGISTIC DISTRIBUTION
Figure 4 illustrates the logistic distribution with varying parameter δ .
B. A DDITIONAL E XPERIMENT S ETTINGS F OR R EAL D ATA
We use BERT pretrained models and code from the library pytorch-transformers .
All models were implemented using Pytorch.
The output of BERT is a R 768 vector, Z is a R 50 vector.
The dimension of Z (50) was obtained by cross validation.
D , C bulk and C ext are MLP composed of 3 layers and uses a ReLu as activation function.
Decoders D bulk and D ext are GRU composed of 6 layers.
For training the batch size is set to 64 and we set λ 1 = (1 − (cid:98) P ( || Z || ≥
|| Z (cid:98) κn (cid:99) || ) − 1 , λ 2 = (cid:98) P ( || Z || ≥
|| Z (cid:98) κn (cid:99) || ) − 1 , λ 3 = 10 − 3 .
All neural networks have been optimized with Adam (Kingma & Ba (2014)).
For generation the maximum dilation factor λ is set to 1 . 5 since we observe that with greater values a stuttering phenomenon appears. [Figure] [Figure]
B. A DDITIONAL
[Figure]
Dilation invariance of C ext : Figure 5a shows the evolution of the prediction for each sample of the test set applying the dilation. Predictions of C bulk change for some samples resulting in bichromatic columns. Figure 5b focuses on the samples from T and is exclusively composed with monochromatic columns demonstrating that the predicted labels from C ext do not change, C ext is homothetic invariant. With no assumption on C ext , this classiﬁer enjoys favorable theoretical properties since it belongs to the class of angular classiﬁers at the end of the training phase.
[Table]
[Figure] [Table] B. A DDITIONAL E XPERIMENTS ON A MAZON D ATA
Experiments on the Amazon dataset were conducted similarly to to the ones from the Yelp dataset from subsection 4.1, similar conclusions can be drawn from Figure 6.
[Figure]
B. Q UALITATIVE E VALUATION
For the qualitative evaluation, we compute the Krippendorff’s alpha coefﬁcient Krippendorff (2018) to evaluate the inter-annotator agreement.
We obtain α = 0 .
41 for the grammar analysis and α = 0 .for
the sentiment analysis.
If α = 1 it indicates perfect reliability; if α =
it indicates the absence of reliability.
Such values of α can be considered as relatively high given our huge number of annotators (126 for the grammar and 134 for the sentiment analysis)
. B. A DDITIONAL GENERATED SENTENCES
We present in Table 4 additional extreme sentences.
C A LINGUISTIC POINT OF VIEW OF EXTREMES
In this section, we provide a comparative linguistic analysis of the extreme samples in case of sentiment classiﬁcation through the affect scores provided by the lexicon for English words from
Mohammad (2017).
To better understand the content of extreme and non extreme text samples in experiments conducted in section 4, the maximum and the minimum of the affect score for each sentence are studied (see
Figure 7).
For both Yelp and Amazon datasets, one can notice that the violin plot of the maximum of the affect score is wider among the extremes than in the bulk.
This illustrates that extreme tend to have both a more positive and a more negative affect score than the non extreme observations.
Same goes for the minimum affect score of extremes whose violin plot tends to be larger among the extremes compared to the violin plot of the minimum affect score on the bulk.
[Figure]