U NSUPERVISED L EARNING OF S ENTENCE R EPRESENTATIONS U SING S EQUENCE C ONSISTENCY
Anonymous authors
Paper under double-blind review
A BSTRACT
Computing universal distributed representations of sentences is a fundamental task in natural language processing.
We propose ConsSent, a simple yet surprisingly powerful unsupervised method to learn such representations by enforcing consistency constraints on sequences of tokens.
We consider two classes of such constraints – sequences that form a sentence and between two sequences that form a sentence when merged.
We learn sentence encoders by training them to distinguish between consistent and inconsistent examples, which are generated by random perturbations in six different ways.
Extensive evaluation on several transfer learning and linguistic probing tasks shows improved performance over strong unsupervised and supervised baselines, substantially surpassing them in several cases. Our best results are achieved by training sentence encoders in a multitask setting and by an ensemble of encoders trained on the individual tasks.
I NTRODUCTION
In natural language processing, the use of distributed representations has become standard through the effective use of word embeddings.
In a wide range of NLP tasks, it is beneﬁcial to initialize the word embeddings with ones learned from large text corpora like word2vec (Mikolov et al.
or GLoVe (Pennington et al.
and tune them as a part of a target task e.g. text classiﬁcation.
It is therefore a natural question to ask whether such standardized representations of whole sentences that can be widely used in downstream tasks is possible.
There are several of approaches to this problem.
Taking cue from word2vec, an unsupervised learning approach is taken by SkipThought (Kiros et al. (2015)), FastSent (Hill et al.
and QuickThoughts (Logeswaran & Lee (2018)), exploiting the closeness of adjacent sentences in a text corpus. Models trained using a language modeling objective have also been shown to produce excellent sentence representations (Ruder & Howard (2018); Radford (2018)).
More recently, the work of Conneau et al.
(2017) takes a supervised learning approach.
They train a sentence encoder on large scale natural language inference datasets (Bowman et al.
; Williams et al.
and show that the learned encoding transfers well to to a set of transfer tasks.
This is reminiscent of the approach taken by ImageNet (Deng et al.
in the computer vision community.
In Subramanian et al.
(2018), the authors train a sentence encoder on multiple tasks to get improved performance.
In this paper, we take a slightly different unsupervised approach to learning sentence representations.
We deﬁne a sequence of tokens to be consistent if they form a valid sentence.
While deﬁning consistency precisely is difﬁcult, we generate approximately inconsistent sequences of tokens by randomly perturbing consistent ones.
We extend the notion of consistency to pairs of sequences – two sequences are deﬁned to be consistent if they can me merged to form a consistent sequence.
We then train a sentence encoder to discriminate between consistent and inconsistent sequences or pairs of sequences.
Note that external supervised labels are not required for training as we generate our own labels using the notion of consistency. We call our proposed method ConsSent .
We generate inconsistent sequences in four different ways – by deleting, permuting, inserting and replacing random tokens in a consistent sequence.
For pairs of sequences, we generate inconsistent examples by randomly partitioning sentences into two parts and pairing parts from two different sentences, where the method of partitioning can be contiguous or non-contiguous.
We vary the complexity of the training tasks by choosing a different number of tokens and pairs to generate inconsistent examples and train several sentence encoders on a set of unordered sentences from the Billionword corpus (Chelba et al.
We also combine the training tasks and encoders in two different ways – one by training an encoder in a multitask setting and the other by creating an ensemble of models trained on the individual tasks.
We evaluate the trained sentence encoders on a set of transfer learning and linguistic probing tasks encapsulated in the SentEval benchmark Conneau & Kiela (2018). Our models achieve strong results in both settings, improving upon existing unsupervised and supervised methods of learning sentence representations in several cases. Averaged over all the tasks in SentEval, the multitask encoder and ensemble model signiﬁcantly outperform sentence encoders trained with a SkipThought objective and language modeling objective.  R ELATED W ORK
Learning general sentence encoders is a fundamental problem in NLP and there is a long list of works that addresses this problem.
To begin with, an untrained BiLSTM model with max pooling of the intermediate states performs fairly well on several transfer and linguistic probing tasks (Conneau et al.
The next set of simple models consists of a bag-of-words approach using learned word embeddings like GloVe (Pennington et al.
or FastText (Joulin et al.
(2017)), where a simple average of the word embeddings in a sentence deﬁne the sentence embedding.
Although very fast, these approaches are limited by the dimensions of the word embeddings and do not take into account the order of the words.
Due to the availability of practically unlimited textual data, learning sentence encoders using unsupervised learning is an attractive proposition.
The SkipThought model of Kiros et al.
(2015) learns sentence encoders by using an encoder-decoder architecture.
Exploiting the relatedness inherent in adjacent sentences, the model is trained by using the encoder to encode a particular sentence and then using the decoder to decode words in adjacent sentences.
This approach is directly inspired by a similar objective for learning word embeddings like word2vec (Mikolov et al.
In more recent work, Logeswaran & Lee (2018) propose QuickThoughts, which uses a form of discriminative training on encodings of sentences, by biasing the encodings of adjacent sentences to be closer to each other than non-adjacent sentences, where closeness is deﬁned by the dot product.
The Bag-of-Words approach is developed further in the FastSent model (Hill et al.
(2016)) which uses a bag-of-words representation of a sentence to predict words in adjacent sentences.
In work by Arora et al.
(2017) it is shown that a simple post processing of the average word-embeddings can perform comparably or better than skip-thought like objectives.
The work of Pagliardini et al.
(2018) where the authors use n-gram features for unsupervised learning is also relevant.
The other notable unsupervised approach is that of using a denoising autoencoder (Hill et al.
Language modeling is a very natural task for natural language for which models can be trained in an unsupervised manner.
The context encoder trained on a large corpus with a language modeling objective can be used to generate sentence representations by encoding the sentence as a context.
Work by Ruder & Howard (2018) and Radford (2018) has shown that training language models on large text corpora produce excellent sentence encoders that perform very well on downstream tasks.
The recent work of Devlin et al.
(2018) push this approach even further by training large transformers (Vaswani et al. on very large text corpora using a slightly modiﬁed masked language modeling objective. The language modeling based approaches typically require corpora with ordered sentences or document level corpora, which is not a requirement in our case. Supervised learning has been used to learn better sentence encoders.
Conneau et al.
(2017) use the SNLI (Bowman et al.
and MultiNLI (Williams et al.
(2018)) corpus to train a sentence encoder on the natural language inference task that performs well on several transfer tasks.
Another domain where large datasets for supervised training is available is machine translation and the work of McCann et al.
(2017) exploits this in learning sentence encoders by training it on the machine translation task.
Finally, Subramanian et al.
(2018) combine several supervised and unsupervised objectives in a multi-task framework to obtain some of the best results on learning general sentence representations.
Our approach is based on automatically generating (possibly noisy) training data by perturbing sentences.
Such an approach was used by Wagner et al.
(2009) to train a classiﬁer to judge the grammat- icality of sentences.
The ungrammatical sentences were generated by, among other things, dropping and inserting words.
Recent work by Warstadt et al.
(2018) extend this approach by using neural network classiﬁers.
Finally, in parallel to our work, a recent report Ranjan et al.
(2018) uses word dropping and word permutation to generate fake sentences and learn sentence encoders.
Our work is substantially more general and exhaustive.
C ONSISTENT S EQUENCES AND D ISCRIMINATIVE T RAINING
Let S = { w 1 , w 2 , · · · , w n } be an ordered sequence of n tokens.
We deﬁne this sequence to be consistent if the tokens form a valid sentence.
Let E be an encoder that encodes any sequence of tokens into a ﬁxed length distributed representation.
Our goal is to train E to discriminate between consistent sequences and inconsistent ones.
We argue that such an encoder will have to encapsulate many structural properties of sentences, thereby resulting in a good sentence representation.
Whether a sequence of tokens S is consistent is a notoriously hard problem to solve.
We take a slightly different approach.
We start from a consistent sequence (e.g. from some text corpus) and introduce random perturbations to generate sequences S (cid:48) that are not consistent.
We take inspiration from the standard operations in string edit distance and consider the following variations for generating S (cid:48) .
• ConsSent-D( k ): We pick k random tokens in S and delete them.
• ConsSent-P
( k ): We pick k random tokens in S and permute them randomly (avoiding the identity permutation).
• ConsSent-I( k ): We pick k random tokens not from S and insert them at random positions in S .
• ConsSent-R( k ): We pick k random tokens in S and replace them with other random tokens not in S .
It is important to note that it is possible that in some cases S (cid:48) will itself form a valid sentence and hence violate the deﬁnition of inconsistency.
We do not address this issue and assume that such cases will be a relatively few and will not inﬂuence the encoder in a substantial manner.
Also, with larger values of k , the chance of this happening goes down.
We train E to distinguish between S and S (cid:48) by using a binary classiﬁer on top of the representations generated by E .
We extend the deﬁnition of consistency to pairs of sequences.
Given two sequences, we deﬁne them to be consistent if they can be merged into a consistent sequence without changing the order of the tokens.
Similar to the above deﬁnitions, we generate consistent and inconsistent pairs by starting from a consistent sequence and splitting them into two parts.
We consider the following variations.
• ConsSent-N( k ): If n is the number of tokens in a sequence S 1 , let S 11 be a random subsequence of S 1 , and let S 2 1 = S 1 \ S 1 1 be the complementary subsequence.
For a consistent sequence S 1 , S 11 and S 2 1 form a consistent pairs of sequences.
Let S 12 and S 2 2 be a partition for a different consistent sequence S 2 , such that S 21 (cid:54) =
S 22 .
Then S 11 and S 22 form an inconsistent pair of sequences, by virtue of the fact they belong to two different consistent sequences.
We can vary the complexity of the encoder E by training it to discriminate between a consistent pair ( S 11 , S 21 ) and k − 1 other inconsistent pairs ( S 11 , S 22 ) , ( S 11 , S 23 ) , · · · , ( S 11 , S 2 k ) for different values of k .
It is possible to pose the task of discriminating the consistent pair ( S 11 , S 21 ) from the k −inconsistent pairs as a classiﬁcation problem with a classiﬁcation layer applied to encodings of the pairs.
But this introduces additional parameters which is avoidable for sentence pairs.
Instead, we train E by enforcing the constraint that
[Equation]
In other words, we train the encoder to place the representations of consistent pairs of sequences closer in terms of dot product than inconsistent pairs.
A similar procedure was also used in training sentence representations in (Logeswaran & Lee (2018)), but with whole sentences appearing adjacent to each other in a larger body of text.
Our method does not require the individual sentences to be ordered for training.
Note that such kind of training is not possible for classifying single sequences.
[Table]
Maya
[Table] ConsSent
[Table]
• ConsSent-C( k ): We generate S 11 and S 21 from S 1 by partitioning it at a random point.
Thus both S 11 and S 21 are contiguous subsequences of S 1 .
If S 1 is consistent, then the two partitioned sequences form a consistent pair.
We generate inconsistent pairs by pairing Swith k − 1 other S 2 j originating from the partition of different consistent sequences S j
In addition to training an encoder on each individual task, we also consider training encoders in a multitask setting.
We split the tasks into two groups, the ﬁrst containing ConsSent-D( k ), ConsSentP( k ), ConsSent-I( k ) and ConsSent-R( k ) and the other containing ConsSent-N( k ) and ConsSentC( k ).
We then train an encoder E 1 for the ﬁrst group and another encoder E 2 for the second group, cycling through the tasks in a round robin manner within each group.
Note that for the ﬁrst group, we use different classiﬁcation layers for the four different tasks.
After training, the sentence representations from E 1 and E 2 is concatenated to generate the ﬁnal representation for any sentence.
We denote this model with multitask training by ConsSent-MT( k ). In {+Table
+} we show toy positive and negative training examples for each of these training tasks.
The choice of the encoder E is important to generate good sentence representations.
Following Conneau et al.
(2017), we use a bidirectional LSTM to process a sequence of tokens and take a max-pool of the intermediate hidden states to compute a distributed representation.
T RAINING AND E VALUATION
We train our models on the Billionword corpus (Chelba et al.
We use the ﬁrst 50 shards of the training set (approximately 15 million sentences) for training and 50000 sentences from the validation set for validation.
For ConsSent-D( k ), ConsSent-P( k ), ConsSent-I( k ) and ConsSent-R( k ) we delete, permute, insert or replace k tokens with a probability of 0.5.
This produces roughly an equal number of consistent and inconsistent sequences.
For ConsSent- { D,I,K } ( k ) we sweep over k ∈ { 1 , 2 , 3 , 4 , 5 } and for ConsSent-P( k ) we sweep over k ∈ { 2 , 3 , 4 , 5 , 6 } to train a total ofencoders.
In the case of ConsSent-N( k ), for each consistent sequence S 1 , we partition it into two subsequences by randomly picking a token to be in the ﬁrst part S 11 with probability 0.5.
The remaining tokens go into S 21 .
We pick ( k − 1) other random consistent sequences S 2 , · · · S k and do the same.
For
ConsSent-C( k ), for each consistent sequence S 1 , we pick i ∈ { 2 , · · · , n − 1 } uniformly at random and partition it at i to produce S 11 and S 21 .
The remainder of the training procedure is the same as ConsSent-N( k ).
For both these models, we sweep over k ∈ { 2 , 3 , 4 , 5 , 6 } to train a total ofencoders.
For ConsSent-MT( k ) , we sweep over k ∈ { 2 , 3 , 4 , 5 , 6 } . Overall, we train 35 encoders, 30 on the six individual tasks and 5 in the multitask setting. As described in the next section, we also consider an ensemble of models trained on different individual tasks.
We train the BiLSTM-Max encoder E with a hidden dimension of 2048, resulting in 4096 dimensional sentence representations. For ConsSent-MT( k ), we use a hidden dimensions of 1024 for both E 1 and E 2 , resulting in a sentence representation of 4096 dimensions.
For ConsSent-D( k ), ConsSent-P( k ), ConsSent-I( k ) and ConsSent-R( k ), the sentence representations are passed through two linear layers of size 512 before the classiﬁcation Softmax.
For ConsSent-N( k ) and ConsSentC( k ), we pair S 11 with ( k − 1) random S 2 j from within the same minibatch to generate the inconsistent pairs.
For optimization, we use SGD with an initial learning rate of 0.1 which is decayed by 0. after every epoch or by 0.2 if there is a drop in the validation accuracy.
Gradients are clipped to a maximum norm of 5.0 and we train for a maximum of 20 epochs.
We evaluate the sentence encodings using the SentEval benchmark (Conneau & Kiela (2018)).
This benchmark consists of two sets of tasks related to transfer learning and predicting linguistic properties of sentences.
In the ﬁrst set, there are 6 text classiﬁcation tasks (MR, CR, SUBJ, MPQA, SST,
TREC), one task on paraphrase detection (MRPC) and one on entailment classiﬁcation (SICK-E).
All these 8 tasks have accuracy as their performance measure.
There are two other tasks on estimating the semantic relatedness of two sentences (SICK-R and STSB) for which the performance measure is Pearson correlation (expressed as percentage) between the estimated similarity scores and ground truth scores.
For each of these datasets, the learned ConsSent sequence encoders are used to produce representations of sentences.
These representations are then used for classiﬁcation or score estimation using a logistic regression layer.
We also use a L2 regularizer on the weights of the logistic layer whose coefﬁcient is tuned using the validation sets.
The goal of testing ConsSent on these tasks is to evaluate the quality of the sentence encoders on a wide variety of downstream tasks with limited training data.
The second set of tasks probes for 10 different linguistic properties of sentences.
These include tasks like predicting which of a set of target words appears in a sentence (WordContent), the number of the subject in the main clause i.e. whether the subject is singular or plural (SubjNum), depth of the syntactic tree (TreeDepth) and length of the sentence quantized into a few bins (SentLen).
Some of these properties are syntactic in nature, while some require deeper understanding of the semantics of a sentence.
The goal of testing ConsSent on these tasks is to evaluate how much linguistic information is captured by the encoders.
For each of the tasks, the representations produced by the
ConsSent encoders are input to a classiﬁer with a linear layer followed by Sigmoid followed by a classiﬁcation layer.
We tune the classiﬁer on the validation sets by varying the dimension of the linear layer in [50 , 100 , 200] and the dropout before the classiﬁcation layer in [0 , 0 . 1 , 0 . 2] .
For more details on these tasks, please refer to Conneau & Kiela (2018) and Conneau et al.
R ESULTS ON T RANSFER T ASKS
In Table 2 we present results on each of the transfer tasks in SentEval.
In addition to the accuracy and correlation scores, we also report an average of all the 10 scores in the last column.
We only show the best performing model (out of 5) for each of the six methods, based on the average validation performance over all the tasks in SentEval.
ConsSent-N(3), which discriminate between pairs of sequences, performs the best on an average for any single task.
ConsSent-MT(3) achieves a similar average performance, also achieving the best scores on ﬁve of the 10 transfer tasks among our models.
Among the methods that classify single sequences, ConsSent-R(2) performs the best, second only to ConsSent-N. ConsSent-C is dominated by ConsSent-N in most cases, while ConsSent-D and ConsSent-P perform the worst. Notably, all the methods perform better than SkipThought-LN (Kiros et al. on an average and on most individual tasks.
We also compare our results with the recently proposed QuickThoughts model Logeswaran & Lee (2018), which is itself based on a SkipThought like objective. ConsSent-MT(3) outperforms QuickThoughts on MPQA (+0.2) and TREC(+0.8) and ConsSentN(3) on MRPC(+0.8), while the latter performs better in the other tasks. It is important to note the differences between our models and QuickThoughts. Our approach only requires a set of unordered sentences, while QuickThoughts requires ordered sentences and this is crucial for its training.
Further, our results were obtained by training on about 15M sentences, while QuickThoughts was trained on a much larger corpus of 181M sentences. We use a ﬁnal sentence representation of 4096 dimensions while QuickThoughts uses 4800 dimensions. As a baseline, we also train a LSTM (with 4096 hidden dimensions and 512 word embedding dimensions) language model on the same dataset that was used to train our models. The last hidden state of the trained LSTM acting on a sentence is then used as the sentence representation.
As shown in Table 2, its performance is generally much worse than any of our models.
This is not surprising as sentence encoders based on language models tend to work better when trained on large spans of text encompassing several sentences, as has been observed in Radford (2018).
[Table]
ConsSent-R(2) 79.9 84.3 93.
ConsSent-MT(3) 80.2 84.3 94.
Ensemble 81.6 85.1 94.
Supervised Methods
InferSent 81.1 86.3 92. MultiTask
[Table]
[Table] [Table] [Table] [Table]
[Table]
k or k −
A cc u r ac y
CS-C
CS-N
CS-D CS-P
CS-I
CS-R (a) TREC          k or k −
A cc u r ac y
CS-C CS-N
CS-D CS-P
CS-I
CS-R (b) MRPC            k or k −
P ea r s on
C o rr e l a ti on CS-C CS-N CS-D CS-P
CS-I
CS-R (c) STSB
Our best results are achieved by an ensemble of the six individual models. The predictions of the ensemble model are calculated by weighting the predictions of the individual models by the normalized validation performance on each of the tasks in SentEval. The ensemble model shows improvements in all the tasks, with an average improvement of almost +1.6 over the best single model ConsSent-N(3) (or ConsSent-MT(3)).
It also performs better than the supervised InferSent model and comparably to QuickThoughts.
Interestingly, choosing the ensemble by using the best six models (out of the set of 30 models) based on validation performance gave slightly worse results. This empirically shows that the individual training tasks bias the sentence encoders in slightly different ways, and an ensemble of models can beneﬁt from all of them. We compare the relative performance of the encoders with varying values of k for three of the transfer tasks - TREC, MRPC and STSB in Fig. For TREC and MRPC, which are classiﬁcation tasks, there is roughly an inverted V shaped trend with some intermediate value of k giving the best
[Table]
[Table] [Table] [Table] [Table]
[Table]
ConsSent-I(3)
[Table]
[Table]
[Table] [Table]
ConsSent-R(2) 86.9 92.
[Table] [Table]
[Table]
ConsSent-MT(3) 86.
[Table]
[Table]
Ensemble
[Table]
[Table]
Supervised Methods
Seq2Tree [Table]
NMTEn-Fr
[Table]
k or k −
A cc u r ac y
CS-C
CS-N
CS-D CS-P
CS-I
CS-R (a) WordContent           k or k −
A cc u r ac y
CS-C
CS-N
CS-D CS-P
CS-I
CS-R (b) BigramShift          k or k −
A cc u r ac y
CS-C
CS-N
CS-D CS-P
CS-I
CS-R (c) SubjNum results for ConSent-D,P,I,R. Note that for smaller values of k , the encoders are exposed to negative examples that are are relatively similar to the positive ones and hence the discriminative training can be more noisy.
On the other hand, as k increases, the encoders may latch onto superﬁcial patterns and hence not generalize well.
For ConsSent-C,N the trends are less clear for TREC but are closer to an inverted V for MRPC.
For the semantic scoring task of STSB, the trend lines show no clear pattern.
R ESULTS ON L INGUISTIC P ROBING T ASKS
We present results on the 10 linguistic probing tasks in SentEval in Table 3.
All the encoders perform surprisingly well on most of the tasks, with the best ones ConsSent-D(5) and ConsSent-
C o n s S e n tC
C o n s S e n tN
C o n s S e n tD
C o n s S e n tP
C o n s S e n tI
C o n s S e n tR
ConsSent-MT(3) (82.4)
Ensemble (84.7)
P e r ce n t a g e
P(3) attaining an average score of 80.5.
ConsSent-D(5) achieves the best score in ﬁve tasks among our models.
ConsSent-MT(3) performs slightly worse, with an average score of 80.3.
In general, encoders trained on single sequences perform better than the ones trained using pairs of sequences on the linguistic probing tasks. Our methods perform signiﬁcantly better than the other unsupervised methods including
SkipThought and QuickThoughts. ConsSent-MT(3) achieves almost +7.0 points more average score than QuickThoughts, a model which does particularly well in the transfer tasks. Its performance is signiﬁcantly better than even the supervised baseline results trained on machine translation (+4.0) and natural language entailment (+9.0) in Conneau et al. The performance of a third method
Seq2Tree using a gated convolutional network (GCN) is however signiﬁcantly better than the ConsSent encoders (except on the WordContent task).
We have not experimented with a GCN encoder and it is possible that such an encoder may give better results in our case too.
The sentence encoder trained on the language modeling objective does not perform well, with an average score of less than 60.
Here again, our best results are obtained by the ensemble of six models, which shows strong improvements over the individual models across all the linguistic probing tasks, achieving an average score of 83.3, almost +2.8 more than the best single model ConsSent-D(5).
In Fig.
2, we show the trends observed with varying values of k for the individual encoders on three probing tasks –
WordContent, BigramShift and SubjNum.
C OMPARISON ACROSS T RANSFER AND L INGUISTIC P ROBING TASKS
In this section, we compare the ConsSent models across the all the tasks in SentEval.
In Fig.3 we plot the average performance of the encoders trained on the individual tasks, the multitask model and the ensemble model.
Clearly, the sentence encoders trained on single sentences (ConsSent-D,P,I,R) perform better overall than those trained on pairs of sentences (ConsSent-C,N).
In the former group, the ConsSent-R encoders tend to perform slightly better than the others, with ConsSent-R(2) achieving the best overall score of 81.9. The beneﬁts of combining multiple tasks is borne out by the fact that ConsSent-MT(3) achieves the best score of 82.4 for a single encoder.
The gains are even higher with the ensemble model, which achieves 84.7 or +2.8 points higher than the best single model.
Each of ConsSent-R(2), MultiTask and the ensemble model achieve signiﬁcantly better scores than existing unsupervised methods, including QuickThoughts, SkipThought and a baseline language model based encoder trained on our dataset.