F AKE S ENTENCE D ETECTION AS A T RAINING T ASK FOR S ENTENCE E NCODING
Anonymous authors
Paper under double-blind review
A BSTRACT
Sentence encoders are typically trained on generative language modeling tasks with large unlabeled datasets.
While these encoders achieve strong results on many sentence-level tasks, they are difﬁcult to train with long training cycles.
We introduce fake sentence detection as a new discriminative training task for learning sentence encoders.
We automatically generate fake sentences by corrupting original sentences from a source collection and train the encoders to produce representations that are effective at detecting fake sentences.
This binary classiﬁcation task turns to be quite efﬁcient for training sentence encoders.
We compare a basic BiLSTM encoder trained on this task with strong sentence encoding models (Skip-thought and FastSent) trained on a language modeling task.
We ﬁnd that the BiLSTM trains much faster on fake sentence detection (20 hours instead of weeks) using smaller amounts of data (1M instead of 64M sentences).
Further analysis shows the learned representations also capture many syntactic and semantic properties expected from good sentence representations.
I NTRODUCTION
Universal sentence encoding is a way to scale language processing tasks, especially when the target training data is limited.
Solutions to sentence-level language processing tasks can be seen as consisting of two parts – one that creates a generic representation which approximates the meaning of the sentence, and another that uses this representations to make the target classiﬁcations.
The idea behind universal sentence encoding is to learn generic sentence representations from unlabeled texts, which are often much larger and easier to obtain than the training data for the target tasks.
A good universal encoding is one which is effective for training downstream target tasks.
The success of language modeling ideas for learning word representations has inspired similar ideas for learning universal sentence representations.
Skip-thought model (Kiros et al., 2015) uses a Bidirectional LSTM (BiLSTM) encoder, which encodes a given sentence, and uses the encoding to generate neighboring sentences.
Following up on this general idea, the FastSent model (Hill et al., 2016) simpliﬁes the task and the encoder further by learning to output the average word embeddings so that they can predict words in the neighboring sentences (but not necessarily in a sequence).
This language modeling based training is undesirable in two respects.
(1) Predicting neighboring sentences is a difﬁcult generative training task.
The task requires having a large output space, at least the size of the vocabulary.
It also requires a complex decoding model such as an LSTM decoder with millions of parameters.
Effective training requires a large amount of training data with long training cycles.
Indeed Skip-thought requires tens of millions of training sentences and it takes more than a week to train.
(2) Even minor changes to a sentence can drastically change its meaning, a phenomenon that needs to be modeled for many NLP tasks.
Language model based training essentially relies on the training sample for this kind of generalization, i.e., the training text collection should include many instances of sentences that have only minor lexical differences but found in completely different contexts.
Larger the dataset the more likely it is for the encoder to learn how to model these ﬁne grained distinctions.
In this work, we introduce a discriminative training task fake sentence detection to address these challenges.
The main idea is to generate fake sentences by corrupting an original sentence.
We use two methods to generate fake sentences: word shufﬂing where we swap the positions of two words at random and word dropping , where we drop a word at random from the original sentence.
The resulting fake sentences are mostly similar to the original sentences—a fake sentence differs from its source in at most two word positions.
We create the training corpus from a source corpus of unlabeled sentences.
For each sentence in the source corpus we add multiple fake sentences by corrupting the source sentence.
This training task has three key advantages: (1) This binary classiﬁcation task can be modeled with fewer parameters in the output layer and can be trained more efﬁciently compared to language modeling based training tasks.
(2) From a language standpoint, the task forces the encoder to track both syntax and semantics.
For instance, swapping words within a sentence can break the syntax (e.g., “John landed in Chicago on Friday” versus “John landed Chicago in on Friday”), and break or alter the semantics; it can lead to an incoherent or less plausible sentence (e.g., “John landed in
Chicago on Friday” versus “Chicago landed in John on Friday”).
The task explicitly forces the encoder to capture big shifts in meaning that arise from small changes to a sentence.
In particular, since the discrimination is really dependent on a small change to an original sentence, encoders cannot get away with simple aggregation—they need to model compositional aspects well enough to be able to detect small but semantically shifts in sentence constructions.
In overview, we train a bidirectional LSTM (BiLSTM) as our universal encoder and use a threelayer feed-forward network that uses the encoded sentence to predict if it is fake or real.
We then evaluate this trained encoder without any further tuning on multiple sentence-level tasks and use probing tasks (Conneau et al., 2018) to study the syntactic and semantic properties in the encoded representations.
In summary, this paper makes the following main contributions:
Introduces fake sentence detection as an unsupervised training task for learning sentence encoders that can distinguish between small changes in mostly similar sentences.
Provides an empirical evaluation on multiple sentence-level tasks showing representations trained on the fake sentence tasks outperform a strong baseline model trained on language modeling tasks, even when training on small amounts of data (1M vs. 64M sentences) reducing training time from weeks to within 20 hours.
Demonstrates that the fake sentence training produces encoded representations that are better at capturing many linguistic properties compared to language model training.
M OTIVATION
Many sentence-level prediction tasks require learning a function to estimate the probability of a label y given a sentence x .
This function can be parameterized as P ( y | x, θ ) , and the parameter vector θ can be learned by minimizing the negative conditional log likelihood of the training data D , i.e., min θ − (cid:80) ( x,y ) ∈D
log( P ( y | x, θ )) .
However, the amount of training data is limited in many cases, so learning θ by optimizing the conditional probability leads to overﬁtting, especially if one uses a deep neural network with millions of parameters.
One approach to address this problem is to break the task into two sub-tasks each with its own set of parameters: (1) sentence encoding—produce an effective low-dimensional representation of the sentence enc ( x, θ 1 ) , (2) target prediction—estimate probability of the label y conditioned on the encoding P ( y | x, enc ( x, θ 1 ) , θ 2 ) .
The dimension of θ 2 is much smaller than the dimension of θ and it can be learned with the provided labeled training data D .
For the encoder function enc ( x, θ 1 ) , it is hoped that the parameter vector θ 1 can be learned or pre-trained with a different sentence-level task using other data, which is possibly unlabeled, but can be easily collected in large quantity.
The encoder can be trained on the auxiliary task as part of a prediction function f which has its own set of parameters θ 3 .
The encoder parameters θ 1 and the auxiliary task parameters θ 3 are learned to minimize the loss on the auxiliary task: (cid:80) x ∈U L aux ( f ( enc ( x, θ 1 ) , θ 3 )) .
In this work, we are interested in learning an universal sentence encoder that can be subsequently used for various downstream sentence-level tasks.
One reasonable choice for the auxiliary loss function would be the negative log likelihood of the data, i.e., − log P ( x
| θ 1 , θ 3 ) or equivalently − log( P ( enc ( x, θ 1 ) | θ 1 , θ 3 )) .
Recall for the downstream task, we will seek the parameter vector θto minimize the negative conditional log likelihood − log( P ( y | enc ( x, θ 1 ) , θ 1 , θ 2 )) .
The combination of two loss functions is: − log( P ( y | enc ( x, θ 1 ) , θ 1 , θ 2 ))
− log( P ( enc ( x, θ 1 ) | θ 1 , θ 3 )) .
This is equivalent to − log( P ( y, enc ( x, θ 1 )
| θ 1 , θ 2 , θ 3 )) .
In other words, we are maximizing the joint probability of observing the label y and the encoding vector enc ( x, θ 1 ) .
Optimizing the joint probability of the label and data corresponds to having a generative classiﬁer, as opposed to the discriminative classiﬁer that optimizes a conditional probability.
As shown by (Ng & Jordan, 2002), a generative classiﬁer converges faster to its asymptotic behavior than a discriminative classiﬁer does.
A generative approach is particularly useful when there is limited amount of training data.
Thus, we propose to learn a sentence encoder by maximizing the data likelihood P ( enc ( x, θ 1 ) | θ 1 , θ 3 ) , and we expect it to be an useful universal sentence encoder, especially when there is limited amount of labeled training data for downstream sentence-level task.
It is surprisingly simple to learn an encoder to maximize the data likelihood P ( enc ( x, θ 1 ) | θ 1 , θ 3 ) , and this paper proposes exactly that.
We can parameterize this probability using a simple function such as a log-linear model:
[Equation]
where the functional form of the encoder enc is an BiLSTM, and Z is a normalization constant to ensure a valid probability function.
[Equation]
where U is the set of real sentences, and V is the set of fake sentences.
Our objective is to learn θ 1 , θ 3 to maximize the quantity in Equation 1.
This is equivalent to maximizing (cid:80) x ∈U exp( − θ T 3 enc ( x, θ 1 )) while minimizing (cid:80) x ∈V exp( − θ T 3 enc ( x, θ 1 )) .
This can simply be done by training a classiﬁer to separate real sentences from fake sentences.
In this work, we approximate the space of fake sentences by randomly swapping or dropping words from real sentences.
Our method has some connection to the GAN framework (Goodfellow et al., 2014), where there is a generator that attempts to generate realistic data and a discriminator that distinguishes between real and generated data instances.
As shown by Goodfellow et al.
(2014), this approach will converge to a solution where the output of the discriminator is the estimated value for a data point to be real.
In our case, we do not need to learn the generator and the discriminator in an adversarial manner because we already have a ‘strong’ generator that generates highly realistic sentences, simply by shufﬂing or dropping words.
Furthermore, our method can be thought as a generative model, deﬁned and trained using a discriminative function.
This aspect is related to the recently proposed Introspective Neural Networks (Lazarow et al., 2017).
There are some similarities between the proposed method and Skip-thought sentence encoder.
The training task of Skip-thought involves predicting the next or previous sentence z that follows or precedes the current encoded sentence x .
This sentence language model is realized using an LSTM decoder, which sequentially generates a word at each position i conditioning on the previous encoded sentence enc ( x ) , and the words that have been decoded so far ( z 1 , · · · , z i − 1 ).
This model can be seen as maximizing the conditional probability:
[Equation]
Interestingly, both Skip-thought and our proposed method model the unlabeled data U , a sequence of sentences u 1 , u 2 , · · · , u m , and aim to optimize for the joint probability distribution: P ( U| θ 1 , θ 3 ) .
In the case of Skip-thought, there is an underlying Markovian assumption that the probability of a sentence only depends on the previous sentence.
That is to approximate P ( U| θ 1 , θ 3 ) by (cid:81) k P ( u k | enc ( u k − 1 , θ 1 ) , θ 3 ) .
In the case of the proposed method, we assume the sentences are independent of one another, so P
( U| θ 1 , θ 3 ) can be approximated as (cid:81) k P ( enc ( u k , θ 1 )
| θ 1 , θ 3 ) .
Both methods are generative models of the data, and both are useful sentence encoders for the downstream tasks, as will be seen in the experiment sections.
However, Skip-thought model has several disadvantages compared to the proposed method: (1)
The architecture of Skip-thought model is much more complex.
Skip-thought model requires an
LSTM decoder with millions of parameters.
Furthermore, the need for computing the probability of individual words requires having a large output space, as large as the size of the vocabulary.
This further increases then number of parameters and the complexity of the overall architecture.
The modeling task of Skip-thought is much harder than that of the proposed method.
Skipthought requires a probability model for both complete and incomplete sentences, and this much harder than modeling the probability of complete sentences alone (as proposed here).
Due to the complexity of the architecture and the difﬁculty of the modeling task, Skip-thought requires much more training data and longer training time.
(4) Skip-thought is a purely generative model and it is only trained using the real data.
Meanwhile, the proposed method is a generative model deﬁned via a discriminative function.
This function can be trained using both real and fake data, where the fake data can be selectively chosen to be very hard examples.
Speciﬁcally, we use word shufﬂing and word dropping to generate hard examples that do not differ from the original sentences.
This enables the sentence encoder to focus on the subtle semantics information.
R ELATED W ORK
Previous sentence encoding approaches can be broadly classiﬁed as supervised (Conneau et al., 2017; Cer et al., 2018; Marcheggiani & Titov, 2017; Wieting et al., 2015), unsupervised (Kiros et al., 2015; Hill et al., 2016) or semi-supervised approaches (Peters et al., 2018; Dai & Le, 2015;
Socher et al., 2011; Clark et al., 2018).
The unsupervised approaches extend the skip-gram (Mikolov et al., 2013) to the sentence level, and use the sentence embedding to predict the adjacent sentences.
Skip-thought (Kiros et al., 2015) uses a BiLSTM encoder to obtain a ﬁxed length embedding for a sentence, and uses a BiLSTM decoder to predict adjacent sentences.
Training Skip-thought model is expensive, and even one epoch of training on the Toronto BookCorpus (Zhu et al., 2015) dataset takes more than two weeks (Hill et al., 2016) on a single GPU.
FastSent (Hill et al., 2016) uses embeddings of a sentence to predict words from the adjacent sentences.
A sentence is represented by simply summing up the word representation of all the words in the sentence.
FastSent requires less training time than Skip-thought, but FastSent has lower performance on most downstream tasks.
The supervised approaches train the encoders on tasks such as NLI and use transfer learning to adapt the learned encoders to different downstream tasks (Conneau et al., 2017).
Some semi-supervised approaches (Peters et al., 2018; Dai & Le, 2015; Socher et al., 2011) train sentence encoders on large unlabeled datasets, and do a task speciﬁc adaptation using labeled data, while some other approaches such as Cross-View Training(CVT)
(Clark et al., 2018) jointly train the sentence encoder on the labeled and unlabeled data.
CVT applies regular supervised learning on the labeled data, and for the unlabeled data, it enforces consistency across the outputs obtained for partial versions of the input sentence.
In this work, we propose an unsupervised sentence encoder that takes around 20 hours to train on a single GPU, and outperforms Skip-thought and FastSent encoders on multiple downstream tasks.
Unlike the previous unsupervised approaches, we use the binary task of real versus fake sentence classiﬁcation to train a BiLSTM based sentence encoder.
T RAINING T ASKS FOR E NCODERS
We propose a discriminative task for training sentence encoders.
The key bottleneck in training sentence encoders is the need for large amounts of labeled data.
Prior work use language modeling as a purely generative training task that models the generation of unlabeled text data.
The encoder is trained to produce sentence representations which are effective at either generating neighboring sentences (e.g., Skip-thought (Kiros et al., 2015) or to predict the words in the neighboring sentences (Hill et al., 2016).
As we outlined in Section 2 this purely generative training raises multiple challenges.
Instead, we propose fake sentence detection, a task that targets the same overall generative model, one that generates unlabeled text, but is trained discriminatively.
The task requires making a single prediction over an input sentence.
In particular, we propose to learn a sentence encoder by training a sequential model to solve the binary classiﬁcation task of detecting whether a given input sentence is fake or real.
This real-fake sentence classiﬁcation task would perhaps be trivial if the fake sentences look very different from the real sentences.
We propose two simple methods to generate noisy sentences which look mostly similar to real sentences.
We describe the noisy sentence generation strategies in Section 4.1.
Thus, we create a labeled dataset of real and fake sentences, and train a sequential model to distinguish between real and fake sentences, which results in a model whose classiﬁcation layer has far fewer parameters than previous language model based encoders.
Our model architecture is described in Section 4.2.
4. F AKE S ENTENCE G ENERATION
For a sentence X = w 1 , w 2 , . . .
, w n comprising of n words, we consider two strategies to generate a noisy version of the sentence: 1) WordShufﬂe : randomly sample two indices i and j corresponding to words w i and w j in X , and shufﬂe the words to obtain the noisy sentence ˆ X .
Noisy sentence
ˆ
X would be of the same length as the original sentence X . 2)
WordDrop : randomly pick one index i corresponding to word w i and drop the word from the sentence to obtain ˆ X .
Note there can be many variants for this strategy but here we experiment with this basic choice.
[Figure]
Figure 1 shows the proposed architecture of our fake sentence classiﬁer with an encoder and a Multilayer Perceptron(MLP) with 2 hidden layers.
We do not use any nonlinearity after these hidden layers, hence, these layers correspond to the single linear layer represented by θ 3 in Equation 1.
Another motivation behind not using the nonlinearity is to keep the classiﬁer simple enought so that the encoder is forced to learn useful representation while training for the fake sentence classiﬁcation task.
The encoder consists of a bidirectional LSTM followed by a max pooling layer.
Given a sentence X = w 1 , w 2 , . . .
, w n consisting of n words, the forward LSTM of the encoder generates a hidden state vector −→ h t .
Similarly, the backward LSTM generates the hidden state ←− h t .
The forward and backward hidden states are concatenated to get u t
= ( −→ h t , ←− h t ).
Processing the entire sentence results in n such vectors, and we apply max-pooling to these concatenated hidden states to obtain vector z .
z serves as a ﬁxed length encoding of the sentence X , which we then use as input to a
MLP for classifying the sentence into real/fake classes.
E VALUATION S ETUP
Downstream Tasks:
We compare the sentence encoders trained on a large collection (BookCorpus (Zhu et al., 2015)) by testing them on multiple sentence level classiﬁcation tasks (MR, CR,
SUBJ, MPQA, TREC, SST) and one NLI task deﬁned over sentence-pairs (SICK).
We also eval-
[Table] [Table]
use the same evaluation protocol and dataset split as (Karpathy & Fei-Fei, 2015;
Conneau et al., 2017).
Table 1 lists the classiﬁcation tasks and the datasets.
We also compare the sentence representations for how well they capture important syntactic and semantic properties using probing classiﬁcation tasks (Conneau et al., 2018).
For all downstream and probing tasks, we use the encoders to obtain representation for all the sentences, and train logistic regression classiﬁers on the training split.
We tune the L 2 -norm regularizer using the validation split, and report the results on the test split.
Training Corpus: The FastSent and Skip-thought encoders are trained on the full Toronto BookCorpus of 64M sentences (Zhu et al., 2015).
Our models, however, train on a much smaller subset of only 1M sentences.
[Table]
ordination is inverted.
Sentence Encoder Implementation: Our sentence encoder architecture is the same as the
BiLSTM-max model (Conneau et al., 2017).
We represent words using 300-d pretrained Glove embeddings (Pennington et al., 2014).
We use a single layer BiLSTM model, with 2048-d hidden states.
The MLP classiﬁer we use for fake sentence detection has two hidden layers with 1024 and 512 neurons.
We train separate models for word drop and word shufﬂe.
The models are trained for 15 epochs with a batch size of 64 using SGD algorithm, when training converges with a validation
[Table]
set accuracy of 89 for word shufﬂe.
The entire training completes in less than 20 hours on a single
GPU machine.
Baseline Approaches:
We compare our results with previous unsupervised sentences encoders,
Skip-thought (Kiros et al., 2015), FastSent (Hill et al., 2016), Quick-thought(MC-QT) (Logeswaran & Lee, 2018), and bag-of-words baseline NB-SVM (Wang & Manning, 2012).
We use the FastSent and Skip-thought results trained on the full BookCorpus as mentioned in (Conneau et al., 2017).
We also report Skip-thought results when trained on a smaller 1M sentence subset.
R ESULTS
Classiﬁcation and NLI: Results are shown in Table 2.
Both fake sentence training tasks yield better performance on ﬁve out of the seven language tasks when compared to Skip-thought (full), i.e., even when it is trained on the full BookCorpus.
Word drop and word shufﬂe performances are mostly comparable.
Skip-thought (1M) row shows that training on a sentence-level language modeling task can fare substantially worse when trained on a smaller subset of data.
FastSent, while easier to train and has faster training cycles, is better than Skip-thought (1M) but is worse than the full Skip-thought model.
Further gain in performance can be obtained by validating the performance of the encoder on downstream tasks after each epoch of training, and picking the encoder with best performance on the validation set for the downstream tasks.
For MR and SUBJ tasks, this leads to accuracy of 80.3 and 93.3 respectively.
For the remaining tasks, performance gain is less signiﬁcant.
Image-Caption Retrieval: On both caption and image retrieval tasks (last 2 columns of Table 2), fake sentence training with word dropping and word shufﬂe are better than the published Skipthought results.
Linguistic Properties: Table 3 compares sentence encoders for various linguistic properties using the recently proposed probing tasks (Conneau et al., 2018).
The goal of each task is to use the input sentence encoding to predict a particular syntactic or semantic property of the original sentence it encodes (e.g., predict if the sentence contains a speciﬁc word).
Encodings from fake sentence training score higher in six out of the ten tasks.
A key property of the discriminative training is that it forces the encoder to be sensitive to small changes that can induce large meaning shifts.
WordShufﬂe encodings turn out to perform significantly better on probing tasks that are related to the sensitivity property.
Semantic odd man out requires identifying word insertions that are incompatible in the context, tracking word content requires knowing if a particular word is present or not, and bigram shift (BShift) requires knowing if words in adjacent positions are swapped (a subset of the training task for WordShufﬂe).
Tableillustrates examples where fake sentence training is able to distinguish sentence meaning changes that results from a small change in its surface form.
Sentence Lengths We analyze the performance of fake sentence and Skip-thought models to understand how the models behave for sentences of different lengths.
Figure 2 compares the performance of fake sentence and Skip-thought encoders binned by length.
It turns out that fake sentence training performs better on longer sentences on MR but not so on the SST, even though both are sentiment tasks.
The analysis for other tasks (not shown here) also indicate that fake sentence is better for most sentence lengths but do not indicate a clear trend with respect to increasing sentence lengths.
[Figure] [Figure]
Relation between Fake Sentence Classiﬁcation and Downstream Tasks In ﬁgure
[Figure]
we compare the test performance on fake sentence classiﬁcation task(WordShufﬂe) with that on multiple downstream tasks.
We notice that the encoder resulting in good performance on the fake sentence classiﬁcation task results in good performance on the downstream task.
C ONCLUSIONS
The effectiveness of universal sentence encoders depends both on the architecture of the encoders as well as the training task itself.
Language modeling is a suitable task as it ﬁts the generative goal of modeling the distribution of language (text data).
However, tackling this in a purely generative fashion (i.e., actually generating sentences) requires large models and long training times.
Instead, we introduced a discriminative formulation of the generative task called fake sentence detection.
The sentence encoders are trained to produce representations which are effective at detecting if a given sentence is an original or a fake.
This leads to better performance on downstream tasks and is able to represent semantic and syntactic properties, while also reducing the amount of training needed.
As future work, the discriminative setup opens up the possibility for the training to be inﬂuenced by a speciﬁc downstream target task.
In particular, we can create negative samples (fake sentences) that are focused towards those speciﬁc phenomena relevant to the target task.