Improving Sentence Representations with Multi-view Frameworks
Anonymous Author(s)
AfﬁliationAddress email
Abstract
Multi-view learning can provide self-supervision when different views are available of the same data.
Distributional hypothesis provides another form of usefulself-supervision from adjacent sentences which are plentiful in large unlabelledcorpora.
Motivated by the asymmetry in the two hemispheres of the human brainas well as the observation that different learning architectures tend to emphasisedifferent aspects of sentence meaning, we present two multi-view frameworks forlearning sentence representations in an unsupervised fashion.
One framework usesa generative objective and the other a discriminative one.
In both frameworks,the ﬁnal representation is an ensemble of two views, in which, one view encodesthe input sentence with a Recurrent Neural Network (RNN), and the other viewencodes it with a simple linear model.
We show that, after learning, the vectorsproduced by our multi-view frameworks provide improved representations overtheir single-view learned counterparts, and the combination of different views givesrepresentational improvement over each view and demonstrates solid transferabilityon standard downstream tasks.
Introduction
Multi-view learning methods provide the ability to extract information from different views of thedata and enable self-supervised learning of useful features for future prediction when annotated datais not available [18].
Minimising the disagreement among multiple views helps the model to learnrich feature representations of the data and, also after training, the ensemble of the feature vectorsfrom multiple views can provide an even stronger generalisation ability.
Distributional hypothesis [24] noted that words that occur in similar contexts tend to have similarmeaning [53] and distributional similarity [21] consolidated this idea by stating that the meaningof a word can be determined by the company it has.
The hypothesis has been widely used inmachine learning community to learn vector representations of human languages.
Models built upondistributional similarity don’t explicitly require human-annotated training data; the supervision comesfrom the semantic continuity of language data, such as text and speech.
Large quantities of annotated data are often hard to obtain.
Our goal is to propose learning algorithmsbuilt upon the ideas of multi-view learning and distributional hypothesis to learn from unlabelleddata.
We draw inspiration from the lateralisation and asymmetry in information processing of thetwo hemispheres of the human brain where, for most adults, sequential processing dominates the lefthemisphere, and the right hemisphere has a focus on parallel processing [11], but both hemisphereshave been shown to have roles in literal and non-literal language comprehension [16, 17].
Our proposed multi-view frameworks aim to leverage the functionality of both RNN-based models,which have been widely applied in sentiment analysis tasks [58], and the linear/log-linear models,which have excelled at capturing attributional similarities of words and sentences [6, 7, 26, 53] for learning sentence representations.
Previous work on unsupervised sentence representation learningbased on distributional hypothesis can be roughly categorised into two types:
Generative objective: These models generally follow the encoder-decoder structure.
The encoderlearns to produce a vector representation for the current input, and the decoder learns to generatesentences in the adjacent context given the produced vector [31, 26, 22, 52].
The idea is straightforward, yet its scalability for very large corpora is hindered by the slow decoding process thatdominates training time, and also the decoder in each model is discarded after learning as the qualityof generated sequences is not the main concern, which is a waste of parameters and learning effort.
Our ﬁrst multi-view framework has a generative objective and uses an RNN as the encoder and aninvertible linear projection as the decoder.
The training time is drastically reduced as the decoderis simple, and the decoder is also utilised after learning.
A regularisation is applied on the lineardecoder to enforce invertibility, so that after learning, the inverse of the decoder can be applied as alinear encoder in addition to the RNN encoder.
Discriminative Objective: In these models, a classiﬁer is learnt on top of the encoders to distinguishadjacent sentences from those that are not [34, 28, 43, 36]; these models make a prediction using apredeﬁned differential similarity function on the representations of the input sentence pairs or triplets.
Our second multi-view framework has a discriminative objective and uses an RNN encoder and alinear encoder; it learns to maximise agreement among adjacent sentences.
Compared to earlierwork on multi-view learning [18, 19] that takes data from various sources or splits data into disjointpopulations, our framework processes the exact same input data in two distinctive ways.
Having twodistinctive information processing views encourages the model to encode different aspects of an inputsentence, and is beneﬁcial to the future use of the learnt representations.
Our contribution is threefold:
Two multi-view frameworks for learning sentence representations are proposed, in which
oneframework uses a generative objective and the other one adopts a discriminative objective.
Two encoding functions, an RNN and a linear model, are learnt in both frameworks.2.
The results show that in both frameworks, the ensemble of two views provide better resultsthan each view alone, and especially, aligning two views by the invertible contraint inthe generative approach and by maximising the agreement in the discriminative objectiveimproves each view in our efﬁciently trained multi-view frameworks.3.
Models trained under our proposed frameworks achieve good performance on the unsupervised tasks, and overall outperform existing unsupervised transfer learning models, andarmed with various pooling functions, they also show solid results on supervised tasks,which are either comparable to or better than those of the best unsupervised transfer model.
It is shown in [26] that the consistency between supervised and unsupervised evaluation tasks is muchlower than that within either supervised or unsupervised evaluation tasks alone and that a modelthat performs well on supervised evaluation tasks may fail on unsupervised tasks.
[15] subsequentlyshowed that, with a labelled training corpus, such as SNLI [10] and MultiNLI [57], the resultingrepresentations of the sentences from the trained model excel in both supervised and unsupervisedtasks.
Our model is able to achieve good results on both groups of tasks without labelled information.
Model Architecture
Our goal is to marry RNN-based sentence encoder and the avg-on-word-vectors sentence encoderinto multi-view frameworks with simple training objectives.
The motivation for the idea is that, as mentioned in the prior work, RNN-based encoders process thesentences sequentially, and are able to capture complex syntactic interactions, while the avg-on-wordvectors encoder has been shown to be good at capturing the coarse meaning of a sentence whichcould be useful for ﬁnding paradigmatic parallels [53].
We present two multi-view frameworks to learn two different sentence encoders; after learning, thevectors produced from two encoders of the same input sentence are used to compose the sentencerepresentation.
The details of our learning frameworks are described as follows:
2.1 Encoders
In our multi-view frameworks, we ﬁrst introduce two encoders that, after learning, can be used tobuild sentence representations.
One encoder is a bi-directional Gated Recurrent Unit [12] f ( s ; φ ) ,where s is the input sentence and φ is the parameter vector in the GRU.
During learning, onlyhidden state at the last time step is sent to the next stage in learning.
The other encoder is a linearavg-on-word-vectors model g ( s ; W ) , which basically transforms word vectors in a sentence by alearnable weight matrix W and outputs an averaged vector.
2.2 Generative Objective
Given the ﬁnding [52] that neither an autoregressive nor an RNN decoder is necessary for learningsentence representations that excel on downstream tasks, our learning framework only learns topredict words in the next sentence.
The framework has an RNN encoder f , and a linear decoder h .
Given an input sentence s i , the encoderproduces a vector z fi = f
( s i ; φ ) , and the decoder h projects the vector to x i
= h ( z fi ; U ) =
Uz fi ,which has the same dimension as the word vectors v w .
Negative sampling is applied to calculate thelikelihood of generating the j -th word in the ( i + 1) -th sentence, shown in Eq.
[Equation]
(1) where v w k are pretrained word vectors for w k , the empirical distribution P e ( w ) is the unigramdistribution raised to power 0.75 [41], and K is the number of negative samples.
The learningobjective is to maximise the likelihood for words in all sentences in the training corpus.
Ideally, the inverse of h should be easy to compute so that during testing we can set g
= h − 1 .
As h isa linear projection, the simplest situation is when U is an orthogonal matrix and its inverse is equalto its transpose.
Often, as the dimensionality of vector z fi doesn’t necessarily need to match that ofword vectors v w , U is not a square matrix 1 .
To enforce invertibility on U , a row-wise orthonormal regularisation on U is applied during training,which leads to UU (cid:62)
=
I , where I is the identity matrix, thus the inverse function is simplyg − 1 ( x ) =
U (cid:62) x , which is easily computed.
The regularisation formula is || UU (cid:62) −
I || F , where|| · || F is the Frobenius norm.
Speciﬁcally, the update rule [13] for the regularisation is:
[Equation]
where β is set to 0 .
After learning, we set W =
U (cid:62) , then the inverse of the decoder h becomesthe encoder g .
Compared to prior work with generative objective, our framework reuses the decodingfunction rather than ignoring it for building sentence representations after learning, thus informationencoded in the decoder is also utilised.
2.3 Discriminative Objective
Our multi-view framework with discriminative objective learns to maximise the agreement betweenthe representations of a sentence pair across two views if one sentence in the pair is in the neighbourhood of the other one.
An RNN encoder f
( s ; φ ) and a linear avg-on-word-vectors g ( s ; W ) producea vector representation z fi and z gi for i -th sentence respectively.
The agreement between two views ofa sentence pair ( s i , s j ) is deﬁned as a ij = a ji = cos( z fi , z gj ) + cos( z gi , z fj ) .
The training objectiveis to minimise the loss function:
[Equation]
where
[Equation]
where τ is the trainable temperature term, which is essential for exaggerating the difference betweenadjacent sentences and those that are not.
The neighbourhood/context window c , and the batch size
N are hyperparameters.
The choice of cosine similarity based loss is based on the observations in [53] that, of word vectorsderived from distributional similarity, vector length tends to correlate with frequency of words, thusangular distance captures more important meaning-related information.
Also, since our model isunsupervised/self-supervised, whatever similarity there is between neighbouring sentences is what islearnt as important for meaning.
2.4 Postprocessing
The postprocessing step proposed in [7], which removes the top principal component of a batch ofrepresentations, is applied on produced representations from f and g respectively after learning witha ﬁnal l 2 normalisation.
In addition, in our multi-view framework with discriminative objective, in order to reduce thediscrepancy between training and testing, the top principal component is estimated by power iterationmethod [42] and removed during learning.
Experimental Design
Three unlabelled corpora from different genres are used in our experiments, including BookCorpus[60], UMBC News [23] and Amazon Book Review 2 [38]; six models are trained separately on each ofthree corpora with each of two objectives.
The summary statistics of the three corpora can be foundin Table 1.
Adam optimiser [30] and gradient clipping [46] are applied for stable training.
Pretrainedword vectors, fastText [9], are used in our frameworks and ﬁxed during learning.
[Table]
vectors in i -th sentence, and H i refers to hidden states at all time steps produced by f .
[Table]
Bi-GRU f : z fi [ max ( H i ); mean ( H i ); min ( H i ); h M i
i ] mean ( H i )
Linear g : z gi [ max ( W X i ); mean ( W X i ); min ( W X i )] mean ( W X i )
[Table]
All of our experiments including training and testing are done in PyTorch [47].
The modiﬁed SentEval[14] package with the step that removes the ﬁrst principal component is used to evaluate our modelson the downstream tasks.
Hyperparameters, including negative samples K in the framework withgenerative objective, context window c in the one with discriminative objective, are tuned onlyon the averaged performance on STS14 of the model trained on the BookCorpus; STS14/G1 and
STS14/D1 results are thus marked with a (cid:63) in Table 3 and Table 4 to indicate possible overﬁtting onthat dataset/model only.
Batch size N and dimension d in both frameworks are set to be the same forfair comparison.
Hyperparameters are summarised in supplementary material.
[Table] [Table]
3.1 Unsupervised Evaluation - Textual Similarity Tasks
Representation: For a given sentence input s with M words, suggested by [48, 33], the representationis calculated as z = (cid:0) ˆ
z f
+ ˆ z g (cid:1) / 2 , where ˆ z refers to the post-processed and normalised vector, andis mentioned in Table 2.
Tasks :
The unsupervised tasks include ﬁve tasks from SemEval Semantic Textual Similarity (STS)in 2012-2016 [1, 2, 3, 4, 5] and the SemEval2014 Semantic Relatedness task (SICK-R)
Comparison : We compare our models with: • Unsupervised transfer learning : We selected models with strong results from related work, includingfastText, fastText+WR.• Semi-supervised transfer learning : The word vectors are pretrained on each task [55] without labels,and word vectors are averaged to serve as the vector representation for a given sentence [7].
• Supervised transfer learning : ParaNMT [56] is included as a supervised transfer learning method as the data collection stage requires a neural machine translation system trained in supervised fashion.
and the InferSent
trained on SNLI and MultiNLI [ ] is included as well.
The results are presented in Table 3.
Since the performance of FastSent [26] and QT [36] were onlyevaluated on STS14, we compare to their results in Table 4.
All models trained with our learning frameworks outperform other unsupervised and semi supervisedtransfer learning methods, and the model trained on the UMBC News Corpus with discriminativeobjective gives the best performance likely because the STS tasks contain multiple news- andheadlines-related datasets which is well matched by the domain of the UMBC News Corpus.
3.2 Supervised Evaluation
The evaluation on these tasks involves learning a linear model on top of the learnt sentence representations produced by the model.
Since a linear model is capable of selecting the most relevantdimensions in the feature vectors to make predictions, it is preferred to concatenate various types ofrepresentations to form a richer, and possibly more redundant feature vector, which allows the linearmodel to explore the combination of different aspects of encoder functions to provide better results.
Representation: Inspired by prior work [39, 49], the representation z f is calculated by concatenatingthe outputs from the global mean-, max- and min-pooling on top of the hidden states H , and the lasthidden state, and z g is calculated with three pooling functions as well.
The post-processing and thenormalisation step is applied individually.
These two representations are concatenated to form a ﬁnalsentence representation.
Table 2 presents the details.
Tasks : Semantic relatedness (SICK) [37], paraphrase detection (MRPC) [20], question-type classiﬁcation (TREC) [35], movie review sentiment (MR) [45], Stanford Sentiment Treebank (SST)[50], customer product reviews (CR) [27], subjectivity/objectivity classiﬁcation (SUBJ) [44], opinionpolarity (MPQA) [54].
The results are presented in Table 5.
Comparison : Our results as well as related results of supervised task-dependent training models,supervised transfer learning models, and unsupervised transfer learning models are presented in Table5.
Note that, for fair comparison, we collect the results of the best single model (MC-QT) trained on
BookCorpus in [36].
Six models trained with our learning frameworks either outperform other existing methods, or achievesimilar results on some tasks.
The model trained on the Amazon Book Review gives the bestperformance on sentiment analysis tasks, since the corpus conveys strong sentiment information.
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table] [Table] AdaSent - - - - 92. TF-KLD - - - 80.4/85.
SWEM- concat - - - 71.5/81.
Supervised training - Transfer learning
InferSent <88.
Unsupervised training with unordered sentences
ParagraphVec - - 72.9/81.
GloVe+WR86.
fastText+bow - - - 73.4/81.
SDAE - - 73.7/80.
Unsupervised training with ordered sentences
FastSent - - 72.2/80.
Skip-thought 85.
CNN-LSTM † - 86.
DiscSent ‡ - - 75.0/ - 87.
DisSent ‡ - 79.
MC-QT 86.
Multi-view G 3.
Multi-view G  87.
Multi-view G  87.
Multi-view D  87.
Multi-view D 8.
Multi-view D  87.
[Table]
Discussion
4.1 Ensemble in Multi-view Frameworks
In both frameworks, RNN encoder and linear encoder perform well on all tasks, and generativeobjective and discriminative objective give similar performance.
In general, an ensemble of the representations generated from two distinct encoding functionsperforms even better.
The two encoding functions, f and g , have naturally different behaviour.
Withdistributional similarity [21], our multi-view frameworks help the two encoding functions to learnmore generalised representations.
Therefore, f and g encode the input sentence with emphasis ondifferent aspects, and the subsequently trained linear model for each of the supervised downstreamtasks beneﬁts from this diversity leading to better predictions.
4.2 Generative Objective: Regularisation on Invertibility
The orthonormal regularisation applied on the linear decoder to enforce invertibility in our multi-viewframework encourages the vector representations produced by f and those by h − 1 , which is g intesting, to align with each other.
A direct comparison is to train our multi-view framework without the invertible constraint, and still directly use U (cid:62) as an additional encoder in testing.
The results ofour framework with and without the invertible constraint are presented in Table 6.
The ensemble method of two views, f and g , on unsupervised evaluation tasks (STS12-16 and
SICK14) is averaging, which beneﬁts from aligning representations from f and g by applyinginvertible constraint, and the RNN encoder f gets improved on unsupervised tasks by learningto align with g .
On supervised evaluation tasks, as the ensemble method is concatenation and alinear model is applied on top of the concatenated representations, as long as the encoders in twoviews process sentences distinctively, the linear classiﬁer is capable of picking relevant featuredimensions from both views to make good predictions, thus there is no signiﬁcant difference betweenour multi-view framework with and without invertible constraint.
4.3 Discriminative Objective: Multi-view vs. Single-view
In order to determine if the multi-view framework with two different views/encoding functions ishelping the learning, we compare our framework with discriminative objective to other reasonablevariants, including the multi-view model with two functions of the same type but parametrisedindependently, either two f -s or two g -s, and the single-view model with only one f or g .
Table 6presents
the results of the models trained on UMBC Corpus.
In our multi-view learning with f and g , the two encoding functions improve each other’s view.
As illustrated in previous work, and speciﬁcally emphasised in [26], linear/log-linear models, whichinclude g in our model, produce better representations for STS tasks than RNN-based models do.
Thesame ﬁnding can be observed in Table 6 as well, where g consistently provides better results on STStasks than f does.
In addition, as we expected, in our multi-view learning with f and g , g improvesthe performance of f on STS tasks.
With maximising the agreement between the representationsgenerated from f and g , we also see in the table that f improves g on supervised evaluation tasks.
Compared with the ensemble of two multi-view models, each with two encoding functions of thesame type, our multi-view framework with f and g provides slightly better results on STS tasks, andsimilar results on supervised evaluation tasks, while our model has much higher training efﬁciency.
Compared with the ensemble of two single-view models, each with only one encoding function, thematching between f and g in our multi-view model produces better results.
Conclusion
We proposed multi-view sentence representation learning frameworks with generative and discriminative objectives; each framework combines an RNN-based encoder and an average-on-word-vectorslinear encoder and can be efﬁciently trained within a few hours on a large unlabelled corpus.
Theexperiments were conducted on three large unlabelled corpora, and meaningful comparisons weremade to demonstrate the generalisation ability and the transferability of our learning frameworks,and also to consolidate our claim.
The produced sentence representations outperform existing unsupervised transfer methods on unsupervised evaluation tasks, and match the performance of the bestunsupervised model on supervised evaluation tasks.
As presented in our experiments, the ensemble of two views leveraged the advantages of both views,and provides rich semantic information of the input sentence, also the multi-view learning helpseach view to produce better representations than single-view learning.
Meanwhile, our experimentalresults also support the ﬁnding in [26] that linear/log-linear models ( g in our frameworks) tend towork better on the unsupervised tasks, while RNN-based models ( f in our frameworks) generallyperform better on the supervised tasks.
Future work should explore the impact of having variousencoding architectures and learning under the multi-view framework.
Our multi-view learning frameworks were inspired by the asymmetric information processing in thetwo hemispheres of the human brain, in which for most adults, the left hemisphere contributes tosequential processing, including primarily language understanding, and the right one carries out moreparallel processing, including visual spatial understanding [11].
The experimental results raise anintriguing hypothesis about how these two types of information processing may complementarilyhelp learning.
Evaluation tasks
The details, including size of each dataset and number of classes, about the evaluation tasks arepresented below 1 .
[Table]
Power Iteration
The Power Iteration was proposed in [42], and it is an efﬁcient algorithm for estimating the topeigenvector of a given covariance matrix.
Here, it is used to estimate the top principal componentfrom the representations produced from f and g separately.
We omit the superscription here, sincethe same step is applied to both f and g .
Suppose there is a batch of representations Z =
[ z 1 , z 2 ..., z N ] ∈ R 2 d × N from either f or g , the
Power Iteration method is applied here to estimate the top eigenvector of the covariance matrix 2 :
C = ZZ (cid:62) , and it is described in Algorithm 1:
Algorithm the First Principal Component [42]
Input: Covariance matrix C ∈ R 2 d × 2 d , number of iterations T
Output: First principal component u ∈ R 2 d 1: Initialise a unit length vector u ∈ R 2 d 2: for t ← 1 , T do 3: u ← Cu , 4:
[Equation]
In our experiments T is set to be
Training & Model Details
The hyperparameters we need to tune include the batch size N , the dimension of the GRU encoderd , and the context window c , and the number of negative samples K .
The results we presented inthis paper is based on the model trained with N = 512 , d = 1024 .
Speciﬁcally, in discriminativeobjective, the context window is set c = 3 , and in generative objective, the number of negativesamples is set K = 5 .
It takes up to 8GB on a GTX 1080Ti GPU.
The initial learning rate is 5 × 10 − 4 , and we didn’t anneal the learning rate through the training.
Allweights in the model are initialised using the method proposed in [25], and all gates in the bi-GRUare initialised to 1, and all biases in the single-layer neural network are zeroed before training.
Theword vectors are ﬁxed to be those in the FastText [9], and we don’t ﬁnetune them.
Words that arenot in the FastText’s vocabulary are ﬁxed to 0 vectors through training.
The temperature term isinitialised as 1 , and is tuned by the gradient descent during training.
The temperature term is used to convert the agreement a ij to a probability distribution p ij in Eq.1 in the main paper.
In our experiments, τ is a trainable parameter initialised to 1 that decreasedconsistently through training.
Another model trained with ﬁxed τ set to the ﬁnal value performedsimilarly.
Number of Parameters
The number of parameters of each of the selected models is:
Ours: ≈ 8 . 8 M2.
Quick-thought [36]: ≈ 19 . 8 M3.
Skip-thought [31]:
≈ 57 .
7 M