N O T RAINING R EQUIRED : E XPLORING R ANDOM E NCODERS FOR S ENTENCE C LASSIFICATION
Anonymous authors
Paper under double-blind review
A BSTRACT
We explore various methods for computing sentence representations from pretrained word embeddings without any training , i e., using nothing but random parameterizations.
Our aim is to put sentence embeddings on more solid footing by 1) looking at how much modern sentence embeddings gain over random methods—as it turns out, surprisingly little; and by 2) providing the ﬁeld with more appropriate baselines going forward—which are, as it turns out, quite strong.
We also make important observations about proper experimental protocol for sentence classiﬁcation evaluation, together with recommendations for future research
I NTRODUCTION
A sentence embedding is a vector representation of the meaning of a sentence, often created by a transformation of word embeddings through a compositional function. This function is often nonlinear and recurrent in nature, and usually the word embeddings are initialized with pre-trained embeddings.
Well-known examples of sentence embeddings include SkipThought (Kiros et al., 2015) and InferSent (Conneau et al., 2017).
The purpose of sentence embeddings is to have the meaning of a sentence encoded into a compact representation where it is readily available for downstream applications.
For instance it could provide features for training a classiﬁer or a distant metric could be applied to two representations to compute the semantic similarity of their corresponding sentences.
Sentence embeddings can be trained with either an unsupervised or supervised objective, and they are often evaluated using transfer tasks, where a ahallow classiﬁer is trained on top of the learned sentence encoder (which is kept ﬁxed).
There has been a lot of recent interest in trying to understand better what these sentence embeddings learn (Adi et al., 2016; Linzen et al., 2016; Conneau et al., 2018; Zhu et al., 2018).
The natural language processing community does not yet have a clear grasp on the relationship between word and sentence embeddings: it is unclear how much trained sentence-encoding architectures improve over the raw word embeddings, and what aspect of such architectures is responsible for any improvement.
Indeed, state-of-the-art word embeddings on their own perform quite well with simple pooling mechanisms, as reported by Wieting et al. ; Arora et al. (2017) and Shen et al.
Given the tremendous pace of research on sentence representations, it is important to establish solid baselines for others to build on.
It has been observed that bidirectional LSTMs with max-pooling perform surprisingly well even without any training whatsoever (Conneau et al., 2017; 2018), leading to claims that such architectures “encode priors that are intrinsically good for sentence representations” (Conneau et al., 2018), similar to convolutional networks for images (Ulyanov et al., 2017).
Inspired by these observations, we propose to examine the following question: given a set of word embeddings, how can we maximize classiﬁcation accuracy on the transfer tasks without any training , i.e. without updating any parameters except for those in the transfer task-speciﬁc linear classiﬁer trained on top of the representation.
SkipThought famously took around one month to train, while InferSent requires large amounts of annotated data—we examine to what extent we can match the performance of these systems by exploring different ways for combining nothing but the pre-trained word embeddings.
We go down a well-paved avenue of exploration in the machine learning research community, and exploit an insight originally due to Cover (1965): “A complex pattern-classiﬁcation problem, cast in a high-dimensional space nonlinearly, is more likely to be linearly separable than in a lowdimensional space, provided that the space is not densely populated.”
That is, we examine three of for obtaining randomly computed representations from trained
Our goal is not to obtain a new state of the art, but to put current state of the art methods on more solid footing by 1) looking at how much they gain compared to random methods; and 2) providing the ﬁeld with more solid baselines going forward.
We make several important observations about proper experimental protocol for sentence classiﬁcation evaluation; and ﬁnish with a list of takeaway recommendations.
R ELATED W ORK
Sentence embeddings are receiving a lot of attention.
Many approaches have been proposed, varying in their use of both training data and training objectives.
Methods include autoencoders (Socher et al., 2011; Hill et al., 2016) and other learning frameworks using raw text (Le & Mikolov, 2014;
Pham et al., 2015; Jernite et al., 2017; Pagliardini et al., 2017), a collection of books (Kiros et al., 2015), labelled entailment corpora (Conneau et al., 2017), image-caption data (Kiela et al., 2017), raw text labelled with discourse relations (Nie et al., 2017), or parallel corpora (Wieting & Gimpel, 2017).
Multi-task combinations of these approaches (Subramanian et al., 2018; Cer et al., 2018) have also been proposed.
Progress has been swift, but lately we have started to observe some troubling trends in how research is conducted, in particular with respect to properly identifying the sources of empirical gains (see also Lipton & Steinhardt (2018)).
There was an issue with non-standard evaluation methods, for which SentEval (Conneau & Kiela, 2018) and then GLUE (Wang et al., 2018) were created.
One often overlooked aspect of sentence representation evaluation, for example, is that logistic regression classiﬁers and multi-layer perceptrons (MLP) are not the same thing.
To single out an example, the recent paper by Shen et al.
(2018), which aims to “give baselines more love”, does not compare against LSTMs with the exact same pre-processing and range of hyperparameters, in effect ignoring its own baselines, and uses a custom designed MLP, sweeping over many hyperparameters unique to their setup with different embedding dimensionsionalities.
Even when comparing InferSent and SkipThought, it is not entirely clear where differences come from: the better pre-trained word embeddings; the different architecture; the different objective; the layer normalization—e.g. what would happen if we trained a bidirectional LSTM with max-pooling using GloVe embeddings (i.e., InferSent’s architecture) with a SkipThought objective or added layer normalization to InferSent?
The nowadays surprisingly poor performance of the models in Hill et al.
(2016) can at least partly be explained because 1) they use poorer (older) word embeddings; and 2) FastSent sentence representations are of the same dimensionality as the input word embeddings, while they are compared in the same table to much higher-dimensional representations.
Obviously, a logistic regression classiﬁer on top of a higherdimensional input has more parameters too, giving such models an unfair advantage.
In part, doing such in-full comparisons is simply not feasible, and often not appreciated by reviewers anyway, so we can hardly blame the authors of these papers.
That said, we wholeheartedly agree that baselines need more love, beﬁtting a good tradition in NLP (Wang & Manning, 2012): with this work we hope to establish even stronger baselines for future work and try to estimate how much performance is being added by training sentence embeddings on top of pre-trained word embeddings.
There has been a lot of recent interest in trying to understand what linguistic knowledge is encoded in word and sentence embeddings, for instance in machine translation (Belinkov et al., 2017; Sennrich, 2016; Dalvi et al., 2017), with a focus on evaluating RNNs or LSTMs (Linzen et al., 2016;
Hupkes et al., 2018) or even sequence-to-sequence models (Lake & Baroni, 2018).
Various probing tasks (Ettinger et al., 2016; Adi et al., 2016; Conneau et al., 2018) were designed to try to understand what you can “cram into a vector” for representing sentence meaning.
We show that a lot of information may be crammed into vectors using randomly parameterized combinations of pre-trained word embeddings: that is, most of the power in modern NLP systems is derived from having high-quality word embeddings, rather than from having better encoders.
The idea of using random weights is almost as old as neural networks, ultimately going back to ideas in multi-layer perceptrons with ﬁxed randomly initialized ﬁrst layers (Gamba et al., 1961; Borsellino & Gamba, 1961; Baum, 1988), or what Minsky and Papert call Gamba perceptrons (Minsky &
Papert, 2017).
The idea of ﬁxing a subset of the network was made more explicit in (Schmidt et al., 1992; Pao et al., 1994), which some people have started to call extreme learning machines (Huang et al., 2006).
Random features in machine learning are often used for low-rank approximation (Vempala, 2005), as per the Johnson-Lindenstrauss lemma; exploiting the useful properties of random matrices (Mehta, 2004).
Random “kitchen sink” features have become a seminal approach in the machine learning literature (Rahimi & Recht, 2008; 2009).
Similar ideas underlie e.g. double-stochastic gradient methods (Dai et al., 2014).
In fact, it is well-known that random weights do well, as for example shown in computer vision with respect to convnets (Jarrett et al., 2009; Saxe et al., 2011).
Similarly, the importance of random initializations has been examined in depth (Sutskever et al., 2013).
In our case, we use random projections for higher-rank feature expansion of low-rank dense pre-trained word embeddings, exploiting Cover’s theorem (Cover, 1965).
An encoder like this does not require any training, unlike other sentence encoders such as SkipThought and InferSent.
Comparing those methods to our random sentence encoders provides valuable insight into how much of a performance improvement we have actually gained from training for a long time (in the case of SkipThought) or training on expensive annotated data (in the case of InferSent which is trained on the Stanford Natural Language Inference (SNLI) Corpus Bowman et al. (2015), a large textual entailment dataset.).
The same idea of using ﬁxed random computations underlies reservoir computing (Lukoˇseviˇcius &
Jaeger, 2009) and echo-state networks (Jaeger, 2001).
In reservoir computing, inputs are fed into a ﬁxed, random, dynamical system called a reservoir that maps the input into a high dimensional space.
Then a trainable linear transformation of this high dimensional space is learned to predict some output signal.
Echo-state networks are a speciﬁc type of reservoir computing and are further described in Section 3.1.3.
Reservoir computing has been used sparingly in Natural Language Processing (NLP).
In Daubigney et al.
(2013), Tong et al.
(2007), Frank (2006a), Frank (2006b), Hinaut & Dominey (2012)
A PPROACH
In this paper, we explore three architectures that produce sentence embeddings from pre-trained word embeddings, without requiring any training of the encoder itself.
These sentence embeddings are then used as features for a collection of downstream tasks.
The downstream tasks are all trained with a logistic regression classiﬁer using the default settings of the SentEval framework (Conneau & Kiela, 2018).
The parameters of this classiﬁer are the only ones that are updated during training (see Section 3.2 below).
3. R ANDOM S ENTENCE E NCODERS
We are concerned with obtaining a good sentence representation h that is computed using some function f
parameterized by θ over pre-trained input word embeddings e ∈ L , i.e. h = f θ ( e 1 , . .
e n ) where e i is the embedding for the i -th word in a sentence of length n .
Typically, sentence encoders learn θ , after which it is kept ﬁxed for the transfer tasks.
InferSent represents a sentence as f = max(BiLSTM( e 1 , . .
e n )) and optimizes the parameters using a supervised cross-entropy objective for predicting one of three labels from a combination of two sentence representations: entailment, neutral or contradictive.
SkipThought represents a sentence as f = GRU n ( e 1 , . .
e n ) , with the objective of being able to decode the previous and next utterance using negative log-likelihood from the ﬁnal (i.e., n -th) hidden state.
InferSent requires large amounts of expensive annotation, while SkipThought takes a very long time to train.
Here, we examine different ways of parameterizing f for representing the sentence meaning, without any training of θ .
This means we do not require any labels for supervised training, nor do we need to train the sentence encoder for a long time with an unsupervised objective.
We experiment with three methods for computing h :
Bag of random embedding projections, Random
LSTMs, and Echo State Networks.
In this section, we describe the methods in more detail.
In the following sections, we show that our methods lead to surprisingly good results, shedding new light on sentence representations, and establishing strong baselines for future work.
3.1. B AG OF RANDOM EMBEDDING PROJECTIONS (BOREP)
The ﬁrst family of architectures we explore consists of simply applying a single random projection in a standard bag-of-words (or more accurately, bag-of-embeddings) model.
We randomly initialize a matrix W ∈ R D × d , where D is the dimension of the projection and d is the dimension of our input word embedding.
The values for the matrix are sampled uniformly from [ − 1 √ d , 1 √ d ] , which is a standard initialization heuristic used in neural networks (Glorot & Bengio, 2010).
The sentence representation is then obtained as follows:
[Equation]
where f pool is some pooling function, e.g. f pool ( x ) =
(cid:80) ( x ) , f pool ( x ) = max( x ) (max pooling) or f pool ( x ) =
| x | − 1 (cid:80) ( x ) (mean pooling).
Optionally, we impose a nonlinearity max (0 , h ) .
We experimented with imposing positional encoding for the word embeddings, but did not ﬁnd this to help.
3.1. R ANDOM LSTM S
Following InferSent, we employ bidirectional LSTMs, but in our case without any training.
Conneau et al.
(2017) reported good performance for the random LSTM model on the transfer tasks.
The
LSTM weight matrices and their corresponding biases are initialized uniformly at random from [ − 1 √ d , 1 √ d ] , where d is the hidden size of the LSTM.
In other words, the architecture here is the same as that of InferSent modulo the type of pooling used:
[Equation]
3.1. E CHO S TATE N ETWORKS
Echo State Networks (ESNs) (Jaeger, 2001) were primarily designed for sequence prediction problems, where given a sequence X , we predict a label y for each step in the sequence.
The goal is to minimize the error between the predicted ˆ y and the target y at each timestep.
Formally, an ESN is described using the following update equations:
[Equation]
where W i , W h , and b i are randomly initialized and are not updated during training.
The parameter α ∈ (0 , 1] governs the extent to which the previous state representation is allowed to leak into the current state.
The only learned parameters in an ESN are the ﬁnal weight matrix, W o and corresponding bias b o , which are together used to compute a prediction ˆ y i for the i th label
y
i :
[Equation]
We diverge from the typical per-timestep ESN setting, and instead use the ESN to produce a random representation of a sentence.
We use a bidirectional ESN, where the reservoir states, h i , are concatenated for both directions.
We then pool over these states to obtain the sentence representation:
[Equation]
The property of echo state networks that sets them apart from randomly initialized classical recurrent networks, and allows for better performance, is the echo state property .
The echo state property (Jaeger, 2001) claims that the state of the reservoir should be determined uniquely from the input history, and the effects of a given state asymptotically diminish in favor of more recent states.
In practice, one can satisfy the echo state property in most cases by ensuring that the spectral radius of W h is less than 1 (Lukoˇseviˇcius & Jaeger, 2009).
The spectral radius, i.e., the maximal absolute eigenvalue of W h , is one of many hyperparameters to be tuned when using ESNs.
Others include the [Table] activation function, the amount of leaking between states, the sparsity of W h , whether to concatenate the inputs to the reservoir states, how to sample the values for W i and other factors.
Luko ˇ sevi ˇ cius & Jaeger (2009) gives a good overview of what hyperparameters are most critical when designing
ESNs.
3. E VALUATION
In our experiments, we evaluate on a standard sentence representation benchmark using SentEval (Conneau & Kiela, 2018).
SentEval allows for evaluation on both downstream NLP datasets as well as probing tasks, which measure how accurately a representation can predict linguistic information about a given sentence.
The set of downstream tasks we use for evaluation comprises sentiment analysis (MR, SST), question-type (TREC), product reviews (CR), subjectivity (SUBJ), opinion polarity (MPQA), paraphrasing (MRPC), entailment (SICK-E, SNLI) and semantic relatedness (SICK-R, STSB).
The probing tasks consist of those in Conneau et al.
We use the default SentEval settings (see Appendix A).
R ESULTS
We compare primarily to two well-studied sentence embedding models, InferSent (Conneau et al., 2017) and SkipThought (Kiros et al., 2015) with layer normalization (Ba et al., 2016).
We point out that there are recently introduced multi-task sentence encoders that improve performance further, but these either do not use pre-trained word embeddings (GenSen (Subramanian et al., 2018)), or don’t use SentEval (Universal Sentence Encoders (Cer et al., 2018)).
Since both architectures are inspired by InferSent and SkipThought, and combine their respective supervised and unsupervised objectives, we limit our comparison to the original models.
We compute the average accuracy/Pearson’s r , along with the standard deviation, over 5 different seeds for the random methods, and tune on validation for each task.
See Appendix A for a discussion of the used hyperparameters.
Table 1 reports the results on the selected SentEval benchmark tasks, where all models havedimensions (with the exception of SkipThought, which has 4800 ).
We compare to three different
InferSent models: the results from the paper, which had non-standard pooling over padding symbols (InferSent-1); the results from the InferSent GitHub, 2 with ﬁxed padding, using FastText instead of
GloVe (InferSent-2); the results from an InferSent model we trained ourselves, 3 with ﬁxed padding, [Table] using GloVe embeddings (InferSent-3) (see Appendix C for a more detailed discussion of padding and pooling).
4 Note that the comparison to layer-normalized SkipThought is not entirely fair, because it uses different (and older) word embeddings, but a higher dimensionality.
We hypothesize that SkipThought might do a lot better if it had been trained with better pre-trained word embeddings.
First of all, we observe that all random sentence encoders generally improve over bag-ofembeddings.
This is not entirely surprising, but it is important to note that the proper baseline for an n -dimensional sentence encoder is an n -dimensional BOREP representation, not an ( m < n ) - dimensional BOE representation.
BOREP does markedly better than BOE, constituting a much stronger baseline (and requiring no additional computation besides a simple random projection).
When comparing the random sentence encoders, we observe that ESNs outperform BOREP and
RandLSTM on all tasks.
It is unclear whether (randomly initialized) LSTMs exhibit the echo state property, but the main reason for the improvement is likely that in our experiments ESNs had more hyperparameters available for tuning.
When comparing to InferSent, in which case we should look at InferSent-3 in particular (as it has ﬁxed padding and also uses GloVe embeddings), we do see a clear difference on some of the tasks, showing that training does in fact help.
The performance gains over the random methods, however, are not as big as we might have hoped, given that InferSent requires annotated data and takes time to train, while the random sentence encoders can be applied immediately.
For SkipThought, we discern a similar pattern, where the gain over random methods (which do have better word embeddings) is even smaller.
While SkipThought took a very long time to train, in the case of SICK-E you would actually even be better off simply using BOREP, while ESN outperforms SkipThought on ﬁve of the 10 tasks.
Note that in these experiments we do model selection over per-task validation set performance, but
shows that the method is robust, as we could also have used the best-overall model on validation and obtained similar results.
Keep in mind that the point of these results is not that random methods are better than these other encoders, but rather that we can get very close and sometimes even outperform those methods without any training at all, from just using the pre-trained word embeddings.
4. T AKING C OVER TO THE M AX
If we take Cover’s theorem to the limit, we can project to an even higher-dimensional representation as long as we can still easily ﬁt things onto a modern GPU: hence, we project to 4096 ×( 24576 ) dimensions instead of the 4096 dimensions we used in Table 1.
In order to make for a fair comparison, we can also randomly project InferSent and SkipThought representations to the same dimensionality and examine performance.
[Table] Table 2 shows the results.
Interestingly, the gap seems to get smaller, and the projection in fact appears to be detrimental to InferSent and SkipThought performance.
The numbers reported in the table are competitive with (older) much more sophisticated trained methods.
Simply maximizing the number of dimensions, however, might lead to overﬁtting, so we also analyze how performance changes as a function of the dimensionality of the sentence embeddings: we sample random models for a range of dimensions, { 512, 1024, 2048, 4096, 8192, 12288, 24576 } , and train models for BOREP, random LSTMs, and ESNs.
Performance of these models is shown in
Figure 1.
As suggested by Cover’s theorem, as well as earlier ﬁndings in the sentence embedding literature (see e.g. Fig.
5 of Conneau et al.
, we observe that higher dimensionality in most cases leads to better performance.
In some cases it looks like we would have beneﬁted from having even higher dimensionality (e.g. SUBJ, TREC and SST2), while in other cases we can see that the model probably starts to overﬁt (STSB, SICK-E for BOREP).
In general, the trend is up, meaning that a higher-dimensional embeddings leads to better performance.
A NALYSIS
We analyze random sentence embeddings by examining how these embeddings perform on the probing tasks introduced by Conneau et al.
(2018), in order to gauge what properties of sentences they are able to recover.
These probing tasks were introduced in order to provide a framework for as- certaining the linguistic properties of sentence embeddings, comprising three types of information: surface, syntactic and semantic information.
There are two surface information tasks: predicting the correct length bin from 6 equal-width bins sorted by sentence length (Length) and predicting which word is present in the given sentence from a set of 1000 mid-frequency words (Word Content, WC).
Syntactic information comprises 3 tasks: predicting whether a sentence has been perturbed by switching two adjacent words (BShift); the depth of the constituent parse tree of the sentence (TreeDepth); and the topmost constituent sequence of the constituent parse in a 20-way classiﬁcation problem (TopConst).
Finally, there are ﬁve semantic information tasks: predicting the tense of the main-clause verb (Tense); the number of the subject of the main clause (SubjNum); the number of the direct object of the main clause (ObjNum); whether a sentence has been modiﬁed by replacing a noun or verb with another in a way that the newly formed bigrams have similar frequencies to those they replaced (Semantic Odd Man
Out, SOMO); and whether the order of two coordinate clauses has been switched (CoordInv).
Table 3 shows the performance of the random sentence encoders (using the best-overall model tuned on the classiﬁcation validation sets of the SentEval tasks) on these probing tasks along with bag-ofembeddings (BOE), SkipThought-LN, and InferSent.
From the table, we see that ESNs and RandLSTMs outperform BOE and BOREP on most of the tasks, especially those that require knowledge of the order of the words.
This indicates that these models, even though initialized randomly, are capturing order, as one would expect.
We also see that ESNs and InferSent are fairly close on many of the tasks, with Skipthought-LN generally outperforming both.
It seems that random models do best when the tasks largely require picking up on certain words.
We can see which tasks these are by looking at how well BOREP does compared to the recurrent models (WC, Tense, SubjNum, ObjNum are good candidates for this type of task).
In these, random models are all very competitive to the trained encoders. If one looks at the tasks where there is the largest difference between ESN and the best of IS or ST-LN (SOMO, CoordInv, BShift, TopConst) it seems that one thing these all have in common is that they do require sequential knowledge. We say this because both the BOREP baseline lags behind the recurrent models signiﬁcantly for many of these (and is often at the majority-vote baseline) and also the very deﬁnitions of these tasks.
This also makes intuitive sense as well since this is the type of knowledge that is much harder to learn and is not provided by stand-alone word embeddings. Therefore, we’d expect the trained models to have an edge here, which seems to bear out in these experiments.
D ISCUSSION
In light of our ﬁndings, we list several take-away messages with regard to sentence embeddings:
• If you need a baseline for your sentence encoder, don’t just use BOE, use BOREP of the same dimension, and/or a randomly initialized version of your encoder.
• If you are pressed for time and have a small to mid-size dataset, simply randomly project to a very high dimensionality, and proﬁt.
• More dimensions in the encoder is usually better (up to a point).
• If you want to show that your system is better than another system, use the same classiﬁer on top with the same hyperparameters; and use the same word embeddings at the bottom; while having the same sentence embedding dimensionality.
• Be careful with padding, pooling and sorting: you may inadvertently end up favoring certain methods on some tasks, making it harder to identify sources of improvement.
As Rahimi and Recht wrote when reﬂecting on their random kitchen sinks paper 5 :
Its such an easy thing to try.
When they work and I’m feeling good about life, I say “wow, random features are so powerful!
They solved this problem!”
Or if I’m in a more somber mood
Indeed, random sentence encoders are easy to try: they require no training, and should be used as a solid baseline to be compared against when learning sentence encoders that are supposed to capture more than simply what is encoded in the pre-trained word embeddings.
While sentence embeddings constitute a very promising research direction, much of their power appears to come from pre-trained word embeddings, which even random methods can exploit.
The probing analysis revealed that the trained systems are in fact better at some more intricate semantic probing tasks, aspects of which are however apparently not well-reﬂected in the downstream evaluation tasks.
C ONCLUSION
In this work we have sought to put sentence embeddings on more solid footing by examining how much trained sentence encoders improve over random sentence encoders.
As it turns out, differences exist, but are smaller than we would have hoped: in comparison to sentence encoders such as
SkipThought (which was trained for a very long time) and InferSent (which requires large amounts of annotated data), performance improvements are less than 2 points on average over the 10 SentEval tasks.
Therefore one may wonder to what extent sentence encoders are worth the attention they’re receiving.
Hope remains, however, if we as a community start focusing on more sophisticated tasks that require more sophisticated learned representations that cannot merely rely on having good pre-trained word embeddings.
A H YPERPARAMETERS
For all experiments, we attempt to keep the number of tunable hyperparameters to a minimum.
By being judicious with the number of tuning experiments and averaging over different seeds, we provide strong evidence that these architectures are robust and can be competitive with trained (nonrandom) sentence embedding models.
In all experiments, we tune the type of pooling to use.
Different tasks beneﬁt from different types of pooling, and while many pooling mechanisms have been proposed in the literature, we just tune over the most commonly used ones: mean pooling and max pooling.
We use the publicly available 300-dimensional GloVe embeddings (Pennington et al., 2014) trained on Common Crawl for all experiments.
All words that are not in the vocabulary for GloVe are assigned a vector of zeros.
For the ESNs, we only tune whether to use a ReLU or no activation function, 6 the spectral radius from { 0 . 4 , 0 . 6 , 0 . 8 , 1 . 0 } , the range of the uniform distribution for initializing W i where the max distance from zero is selected from { 0 . 01 , 0 . 05 , 0 . 1 , 0 . 2 } , and ﬁnally the fraction of elements in W h that are set to 0, i.e., sparsity, is selected from { 0 , 0 . 25 , 0 . 5 , 0 .
Furthermore, our model did not include a bias term b i .
We chose not to experiment with other possibilities that ESNs provide that could further enhance performance like leaking or concatenating/adding the input embedding to the reservoir state in favor of a simpler model.
We use the default SentEval settings, which are to train with a logistic regression classiﬁer, use a batch size of 64, a maximum number of epochs of 200 with early stopping, 7 no dropout [Table] and use
Adam (Kingma & Ba, 2014) for optimization with a learning rate of 0 .
B T ESTING R OBUSTNESS
[Table]
In order to examine the stability of the random sentence encoders
[Table]
we select the two best overall models by best validation score—one that achieved the highest accuracy score
[Table]
and one that achieved the highest correlation score (as these differed signiﬁcantly)—and examine the results.
The performance of these models is shown in Table 4.
We observe that performance is very stable
[Table]
and that task-speciﬁc tuning yields little or no beneﬁt over the best-overall model, which is beneﬁcial: the good results obtained by sentence encoders are not some
, and the ﬁnding
C P OOLING AND P ADDING
[Table]
is done over padded values instead of over the length of the sentence.
Experiments are split between
Sorted where sentences are sorted in order of length prior to batching and Unsorted where they are not.
We further analyzed how max-pooling over padding affects downstream evaluations and noticed that for this effect to occur, the batch size to produce the embeddings and the order in which sentenced were embedded needed to be a speciﬁc way.
The order in which sentences are embedded in SentEval is not random, as sentences are sorted by length prior to being grouped into batches.
We noticed that upsetting this order, or changing the batch size so that sentences are grouped differently, causes a signiﬁcant change on the downstream performance.
In Table 5, we reproduce this effect for RandLSTM (averaged over 5 seeds) and also include results for InferSent and GenSen using their released code.
The ﬁrst half of the table shows results when max pooling with padding is used and the batches are sorted.
The second half of the table shows performance when the batches are unsorted.
As can be seen by the table, the performance has a signiﬁcant drop-off when the batches are unsorted, especially for MR and CR.
Max pooling over padded values changes negative values in the features of longer sentences to zero.
This is because if the largest value in the hidden representations over the length of the sentence is negative, the padded zeros will be greater.
Thus, longer sentences, when grouped with shorter ones, will have more sparse representations.
We tried to reproduce this effect by using a ReLU, but it didn’t increase performance.
We also checked to see if length was strongly correlated with either class for the problems in Table 5, but found the correlation was low for all binary tasks.
In fact it is 0.0 for CR, one of the tasks most affected by this phenomenon.
It turns out that in several of the datasets, those examples obtaining the sparse representations (becoming sparsed ) occur much more often in certain classes than others.
Table 6 analyzes the distribution of sparsed examples in MR, CR, MPQA, SUBJ, and SST2.
Since MR, CR, MPQA, and SUBJ have no deﬁned training/validation/testing split, the ﬁrst row of the table shows the percentage of the data that becomes sparsed when the sentences are embedded.
The second row is what percentage of those sparsed embeddings are of the positive class.
SST2 is split into a training/validation/test set.
In it, only 3.5% of the training data is sparsed.
However, 94% of the sparsed data has a class 1 label.
In the testing data, 75% of the data is sparsed with 61% of the data having class 1.
Overall, since little training data is actually sparsed, and the class imbalance of the testing data isn’t as skewed as in the other datasets, SST2 is less affected by these sparsed representations as can be seen in the table.
[Table]
[Table]
[Table]
D E XPLORING D IFFERENT I NITIALIZATION S TRATEGIES
We analyzed the effects of different initialization strategies had on performance for BOREP and
RandLSTM.
We experimented with 6 different initialization strategies: 1) Heuristic, which is the approach used in this paper for BOREP and RandLSTM experiments unless otherwise noted, where elements are sampled uniformly from [ − 1 √ d , 1 √ d ] 2) Uniform where parameters are sampled from [ − 0 .
1] , 3) Normal where parameters are sampled from a normal distribution with 0 mean and a standard deviation of 1, 4) Orthogonal where elements are sampled from the heuristic and then the matrices of the parameters are made orthogonal 5)
He initialization (He et al., 2015) and 6) Xavier initialization (Glorot & Bengio, 2010).
We analyzed the effect different initialization strategies had on performance for BOREP and RandLSTM.
BROEP is robust to initialization strategy, with a slight edge for Orthogonal and Xavier initialization.
However, RandLSTM does poorly with Normal initialization and to a lesser degree,
Uniform.
He and Xavier initialization seem to give the best average performance.
[ What else can we say here?
Any reasons for these observations you can think of?
–JW]
E E XPLORING R ANDOM E NCODERS WITH R ANDOM W ORD E MBEDDINGS
We next explore the contributions of pre-trained word embeddings for the BOREP and RandLSTM models.
In these experiments, both the parameters of the architectures and the word embeddings are randomly sampled.
Just as in Section D, we experiment with six ways of initializing the random embeddings for BOE, BOREP, RandLSTM, and a 4096 dimension BOE model.
The parameters of the architectures (if applicable) are sampled in the same way as the random embeddings.
The results are shown in Table 8.
[Table]
[Table]
7 points.
From the table, we see knowledge from the pre-trained word embeddings offers signiﬁcant improvement for most tasks, averaging 5.7 points per task.
It seems to be especially helpful for classiﬁcation, and less-so for those tasks relying on a measure of semantic similarity.
This is sensible because semantic similarity tasks can make use of embeddings appearing in the test data that were not in the training data if the words appear in both sentences in the example.
However, this is not the case for classiﬁcation as an unseen random word embeddings provide very little information.
Interestingly, pooling randomly initialized 4096 dimension embeddings outperforms BOREP and
RandLSTM.
It would be interesting to see the performance of this model when the embeddings are pre-trained instead of random.
Perhaps pooling them would be competitive with trained encoders like Infersent.
It would also be interesting to see what effect large-dimension word embeddings have on both a trained sentence encoder like Infersent as well as random recurrent architectures.
We leave this exploration for future work