AriEL: volume coding for sentence generation
Anonymous authors
Paper under double-blind review
Abstract
Saving sequences of data to a point in a continuous space makes it diﬃcult to retrieve them via random sampling.
Mapping the input to a volume makes it easier, which is the strategy followed by Variational Autoencoders.
However optimizing for prediction and for smoothness, forces them to trade-oﬀ between the two.
We analyze the ability of standard deep learning techniques to generate sentences through representation sampling.
We propose
AriEL, an entropic coding method to construct volumes without the need for extra loss terms, and compare those standard learning techniques with its use of the latent space.
We benchmark on a toy grammar, to automatically evaluate the language learned and generated, and ﬁnd where it is stored in the latent space.
Then, we benchmark on a dataset of human dialogues and using GPT-2 inside AriEL.
Our results indicate that the random access to stored information can be improved, since AriEL is able to generate a wider variety of correct language by randomly sampling the latent space.
This supports the hypothesis that encoding information into volumes, improves retrieval of learned information with random sampling.
Introduction
Representation regularization, through the normalization and bounding of data, representations and gradients, is fundamental to fast deep learning training (Ioﬀe and Szegedy, 2015; Kingma and Welling, 2014; He et al., 2015; Perez et al., 2018).
However, it seldom oﬀers guarantees for boundedness, only encouraging it through initial conditions and loss summands. The ﬁnal conditions, i e. the representations learned, can be empirically explored through the sampling of the latent space. This exercise reveals how often the data not seen during training has no bounded representation, which can be regarded as undesirable if we want architectures that can quickly generalize outside the training bias for successful transfer learning.
However it is diﬃcult to ﬁnd the learned patterns through latent sampling, since typically neural networks map an input to a point in R d (Hochreiter and Schmidhuber, 1997; Vaswani et al., 2017; LeCun et al., 1989).
Some models do map inputs to volumes, to ease retrieval through random sampling. Variational Autoencoders (Kingma and Welling, 2014; Bowman et al., 2016; Chen et al., 2018) encourage volume representations: by encoding an input into a probability distribution that is sampled before decoding, neighbouring points in R d can end up representing one input.
However, it requires two loss summands, a log-prior and a log-likelihood, that ﬁght for two diﬀerent causes.
A smooth and volumetric representation, encouraged by the log-prior regularization, can worsen performance, encouraged by the log-likelihood.
By giving partially up on smoothness, we propose AriEL, a method to construct volumes, without a loss to encourage them.
It maps sentences to volumes in R d for eﬃcient retrieval with random sampling, or a network that operates in its continuous space.
It fuses arithmetic coding (AC)
(Elias and Abramson, 1963) and k-d trees (KdT)
(Bentley, 1975), so we name it
Arithmetic coding and k-d trEes for Language (AriEL).
For simplicity we focus on dialogue language, even if AriEL works with any variable length sequence of symbols.
AriEL can be used as a benchmark to understand natural language processing and generation models use of latent space.
It ﬁlls completely the latent space with the language learned, using information theory, and bounding its representations within [0 , 1] d .
Its language model splits the latent space in volumes, guided by the probability assigned to the next symbol in a sentence.
It can provide an agent with a simpler interface with a pretrained language model, e.g. a GPT-2 (Radford et al., 2019; Wolf et al., 2020), where the agent could choose the optimal d .
We prove how such a volume representation eases the retrieval of stored learned patterns and how to use it to set references for other models.
Our contributions are therefore:
• AriEL, a novel unsupervised volume representation based on arithmetic coding and k-d trees (Section 3.1), to retrieve learned patterns with random sampling; • the HouseQ dataset, consisting of a large context-free grammar and a random bias (Section 3.3), to automatically generate and evaluate sentences generated by trained models, and ﬁnd them in their latent space; • the notion that explicit volume coding (Section 4) can be a useful technique in tasks that involve the generation of sequences of discrete symbols, such as sentences; • the observation that conventional learned codes like AE, VAE or Transformer, do not use the latent space eﬀectively (Section 4), in the AriEL entropic
coding sense.
Related Work
Volume codes: We deﬁne a volume code as two functions, an encoder and a decoder functions, where the encoder maps an input x into a set that contains compact and connected sets of R d (Munkres, 2018), and the decoder maps every point within that set back to x .
It is a distributed representation (Hinton et al., 1984) since the input x is represented by at least one R d point.
We call the volume code implicit , when the volumes are encouraged through a loss term (Bengio et al., 2013; Ng and Jordan, 2002; Kingma and Welling, 2014; Jebara, 2012) and explicit , when the volumes are constructed through the model’s operations, independently from any loss and optimizer choice.
Sentence generation through random sampling: Generative Adversarial Networks (GAN) (Goodfellow et al., 2014) map random samples to a learned generation through a 2-players game procedure.
Yu et al. (2017); Kusner and Hern´andez-Lobato (2016) ; Scialom et al. (2020) signiﬁcantly improved GAN performance in text generation.
Random sampling the latent space is used as well by Variational Autoencoders (VAE) (Kingma and Welling, 2014), to smooth their representations.
Bowman et al. ; Yang et al.
; Li et al. (2021), and others, have reﬁned their performance for text representation.
AriEL can be used as a generator or a discriminator in a GAN, or as an encoder or a decoder in an autoencoder.
However it diﬀers from them in the explicit procedure to construct volumes.
It ﬁlls the entire latent space with the learned patterns, to ease retrieval by uniform sampling.
Arithmetic coding and neural networks: AC is one of the most eﬃcient lossless data compression techniques (Elias and Abramson, 1963; Witten et al., 1987).
AC assigns a sequence to a segment in [0 , 1] with length proportional to its frequency. When converted into bits, frequent symbols take less bits than unfrequent.
AC is used for neural network compression (Wiedemann et al., 2019) and neural networks are used in AC to perform prediction based compression (Jiang et al., 1993; Pasero and Montuori, 2003; Tatwawadi, 2018).
We generalize AC to R d , to combine its properties with the properties of highdimensional spaces, neural networks domain.
K-d trees and neural networks: KdT (Bentley, 1975) is a data structure for storage that can handle diﬀerent types of queries eﬃciently.
It is typically used as a fast approximation to k-nearest neighbours in low dimensions (Friedman et al., 1977).
It gives a binary label to the data with respect to its median.
It moves through the k dimensions of the data and repeats the process.
Neural networks are typically used in conjuction with KdT to reduce the dimensionality of the search space, for KdT to be able to perform queries eﬃciently (Woodbridge et al., 2018;
Yin et al
Methodology 3. AriEL: volume coding of language in continuous spaces
ac
S a | b | aa | ab | ac | bc | abc | bcc aa abc ab b bc bcc a    1 b bc a aa ab ac
Arithmetic Coding
AriEL   b a
[Figure]
d [Figure]
1 b bc bcc a aa ab ac abc
[Figure]
generating context grammar CFG is S → a | b | aa | ab | ac | bc | abc | bcc plot top indicates the frequency of those sentences in the dataset as an extra language of this a dimension within [ 1], and the frequency of the sentence determines the length assigned on AriEL mensional
2D) frequency
The Language Model provides the boundaries where the next symbols are to be found. For a 2D latent space, d = 2, the axis to split to ﬁnd symbol s t is d t = t mod d . d t = 0 ,represent the horizontal and vertical axis.
AriEL maps the sequence/sentence ( s 1 , · · · , s n ) = ( s t ) nt =1 of length n , to a d -dimensional volume of size P (( s 1 , · · · , s n ))
=
Π nt =1 P ( s t | ( s t (cid:48) ) t (cid:48) <t ) in the [0 , 1] d hypercube.
When the symbol is used as a random variable we refer to it as s, s (cid:48) , while s t represents the observed sample at time step t .
The words belong to a ﬁnite vocabulary s ∈ { 1 , · · · , V size } , V size ∈ N .
To adapt KdT to more splits than binary, we split axis d t = t mod d , into V size segments, one for each possible s t . The segment has length proportional to s t probability.
Then we turn to the following axis d t +1 , and continue the process of splitting and turning (ﬁgure 1 and algorithm 1). In ﬁgure 1, s t ∈ { a, b, c, } . The initial token s 1 =
a is given a portion on d 1 of length P ( a ), larger than the portion given to s 1 = b or s 1 = c , since there are less sentences that start with b than with a , and there is none that starts with c : P ( a ) > P ( b ) > P ( c ) = 0. Then, we split d 2 according to the probability of the {+next symbol s 2 . In this case the second most likely symbol after symbol s 1 = a is s 
+}
= c , so ac ends with a larger volume than aa , ab and a .
For sentences longer than d , next symbol is assigned an axis d 3 previously split, but only the volume selected up to t − 1 is further split.
So, the sentence abc takes a portion of ab equal to P ( c | ( ab )), while ‘ ab ’ takes a portion equal to P ( | ( ab )). We estimate language statistics with a Language Model (LM), P LM ( s t | ( s t (cid:48) ) t (cid:48) <t ).
This will approximate the frequency information that makes AC entropically eﬃcient.
The sentence is ﬁnally encoded as the center of the volume bounded by those segments for simplicity, and any point within it is decoded to the same sentence.
The extension to a larger [ a, b ] d hypercube is straightforward, and could provide higher precision, but we restrict ourselves to [0 , 1] d .
AriEL has a computational complexity of O ( nD 2 V size ) for encoding and decoding (algorithm 1), where n is the length of the sequence, D is the dimensionality of the LM latent space, and V size the vocabulary size.
AriEL has a minimum number of sequential operations of O ( n ) for both encoding and decoding, on par with conventional seq2seq recurrent networks.
3. Neural Networks: models and experimental conditions
We compare AriEL to standard approaches to map variable length discrete spaces to ﬁxed length continuous spaces.
These are the sequence to sequence recurrent autoencoders (AE) (Sutskever et al., 2014), their variational version (VAE) (Bowman et al., 2016) and Tran-
AriEL Encoding
Input: sentence: S =
( s t ) nt =
Output: z represents S in [0 , 1] d 1: function AriEL encode ( S ) 2: d = latent space dimension 3:
B low =
zeros ( d ) 4:
B up = ones ( d ) 5: n = length ( S ) 6: for t = 0 , · · · , n − 1 do (cid:46) choose dimension to split 7: d t = t mod d 8:
P next ( s ) =
P LM ( s | ( s t (cid:48) ) t (cid:48) <t ) 9: c low ( s ) =
(cid:80) s
>s (cid:48)
P next ( s (cid:48) )
10: c up ( s ) =
(cid:80) s
>s (cid:48) − 1 P next ( s (cid:48) )
11: a = B up ( d t ) − B low ( d t ) (cid:46) update volume bounds 12:
B up ( d t ) =
B low ( d t ) + a · c up ( s t ) 13:
B low ( d t ) =
B low ( d t )+ a · c low ( s t ) 14: end for (cid:46) represent the volume by its center 15:
z =
( B low + B up ) /
: return z 17: end function AriEL Decoding
Input: z represents S in [0 , 1] d
Output: sentence: S =
( s t ) nt = 1: function AriEL decode ( z ) 2: d = dimension ( z ) 3:
B low =
zeros ( d ) 4:
B up = ones ( d ) 5:
S = (cid:104) START (cid:105) 6: for t = 0 , · · · , n max − 1 do (cid:46) choose dimension to unsplit 7: d t = t mod d 8:
P next ( s ) =
P LM ( s | S ) 9: c low ( s ) =
(cid:80) s
>s (cid:48)
P next ( s (cid:48) )
10: c up ( s ) =
(cid:80) s
>s (cid:48) − 1 P next ( s (cid:48) )
11: a = B up ( d t ) − B low ( d t ) (cid:46) update volume bounds 12:
Bs up ( s )
=
B up ( d i ) + a · c up ( s ) 13:
Bs low ( s ) =
B low ( d t )+ a · c low ( s ) (cid:46) any point in the volume is assigned the symbol s t 14: s t = find s (cid:16) Bs low ( s ) <
z ( d t ) <
Bs up ( s ) (cid:17) 15:
B up ( d t ) =
Bs up ( s t ) 16:
B low ( d t ) =
Bs low ( s t ) 17:
S = S.append ( s t ) 18: end for 19: return S 20: end function former (Vaswani et al., 2017).
We trained them for next word prediction, over the biased train set (section 3.3).
All can be split into an encoder and a decoder that map the sentences to R d and back.
More training details in SM7.
In this work, AriEL’s language model neural network P LM consists of a word embedding of size 64, a 140-unit LSTM, a feedforward layer and a softmax.
At test time the argmax is not applied directly to the softmax, but the latent space is used as the deterministic pointer that chooses the s t , as shown in find deﬁnition in SM10.
However the LM is trained for next time step prediction through cross-entropy.
For AE and VAE, we stack two GRU layers (Cho et al., 2014) with 128 units at both, the encoder and the decoder.
The last encoder layer has either d = 16 units or d = 512 for all methods.
The decoder outputs a softmax.
Transformer (Vaswani et al., 2017) is the state-of-the-art in many S2S problems (Dai et al., 2019;
Radford et al., 2018).
Since at the word level it is a ﬁxed-length representation and it is variable-length at the sentence level, we padded all sentences to the maximum length in the dataset to be able to compare its latent space capacity to the other models.
We take as its latent dimension the connection between the encoder and decoder, with d model size, that will take a value of 16 or 512.
We choose most parameters as in the original work: the number of attention heads n head = 8, the key and value dimension d key = d value = 64, a dropout regularization of 0.1.
We change the stack of identical decoders and encoders to n layers = 2, and the dimension of the inner feed-forward network to d ﬀ = 256 to have a number of parameters similar to the other methods.
On the GuessWhat?!
dataset (De Vries et al., 2017) we tested n layers = 20 to have an amount of parameters comparable for d =to the other methods, but performed worse than n layers = 2, so we report the smaller one. All architectures are tested either for next time-step prediction or sampled from the latent space, and all of them are constrained to a small latent space. All of them are 2-layers of encoder and decoder, except for AriEL, since they improved performance compared to when they had only one layer. All of them have about the same number of parameters.
All of them could be used as Language Models inside AriEL.
3. Datasets: toy and human sentences
We perform our analysis on two datasets.
A toy dataset of sentences generated from a context-free grammar (CFG) and a realistic dataset of sentences written by humans playing a cooperative game.
The toy dataset: we generate questions about objects with a CFG (Supplementary Material 1).
To stress the models we choose a CFG with a large vocabulary and numerous grammar rules, rather than classic alternatives (e.g. REBER, Hochreiter and Schmidhuber (1997)).
All are questions about house objects, so we call it the HouseQ dataset.
We distinguish between unbiased sentences, simply sampled from the CFG, and biased sentences, selected according to an additional structural constraint after being sampled from the CFG.
To do so we generate an adjacency matrix of words that can occur together in a sentence, and we use that to bias the sentences.
Once a sentence is produced from the CFG, if all its words can be together in a sentence judged by the adjacency matrix, the sentence is considered as biased, and unbiased otherwise.
For simplicity the adjacency matrix is a random matrix of zeros and ones, generated once for all the experiments, making sure that some symbols like the , it or ? , can be found in all sentences.
We want to emulate a CFG constrained by realistic scenes, where not all the grammatically correct sentences are semantically correct: e.g. ‘Is it the wooden shower in the kitchen ?’
could be grammatical, but semantically incorrect since it is unusual.
We use it to detect how each learning method extracts the grammar and the roles of each word, despite a bias that makes it harder.
It has a 840 words vocabulary with maximal and mean sentence length of 19 and 9.9 symbols.
We split the biased dataset into 1M train, 10k test and 512 validation sentences, with no overlap between them.
We created 10k unbiased test sentences with the same CFG, only with sentences that do not follow the adjacency matrix.
We train on the biased sentences and we test if they grasped the grammar behind, with the unbiased.
The real dataset: we choose the GuessWhat?!
dataset (De Vries et al., 2017), a dataset of questions asked by humans to solve a cooperative game.
It has a vocabulary of 10, words.
The maximal and mean length of the sentences are of 57 and 5.9 symbols.
3. Evaluation Metrics 3.4. Quantitative evaluations on the toy grammar, HouseQ
Metrics chosen take advantage of the possibilities opened by the CFG and the random bias, that allow to automatically check the grammatical and bias correctness of the language produced.
Standard metrics like perplexity, only check if the word predicted matches in the exact time step the word in a reference sentence.
In our view, it makes it an obscure metric for language quality, used in the lack of a better one for (non CFG) human language.
We measure generation, prediction and generalization quality.
We study networks with a latent dimension of 16 units, to understand their compression limits, and for 512 units, often taken as the default size (Kingma and Welling, 2014; Vaswani et al., 2017).
Generation/Decoding Quality is evaluated with sentences produced randomly sampling the latent space of each decoder.
The sampling is done uniformly within the maximal hyper- cube deﬁned by the encoded biased test sentences.
We sample 10k sentences and evaluate: i) vocabulary coverage (VC) as the ratio of sampled words, over the size of the complete vocabulary; ii) uniqueness (U) as a ratio of unique sampled sentences; and iii) validity (V) as a ratio of valid sampled sentences, deﬁned as the unique and grammatically correct sentences, the most important of our metrics.
Prediction Quality is evaluated by encoding and decoding the 10k biased test sentences as follows: i) prediction accuracy biased (PAB) as a ratio of correctly reconstructed sentences (i.e. all words must match); ii) grammar accuracy (GA) as a ratio of grammatically correct reconstructions (i.e. can be parsed by the CFG, even if the reconstruction is not accurate).
and iii) bias accuracy (BA) as the ratio of reconstructions that keep the training set bias.
Generalization Quality is evaluated using the 10k unbiased test sentences while the embeddings were trained on the biased training set.
The prediction accuracy unbiased (PAU) , as PAB , is the ratio of correctly reconstructed unbiased sentences.
It measures how well the latent space generalizes to grammatically correct sentences outside the training bias.
3.4. Quantitative evaluations on the real dataset, GuessWhat?!
Humans use spontaneously ungrammatical constructions.
We therefore quantify the quality of the language learned with two measures: memorization and subjective interpretability .
Memorization is the percentage of sentences that are unique and are found in the training data, indicating how easy it is to retrieve the learned information.
To measure subjective interpretability , we asked 5 surveyees to evaluate the interpretability of the generated sentences.
They were assigned 5 sentences for each method and d , with each sentence from a diﬀerent training seed.
Samples shown to the surveyees can be seen in SM12.
3.4. Interpolations within AriEL
AriEL organization of language in volumes changes with d , using the same Language Model.
In ﬁgure 2 we show how many valid sentences can be found on a straight line between two random points in [0 , 1] d , AriEL’s latent space.
Valid refers to unique and grammatically correct, as in the toy dataset case.
Since the average is taken over the segment and over the random pairs of points, we call this metric interpolation diversity .
For each d the average is taken over 15 diﬀerent random sentence pairs, and over 100 interpolation steps on the line that connects them.
The Language Model tested is the one trained on the toy grammar.
3.4. Qualitative evaluations
We show (1) samples of reconstruction via next word prediction of unbiased sentences, to understand the generalization capabilities of diﬀerent models ( d = 16, table 3 and S2), (2) generated sentences by uniformly sampling the latent space, after training on the toy dataset, to understand the generation capabilities ( d = 16, table 4 and S3) (3) samples generated by AriEL with a pretrained GPT2 as its language model, for small latent spaces ( d = 1 , 5, table 5 and S4) and long sentences (100 symbols), to see if the folding process and the ﬂoating point precision represent a limitation for AriEL.
To avoid cherry picking, we display the ﬁrst samples produced.
Results 4. Quantitative Evaluations
AriEL improves over the rest for all the 7 measures on the toy data (table 1 and ﬁgure S2), outperforming them by a large margin for validity, i.e. unique and grammatical sentences generated, the most important of the metrics.
Transformer performs remarkably well at not overﬁtting and it is able to reconstruct biased and unbiased sentences better than the other non-AriEL methods, even under-parameterized ( d = 16).
However, it performs very poorly at generating diverse valid sentences by random sampling.
VAE 16 despite the poor generalization to the biased and the unbiased test set, results in the best non-AriEL generator, measured by validity. The conﬂict between log-prior and log-likelihood, encouraged VAE to
[Table] [Figure] [Table]
What?!
Questioner data.
For the real dataset the pattern is repeated: AriEL shows that a larger value of valid sentences is possible. Transformer 16 gave better results when n layers = 2 than n layers = 20, that was tested to increase its learnable parameters from 588K to 2,666K. look for sentences outside the bias, since it was able to produce more grammatically correct sentences, albeit unbiased, than AE.
Increasing the learned parameters ( d = 512), had no eﬀect on Transformer, that was already excellent in several of the metrics, apart from a signiﬁcant improvement in validity.
However, a larger latent space and the increase in number of parameters that followed, prevented AE and VAE from overﬁtting (better PAU and PAB).
When trained on human sentences, on the GuessWhat?!
dataset, AriEL sets again a large memorization and interpretability margin compared to the other approaches.
Generated samples can be found in table S5.
In the interpolation diversity study (ﬁgure 2) we see that for low d , we have to pass through many sentences in between two random points in the latent space, while as we augment the dimensionality, we distribute the sentences in diﬀerent directions.
Therefore we ﬁnd less sentences when we move on a straight line between two random points.
[Table]
Blue means that the incorrect reconstruction complies with the bias and purple means that it is still unbiased.
Most reconstructions seem grammatically correct.
In practice AriEL also made errors.
Some failed reconstructions comply with the training bias, some do not.
Interestingly the errors made by Transformer tend to turn the unbiased input sentence into a biased version.
AE produced only biased sentences whose structure resembled the unbiased ones.
VAE behaved similarly, producing more unbiased sentences.
[Table]
Transformer re veals itself to be hard to control via random sampling of space since it almost never produces sentences method and VAE manage to produce several diﬀerent sentences, the latter producing more non grammatical, but as well more varied grammatical ones
4. Qualitative Evaluations
and S2 show the generalization study.
AE and VAE fail to generalize to the unbiased language, but both keep the structure of the input sentence.
Their behavior improved with d = 512, and its increase of parameters.
In theory, AriEL can reconstruct any sequence by design.
In practice, it failed slightly less often than Transformer.
Both produce reconstructions of the unbiased input at a similar rate (table 3, 1 and ﬁgure S2).
This means that codes for data not seen during training are available for AriEL and Transformer.
Instead, the latent space seems to be taken exclusively by the training set for AE and VAE, since sentences that are not seen during training (e.g. unbiased) cannot be reconstructed.
A very high number (97 . 6 ± 0 . 2%) of AriEL ’s generations are valid (table 4 and 1).
AE and VAE perform well despite the small latent space.
VAE triples AE in validity when d =(table 1).
Transformer struggles to generate grammatical sentences sampling the latent space.
However, increasing d , Transformer, AE and VAE improve in validity, remaining at one third AriEL’s reference.
Samples generated with GPT-2 as AriEL’s LM can be seen in table 5 and S4, for d = 1 , 5 , 10 , 50 , 100.
Despite the large vocabulary of GPT-2, 50K subwords, and the long sentences, 100 symbols, the ﬂoating point precision and the unit hypercube latent space of
AriEL, seem to still be able to generate high quality language.
Discussion
AriEL latent space d is a free parameter.
The size d of the latent space of AriEL can be deﬁned at any time, for a ﬁxed Language Model.
It could therefore be controlled with a learnable parameter or with the activity of another neuron.
As we increase d , the volumes will have more neighbouring volumes that represent diﬀerent sentences (ﬁgure 2).
d =
Where to go from here?
\ n \ n We’re looking for special interests and fans to pay much more attention to when they come in to another site.
If you’re looking for good stuﬀ to read on other sites, then get on your site and read their stories.
If you’re looking for things that work for your community, then look up what their leaderboard is and where they’re based.
If you’re looking for things that make me love them, then say hey.
You’re really doing your
Saul Niguez is not going to let go of his dream of playing for Hillary Clinton.
\ n \ n
The
Florida native will plan to accept the Democratic presidential nomination for president on November 8.
\ n \ n The 38-year-old machinist has called Trump \ ’s presidential nominee a ”post-modern bull - ” on social media.
\ n \ n In an interview with DailyMail.com, Niguez told her you could still name her after Trump.
\ n \ n ”This is my father \ ’s d =
Utility Shells are just as dependent on human labor as steel, yet they had been enormously more eﬃcient than nuclear power in incongruity’s late 70s.
US clients – including the nation’s largest retiree retirement fund – already manufactured many of their own carbonfree cars.
But what about non-node Americans?
How could they simply cool down at home for work for years at a time?
Inventing these subs would seem like a smart thing to do.
But there’s one
Out featured: on Alcoholism, Marriage and Dating: I’am No Timber [Decision note: 19th
August 2013] with Runnymede Colman and producer Dan Patrick \ n \ n Partial transcript as follows: \ n \ n DAVID WEISES:
After 20 years of marriage to my Malcastian partner, we’ve lost our lives to drugs.
And so it’s our hope that by allowing my Malcastian partner to become an alcoholic and grow up to be a successful husband and in high quality samples for a wide range of latent dimensions (SM 11).
Floating point precision does not seem to be of concern up to this limit.
Evidence for volume codes.
The results suggest that AriEL volumes are responsible of its success.
We provided evidence on how volume codes improve the retrieval of information composed of discrete, variable length sequences, by random sampling, compared to other codes.
AriEL generates more valid sentences, an explicit volume code.
VAE is the second on the toy dataset, an implicit volume code, but it is worse than AE on the real dataset.
Transformers are hard to sample from the latent space.
The Transformer has been used in this work in an uncommon way: by sampling its latent space instead of its output space.
Its low validity score in this work reﬂects it.
Our aim was to understand the latent organization of language, so, we do not suggest this is the most eﬀective way to use
Transformer.
Transformer is excellent when sampled in the output space, but it’s diﬃcult to sample from the latent space.
This is so because Transformer represents each word in d dimensions while the other approaches represent a sentence in d dimensions.
Transformer needs a very high dimensional vector to represent a sentence, n · d , where n is the number of words in it.
This makes it hard to ﬁnd sentences by uniform sampling the latent space.
Conclusions
We proposed AriEL, a mapping of language into volumes, that we used as a reference system for language generation.
AriEL fuses arithmetic coding and k-d trees to construct volumes that preserve the statistics of a dataset.
On the one hand, this study helps to realize how much of the latent space lies unused by standard architectures.
On the other hand, when compared to standard techniques it highlights room for improvement in their capacity for generation, prediction and generalization.
AriEL allows us to sample/generate in theory the training set probability distribution and in practice a diverse set of sentences, as demonstrated on the toy, on the human dataset and using GPT-2 as its language model. Supplementary Material Context-free grammar (CFG) used in the experiments
The context free grammar used to generate the biased and unbiased sentences is composed by the following rules: s → q q → qword adjective ’ , ’ adjective ’and ’ adjective ’? ’
q → qword adjective ’and ’ adjective ’? ’
q → qword adjective ’? ’
q → qword ’made ’ ’ of ’ noun material ’? ’
q → qword preposition np ’? ’
q → qword np ’? ’
np → determiner
[Table]
np → determiner
[Table]
[Table]
’and ’ adjective noun
np
[Table]
[Table]
noun np
[Table]
’made
[Table]
’ of
[Table]
noun
[Table]
np
[Table]
Size of the CFG language space
From the CFG used in the experiment
[Table]
it is possible to extract a total of 15,396 distinct grammar rules, some are shown below.
However, for simplicity
[Table]
we deﬁned only 4, related to the number of adjectives in it.
In the case of the unbiased dataset, those rules can produce a total of 9.81e+18 unique sentences.
The total number of unique sentences for the biased dataset is expected to be an order of magnitude smaller
[Table]
Example of sentences generated from the CFG 3. Biased sample sentences
• is it large, light yellow and light ?
• is it white, deep pink and average-sized ?
• is it a light, huge and shallow laminate ?
• is the object average-sized and light ?
• is the object fashionable, ghost white and pale turquoise ?
• is the thing huge, huge and khaki ?
• is the thing small, ignitable and very light ?
• is the object a notable very light orange carpet ?
• is the object this small wood made of facing stone ?
• is the object a textured and combinable ﬂoor cover made of laminate ?
3. Unbiased sample sentences
• is the object the huge tiny lovable guest room ?
• is the object the closed closed transparent textile ?
• is the thing a transparent, narrow and slightly heavy textile ?
• is it steerable, dark orange and light ?
• is it gray, very heavy and textured ?
• is it closed, heavy and moderately light ?
• is it transparent, transformable and moderately light ?
• is the thing average-sized and dark red ?
• is the thing large and deep garage ?
• is it that slightly heavy stucco made of grass ?
CFG Vocabulary
[Table]
Example of classes
[Table]
Location
[Table]
Color red, royal blue, dark gray, sea shell
[Table]
textured
Material
[Table]
wood, textile, leather, carpet, decoration stone
[Table]
Use of latent space
In ﬁgure S1, each dot represents a sentence in the latent space.
In the ﬁrst row the dot in the latent space is passed as input to the decoder, while in the second and third row the dot is the output of the encoder when the biased test sentence is fed at its input.
Two random axis in R d are chosen for the generator, ﬁrst row, while two axis were chosen subjectively among the ﬁrst components of a PCA for the encoder, second and third row.
In every case, the values in the latent space where normalized between zero and one to ease the visualization.
Lines are used to ease the visualization of the clusters of data with their label, since the point clouds overlap and are hard to see.
The curves are constructed as concave hulls of the dots based on their Delaunay triangulation, a method called alpha shapes Edelsbrunner et al.
We can see in ﬁgure S1 (ﬁrst row) how easy it is to ﬁnd grammatical sentences when randomly sampling the latent space for each model.
AriEL practically only generates grammatical sentences and AE and VAE perform reasonably well too, while Transformer fails.
AriEL failures are plot on top, to remark how few they are, while AE, VAE and Transformer failures are plot at the bottom, otherwise they would hide the rest given how numerous they are.
In the same ﬁgure (rows two and three) we can observe how diﬀerent methods structure the input in the latent space, each with prototypical clusters.
The Transformer presents an interesting structure of clusters whose purpose remains unclear.
Interestingly, the encoding maps seem to be more organized than the decoding ones.
All the models seem to cluster data belonging to diﬀerent classes at the encoding, that could be taken advantage of by a learning agent placed in the latent space.
However it seems hard to use the Transformer as a generator module for an agent.
The good performance of AriEL is a consequence of the fact that all the latent space is utilized, and in no directions large gaps can be observed.
This can be seen in the two encoding rows, where the white spaces around the cloud of dots are consequence of the rotation performed by the PCA, otherwise all the space betweenand 1 would be utilized by AriEL.
Transformer mainly yielded ungrammatical sentences, while AE and VAE were able to produce many grammatical sentences (ungrammatical below, otherwise they would cover up the grammatical
Visualization of performance on toy data
G e n e r a t i o n VC
V
U
BA
GA
PAB
PAU
VAE
AriEL AETransformer
G e n e r a l i z a t i o n
P r e d i c t i o n GCVC V UBAGARAGRAU
GC
VC
V
U
BA
GA
PAB
PAU G e n e r a l i z a t i o n
P r e d i c t i o n
GC
GCVC V UBAGARAGRAU
G e n e r a t i o n
Most importantly AriEL outperforms every other method in Generation Validity (V) and it doesn’t require a large latent space to do so ( R 16 similar to R 512 ).
VAE performs remarkably well at generating unique and grammatical sentences (validity, V) when the latent space is small ( R 16 ), probably given the volume-code nature of the method.
Transformer performs exceptionally at not overﬁtting in the reconstruction tasks and generalizing, it manages to cover all grammar rules, even with a very small number of parameters ( R 16 ).
Transformer proved to be an ineﬃcient generator using random sampling as input (validity) but improved with a larger latent space.
For a larger latent space of R 512 , AE and VAE overﬁt less (PAU and PAB) and improve their Generation (V).
Training details
We go through the training data 10 times, in mini-batches of 256 sentences.
We applied teacher forcing (Williams and Zipser, 1989) during training.
We use the Adam (Kingma and Ba, 2015) optimizer with a learning rate of 1e-3 and gradient clipping at 0.5 magnitude.
Learning rate was reduced by a factor of 0.2 if the loss function didn’t decrease within 5 epochs, with a minimum learning rate of 1e-5.
For all RNN-based embeddings, kernel weights used the Xavier uniform initialization (Glorot and Bengio, 2010), while recurrent weights used random orthogonal matrix initialization (Saxe et al., 2014).
Biases are initialized to zero.
Embeddings layers are initialized with a uniform distribution between [-1, 1].
For Transformer the multihead attention matrices and the feedforward module matrices, used the Xavier uniform initialization (Glorot and Bengio, 2010), the beta of the layer normalization uses zeros, and its gamma uses ones for initialization.
AE and VAE are trained with a word dropout of 0.25 at the input, and VAE is trained with KL loss annealing that moves the weight of the KL loss from zero to one during the 7th epoch, similarly to the original work (Bowman et al., 2016).
Our code is in TensorFlow (Abadi et al., 2015).
We run our experiments on an NVIDIA TITAN Xp and an NVIDIA Tesla K40c.
Each experiment took less than one day to converge.
AriEL is an explicit volume code proof
AriEL uses a bounded region of R d , the interval [0 , 1] d , so encoder and decoder map each input to and from a compact set.
Moreover, any sequence x is assigned to a hyper-rectangle (Johnson, 2018) and back.
Since hyper-rectangles cannot be divided into two disjoint nonempty closed sets, they are connected (Munkres, 2018).
Therefore AriEL is a volume code .
AriEL is an explicit volume code since its LM is trained only on a next word prediction loglikelihood loss, without a regularization term that encourages smoothness, and the volumes are constructed by arranging the softmax outputs into a d dimensional grid, operation performed with any choice of loss or optimizer.
More Sentences from the Qualitative Study
[Table]
Blue means that the sentence complies with the bias and purple means that the incorrect reconstruction is still unbiased.
Most reconstructions seem grammatically correct
In practice AriEL also made errors.
Some of its failed reconstructions comply with the training bias, some do not.
Transformer performs remarkably well, and interestingly the errors made tend to turn the unbiased input sentence into a biased version at the output.
AE produced only biased sentences whose structure resembled the unbiased ones.
VAE behaved similarly, producing more unbiased sentences.
[Table]
ﬁnd operation
The ﬁnd operation deﬁnes within which bounds the latent vector is pointing at, and therefore what is the next word selected by AriEL.
This operation is performed every time an axis
d
i is selected.
We deﬁne it with an argmax operation at the end, but other solutions can be deﬁned.
To give it a pseudonim we call it DetRoulette , from Deterministic Roulette .
Algorithm
Input: bounds on axis d
i : Bs low ( s ), Bs up ( s ) ; value of latent space on d
i : z ( d i )
Output: s i is the word z was pointing at, 1: function find s ( Bs low ( s ) <
z ( d i ) <
Bs up ( s ) ) 2: a ( s ) =
Bs up ( s ) − z
( d i ) 3: b ( s ) =
Bs low ( s ) − z
( d i ) 4: logits ( s ) = − a ( s ) · b ( s ) 5: s i = argmax s ( logits ( s )) 6: return
s i 7: end function
[Figure]
Samples from AriEL with GPT-2 as Language Model d =
He must \ ’ve been very clear that the missive was a sign of trouble since the MTA was lukewarm on keeping maintenance intact. ”
It only takes a few seconds for you to walk through the lines and see the notice,” Kris Johnson, who worked on the 14th Street Bus line in January, said of the notice. ”
But we don \ ’t hire people that don \ ’t want to work as long as they work.”
\ n \ n How many times have you seen ”Why $ 15 is $
Dear Reader, As you can imagine, more people are reading The Jerusalem Post than ever before.
Nevertheless, traditional business models are no longer sustainable and high-quality publications, like ours, are being forced to look for new ways to keep going.
Unlike many other news organizations, we have not put up a paywall.
We want to keep our journalism open and accessible and be able to keep providing you with news and analyses from the frontlines of Israel, the Middle East and the Jewish World.
\ n
d =
Melrose Creek is slated for redevelopment in where hundreds of 150-odd residents live in the eﬀort to develop a pedestrian-friendly and quieter downtown in a neighborhood that has been a struggling housing market for their entire lives.
\ n \ n After decades of straddling one of the most culturally open places in
San Francisco, Melrose Creek’s market-driven nature has resulted in popular zoning problems—including a seismic event lifting oﬀ of a tent park and opulent vistas sunbathed against the bicycle-friendly
Light Air Rating/Bearing Temperature.
Can Handle Winds from Down (1 to 15 ft/min. -
30 It should be able to raise to about 53 mph, 35 mph = 24% duty on stiﬀ and T-shielded loads).
Can Handle Medium Dry conditions and Good
Condition \ n \ n \ n Hybrid Extra Performance.”
Two Driven In Woofer Lighting
Stations! 1967-1992 \ n \ n \ n Also applicable to the best gasket material available.
No Gasket Limit forlyst glass.
\ n \ n \ n
d =
In the last three years, items were being traded around the world for less in bitcoin.
Reuters estimates that its stock price dropped 19%.
\ n \ n Yesterday,
China has kind of happened.
On Yahoo Finance, Rory Schrimpf wrote (”A good amount of Chinese billionaires are planning formal plans for a market of bitcoin for a fund,” his Twitter feed says).
The gist itself looks like this: \ n \ n A ”bitcoin meeting” that can potentially all but guarantee big deals in bitcoin might come down to ”
Even though the story wants to be understood, many people who have experienced hardfought battles of adversity are still going to think that what they experienced in the Battle of Telluride will help deﬁne their narratives in a way which will bring this story to the public’s attention and invite new dialogue and reconciliation, news organisations have widely concluded that dreams of revolution come easy to ordinary people who make diﬃcult choices.
What we often forget is that urbanisation has devastating dams, fragmentation, and denial of services and employment
Samples from AriEL trained on GuessWhat?!
AriEL is it silver ?
bowel in white colour ?
is it the trash can ?
is it the steel top in white ?
look like food ?
is it in left ?
Transformer foreground am exactly exactly by left same cat dog by does it the taller ?
bed about bike base base us on ’ electricity poster clothing a clothing clothing clothing clothing clothing clothing clothing clothing ?
is it she is ?
trees stoplight ground ground bird side bikes ? is it that directly ?
AE a person ?
is
[Table]
table
[Table]
are they
[Table]
is
[Table]
VAE is the one one of us ?
something
[Table]
to white
[Table]
are they in the left of the left side of the
[Table]
all white
[Table]