D ISCOVERING L ATENT N ETWORK T OPOLOGY IN C ONTEXTUALIZED R EPRESENTATIONS WITH R ANDOMIZED D YNAMIC P ROGRAMMING
Anonymous authors
Paper under double-blind review
A BSTRACT
The discovery of large-scale discrete latent structures is crucial for understanding the fundamental generative processes of language.
In this work, we use structured latent variables to study the representation space of contextualized embeddings and gain insight into the hidden topology of pretrained language models.
However, existing methods are severely limited by issues of scalability and efﬁciency as working with large combinatorial spaces requires expensive memory consumption.
We address this challenge by proposing a Randomized Dynamic
Programming (RDP) algorithm for the approximate inference of structured models with DP-style exact computation (e.g., Forward-Backward).
Our technique samples a subset of DP paths reducing memory complexity to as small as one percent.
We use RDP to analyze the representation space of pretrained language models, discovering a large-scale latent network in a fully unsupervised way.
The induced latent states not only serve as anchors marking the topology of the space (neighbors and connectivity), but also reveal linguistic properties related to syntax, morphology, and semantics.
We also show that traversing this latent network yields unsupervised paraphrase generation.
I NTRODUCTION
The discovery of large-scale discrete latent structures is crucial for understanding the fundamental generative processes of language, and has been shown useful to various NLP tasks ranging from data-to-text generation (Li & Rush, 2020), summarization (Angelidis et al., 2021), syntactic parsing (Kim et al., 2019), and knowledge graph reasoning (Qu et al., 2020).
In this work, we use latent structures to analyze geometric properties of representation space of pretrained language models (PLMs).
Despite the large volume of recent work analyzing PLMs and proposing various improvements (Rogers et al., 2020), little is known about the topological structure of their representation manifold.
Since such structure cannot be easily observed, it is only natural to resort to latent variables.
Yet scaling discrete combinatorial structures is extremely difﬁcult with multiple modeling and computational challenges (Wainwright & Jordan, 2008).
In this work, we address the computational challenges arising from working with combinatorial structures.
We consider linear-chain CRFs, a popular structured model family (Ma & Hovy, 2016;
Sutton & McCallum, 2006) that uses dynamic programming for exact inference.
Speciﬁcally, we focus on the forward algorithm (Rabiner, 1989), which is widely used to compute the partition function.
Space complexity for this algorithm is O ( TN 2 ) where N is the number of latent states and T the length of the sequence.
It is precisely the N 2 term that becomes problematic when we construct the adjacent gradient graph with automatic differentiation.
DP-based inference algorithms are not optimized for modern computational devices like GPUs and typically work under small-data regimes, with N in the range [10, 100] (Ma & Hovy, 2016;
Wiseman et al., 2018).
With larger N , inference becomes intractable since gradients do not easily ﬁt into GPU memory (Sun et al., 2019).
Our algorithmic contribution is a randomization technique for dynamic programming which allows us to scale N to thousands (possibly more) latent states.
Speciﬁcally, to approximate the partition function, instead of summing over all possible combinations of latent states, we only sum over paths with most probable states, and sample a subset of less likely paths to correct the bias according to a reasonable proposal.
Since we only calculate the sampled path, memory consumption can be reduced to a small controllable budget which is scale invariant.
With a larger memory budget, our method becomes more accurate, and our estimation error smaller.
We thus recast the memory complexity challenge into a tradeoff between memory budget, proposal accuracy, and estimation error.
When applied to linear-chain CRFs, we show that RDP scales the model by two orders of magnitude with memory complexity as small as one percent of the full DP.
Beyond linear-chains,
RDP is applicable to any structured model with DP-style exact inference such as trees (Kim et al., 2019) and semi-Markov models (Li & Rush, 2020), and could also be extended to more general message passing algorithms (Wainwright & Jordan, 2008).
Our analytical contribution is a geometric study of the representation manifold of PLMs, using the proposed RDP algorithm.
We hypothesize that there exist latent anchor embeddings (or landmarks) that describe the manifold topology.
We also expect these anchor states to be informative enough to generate sentences, and their connections to be linguistically meaningful.
We induce latent structures using a VAE with an inference model parameterized by a scaled CRF where state-word relations are modeled by the emission potential and state-state transitions are modeled by the transition matrix.
The connections of words and states together form a latent network.
We use the vector product between contextualized embeddings and state embeddings to parameterize the CRF potentials, bringing together the geometry of the representation space with graphical model inference .
We further show that it is possible to generate paraphrases by traversing the induced network .
Our approach is fully unsupervised and the discovered latent network is intrinsic to the representation manifold, rather than imposed by external supervision, eschewing the criticism of much previous work on supervised probes (Hewitt & Liang, 2019; Chen et al., 2021).
In experiments, we ﬁrst verify the basic properties of RDP (bias-variance) and show its effectiveness for training latent variable models.
We then visualize the discovered network based on BERT Devlin et al.
, demonstrating how states encode information pertaining to syntax, morphology, and semantics.
Finally, we perform unsupervised paraphrase generation by latent network traversal.
R ANDOMIZED D YNAMIC P ROGRAMMING Preliminaries: Speeding Summation by Randomization
To motivate our randomized DP, we start with a simple setting, namely estimating the sum of a sorted list.
Given a sorted list of positive numbers a , naive summation S = a 1 + ..., + a N requires N − 1 addition operations, which is expensive when N is large.
Suppose we wish to reduce the number of addition operations to K 1 <<
N , and we already know that the list is long-tailed (similar to how words in language follow a Zipﬁan distribution such that there are few very high-frequency words that account for most of the tokens in text and many low-frequency words).
Then, we only need to sum over the top K 1 values to get an efﬁcient estimate:
[Equation]
{ a i } Ni =1 sorted, large to small
[Equation]
Clearly, ˆ S 1 underestimates S .
When the summands are “dense”, i.e., not very different from each other, the bias is large because the top K 1 terms do not contribute much to the sum (Fig. 1A).
To correct this bias, we add samples a δ 1 , ..., a δ K 2 from the remaining summands whose indices δ i are sampled from proposal δ i ∼ q =
[ q K 1 +1 , ..., q N ] :
[Equation]
where K 1 + K 2 = K .
Note that this is an unbiased estimator as E [ ˆ S 2 ]
= S , irrespective of how we choose q .
Without any knowledge about a , the simplest proposal would be uniform, no matter what variance it induces.
The more q i correlates with a i , the less variance ˆ S 2 has.
The oracle q i is proportional to a i , under which ˆ S 2 becomes exact ˆ S 2 ≡ S as q δ i = a δ
i / ( a K +
+ ...
+ a N ) for all i .
So, the strategy is to exploit our knowledge about a to construct a correlated proposal q .
Given this estimator, we can also adjust the computation budget in order to reduce variance.
When the distribution is long-tailed, we may increase K 1 as an instance of Rao-Blackwellization (Liu et al., 2019).
When the distribution is not long-tailed (enough), and top K 1 summation underestimates signiﬁcantly, we may increase K 2 to reduce variance, provided we have a fairly accurate q , as an instance of importance sampling.
This procedure is also discussed in Kool et al.
(2020) for
A. Sampled Sum of a sorted list
B
[Figure]
recursion
C. Model
[Figure]
Long-tail case ˜ a t − ˜ a t
BERT x 0 x 1 x 2 xrrr 2 rz
zz 2 z
Decoder x 0 x 1 x 2 x
TopK summand
Sampled summand
Gap to oracle Dropped summand
TopK state
Sampled state
R e p r e s e n t a t i o n
S p ace
Dense case
Encoder
Latent structure in representation space
RDP
Decoder
B ac k p r o p .
Log Z
Paraphrasing
D. Experiment Protocol gradient estimation.
In fact, it is the underlying basis of many Monte Carlo estimators in various settings (Mohamed et al., 2020)
The Sampled Forward Algorithm
Now we will show how estimator ˆ S 2 can be used to scale summation in DP.
Consider a linear chain CRF which deﬁnes a discrete state sequence
z
=
[ z 1 , ..., z T ] , z t ∈ { 1 , ..., N } over an input sentence x
= [ x 1 , ..., x T ] .
Later we will use this
CRF to construct an inference model to discover latent network structures within contextualized representations.
We are interested in the partition function Z which is commonly computed with the
Forward algorithm, a dynamic programming algorithm that sums over the potentials of all possible state sequences.
The core recursion steps are:
[Equation]
where α t ( i ) is the sum of all possible sequences up to step t and at state i , Φ( · , · ) is an N × N transition matrix, and φ ( x t , i ) is the emission potential that models how word x t generates state i .
We assume all potentials are positive for simplicity.
When implemented on GPUs, space complexity is O ( TN 2 ) (see number of edges in the DP graph in Figure 1B) and it is the squared term N 2 that causes memory overﬂows under automatic differentiation (see Appendix B for engineering details).
Our key insight is to recursively use the memory-efﬁcient randomization of Eq.
2 to estimate
Eq.at every step.
Given a proposal ˜ q t for each step t that correlates with summands ˜ a t (we discuss how to construct ˜ q t in the next section), we obtain its top K 1 index and sample K 2 from the rest:
[Equation]
where ˜ q t ( · ) are normalized to construct the categorical.
Compared to Eq. 3, the key recursion of our
Sampled Forward uses the top K 1 index σ t and sampled K 2 index δ t to substitute the full index:
[Equation]
where the oracle proposal q ∗ t is proportional to the actual summand ˜ a t (Eq. 3, a little bit algebra will show this is actually the backward sampling probability p ( z t = i | z t +1 = j ) ) , which is only accessible with the full Forward.
So, we use the proposal weight ˜ q t (Eq. 4) to move the computation outside the DP.
In Fig.
1B, the top K 1 summed terms correspond to black nodes.
The proposal ˜ q t corresponds to black and grey bars, and its distance from the oracle proposal ˜ a t (which is the major source of variance) is highlighted in green.
Sampled indices are shown as blue nodes.
Essentially, our Sampled Forward algorithm restricts the DP computation from the full graph to subgraphs with top and sampled edges, reducing complexity to O ( TK 2 ) where K = K 1 + K 2 .
By varying K , memory complexity becomes a tradeoff between memory budget and estimation error.
By induction, we can show that ˆ Z (Eq. 7) is an unbiased estimator of Z since ∀ t, E [ˆ α t ] = α t .
When implemented in log space, the expected log ˆ
Z is a lower bound of the exact log Z due to Jensen’s inequality, and the variance is (trivially) reduced by log( · ) .
See Appendix for details on implementation (Section C), theoretical analysis (Section A), and extensions to general sum-product structures (Section D).
L ATENT N ETWORK T OPOLOGY IN P RETRAINED L ANGUAGE M ODELS
Latent States within Representation Space We now use the above technique to uncover hidden geometric structures in contextualized representations.
In experiments we work with BERT (Devlin et al., 2019) and GPT2 (Radford et al., 2019), however, our method can be easily applied to other pretrained language models.
Given sentence x = [ x 1 , ..., x T ] , we denote its contextualized representations as [ r 1 , ..., r T ] =
PLM ( x ) .
Representations r for all sentences lie in one manifold M , namely, the representation space of the language model.
We hypothesize there exists a set of latent states s 1 , ..., s M that function as anchors and outline the space topology.
We emphasize that all parameters of the PLM are ﬁxed (i.e., no ﬁne-tuning takes place), so all learned states are intrinsic to M .
We focus on two topological relations: (a) state-word relations, which represent how word embeddings may be summarized by their states and how states can be explained by their corresponding words; and (b) state-state relations, which capture how states interact with each other and how their transitions denote meaningful word combinations.
Taken together, these two relations form a latent network within M (visualized in Fig.
3 and 4).
We adopt a minimal parametrization of the inference network so as to respect the intrinsic structure of the representation manifold without imposing strong assumptions (e.g., via regularization).
Speciﬁcally, for state-word relations, we associate each word embedding r t with a latent state indexed by z t ∈ { 1 , ..., N } (the corresponding embedding of z t is s z t ).
For state-state relations, we assume a transition weight Φ( i, j ) .
Together we have a linear-chain CRF:
[Equation]
where the dot product follows the common practice of ﬁne-tuning contextualized representations.
We use log space for numerical stability.
The probability of a state sequence given a sentence is:
[Equation]
Here, the only learnable parameters are state embeddings: ψ = [ s 1 , ..., s N ] as we try to be faithful to the representation manifold.
Note how this parametrization reconciles space geometry with graphical models.
As N is large, we estimate Z with the proposed Sampled Forward
(Eq. 7).
Constructing the Proposal We now return to proposal ˜ q t (Eq. 4) which we construct based on a common observation that linguistic phenomena are long-tailed:
[Equation]
where φ ( x t , i ) states that only a few states are likely to generate observation x t , which is often the case in NLP (e.g., there are only a few possible POS tags for each word); and Φ( i ) models the prior probability of state i .
This choice stems from the empirical observation that larger L1 norm correlates with larger dot product, and is thus more likely to be inferred.
Essentially, our proposal combines local emissions φ and global prior Φ to approximate the ˜ a t variables (Eq. 3) and bypass their expensive computation.
Inference and Learning
We use amortized variational inference to learn s .
We simply reuse the architecture from previous work Fu et al.
(2020); Li & Rush (2020)
and build a generative model:
[Equation]
where θ denotes the decoder parameters, Dec ( · ) denotes the decoder (we use an LSTM), h t denotes decoder states, and FF ( · ) denotes a feed-forward network.
This autoregressive formulation essentially encourages states to be “generative”, i.e., to generate sentences and themselves.
We will show in experiments how this formulation lends itself to paraphrasing.
We use q ψ directly from Eq.
9 as our variational posterior, and optimize the following β -ELBO objective:
[Equation]
(13) where the β parameter modulates the topology of the latent structure and prevents posterior collapse.
We follow Fu et al.
and use their Gumbel reparameterization to optimize q ψ , which is more stable than the REINFORCE gradient estimator (Li & Rush, 2020).
When integrating RDP with the Gumbel reparameterization, we noticed that the gradient will only pass through the top K 1 and sampled K 2 states, in other words, not all states receive gradients.
In this case, trading K 1 against K 2 amounts to exploration versus exploitation .
A large K 1 means we give gradients to high-conﬁdence states, i.e., we exploit large local emission and global transition potentials.
While increasing K 2 means we explore low-conﬁdence states.
So, by splitting the computation budget between top K 1 and sampled K 2 states, we not only reduce variance for estimating the partition, but also effectively introduce different strategies for searching over the latent space.
R ELATED W ORK Efﬁcient Inference for Structured Latent Variables
There has been substantial interest recently in the application of deep latent variable models (LVMs) to various language related tasks (Wiseman et al., 2018; Li & Rush, 2020), which has also exposed scalability limitations.
Earlier attempts to render CRF models efﬁcient (Sokolovska et al., 2010; Lavergne et al., 2010) either make many stringent assumptions (e.g., sparsity), rely on handcrafted heuristics for bias correction (Jeong et al., 2009), or cannot be easily adapted to modern GPUs with tensorization and parallelization.
Sun et al.
(2019) are closest to our work, however they only consider top K summation and consistently underestimate the partition.
Chiu & Rush (2020) scale HMMs but assume words are clustered beforehand.
Our approach systematically trades computation with proposal accuracy and estimation error (rather than over-compromising for efﬁciency).
Moreover, we do not impose any hard restrictions like sparsity (Correia et al., 2020), and can accommodate dense and long-tailed distributions.
Our method is inspired by randomized automatic differentiation (RAD, Oktay et al., 2020), and can be viewed as RAD applied to the DP computation graph.
Advantageously, our proposal is compatible with existing efﬁcient implementations (like Rush, 2020) since it does not change the computation graph.
Interpretability of Contextualized Representations
There has been a good deal of interest recently in analyzing contextualized representations and the information they encode.
This line of research, collectively known as “Bertology” (Rogers et al., 2020; Hewitt & Manning, 2019), focuses mainly on supervised probing of linguistic properties (Tenney et al., 2019), while the geometric properties of the representations have been less studied (Cai et al., 2021).
A major dilemma facing this work is whether supervised linguistic probes reveal properties intrinsic to the embeddings or imposed by the supervision signal itself (Hewitt & Liang, 2019; Hall Maudslay et al., 2020; Chen et al., 2021).
In this work, we do not use any supervision to ensure that the discovered network is intrinsic to the representation space.
E XPERIMENTS
In this section, we present our experimental results aimed at analyzing RDP and showcasing its practical utility (see Fig. 1).
Speciﬁcally, we (1) verify the basic properties of RDP by estimating the partition function and (2) using it to train the structured latent variable model introduced in Section 3; (3) we then turn our attention to pretrained language models and examine the network induced with our approach and whether it is meaningful; and (4) we generate sentence paraphrases by traversing this network.
For experiments (1, 2, 4), we use (a).
pretrained GPT2 as the encoder since they are more about autoregressive language modeling and generation; (b).
the MSCOCO dataset, a common benchmark for paraphrasing (Fu et al., 2019).
For experiment (2), we use (a).
BERT since it has been the main focus of most previous analytical work (Rogers et al., 2020); (b).
the 20News dataset, a popular benchmark for training latent variable models (Grisel et al.).
Across all experiments, we
[Figure] [Figure]
[Table]
use an LSTM decoder with states identical to the encoder (762 for BERT base and GPT2 as in Wolf et al., 2020).
More details on experiments and model settings can be found in Appendix E.
5. B ASIC P ROPERTIES
We examine the estimation of the partition function for three unit cases, namely dense, intermediate, and long-tailed distributions.
Instead of simulating these unit cases, to make our experiments more realistic, we extract CRFs on-the-ﬂy from different LVM training stages.
We also study the effects of memory budget by setting K to 20, 200, and 400 (corresponding to 1, 10, and 20 percent of the full memory).
We use TopK summation (Sun et al., 2019) as our main baseline.
This method can be viewed as setting K 1 =
K and K 2 = 0 in our framework, i.e., it does not use the random sample.
For training LVMs, We consider 100 and 2,000 latent states.
With 100 states we are able to perform the summation exhaustively which is the same as Fu et al.
Full summation with 2,000 states is intractable, so we only compare with TopK summation and use K = 100 .
Estimating the Partition Function
As shown in Figure 2, TopK summation always underestimates the partition.
The gap is quite large in the dense case (large entropy), which happens at the initial stages of training when the model is not conﬁdent enough.
The long-tailed case represents later training epochs when the model has converged and is more concentrated.
Our method effectively corrects the bias, and works well in all unit cases with signiﬁcantly less memory.
Training Latent Variable Models
We compare different LVMs in Table 1.
Following common practice, we report negative log likelihood (NLL) and perplexity (PPL).
We perform an extensive search over multiple hyperparameters (e.g., β , learning rate, word dropout) across multiple random seeds (3–6) and report the average performance of the best conﬁguration for each method.
Our model performs best in both 100 and 2,000 state settings.
The advantage is modest (as there are no architecture changes, only different training methods) but consistent.
RDP trades off exploitation (i.e., increasing K 1 ) and exploration (i.e., increasing K 2 ) while TopK summation always focuses on the local solutions by passing gradients through top states.
Intuitively, we have the chance of discovering better latent states (i.e., larger likelihood) by randomly searching the unexplored space.
5. D ISCOVERING L ATENT N ETWORKS FROM P RETRAINED E MBEDDINGS
We now discuss how latent structures induced with RDP reveal linguistic properties of contextualized representations.
We focus on BERT Devlin et al.
and set the number of latent states to
[Table]
A A
[Figure]
[Table]
As BERT’s vocabulary size is 32K, one state would approximately handle 15 words in the uniform case, functioning as a type of “meta” word.
After convergence, we use q ψ to sample z for each x in the training set (recall we use the 20News dataset).
These z can be viewed as samples from the aggregated posterior (cid:80) x
q ψ ( z | x ) p (cid:63) ( x ) where p (cid:63) ( x ) denotes the empirical data distribution.
To get a descriptive summary of BERT’s latent topology, we compute the following statistics on z samples: state frequency (Fig. 3, A3); number words corresponding to each state (Fig. 3, A2); number of states corresponding to each word (Fig. 3, A1); and state bigrams (Fig. 4).
We further differentiate stopwords (e.g., in, of, am, is ) from content words.
State-word Relations Figure 3 gives a ﬁrst impression of how latent states spread over the representation space.
Overall, we observe that the joint space is Zipﬁan, and this property characterizes
Joint Visualization of All States
Top 500 States, Connected by Transition Matrix
Top 500 States, Connected by Aggregated Posterior
Transition Interpretation
Corresponding Bigrams - Occurrence
W / o .
S t
o p w o r d s
W .
S t
o p w o r d s
In prepositional phrase 904-
[Table] [Table]
[Table] [Table]
- is
[Table]
[Table] [Table]
- written
[Table]
by
[Table]
11 |
[Table]
by
[Table]
8 | produced
[Table]
6 | followed
[Table]
6 |
[Table]
by
In somewhere 1895in-bosnia 14 | in-
[Table]
9 | in-seattle 9 | in-
[Table]
| in
[Table]
6 | in
[Table]
los 6 | in-azerbaijan
[Table]
- built-
[Table] [Table]
|
[Table]
|
[Table] [Table]
|
[Table]
|
[Table]
243intentionally-
[Table]
3 |
[Table]
basically
[Table]
heavily
[Table]
[Table] [Table]
much-
[Table]
lot
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
Highlighted Transitions
[Figure]
[Figure]
the distribution of words (A1), states (A3), and word occurrence within each state (C).
We also see that the top 500 states account for most word occurrence (A2) while the remaining states model tail phenomena (A3).
We conjecture this number is related to the intrinsic dimension of the data manifold (see Aghajanyan et al. 2021).
The induced states encode multiple linguistic properties (Fig. 3,
C).
Some states are similar to a lexicon entry encoding speciﬁc words and their morphological variants; other states exhibit clustering based on morphological features ( -s , -er , -ly sufﬁx).
We believe this is closely related to the fact that BERT learns embeddings over subwords.
Note that the past tense cluster contains words exhibiting both regular ( -ed sufﬁx) and irregular morphology (e.g., lost and built ).
Finally, we also see that some states are largely semantic, similar to a conventional topic model (e.g., Computer and Medicine clusters).
See Appendix E.6 for more state-word examples.
State-State Relations As shown in Fig.
4, we observe a clear geometric difference between top and tail states.
Most linguistic constructions seem to be captured by the top 500 states (A1).
The connections of top states are visualized in (A2–A4).
From a statistical perspective, the similarity of (A3) and (A4) clearly shows how the empirical prior (encoded by the transition matrix Φ ) matches
[Table]
the aggregated posterior (encoded in the bigram sample from q ψ ), which is an important desideratum of generative modeling (Mathieu et al., 2019).
Note that the number of edges linked to each node, again, follows a Zipﬁan distribution as top nodes have most of the connections.
From a linguistic perspective, we see how the combination of states leads to meaningful syntactic and semantic constructions.
Again, BERT encodes various syntactic conﬁgurations such as to inﬁnitives, passive voice
[Table]
and even manages to distinguish adverbials (e.g., in fact ) from prepositional phrases (e.g., in
Bosnia ).
In general, the latent network seems to have some grasp of syntax, semantic roles, and collocations.
In the following section, we examine whether this inherent knowledge can be harvested for generation.
See Appendix E.7 for more state transition examples.
5. P ARAPHRASING THROUGH N ETWORK T RAVERSAL
We now study how the latent network can be usefully employed to generate paraphrases without access to parallel training instances.
Given a sentence, we generate its paraphrase by conditioning on the input which we represent as a bag-of-words Fu et al.
(2020) and by sampling from latent states.
This amounts to traversing the latent network then ﬁll in the traversal path to assemble a sentence, as visualized in Fig.
4 C. We instantiate our approach with a latent network learned from
GPT2 representations (Radford et al., 2019) and refer to our model collectively as GPTN ET .
We compare against three previous unsupervised models (ﬁrst block in Table 2), including CGMH (Miao et al., 2019), a general-purpose MCMC method for controllable generation; UPSA (Liu et al., 2020), a strong paraphrasing model with simulated annealing, and G UMBEL -CRF (Fu et al., 2020), a template induction model based on a continuous relaxation of the CRF sampling algorithm.
We present GPTN ET variants with 50 and 2,000 states, and show results with RDP and topK, and the full summation for 50 states.
Following previous work, we use iBLEU (Sun & Zhou, 2012) as our main metric, which trades off ﬁdelity to the references (BLEU) and variation from the input (self-BLEU).
Table 2 shows that RDP is superior to TopK and full summation in terms of iBLUE.
GPTNet models do not outperform G UMBEL -CRF or UPSA.
This is expected as these methods are highly tailored to the task and more ﬂexible (e.g., they do not ﬁx the encoder), while we restrict the modeling within the GPT2 representation space (to infer its structure).
So, our results should be viewed as a sanity check demonstrating the latent network is indeed meaningful for generation (see
Appendix E.8 for more generation examples).
C ONCLUSION
In this paper, we have developed a general method for scaling the inference of structured latent variable models with randomized dynamic programming.
It is a useful tool for the visualization and inspection of the intrinsic structure of contextualized representations.
Experiments with BERT reveal the topological structure of its latent space: state-word connections encapsulate syntactic and semantic roles while state-state connections correspond to phrase constructions.
Moreover, traversal over a sequence of states represents underlying sentence structure.
Ethics Statement As this paper inspects the internal structure of pretrained language models, it is likely that it will reveal frequent linguistic patterns encoded in the language model.
Speciﬁcally, the frequent words, phrases, and sentences associated with different gender, ethnic groups, nationality, interest groups, social status, and all other factors, are likely to be revealed by our model.
When calling the generative part of our model for paraphrasing, these differences are likely to exist in the generated sentences (depending on the dataset).
These facts should be considered when using this model.
Reproducibility Statement A step-by-step implementation guide for our randomized forward algorithm is provided in Appendix section C.
The comparison of RDP versus other possible solutions for scaling the structured models is provided in Appendix section B. A detailed description of the model architecture is provided in the Appendix section E.1.
A detailed description of data processing is provided in the Appendix section E.2.
A detailed description of training strategy, hyperparameter search strategy, and model selection, is provided in Appendix section E.3.
A detailed description of visualization procedure is provided in Appendix section E.5.
We will release code after the anonymity period.
Appendix, table of contents
• Section A. Theoretical analysis of the Sampled Forward algorithm.
Bias and variance.
• Section B. Challenges of full DP inference under Automatic Differentiation.
• Section C. Implementation details of Sampled Forward.
• Section D. Extensions of randomized dynamic programming. • Section E. Experimental details. • Section F. Additional experimentsl results. A T HEORETICAL A NALYSIS OF S AMPLED F ORWARD A LGORITHM A. B IAS A NALYSIS
In this section, we discuss the unbiasedness and variance of the Randomized Forward algorithm.
We ﬁrst show that Randomized Forward gives an unbiased estimator of the partition function.
Theorem A.1 (Unbiasedness) .
For all t ∈ [1 , 2 , ..., T ] , the sampled sum ˆ α t (Eq. 6) is an unbiased estimator of the forward variable α t
(Eq. 3).
The ﬁnal sampled sum ˆ Z (Eq. 7) is an unbiased estimator of the partition function Z (Eq. 3).
Proof.
By the Second Principle of Mathematical Induction.
Assume initialization
α 1 ( i ) =
φ ( x 1 , i ) .
Firstly at t = 2 , for all i , we have:
[Equation]
where the second term can be expanded as a masked summation with the index rearranged from σ 2 ,K 1 +1 to σ 2 ,N :
[Equation]
Notice how we re-index the sum.
Now put it back to Eq.
14 to get: E
[Equation]
[ˆ
[Equation]
( i )] =
[Equation]
[Equation] [Equation] This veriﬁes the induction foundation that E q 1 [ˆ α 2 ( i )]
=
α 2 ( i ) for all i .
Now assume for time index t we have ∀ i, E q 1: t − 1 [ˆ α t ( i )]
=
α 2 ( i ) .
Consider t + 1 , we have:
[Equation]
[Equation] [Equation] Note how we decompose the expectation by using the independence: q 1: t = q 1: t − 1 · q t With a similar masked summation trick as A , we have:
[Equation]
This gives us: E
[Equation]
Thus showing ˆ α t is an unbiased estimator for α t at each step t .
Setting t =
T , the last step, gives us
E [ ˆ Z ]
=
Z (details similar to the above).
Corollary A.1.1.
When we change the (sum, product) semiring to the (log-sum-exp, sum) semiring, the expectation of the estimator will become a lower bound of log α t and log Z .
Proof.
Denote l t ( i ) =
log α t ( i ) and Ω the set of sampled indices where | Ω |
= K 2 .
For simplicity, we omit the top K 1 summation and only show the summation of the sample.
Cases where t > 2 can be derived similarly as following:
[Equation]
where Eq. 34 comes from Jensen’s inequality.
Then by induction one can show at everystep, we have E [ˆ l t ( i )]
≤ l t ( i ) .
Although implementation in the log space makes the estimate biased, it reduces the variance exponentially in a rather trivial way.
It also provides numerical stability.
So, in practice we use it for training.
A. V ARIANCE A NALYSIS
Now we analyze variance. We start with the estimator a δ /q δ , δ ∼ Categorical { q K 1 +1 , ... q K N } in
Eq. 2.
Firstly, this is an unbiased estimator of the tail sum:
[Equation]
We have the folling variance arguments:
Theorem A.2 (Variance of the tail estimator) .
[Equation]
) where
S K
[Equation]
=
[Equation]
[Equation]
which says the variance is:
• quadratic to the tail sum S K 1 , which will can be reduced by increasing K 1 as the effect of Rao-Blackwellization.
• approximately quadratic to the gap (cid:15) i , as the differences between the proposal q i and the oracle a
i /S K 1 , which can be reduced by choosing a correlated proposal as the effect of importance sampling.
Optimal zero variance is achieved on
[Equation]
Proof.
The variance is then:
[Equation]
where the ﬁrst term is the second moment:
[Equation]
taking the expection of it we get:
[Equation]
Plug this back, variance is:
[Equation]
To get the optimal proposal
[Equation]
we solve the constrained optimization problem:
[Equation]
By solving the corresponding Lagrangian equation system (omitted here)
[Equation]
we get the optimal value achieved at:
[Equation]
This says zero variance is achieved by a proposal equal to the normalized summands.
Then deﬁne the gap between the proposal and the normalized summands as:
[Equation]
Plug this to the variance expression we get:
[Equation]
Corollary A.2.1.
When increasing the sample size to K 2 , the variance will reduce to
[Equation]
Now we consider the variance of the Sampled Forward algorithm.
An exact computation would give complicated results.
For simpliﬁcation, we give an asymptotic argument with regard to RaoBlackwellization and importance sampling:
Theorem A.3 (Single Step Asymptotic Variance of Sampled Forward) .
At each step, the alpha varible estimator has the following asymptotic variance:
[Equation]
where:
[Equation]
after the top K 1 summands.
This term will reduce if we increase K 1 , as an instance of RaoBlackwellization.
• (cid:15) 2 t ( i ) =
[Equation]
Nj = K 1 +1 (cid:15) 2 t − 1 ( j, i ) /q
t − 1 ( j ) and (cid:15) t − 1 ( j, i ) is the difference between the proposal q t − 1 ( j ) and the oracle proposal in Eq.
This term will reduce if the proposal is more correlated to the oracle, as an instance of Importance Sampling.
Proof.
We start with a simple setting where K 2 = 1 .
At step t + 1 we have the following variance recursion:
[Equation]
This is derived by plugging estimator 6 to the variance Eq.
38 we have just derived.
Denote:
[Equation]
Then we have
[Equation]
where (cid:15) t ( j, i ) is the differences between the proposal and the normalized exact summands at step t state i :
[Equation]
q t
[Equation]
j
[Equation]
Dropping out the ﬁrst order errors and increasing the number of sample to K 2 , we have the asymptotics:
[Equation]
[Equation]
Theorem A.4 (Asymptotic Variance of Sampled Forward Partition Estimation) .
The alpha variable estimators has the following asymptotic variance recurrsion:
[Equation]
Compared with Eq.
55, this expression:
• Uses the product of the factors φ t,K 1 (a function of the sum-prod of the factor at step t ) and the previous step variance V [ˆ α t ] to substitute the α t,K 1 in equation 55.
Again, this term will decrease with a larger K 1 (Rao-Blackwellization).
• (cid:15) 2 t,K 1 ( i ) =
(cid:80) Nj = K 1 +1 (cid:15) 2 t − 1 ( j, i ) /q
t − 1 ( j ) and (cid:15) t − 1 ( j, i ) is the difference between the proposal q t − 1 ( j ) and the oracle proposal in Eq.
This term will reduce if the proposal is more correlated to the oracle, as an instance of Importance Sampling (same as Eq. 55).
Consequently, the partition function has the following asymptotic variance:
[Equation]
[Equation]
When implemented in the log space, the variance is trivially reduced exponentially:
[Equation]
Proof.
{+Firstly for simplcity we assume K 1 = 
+}
and K 2 = 1 .
The the estimator variance is:
[Equation]
Recall:
[Equation]
Let:
[Equation]
Then:
[Equation]
Note that there exist J t such that:
[Equation]
[Equation]
This is because of the bounded gap of the Jensen’s inequality.
Also recall:
[Equation]
[Equation]
So we get:
[Equation]
[Equation]
empirically J t is not the dominate source of variance (but could be in the worst case, depending on the tightness of Jensen’s inequality).
We focus on the second term:
[Equation] Note that a lot of higher-order sum-products are simpliﬁed here. Adding top K 1 summation and increasing the sample size to K 2 leads to variance reduction as: [Equation]
Recursively expand this equation we get: [Equation] Chaning the implementation to the log space we reduce the variance exponentially:
[Equation]
B C HALLENGES OF F ULL DP UNDER A UTOMATIC D IFFERENTIATION
Now we analyze in detail why direct full summation causes memory overﬂow.
Firstly, we note that in a bachiﬁed stochastic gradient descent setting, memory complexity for the forward algorithm is
[Equation]
Where B is the batch size, T is the maximum sequence length after padding, and N is the number of states.
Consider a typical setting where B = 100 , T = 100 , N = 1000 , then the complexity is:
[Equation]
where c is a constant depending on speciﬁc hardware and libraries.
At ﬁrst sight this may seem reasonable with a large GPU memory (e.g., 16G).
Also, one can come up with improvements, such as not storing the intermediate α t variables.
If we only need one single forward pass, such engineering tricks are indeed effective.
In our experiments, when setting B = 50 , T = 15 , and N = 2000 , the actual memory consumption for the forward computation, together with other model components, is about 4G when implemented with PyTorch 1.8.0.
However, this is not the case when working under an automatic differentiation (AD) setting.
If we want to compute the gradients of the partition function (e.g., to optimize the likelihood), we inevitably need to store all the intermediate steps to keep the full computation graph.
Then, the adjacent graph, in principle, has the same complexity as the forward graph.
But in practice, it will be more complicated due to the actual implementation of AD engines.
Following the above example where B = 50 , T = 15 , N = 2000 , we get a memory overﬂow on a 16G memory GPU when calling the PyTorch backward function.
This situation also immediately invalidates many engineering tricks (e.g., we cannot drop the intermediate α t variables anymore), and substantially increases the difﬁculty of coming up with new ones, since we would also be working with the internal mechanism of automatic differentiation engines.
Note that the internal mechanism of these engines is complicated and opaque to general practitioners (e.g., they may change the underlying computation graph for better speed), and many of these engines (like PyTorch) are not optimized for dynamic programming.
In fact, it would be unwise to overwrite the backward computation, even if the AD engine allowed us to do so, as it would signiﬁcantly increase engineering difﬁculty, as we not only need to overwrite the ﬁrst order gradients (like the gradients for the likelihood), but the second order gradients too (e.g., gradients for marginals or reparameterized samples).
In fact, a brute force implementation would not only forego all the advantages of AD engines (like operator-level optimization), but would also require separate implementations for every graph (e.g., chains, trees, semi-Markovs, and IsingPotts), every inference algorithm (e.g., partition, marginal, sampling, reparameterization, entropy), and higher order gradients.
In summary, this is an extremely difﬁcult path that requires clever tricks to efﬁciently improve a large table of already complicated DP algorithms.
This is why we do not interfere with the AD framework and work on more general and efﬁcient algorithms, rather than tailored implementation tricks.
With the randomized DP, we do not re-implement anything .
We only choose the index by the pre-computed proposal, and introduce the re-indexed potentials to existing efﬁcient libraries (like Torch-struct in Rush, 2020).
This is least difﬁcult from an implementation perspective and best exploits the power of efﬁcient structured prediction libraries and AD engines.
C I MPLEMENTATION OF S AMPLED F ORWARD A LGORITHM
Now we discuss how to implement the sample forward in detail.
Our implementation aims to set all computation outside the actual DP, and reuse existing optimized DP libraries.
Speciﬁcally, we:
1. normalize the log potentials; 2.
construct the proposal according to the local potential and a global prior; 3.
obtain the top K 1 index and sample K 2 from the rest, retrieve the potentials according to these indices; 4.
correct the bias before the DP, then insert the corrected potentials to an existing DP implementation.
Now we explain each step in detail.
Potential Normalization is a technique useful for numerical stability (see Fu et al., 2021).
Specifically, in practice, the log-sum-exp function will saturate (it returns the max input value) when the maximum difference between inputs is larger than 10.
So, we normalize all log potentials to be within range [1 , m ] , m < 10 .
Given the emission potential φ and transition potential Φ from Eq. 8, we normalize as:
[Equation]
Then the actual CRF is constructed based on the normalized potentials.
Proposal Construction
Our proposal is constructed based on the normalized local emission and a prior distribution.
Speciﬁcally:
[Equation]
Index Retrieval For each step t , we retrieve the top K 1 index and get a K 2 sized sample from the rest, as is discussed in the main paper:
[Equation]
Note that for simplicity, we use sampling with replacement.
For sampling without replacement, see the procedure in Kool et al.
Conventional Forward Before calling an existing implementation of Forward, we need to correct the bias of sampled terms.
We do this by multiplying the probability of the sample to its emission potential (which becomes addition in log space).
Note that this will be equivalent to multiplying them inside the DP:
[Equation]
We treat any duplicate indices as if they are different, and view the resulting potentials as if they are a new CRF.
We also retrieve the transition potentials at each step according to the chosen index.
So the new CRF has different transitions across steps.
Finally, we run an existing efﬁcient implementation of the Forward algorithm on the new CRF to get the estimated ˆ α and ˆ Z . D E XTENSION OF RDP
In this section, we extend the RDP to more graph structures and more inference operations.
We ﬁsrt discuss a randomized inside algorithm for the partition function estimation for tree-structured hypergraphs, which includes dependency trees and PCFGs.
Then we discuss a randomized entropy
DP estimation on chain-structured graphs. This estimation will call the randomized forward as its subroutine. Finally we discuss howto generalize RDP to the general sum-product algorithm for any graph structure. D. E XTENSIONS TO T REE - STRUCTURED H YPERGRAPHS : R ANDOMIZED I NSIDE
This section discusses how to extend RDP to Tree-structured hypergraphs, including Dependency
TreeCRFs and PCFGs.
We focus on randomizing the Inside algorithm for partition function estimation.
Speciﬁcally, the core recursion of the inside is:
[Equation]
where β [ i, j, k ] is the summation of all subtrees spanning from location i to location j with label k and s jik is the local score and Ω is the full state space.
Suppose Ω is large so we want to sample its subset Ω ij to reduce the summation computation.
Suppose the proposal is q ijk where (cid:80)
k ∈
Ω q ijk = 1 .
Then a randomized inside recursion with K 2 sample is (we omit the top K1 summation for simplicity):
[Equation]
The analysis about bias and variance is similar to the previous analysis (Sec.
A) on the linear-chain case.
D. E XTENSIONS TO R ANDOMIZED E NTROPY DP E STIMATION
This section describes a randomized DP for estimating the entropy of a chain-structured model (HMMs and linear-chain CRFs).
Recall that the core recursion of conventional entropy DP is:
[Equation]
where H t ( i ) is the intermediate conditional entropy end at state i step t .
In our sampled DP, we ﬁrst call the sampled forward to estimate the alpha variables.
Then we re-use the sampled indices for the entropy DP graph, and the core recursion becomes:
[Equation]
where q t is the proposal at step t .
Note how we re-use the estimated alpha variables ˆ α for our entropy
DP and correct the bias of each step.
This section discusses the extension of sampled DP to general graph structures and message-passing algorithms, following the Bethe variational principle (Wainwright & Jordan, 2008).
Recall the general message-passing algorithm computes pseudo marginals by recursively updating the message at each edge:
[Equation]
where M ts ( x s ) denotes the message from node t to node s evaluated at X s = x s , φ st is the edge potential, φ t is the node potential, N ( t ) /s denotes the set of neighbor nodes of t except s , and κ the normalization constant.
At convergence, we have the pseudo marginals
[Equation]
and the Bethe approximation of the log partition function is given by the evaluation of the Bethe variational problem:
[Equation]
where H denotes marginal entropy and I denotes marginal mutual information.
We consider the application of randomization to the computation of pseudo marginals and the Bethe log partition.
Note that the Bethe approximation will be exact if the underlying graph is a tree.
We consider a local proposal ˜ q combined with a global prior τ :
[Equation]
where the prior τ depends on our knowledge of speciﬁc problem structures.
As one can always retreat to uniform proposals, we can explore more advanced choices.
For ˜ q , one may consider: (a) local normalization as ˜( q )( x s ) =
φ ( x s ) / (cid:80) s φ ( x s ) and (b) pre-computed mean-ﬁeld approximations from algorithms like SVI (Hoffman et al., 2013).
For τ , one can further use a frequency-based empirical prior estimated from the data.
To determine which nodes to perform DP on, one may consider the following principles:
• Special nodes of interest depending on their actual meaning.
For example, one may be interested in some x s =
(e.g., if a user is under a bank fraud threat).
• Nodes with large local weight φ ( x s ) .
• Nodes with large global weight τ
( x s ) .
• Loop elimination .
If two nodes have similarly small local and global weight, we could drop those which eliminate loops in the resulting graph.
Also note that we would prefer removing small-weighted nodes for loop elimination.
With the above principles, one may construct three subsets to perform the sum-product:
• Ω 1 including nodes of special interest, where we perform exact computation.
• Ω 2 from the top items of the proposal, where we also perform exact computation.
• Ω 3 by sampling from the remaining items of the proposal (optionally with loop-reduction).
For this set of nodes, one needs to correct the estimation by dividing the proposal probabilty: φ ( x s ) ← φ ( x s ) /q
( x s ) .
After these steps, we treat nodes in Ω 1 (cid:83) Ω 2 (cid:83) Ω 3 as if they are a new model, then feed them to an existing sum-product implementation.
E E XPERIMENT D ETAILS E. M ODEL A RCHITECTURE D ETAILS
For training LVMs, partition function estimation, and paraphrasing, we use the Huggingface checkpoint of GPT2 model 1 since these experimental variables are more about autoregressive language modeling and generation.
For analyzing latent network topologies, we use Huggingface checkpoint of BERT base model 2 since it has been the main focus of most previous analytical work.
The decoder
LSTM is a one-layer LSTM with hidden state size 762, the same as the size of the contextualized embeddings.
It shares the embeddings of the inferred states with the encoder, and uses its own input word embeddings, whose size is also 762.
This architecture sufﬁces for training LVMs and inferring latent networks.
For paraphrase generation, we change the decoder to be conditional by: (a) using the average word embeddings of content words of the input sentence as its initial hidden state (rather than zero hidden states); (b) letting it attend to the embeddings of content words of the input sentence, and copy from them.
This effectively makes this decoder conditional on the BOW of the content words of the input sentence.
Then decoding a paraphrase becomes a neural version of slot ﬁlling: a sequence of latent states by traversing the network becomes a template of the sentence which we ﬁll with words.
E. D ATASET P REPROCESSING
For the MSCOCO dataset, we use the GPT2 tokenizer to process sentences.
As the ofﬁcial websitedoes not release a test split (only a training and a development split), we use the ofﬁcial development split as our test split, and re-split the ofﬁcial training split to be a new training and a development split.
For training LVMs, we only use 1/10 of the training set for quicker experiments (the full dataset takes over more than 20 hours to converge).
For paraphrasing, we use the full dataset.
For the 20News dataset, we use the data from the sklearn website 4 .
We follow its ofﬁcial split and process the sentences with the BERT tokenizer.
E. H YPERPARAMETERS AND T RAINING S TRATEGIES
To get meaningful convergence of the latent space without posterior collapse, the following techniques are important: (a).
set β in the correct range.
A large β force the posterior to collapse to a uniform prior, while a small β encourages the posterior to collapse to a Dirac distribution.
(b).
use word dropout in initial training epochs, otherwise the decoder may ignore the latent code.
(c).
use potential normalization, otherwise the logsumexp function used in the forward algorithm may saturate and only return the max of the input, this would consequently lead the posterior to collapse to few “activated” states and other states will never be inferred.
We further note that a full DP based entropy calculation will cause memory overﬂow with automatic differentiation.
So we approximate it with local emission entropy, i.e., the sum of the entropy of the normalized local emission factors.
Compared with the full entropy, this approximation does not directly inﬂuence the transition matrix, but encourages more activated states and mitigates the posterior collapse problem.
To get a full entropy regularization, one can further regularize the transition matrix towards an all-one matrix, or extend our randomized DP to entropy computation.
As the current local entropy regularization is already effective for inducing a meaningful posterior, we leave the full entropy computation to future work.
Our reported numbers are averaged over “good enough” runs.
Speciﬁcally, for all hyperparameter conﬁgurations, we ﬁrst run three random seeds and get the mean and standard deviation of the performance metrics.
If the standard deviation is small enough, we report mean performance metrics (NLL, PPL, and iBLEU) and the standard deviation.
If the standard deviation is large, we run extra ﬁve seeds.
Then we drop runs with bad performance (usually 2-3), and compute the mean and standard deviation of the performance metrics again.
In total, we experiment more than 100 runs over more than 7 hyperparameters: (a).
learning rate 10 − 3 , 10 − 4 , 5 × 10 − 4 ; (b).
optimizer: SGD,
Adam, AdamW; (c).
dropout: 0.2, 0.3, 0.4; (d).
β variable: 10 − 4 , 5 × 10 − 4 , 10 − 3 , 10 − 2 ; (e).
Gumbel CRF and that different models may achieve best performance with different combinations.
We make sure that all models are searched over the [Table] hyperparameter combinations, and report average performance metrics over multiple seeds of the best hyperparameter conﬁguration for each model.
E. T IME C OMPLEXITY A NALYSIS
Table 3 shows the actual efﬁciency comparison of our method.
Time is measured by seconds per batch.
For experiments with 500 and 2,000 states, we set K = 100 .
*When the number of states is small, our method underperforms due to the overhead of constructing the proposal.
However, its advantage becomes clear as the number of states becomes large.
E. V ISUALIZATION P ROCEDURE
This section describes how we produce Fig.
3(B1-2) and Fig.
4(A1-4).
We use the sklearn implementation 5 of tsne (Van der Maaten & Hinton, 2008).
For Fig.
3(B1-2), the word embeddings are obtained by sampling 8,000 contextualized embeddings from the full word occurrences in the 20News training set.
Then we put the sampled word embeddings and the 2,000 states into the tsne function.
The perplexity is set to be 30.
An important operation we use, for better seperating the states from each other, it to manually set the distances between states to be large, otherwise the states would be concentrated in a sub-region, rather than spread over words.
Fig.
4 is produced similarly, except we do not use word embeddings as background, and change the perplexity to be 5.
For Fig. 4(A3), we connect the states if their transition potential is larger than a threshold.
For Fig. 4(A4), we connect the states if their bigram frequency is larger than a threshold.
All our decisions of hyperparameters are for the purpose of clear visualization which includes reducing overlapping, overcrowding, and other issues.
We further note that no single visualization method can reveal the full structure of high-dimensional data, and any projection to 2-D plain inevitably induces information loss.
We leave the investigation of better visualization methods to future work.
E. S TATE - WORD P AIR E XAMPLES
See Tables 8 and 9 for more sample.
E. S TATE - STATE T RANSITION E XAMPLES
See Table 10 for transitions involving stopwords and Table 11 for transitions without stopwords.
Also note their differences as transitions involving stopwords are more about syntactic constructions and transitions without stopwords are more about speciﬁc meaning.
E. S TATE T RAVERSAL E XAMPLES
See Table 12 for more sample. [Table] [Table]
F A DDITIONAL E XPERIMENT R ESULTS F. RDP V . S . T OP K MSE C OMPARISON
Table 4 shows the mean square error (MSE) comparison between RDP and TopK on dense, intermediate and long-tail distributions. Again, MSE for RDP is signiﬁcantly smaller with less memory consumption . F. MSE C OMPARISON OVER P ROPOSALS Table 5 shows the mean square error (MSE) comparison between different proposals.
Our proposed local and global proposal outperforms a baseline uniform proposal on all distributions (dense, intermediate and long-tail)
Figure 5 shows state frequency with different N. The long-tail behavior becomes clearer only when N is large enough (larger than 500 in our case).
F. C ONTROLLING K 1 V . S . K 2 R ATIO
Figure 6 show state frequency with different K 1 /K 2 ratios at different N. We highlight that when K 2 = 0 , a pure topK summation approach would lead to posterior collapse where there exist inactive states that do not have any density. We also notice that an increasing K 2 consistently increases the frequency of tail states.
This observation can be explained by the exploitation-exploration tradeoff, where increasing K 1 would encourage the exploitation of states already conﬁdent enough during training (consequently leading to high-frequency head states after convergence) while increasing K 2 would encourage exploring states that are not conﬁdent enough during training, leading to a larger tail frequency after convergence. F. C OMPARISON TO R ANDOMLY I NITIALIZED B ERT Figure 7 shows the comparison of reconstruction NLL ( − log p ( x t | z t , · ) ) between a randomly initialized BERT and a pretrained BERT.
States induced from pretrained BERT are more meaningful than random, making it easier to reconstruct words based on their corresponding states.
[Figure] [Table]
F. R ANDOMIZED I NSIDE A LGORITHM FOR T REE - STRUCTURED H YPERGRAPH
We simulate three unit cases of treecrfs (dense, intermediate and long-tail) by controlling the entropy. We set the number of latent states to 2000. We use a uniform proposal for comparison, and set K(top K sum size) = K 2 = 50 .
We use RDP with 20, 200, 400 states
[Table]
which correspond to 1%, 10%
[Table]
and 20% of the full number of states. We use topK summation as our baseline with 500 and 1000 states (25% and 50% of the full number of states). Figure 8 shows performance of RDP to tree structured hypergraph. Our method still outperforms topK summation (grey lines) with less memory (as our estimates are visually closer to the red line,
i [Table]
e., the true partition).
Table 6 shows the MSE of our method compared to topK summation
[Table]
and the results are consistent with Figure 8. F. R ANDOMIZED E NTROPY DP A LGORITHM Figure 9 shows the application of RDP to entropy estimation of linear-chain CRFs. Our method consistently outperforms topK summation (grey lines) with less memory (as our estimates are visually closer to the red line, i.e., the true partition).
Table 7 shows the MSE of our method compared to topK summation, and the results are consistent with Figure 9.
N
=
=
K2 =
[Equation]
[Table]
[Figure]
[Figure]
[Table]
[Table]
[Table]
[Table]
[Table] [Table] [Table]
[Table] [Table] [Table]
[Table]
lines
[Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table] [Table] [Table]
[Table]
dial
[Table]
contend-
[Table]
rough
[Table]
contact
[Table] [Table]
bash
[Table]
favor
[Table] [Table]
western-reserve
[Table]
case-
[Table] [Table]
ohio-state
[Table]
united-states
[Table]
penn-state
[Table]
african-american
[Table]
north-american
[Table]
-eastern
[Table]
polytechnic-state
[Table]
north-carolina1912world-series
[Table]
home-plate
[Table]
division-winner
[Table]
runs-scored
[Table]
batting-average
[Table]
game-winner
[Table]
sports-##channel
[Table]
plate-umpire 3 ; baseball-players 3 ; league-baseball1461- ##l-bus 5 ; bit-color
[Table]
3d-graphics
[Table]
##p-posting 3 ; computer-graphics 3 ; wire-##frame 3 ; bit- graphics
[Table]
##eg-ﬁle
[Table]
access-encryption
[Table]
##frame-graphics 2 ; ﬁle-format123health-care 10 ; high-school
[Table]
es-##crow
[Table]
key-es
[Table]
high-power
[Table]
local-bus
[Table]
low-level
[Table]
high-speed
[Table]
minor-league
[Table]
health-service
[Table]
regular-season
[Table]
mother-##board1702mile-##age 8 ; engine-compartment 5 ; semi-auto 5 ; manual-transmission 5 ; drive-power 4 ; door-car 3 ; passenger-cars 3 ; sports-car 3 ; shaft-drive 3 ; mini-##van 3 ; speed-manual1874ﬂoppy-disk 11 ; jp-##eg 11 ; encryption-algorithm 8 ; ##per-chip 7 ; ##mb-ram 7 ; ##ga-card 6 ; encryption-devices 5 ; silicon-graphics 4 ; disk-drive 4 ; ﬂoppy-drive1208atheist-##s 43 ; homosexual-##s 12 ; fed-##s 9 ; libertarian-##s 8 ; ##eri-##s 7 ; ##tile-##s 7 ; azerbaijani-##s
[Table]
##tar-##s
[Table]
mormon-##s
[Table]
sniper-##s 5 ; physicist-##s1710power-supply 5 ; atomic-energy 4 ; water-ice
[Table]
power-cord
[Table]
##com-telecom 3 ; light-pollution 3 ; light-bulb 3 ; radio-station 3 ; radio-##us 3 ; air-conditioning 3 ; light-##wave1080public-access
[Table]
via-anonymous
[Table]
private-sector
[Table]
available-via
[Table]
general-public
[Table]
community- outreach 4 ; public-domain
[Table]
personal-freedom 3 ; private-property 3 ; private-activities254jimmy-carter 9 ; george-bush
[Table]
bill-clinton
[Table]
bryan-murray
[Table]
joe-carter
[Table]
henry-spencer
[Table]
bill-james
[Table]
janet-reno
[Table]
craig-holland
[Table]
clayton-cramer
[Table]
##zie-smith1571ms-windows 24 ; windows-nt 12 ; ibm-pc 10 ; ms-##dos 7 ; unix-machine 6 ; microsoft-windows 5 ; windows-applications 4 ; run-windows 3 ; apple-monitor 3 ; mac-##s 3 ; desktop-machine66- ##ian-1919 3 ; energy-signature 2 ; charlotte-##sville 2 ; environment-variables 2 ; duty-cycle 2 ; second-period 2 ; spin-state 2 ; power-consumption 2 ; inter-##mission 2 ; power-play1683worth-##while 4 ; nominal-fee 4 ; get-paid 3 ; risk-factors 3 ; scholarship-fund 2 ; cost-$ 2 ; tax-dollars 2 ; beneﬁcial-item 2 ; bank-account 2 ; take-responsibility1579m-sorry 5 ; news-reports 4 ; heard-anything 4 ; ran-##ting 3 ; short-story 3 ; news-reporters 3 ; press- conference 3 ; heard-something 3 ; tv-coverage 2 ; horror-stories 2 ; heard-horror1656urbana-champaign 3 ; peace-talks 3 ; acceptable-solutions 2 ; marriage-partner 2 ; intercontinental- meetings 2 ; interested-parties 2 ; conference-calls
[Table] [Table]