Dynamic Programming in Rank Space: Scaling Structured Inference with Low-Rank HMMs and PCFGs
Abstract
Hidden Markov Models (HMMs) and Probabilistic Context-Free Grammars (PCFGs) are widely used structured models, both of which can be represented as factor graph grammars (FGGs), a powerful formalism capable of describing a wide range of models.
Recent research found it beneﬁcial to use large state spaces for HMMs and PCFGs.
However, inference with large state spaces is computationally demanding, especially for PCFGs.
To tackle this challenge, we leverage tensor rank decomposition (aka.
CPD) to decrease inference computational complexities for a subset of FGGs subsuming HMMs and PCFGs.
We apply CPD on the factors of an FGG and then construct a new FGG deﬁned in the rank space.
Inference with the new FGG produces the same result but has a lower time complexity when the rank size is smaller than the state size.
We conduct experiments on HMM language modeling and unsupervised PCFG parsing, showing better performance than previous work.
Our code is publicly available at https://github.com/
VPeterV/RankSpace-Models .
Introduction
Hidden Markov Models (HMMs) and Probabilistic
Context-Free Grammars (PCFGs) are widely used structured models in natural language processing.
They can both be represented as factor graph grammars (FGGs) (Chiang and Riley, 2020), which are a powerful tool to describe a wide range of models, allowing exact and tractable inference in most situations of interest. Over-parameterization has been shown beneﬁcial in facilitating optimization of deep networks (Arora et al., 2018; Xu et al., 2018; Du et al., 2019). Buhai et al.
(2020) found that overparameterization is also helpful in learning latent variable models by increasing the number of hidden states.
Buhai et al. (2020) argued that it is important to study over-parameterization in structured settings because structured latent variable models are more suitable to model real-word phenomena which exhibit complex dependencies. HMMs and PCFGs are typical structured latent variable models, and recently researchers have found it beneﬁcial to use large state spaces for HMMs and PCFGs (Dedieu et al., 2019; Chiu and Rush, 2020;
Yang et al., 2021b; Chiu et al., 2021).
However, structured inference with large state spaces is computationally demanding, especially for PCFGs, pushing researchers to develop methods to decrease the computational complexities.
Chiu and Rush (2020) propose a neural VL-HMM with 2 15 states for language modeling, narrowing down the performance gap between HMMs and LSTMs. They follow
Dedieu et al. to impose a strong sparsity constraint (i.e., each hidden state can only generate a small subset of terminal symbols) to decrease the time complexity of the forward algorithm, thus requiring pre-clustering of terminal symbols.
Yang et al.
(2021b) use a large state space for neural PCFG induction and achieve superior unsupervised constituency parsing performance. They follow Cohen et al.
to use tensor rank decomposition (aka.
canonical-polyadic decomposition (CPD) (Rabanser et al., 2017)) to decrease the computational complexity of the inside algorithm, but only scale the state size from tens to hundreds because the resulting complexity is still high.
Chiu et al.
(2021) use tensor matricization and low-rank matrix decomposition to accelerate structured inference on chain and tree structure models.
However, their method has an even higher complexity than
Yang et al.
(2021b) on PCFGs.
Recently, Fu and
Lapata (2021) propose a family of randomized dynamic programming algorithms to scale structured models to tens of thousands of states, which is orthogonal to the aforementioned low-rank-based approaches as the former performs approximate inference whereas the latter perform exact inference.
In this work, we propose a new low-rank-based approach to scale structured inference, which can be described by FGG notations intuitively.
We ﬁrst provide an intuitive and unifying perspective toward the work of Yang et al.
(2021b)
and Chiu et al.
(2021), showing that their low-rank decompositionbased models can be viewed as decomposing large factors in an FGG—e.g., the binary rule probability tensor in PCFGs— into several smaller factors connected by new “rank” nodes.
Then we target at a subset of FGGs—which we refer to as B-FGGs— subsuming all models considered by Chiu et al.
(2021), whereby the inference algorithms can be formulated via B-graphs (Gallo et al., 1993; Klein and Manning, 2001).
We propose a novel framework to support a family of inference algorithms in the rank space for B-FGGs.
Within the framework, we apply CPD on the factors of a B-FGG and then construct a new B-FGG deﬁned in the rank space by marginalizing all the state nodes.
Inference with the new B-FGG has the same result and a lower time complexity if the rank size is smaller than the state size.
We conduct experiments in unsupervised PCFG parsing and HMM language modeling.
For PCFG induction, we manage to use 20 times more hidden states than Yang et al.
(2021b)
, obtaining much better unsupervised parsing performance.
For HMM language modeling, we achieve lower perplexity and lower inference complexity than Chiu et al.
Background 2. Factor graph grammar
Factor graphs are ﬁxed-sized and thus incapable of modeling substructures that repeat a variable number of times.
Chiang and Riley (2020) propose factor graph grammars (FGGs) to overcome this limitation, which are expressive enough to subsume HMMs and PCFGs.
The main purpose of introducing FGGs in this work is to facilitate more intuitive presentation of our method, and to enable generalization beyond HMMs and PCFGs.
2.1. Basics
We display necessary notations and concepts of
FGGs (Chiang and Riley, 2020, Def. 1,2,5,6,8).
Deﬁnition 1.
A hypergraph is a tuple (cid:0) V, E, att, lab V , lab E (cid:1) where
• V and E are ﬁnite set of nodes and hyperedges.
• att : E → V (cid:63) maps each hyperedge to zero or more (not necessarily distinct) endpoint nodes.
• lab V : V → L V assigns labels to nodes.
• lab E : E → L E assigns labels to edges.
Deﬁnition 2.
A factor graph is a hypergraph with mappings Ω and F where
• Ω maps node labels to sets of possible values.
Ω( v ) (cid:44) Ω( lab V ( v )) .
• F maps edge labels to functions.
F ( e ) (cid:44) F ( lab E ( e )) is of type Ω( v 1 ) × · · · × Ω( v k ) where att ( e ) = v 1 · · · v k .
In the terminology of factor graphs, a node v with its domain Ω( v ) is a variable , and an hyperedge e with F ( e ) is a factor .
We typically use T , N , O to denote hidden state, nonterminal state and observation variables for HMMs and PCFGs.
Deﬁnition 3.
A hypergraph fragment is a tuple ( V, E, att, lab V , lab E , ext ) where
• ( V, E, att, lab V , lab E ) is a hypergraph.
• ext ∈ V (cid:63) is a set of zero or more external nodes and each of which can be seen as a
connecting point of this
hypergraph fragment with another fragment.
Deﬁnition 4.
A hyperedge replacement graph
grammar (HRG) (Drewes et al., 1997) is a tuple ( N, T, P, S )
where
• N, T ⊂
L E is ﬁnite set of nonterminal and terminal symbols.
N ∩ T = ∅ .
• P is a ﬁnite set of rules ( X → R ) where X ∈ N and R is a hypergraph fragment with edge labels in N ∪ T 1 .
• S ∈ N is the start symbol.
Deﬁnition
A HRG with mapping Ω , F (Def.
2) is referred to as an FGG.
In particular, F is deﬁned on terminal edge labels T only.
Notations.
• N : variable node.
N : external node.
• X e : hyperedge e with label X ∈ N .
indicates zero or more endpoint nodes.
S π 1 −→
T 1 T 1 = bos X 2 , (0) T X , ( i − 1) π 2 −→ T
T
Op ( T 2 | T 1 ) p ( O 3 | T 2 )
X 4 , ( i )
O 3 = w i −
T
X , ( n ) π 3 −→ T
T 2 p ( T 2 | T 1 )
T 2 = eos (a)
S (cid:48)
π 4 −→
N
X 2 , (0 ,n )
N
X , ( l − 1 ,l ) π 5 −→
N
Op ( N 1 → O 2 )
O 2 = w l −
N
X , ( i,j ) π 6 −→
N
N
Np ( N 1 → N 2 N 3 )
X 4 , ( i,k )
X 5 , ( k,j ) (b)
Fig.
1 illustrates HGG representations of HMM and PCFG.
Generative story.
An FGG starts with S , repeatedly selects
X e and uses rule X → R from P to replace e with R , until no
X e exists.
2.1. Conjunction
The conjunction operation (Chiang and Riley, 2020,
Sec. 4) allows modularizing an FGG into two parts, one deﬁning the model and the other deﬁning a query.
In this paper, we only consider querying the observed sentence w 0 , · · · , w n − 1 , which is exempliﬁed by the red part of Fig.
We sometimes omit the red part without further elaboration.
2.1. Inference
Denote ξ as an assignment of all variables, Ξ D as the set of all assignments of factor graph D , and
D ( G ) as the set of all derivations of an FGG G , i.e., all factor graphs generated by G .
an FGG G assigns a score w G ( D, ξ ) to each D ∈ D ( G ) along with each ξ ∈ Ξ D .
A factor graph D ∈ D ( G ) assigns a score w D ( ξ ) to each ξ ∈ Ξ D : w D ( ξ ) = (cid:89)
[Equation] [Equation]
[Equation]
(1) with att ( e ) = v 1 · · · v k .
Notably, w D ( ξ ) (cid:44) w G ( D, ξ ) .
The inference problem is to compute the sum-product of G :
[Equation]
To obtain Z G , the key difﬁculty is in the marginalization over all derivations, since (cid:80) ξ ∈
Ξ D w D ( ξ ) can be obtained by running standard variable elimination (VE) on factor graph D .
To tackle this,
Chiang and Riley (2020, Thm. 15) propose an extended VE.
For each X ∈ N, ξ ∈ Ξ X 2 , deﬁne P X as all rules in P with left-hand side X , and then deﬁne:
[Equation]
for each rhs
R
[Equation]
E T , att, lab V , lab E , ext ) , where E N , E T consist of nonterminal/terminal-labeled edges only, and τ R ( ξ ) is given by:
[Equation]
This deﬁnes a recursive formula for computing ψ S , i.e., Z G .
Next, we will show how Eq.
3recover the well-known inside algorithm.
Example: the inside algorithm.
Consider π 6 in
Fig.1(b).
All possible fragments R (rhs of π 6 ) differs in the value of k , i.e., the splitting point, so we use R k to distinguish them.
Then Eq. 3 becomes:
[Equation]
Putting values into Eq. 4:
[Equation]
where p denotes FGG rule probability p ( N 1 →
N 2 N 3 ) .
It is easy to see that ψ X i,k is exactly the inside score of span [ i, k ) , and Eq.
5-6 recovers the recursive formula of the inside algorithm.
Remark.
Eq.
4 can be viewed as unidirectional (from e ∈ E N to external nodes) belief propagation (BP) in the factor graph fragment R , where the incoming message is ψ lab E ( e ) for e ∈ E N , and the outcome of Eq.
4 can be viewed as the message passed to the external nodes.
The time complexity of message updates grows exponentially with the number of variables in the factors.
Therefore, to decrease inference complexity, one may decompose large factors into smaller factors connected by new nodes, as shown in the next subsection.
2. Tensor rank decomposition on factors
Consider a factor F ( e ) (Def. 2), it can be represented as an order- k tensor in R N 1 ×···× N k where
N i (cid:44) | Ω( v i ) | .
We can use tensor rank decomposition (aka. CPD) to decompose F ( e ) into a weighted sum of outer products of vectors:
[Equation]
where r is the rank size; w qe k ∈ R N k ; ⊗ is outer product; λ q is weight, which can be absorbed into { w qe k } and we omit it throughout the paper.
Dupty and Lee (2020, Sec. 4.1) show that BP can be written in the following matrix form when applying CPD on factors:
[Equation]
where m ei ∈ R N i is factor-to-node message; n ie ∈ R
N i is node-to-factor message; N ( · ) indicates neighborhood ; W e [Equation]
R r × m ; (cid:12) is element-wise product.
We remark that this amounts to replacing the large factor F ( e ) with smaller factors { F ( e i ) } connected by a new node
R that represents rank, where each F ( e i ) can be represented as W e i .
Fig.
2 illustrates this intuition.
We refer to R as rank nodes and others as state nodes thereafter. vvv .. v k F ( e ) → vvv .. v k R
F ( e 1 )
F ( e 2 ) F ( e k ) [Figure]  Low-rank structured inference
In this section, we recover the accelerated inside algorithms of TD-PCFG (Cohen et al., 2013; Yang et al., 2021b) and LPCFG (Chiu et al., 2021) in an intuitive and unifying manner using the FGG notations.
The accelerated forward algorithm of
LHMM (Chiu et al., 2021) can be derived similarly.
Denote T ∈ R m × m × m as the tensor representation of p ( N 1 → N 2 N 3 ) , and α i,j ∈ R m as the inside score of span [ i, j ) .
Cohen et al.
(2013) and Yang et al.
(2021b) use CPD to decompose T , i.e., let T =
(cid:80) rq =1 u q ⊗ v q ⊗ w q where u q , v q , w q ∈ R m .
Denote U , V , W ∈ R r × m as the resulting matrices of stacking all u q , v q , w q ,
Cohen et al.
(2013) derived the recursive form:
[Equation]
Eq.
9 can be derived automatically by combining
Eq.
7 (or Fig.
3 (a)) and Eq.
Cohen et al.
(2013) note that U T can be extracted to the front of the summation (Eq. 10), and V α
i,k , W α k,j can be cached and reused, leading to further complexity reduction.
The resulting inside algorithm time complexity is O ( n 3 r + n 2 mr ) .
Recently, Chiu et al.
(2021) use low-rank matrix decomposition to accelerate PCFG inference.
They ﬁrst perform tensor matricization to ﬂatten T to
T (cid:48)
∈
R m × m 2 , and then let T (cid:48)
=
U T V where
U ∈ R r × m , V ∈ R r × m 2 .
By un-ﬂattening V to
V (cid:48)
∈ R r × m × m , their accelerated inside algorithm has the following recursive form:
[Equation]
Eq.
11 can be derived by combining Fig.
3 (b) and
Eq.
The resulting inside time complexity is
O ( n 3 m 2 r + n 2 mr ) , which is higher than that of
TD-PCFG.
When learning a PCFG and a HMM, there is no need to ﬁrst learn T and then perform decomposition on T .
Instead, one can learn the decomposed matrices (e.g., U , V ) to learn T implicitly.
During inference, one can follow Eq.
10 or 12 without the need to reconstruct T .
Validity of probability.
The remaining problem is to ensure that T is a valid probability tensor (i.e., being nonnegative and properly normalized) when learning it implicitly.
Yang et al. (2021b) essentially transform Fig. 3(a) into a Bayesian network, adding directed arrows N 1 → R , R → N 2 , R →
N 3 .
This is equivalent to requiring that V , W are nonnegative and column-wise normalized and U is nonnegative and row-wise normalized, as described in Yang et al.
(2021b, Thm. 1). One can apply the Softmax re-parameterization to enforce such requirement, which is more convenient in endto-end learning. Chiu et al. (2021) replace the local normalization of Yang et al.
(2021b)
with global normalization, and we refer readers to their paper for more details.
We adopt the strategy of Yang et al.
(2021b) in this work.
Rank-space modeling and inference 4. Rank-space inference with B-FGGs
Interestingly, when applying CPD on factors and if the rank size is smaller than the state size, we can even obtain better inference time complexities for a subset of FGGs which we refer to as B-FGGs.
We call a hyperedge a B-edge if its head contains exactly one node.
B-graphs (Gallo et al., 1993) are a subset of directed hypergraphs whose hyperedges are all B-edges.
Many dynamic programming algorithms can be formulated through B-graphs (Klein and Manning, 2001; Huang, 2008; Azuma et al., 2017; Chiu et al., 2021; Fu and Lapata, 2021), including the inference algorithms of many structured models, e.g., HMMs, Hidden Semi-Markov
Models (HSMMs), and PCFGs.
We follow the concept of B-graphs to deﬁne B-FGGs.
Deﬁnition 6.
A hypergraph fragment is a Bhypergraph fragment iff.
there is exactly one external node and there is no nonterminal-labeled hyperedge connecting to it.
An FGG is a B-FGG iff.
all rhs of its rules are B-hypergraph fragments.
It is easy to see that the aforementioned models are subsumed by B-FGGs.
We can design a family of accelerated inference algorithms for B-FGGs based on the following strategy.
(1) If there are multiple factors within a hypergraph fragment, merge them into a single factor.
Then apply CPD on the single factor, thereby introducing rank nodes.
Find repeated substructures that take rank nodes as external nodes.
Marginalize all state nodes to derive new rules.
(3) Design new inference algorithms that can be carried out in the rank space based on the general-purpose FGG inference algorithm and the derived new rules.
We give two examples, the rank-space inside algorithm and the rank-space forward algorithm, in the following two subsections to help readers understand this strategy.
4. The rank-space inside algorithm
Consider an B-FGG G shown in Fig.1(b) and replace the rhs of π 6 with Fig. 3(a), i.e., we use
CPD to decompose binary rule probability tensor.
Besides U , V , W ∈ R r × m deﬁned in Sec.
3, we deﬁne the start rule probability vector as s ∈ R m × 1 , and the unary rule probability matrix as E ∈ R o × m where o is the vocabulary size.
Fig. 4(a) is an example (partial) factor graph D generated by G .
We highlight substructures of interest with dashed rectangles.
Each substructure consists of a node N and two factors connecting to it.
N is an external node connecting two hypergraph fragments which contain the two factors respectively.
For each substructure, we can marginalize the state node N out, merging the two factors into a single one.
After marginalizing all state nodes, we obtain a (partial) factor graph D (cid:48) shown in the right of Fig. 4(a)
where
H =
VU T , I = WU T , J = VE T , K = WE T , L = ( Us ) T .
We denote this transformation as M ( D ) =
D (cid:48) . It is worth men- N R
N
N
R N N N 7 N
O
O
R
R
R
R
R
R
R
O
O
R...
L
H
I
H
K
J
I ...
(a)
R
R
R
H
I
X 4 , ( i,k )
X 5 , ( k,j )
R
R
O
H
K
X 4 , ( i,j − 1) O 3 = w j −
R
L
X 2 , (0 ,n )
R
X , ( i,j )
S ππππππ
R
O
R
J
I
O 2 =
w i X 5 , ( i + 1 ,j )
R
O
O
J
K
O 2 = w i
O 3 = w i +
O
O 1 = w(b) tioning that H , I , J , K , L are computed only once and then reused multiple times during inference, which is the key to reduce the time complexity.
Then we deﬁne a new B-FGG G (cid:48) with rules shown in Fig 4(b).
It is easy to verify that for each D ∈ D ( G ) , we have M ( D ) ∈ D ( G (cid:48) ) , and vice versa.
Moreover, we have:
[Equation]
( M ( D ) , ξ ) because marginalizing hidden variables does not affect the result of sum-product inference.
Therefore,
Z G =
Z G (cid:48)
(Eq. 2).
We can easily derive the inference (inside) algorithm of G (cid:48) by following Eq.
3-4 and Fig. 4(b) 3 .
Let α i,j ∈ R r denote the rank-space inside score for span [ i, j ) .
When j > i + 2 : α i,j = from π 1 of Fig. 4(b) (cid:122)
[Equation]
(cid:12) I α i +1 ,j (cid:124)
[Equation]
from
[Equation]
H α
i [Equation] (cid:12) K : [Equation]
w
[Equation]
(cid:124) (cid:123)(cid:122) (cid:125) from πand when j = i + 2 , α i,j = J : ,w i (cid:12) K : ,w i +1 (from π 4 ) .
w j is the index of the j -th word of the input sentence in the vocabulary; A : ,j indicates the j -th column of A .
We note that, similar to Cohen et al.
(2013), we can cache H α i,k , I α k,j and reuse them to further accelerate inference 4 .
Denote α Li,j , α Ri,j ∈ R r as the inside scores of span [ i, j ) serving as a left/right child of a larger span.
Then we have: α Li,i +1 =
K : ,i α Ri,i +1 = J : ,i α Li,j = H α i,j
[Equation]
and ﬁnally, Z G (cid:48)
=
L α 0 ,n .
We minimize − log Z G (cid:48) using mini-batch gradient descent for unsupervised learning.
The resulting inference complexity is
O ( n 3 r + n 2 r 2 ) 5 , which is lower than O ( n 3 r + n 2 mr ) of TD-PCFG when r < m , enabling the use of a large state space for PCFGs in the low-rank setting.
The key difference between the rank-space inference and the original state-space inference is that they follow different variable elimination orders.
The former marginalizes all state nodes before performing inference and marginalizes rank nodes from bottom up during inference; whereas the later marginalizes both state and rank nodes alternately from bottom up during inference.
Parsing. Low-rank inference does not support the Viterbi semiring 6 , inhibiting the use of CYK T T
Op ( T 2 | T 1 ) p ( O 3 | T 2 ) X T R
T
O
U
V
W
X(a)
T 1 T 1 = bos
R
T
O
R
T
O
R
T
O T 3 = eos (b) R R
R
O
O
O(c) decoding.
Therefore, we follow Yang et al.
(2021b) to use Minimum Bayes-Risk decoding (Goodman, 1996).
Speciﬁcally, we estimate the span marginals using auto-differentiation (Eisner, 2016; Rush, 2020), which has the same complexity as the inside algorithm.
Then we use the CYK algorithm to ﬁnd the ﬁnal parse with the maximum number of expected spans in O ( n 3 ) time, similar to Smith and Eisner (2006). Implementation.
The implementation of the inside algorithm greatly inﬂuences the actual running speed.
First, O ( n 2 ) out of O ( n 3 ) can be computed in parallel using parallel parsing techniques (Yi et al., 2011; Canny et al., 2013; Zhang et al., 2020; Rush, 2020). In this work, we adapt the efﬁcient implementation of Zhang et al. (2020) for fast inside computation. Second, we adopt the log-einsum-exp trick (Peharz et al., 2020) to avoid expensive log-sum-exp operations on high-dimensional vectors, which reduces both GPU memory usage and total running time.
4. The rank-space forward algorithm
Consider an B-FGG G shown in Fig.
1 (a).
We replace the rhs of π 2 by the hypergraph fragment in the right of Fig. 5(a), i.e., we merge the factor p ( T 2 | T 1 ) and p ( O 3 | T 2 ) into a single factor, which can be represented as T ∈ R m × m × o and can be decomposed into three matrices U , V ∈
R r × m , W ∈ R r × o via CPD, where m/o/r is the state/vocabulary/rank size.
Fig.
5(b) gives an example factor graph of HMMs with sentences of length 3.
Similar to previous subsection, we marginalize state nodes T to construct a new B-FGG G (cid:48) .
The rule set of G (cid:48) can be obtained by replacing all variable nodes T with R and modifying all factors accordingly, as one can easily infer from Fig.
5(c).
Inference with G (cid:48) simply coincides with the forward algorithm, which has a O ( nr 2 ) time complexity and is lower than O ( nmr ) of LHMM (Chiu et al., 2021)
when r < m .
4. Neural parameterization
We use neural networks to produce probabilities for all factors, which has been shown to beneﬁt learning and unsupervised induction of syntactic structures (Jiang et al., 2016; He et al., 2018; Kim et al., 2019; Han et al., 2019; Jin et al., 2019; Zhu et al., 2020; Yang et al., 2020, 2021b; Zhao and Titov, 2020; Zhang et al., 2021; Chiu and Rush, 2020; Chiu et al., 2021; Kim, 2021).
We use the neural parameterization of Yang et al.
(2021b)
with slight modiﬁcations.
We show the details in Appd.
A and Appd.
B.
Experiments 5. Unsupervised parsing with PCFGs Setting.
We evaluate our model on Penn Treebank (PTB) (Marcus et al., 1994).
Our implementation is based on the open-sourced code of Yang et al.
(2021b)[-
-]
and we use the same setting as theirs.
For all experiments, we set the ratio of nonterminal number to the preterminal number to 1:2 8 which is the common practise.
We set the rank size to 1000.
We show other details in Appd.
C and D.
Main result.
Table 1 shows the result on PTB.
Among previous unsupervised PCFG models, TNPCFG (Yang et al., 2021b) uses the largest number of states (500 perterminals and 250 nonterminals).
Our model is able to use much more states thanks to our new inside algorithm with lower time complexity, surpassing all previous PCFG-based models by a large margin and achieving a new state-of-the-art in unsupervised constituency parsing in terms of sentence-level F1 score on PTB.
Model
S-F
N-PCFG (Kim et al., 2019)
C-PCFG (Kim et al., 2019) 55.
NL-PCFG (Zhu et al., 2020) 55.
TN-PCFG (Yang et al., [Table] NBL-PCFG (Yang [Table]
al.,
[Table]
Ours with 9000 PTs and
[Table]
For reference
Constituency test (Cao et al., 2020) 62.
S-DIORA
[Table]
Drozdov et al., 2020) 57.
StructFormer (Shen et al., 2021) 54.
DIORA+span constraint (Xu et al., 2021) 61.
[Figure]
e x it y
[Figure]
Ablation study.
Fig.
6 shows the change of the sentence-level F1 scores and perplexity with the change of the number of preterminals.
As we can see, when increasing the state, the perplexity tends to decrease while the F1 score tends to increase, validating the effectiveness of using large state spaces for neural PCFG induction.
5. HMM language modeling Setting.
We conduct the language modeling experiment also on PTB.
Our implementation is based
Model [Table] [Table] on the open-sourced code of Chiu et al.
We set the rank size to 4096.
See Appd.
C and D for more details.
Main result.
Table 2 shows the perplexity on the
PTB validation and test sets.
As discussed earlier, VL-HMM (Chiu and Rush, 2020) imposes strong sparsity constraint to decrease the time complexity of the forward algorithm and requires preclustering of terminal symbols.
Speciﬁcally, VLHMM uses Brown clustering (Brown et al., 1992), introducing external information to improve performance.
Replacing Brown clustering with uniform clustering leads to a 10 point increase in perplexity on the PTB validation set.
LHMM (Chiu et al., 2021) and our model only impose low-rank constraint without using any external information and are thus more comparable.
Our method outperforms LHMM by 4 . 8 point when using the same state number (i.e., 2 14 ), and it can use more states thanks to our lower inference time complexity.
Ablation study.
As we can see in Table 3, the perplexity tends to decrease when increasing the state number, validating the effectiveness of using more states for neural HMM language modeling. Discussion. It is interesting to note that our
HMM model is roughly equivalent to another
HMM with interchanged rank and state sizes as can be seen in Fig.5(c).
To verify this equivalence, we run LHMM in the original state space with 2048 states and rank 2 15 . The resulting perplexity is 133.49 on average on the PTB test set, which is worse than that of ours (126.4). We leave further experimentation and analyses of this discrepancy for future work.  Related work
Tensor and matrix decomposition have been used to decrease time and space complexities of probabilistic inference algorithms.
Siddiqi et al.
(2010) propose a reduced-rank HMM whereby the forward algorithm can be carried out in the rank space, which is similar to our model, but our method is more general.
Cohen and Collins (2012); Cohen et al.
(2013) use CPD for fast (latent-variable) PCFG parsing, but they do not leverage CPD for fast learning and they need to actually perform CPD on existing probability tensors.
Rabusseau et al.
(2016) use low-rank approximation method to learn weighted tree automata, which subsumes PCFGs and latentvariable PCFGs.
Our method can subsume more models.
Yang et al.
(2021b,a) propose CPD-based neural parameterizations for (lexicalized) PCFGs.
Yang et al.
(2021b) aim at scaling PCFG inference.
We achieve better time complexity than theirs and hence can use much more hidden states.
Yang et al.
(2021a) aims to decrease the complexity of lexicalized PCFG parsing, which can also be described within our framework.
Chiu et al.
(2021) use low-rank matrix decomposition, which can be viewed as CPD on order-2 tensors, to accelerate inference on chain and tree structure models including HMMs and PCFGs.
However, their method is only efﬁcient when the parameter tensors are of order 2, e.g., in HMMs and HSMMs.
Our method leverages full CPD, thus enabling efﬁcient inference with higher-order factors, e.g., in PCFGs.
Our method can be applied to all models considered by Chiu et al.
(2021), performing inference in the rank-space with lower complexities.
Besides HMMs and PCFGs, Wrigley et al.
(2017) propose an efﬁcient sampling-based junction-tree algorithm using CPD to decompose high-order factors.
Dupty and Lee (2020) also use
CPD to decompose high-order factors for fast belief propagation.
Yang and Tu (2022) use CPD to decompose second-order factors in semantic dependency parsing to accelerate second-order parsing with mean-ﬁeld inference.
Besides CPD, Ducamp et al.
(2020) use tensor train decomposition for fast and scalable message passing in Bayesian networks.
Bonnevie and Schmidt (2021) leverage matrix product states (i.e., tensor trains) for scalable discrete probabilistic inference.
Miller et al.
(2021) leverage tensor networks for fast sequential probabilistic inference.
Conclusion and future work
In this work, we leveraged tensor rank decomposition (CPD) for low-rank scaling of structured inference.
We showed that CPD amounts to decomposing a large factor into several smaller factors connected by a new rank node, and gave a unifying perspective towards previous low-rank structured models (Yang et al., 2021b; Chiu et al., 2021).
We also presented a novel framework to design a family of rank-space inference algorithms for B-FGGs, a subset of FGGs which subsume most structured models of interest to the NLP community.
We have shown the application of our method in scaling
PCFG and HMM inference, and experiments on unsupervised parsing and language modeling validate the effectiveness of using large state spaces facilitated by our method.
We believe our framework can be applied to many other models which have high inference time complexity and are subsumed by B-FGGs, including lexicalized PCFGs, quasi-synchronous contextfree grammars (QCFGs), etc.
A direct application of our method is to decrease the inference complexity of the neural QCFG (Kim, 2021)
Acknowledgments
This work was supported by the National Natural
Science Foundation of China (61976139).
A Neural parameterization of PCFGs
In this section, we give the full parameterization of PCFGs.
We follow Yang et al.
(2021b) with slight modiﬁcations for generations of U , V , W ∈
R m × r in 4.2.
We use the same MLPs with two residual layers as Yang et al.
(2021b):
[Equation]
where Σ is the vocabulary set, H is the set of rank, N is a ﬁnite set of nonterminals,
[Equation]
The main differences of neural parameterization between ours and previous work are that we make the projection parameter u H shared among
U , V , and U .
B Neural parameterization of HMMs
In this section, we give the full parameterization of
HMMs, which is similar to PCFGs’ parameterization.
Deﬁne s as start probability for HMMs.
And the deﬁnitions of U , V , W are same as deﬁnitions in 4 . 3 :
[Equation]
where S is a ﬁnite set of states, H is the set of rank, Σ is vocabulary set.
C Data details
Penn Treebank (PTB) (Marcus et al., 1994) 10 consists of 929k training words, 73k validation words, and 82k test words, with a vocabulary of size 10k.
For PCFGs, we follow Yang et al.
(2021b)
and use their code to preprocess dataset.
This processing discards punctuation and lowercases all tokens with 10k most frequent words as the vocabulary.
The splits of the dataset are: 2-21 for training,for validation and 23 for test.
For HMMs, we follow Chiu et al.
and use their code to preprocess dataset.
We lowercase all words and substitutes OOV words with UNKs.
EOS tokens have been inserted after each sentence.
D Experimental details
For PCFGs, we use Xavier normal initialization to initialize the weights in h i and f i .
We optimize our model using Adam optimizer with β 1 = 0 .
75 , β 2 = 0 .
999 , and the learning rate 0 .
002 , setting the dimension of all embeddings to 256 .
For HMMs, we initialize all parameters by
Xavier normal initialization except for w s and w w .
We use AdamW optimizer with β 1 = 0 .
99 , β 2 = 0 .
999 , and the learning rate 0 .
001 , and a max grad norm of 5 .
We use dropout rate of 0 . 1 to dropout w s and U , V in HMMs.
We train for 30 epochs with a max batch size of 256 tokens, and reduce the learning by multiplying 12 if the validation perplexity fails to improve after 2 evaluations.
Evaluations are performed one time per epoch.
We follow
Chiu et al.
(2021) to shufﬂe sentences and leverage bucket iterator, where batch of sentences are drawn from buckets containing sentences of similar lengths to minizing padding.
We run all experiments on NVIDIA TITAN RTX and NVIDIA RTX 2080ti and all experimental results are averaged from four runs.