On the Uncomputability of Partition Functions In Energy-Based Sequence Models
Abstract
In this paper, we argue that energy-based sequence models backed by expressive parametric families can result in uncomputable and inapproximable partition functions.
Among other things, this makes model selection — and therefore learning model parameters — not only diﬃcult, but generally undecidable .
The reason is that there are no good deterministic or randomized estimates of partition functions.
Speciﬁcally, we exhibit a pathological example where under common assumptions, no useful importance sampling estimates of the partition function can guarantee to have variance bounded below a rational number.
As alternatives, we consider sequence model families whose partition functions are computable (if they exist), but at the cost of reduced expressiveness.
Our theoretical results suggest that statistical procedures with asymptotic guarantees and sheer (but ﬁnite) amounts of compute are not the only things that make sequence modeling work; computability concerns must not be neglected as we consider more expressive model parametrizations.
Introduction
Modeling discrete sequences is central to natural language processing and bioinformatics.
Many common parametric sequence models ˜ 𝑝 are (or can be cast as) energy-based (LeCun et al., 2006): they yield a weight ˜ 𝑝 ( 𝒙 ) for any given string 𝒙 .
Although energy-based models (EBMs) of sequences need not represent probability distributions, they have been proposed as such to counter the inexpressivity of autoregressive sequence models (Bakhtin et al., 2021; Lin et al., 2021, §4.1).
EBMs with ﬁnite partition functions 𝑍 ˜ 𝑝 (cid:44) (cid:205) 𝒙 ˜ 𝑝 ( 𝒙 ) ∈ R > 0 deﬁne distributions 𝑝 ( 𝒙 ) (cid:44) ˜ 𝑝 ( 𝒙 ) / 𝑍 ˜ 𝑝 over strings, which are often queried under the principle of probabilistic inference (Ghahramani, 2015).
Energy-based sequence models are often parametrized as neural models.
Each parameter vector 𝜽 in some parametric neural network family 𝚯 ⊆ R 𝑑 identiﬁes a parametric model ˜ 𝑝 𝜽 , which then deﬁnes a parametric distribution over strings 𝑝 𝜽 , assuming 𝑍 ˜ 𝑝 𝜽 exists (Chen et al., 2018).
Contemporary neural networks have been shown to be very powerful.
In particular, some popular parametric neural sequence model families, such as RNNs (Siegelmann & Sontag, 1995) and Transformers (Bhattamishra et al., 2020; Pérez et al., 2021), have been formally shown to recognize recursively enumerable languages: given any Turing machine 𝑀 , there is a parameter vector 𝜽 ∈ 𝚯 such that the parametrized sequence model 𝑁 𝜽 recognizes the same language as 𝑀 does.
In other words, these sequence model families are
Turing-complete .
It is therefore intuitive that energy-based sequence models, backed by such powerful neural networks, form expressive families of string distributions (§3.1): for any decision algorithm 𝑀 , there is a parameter vector 𝜽 𝑀 in a Turing-complete parametric family 𝚯 , such that 𝑝 𝜽 𝑀 exists, and 𝑝 𝜽 𝑀 ( 𝒙 ) is high if and only if 𝑀 accepts 𝒙 .
It would seem assuring to work with such expressive family of sequence distributions, 𝚯 : assuming the true string probabilities indeed can be computed in polytime, 𝚯 is well-speciﬁed.
Moreover, one may assume that given we can sample from 𝑝 𝜽 𝑀 , we would be able to use consistent estimators to ﬁnd 𝜽 (cid:48) ∈ 𝚯 , where 𝜽 (cid:48)
≈ 𝜽 𝑀 .
Unfortunately, we ﬁnd that with such an expressive distribution family 𝚯 , whether the identiﬁability assumption holds — required for most consistent estimators — itself is undecidable (Turing, 1937).
Moreover, model selection on any held-out data is also undecidable.
Even worse, we show that there exists a vector 𝜽 (cid:48) ∈ 𝚯 , such that as long as a parametric family 𝚯 (cid:48) ⊆ 𝚯 𝜽 (cid:48) , model selection will not be possible for 𝚯 (cid:48) either — even when 𝚯 (cid:48) itself is not necessarily expressive ( e.g. , 𝚯 (cid:48) can be ﬁxed-size Transformer EBMs, which cannot parametrize all EBMs that require more parameters).
We construct one such ‘pathological’ distribution 𝑝 𝜽 (cid:48) as a Transformer EBM.
These negative results stem from the uncomputability and inapproximability of 𝑍 .
A main technical contribution of this paper is that there is no algorithm (either deterministic or randomized) that can approximate 𝑍 well.
An immediate consequence is that sampling-based statistical procedures are not useful in this scenario, either, as long as they terminate in ﬁnite time. However, we will see that for less expressive model families, such uncomputability issues do not arise.
Our negative results are summarized in Table A.1.
To ensure that model selection is still possible (such that we can compare diﬀerent parameter vectors with conﬁdence), 2 we have no choice but to resort to less expressive string distributions.
The paper is structured as follows.
In §2 we review deﬁnitions and known results of weighted languages, sequence model families, and formalize weighted Turing machines, EC-complete parametric families, and computable estimates.
In §§3–5 we describe our main technical results: there exist pathological EBM sequence models that have uncomputable partition functions, which cannot be approximated well under stochastic estimates, and do not have asymptotic estimates that have any good guarantees.
In §6 we that argue our negative results make model selection impossible for expressive model families, and discuss why common estimation methods fail.
Finally, we discuss three ‘palliative’ parametrization choices in §7, which all guarantee computable partition functions, at the cost of expressiveness.
Background 2.1 Energy-based sequence models
Energy-based models (EBMs) of sequences (LeCun et al., 2006; Rosenfeld et al., 2001; Sandbank, 2008; Huang et al., 2018;
Bakhtin et al., 2021) are a family of discrete sequence models.
Given a string 𝒙 ∈ 𝑉 ∗ over a ﬁnite vocabulary 𝑉 , an EBM ˜ 𝑝 computes string weight ˜ 𝑝 ( 𝒙 ) , but not the (normalized) string probability 𝑝 ( 𝒙 ) =
˜ 𝑝 ( 𝒙 ) / (cid:205) 𝒙 ˜ 𝑝
( 𝒙 ) .
In this work, we focus on EBMs whose weight is eﬃciently computable, i.e. , polytime in the length of 𝒙 .
Previous work Bakhtin et al.
(2021); Lin et al.
(2021) showed that EBMs deﬁne expressive distributions; but this requires normalization.
While EBMs are often intractable to normalize ( e.g. , the Ising model), the inﬁnite domain of ﬁnite strings opens the door to uncomputable probabilities, which invite the impossibility of model selection and comparison.
2.2 Weighted languages
EBMs give weights to strings.
Here we formally characterize these weighted strings as a weighted language .
An unweighted language 𝐿 ⊆
𝑉 ∗ is a set of ﬁnite strings 𝒙 over a ﬁnite vocabulary 𝑉 .
A weighted language ˜ 𝑝 is a function ˜ 𝑝 : 𝑉 ∗ → R ≥ 0 .
In this work, we discuss Boolean languages, such that 𝑉 = B (cid:44) { 0 , 1 } .
We also focus on weighted languages where distributions over strings exist.
Following
Lin et al.
(2021), we say such weighted languages ˜ 𝑝 are normalizable : 𝑍 ˜ 𝑝 (cid:44) (cid:205) 𝒙 ∈ B ∗ ˜ 𝑝 ( 𝒙 ) ∈ R > 0 .
𝑍 ˜ 𝑝 is also called the partition function of ˜ 𝑝 .
4 We can then normalize ˜ 𝑝 into a distribution 𝑝 over B ∗ such that 𝑝 ( 𝒙 ) =
˜ 𝑝 ( 𝒙 )/ 𝑍 ˜ 𝑝 , and thereby (cid:205) 𝒙 ∈ B ∗ 𝑝 ( 𝒙 )
= 1.
The eﬃciently computable (weighted) languages (EC; Lin et al., 2021) are those weighted languages ˜ 𝑝 , where a string’s weight (which must be non-negative) is a polytime function of the string.
Weighted languages deﬁned by most EBM sequence models fall into this class, since they score every string in ﬁnite time (and usually in polytime); and the scores ultimately are computed by some algorithm.
Intuitively, string weights of an EC language can be obtained as a side product from a weighted Turing machine that recognizes 𝒙 (in polytime).
However, Lin et al.
(2021) did not precisely describe how a Turing machine maps input string 𝒙 to its (rational) weight ˜ 𝑝 ( 𝒙 ) in ﬁnite time.
In this work, such a construction (out of many possible ones) is necessary, as we need to show that this string weighting can be done by parametric sequence model families (see Appendix B).
2.3 Locally normalized distributions
A popular subclass of weighted languages is locally normalized weighted languages (LN), where conditional local distributions given preﬁx ˆ 𝒙 : 𝑝 (· | ˆ 𝒙 ) can be computed in ﬁnite time.
Since they automatically deﬁne a distribution over strings, we use the term locally normalized distributions interchangeably.
If 𝑝 (· | ˆ 𝒙 ) can be computed in polynomial time, we call such distributions eﬃciently locally normalized distributions (ELN) — this is the weighted language class of most autoregressive models, just as its superset EC is the weighted language class of most energy-based sequence models.
Locally normalized distributions over sequences have ˜ 𝑝 ( 𝒙 ) = 𝑝 ( 𝒙 ) , and 𝑍 = (cid:205) 𝒙 ∈ B ∗ ˜ 𝑝 ( 𝒙 ) = 1.
They are consistent : the probability that a string is inﬁnitely long under such distributions is zero.
Equivalently, given any 𝜖 > 0, we can approximate 𝑍 with a ﬁnite sum of string weights:
Proposition 1 (Consistency of LN distributions (Booth & Thompson, 1973; Chen et al., 2018)) .
Let 𝑝 ∈ LN be a locally normalized distribution over strings.
All strings of a given length or longer have their probabilities bounded.
That is, for all positive real numbers 𝜖 , there is a length 𝑛 at which all strings 𝒙 of length at least 𝑛 have 𝑝 ( 𝒙 ) < 𝜖 .
The consistency property of locally normalized distributions implies that they have an exact sampling procedure that almost surely terminates in ﬁnite time.
Therefore, they (and in particular ELN distributions) are an attractive choice when we need to sample from the distribution they deﬁne, e.g. , in sampling-based parameter estimation procedures.
2.4 EC-complete parametric families
We have introduced energy-based sequence models (§2.1) and their characterization as weighted languages (§2.2).
However, we usually do not work with weighted Turing machines directly in machine learning.
Here the most common models of computation are (neural) sequence model families , such as RNNs and Transformers.
While these computational models can appear quite dissimilar to state-machine-based models of computation ( e.g. , Turing machines), they have been shown to possess the same computation power (Siegelmann & Sontag, 1995; Pérez et al., 2021).
That is, they are Turing-complete .
Just as we extend the deﬁnition of Turing machines to weighted Turing machines, we likewise formalize weighted sequence model families.
We thus introduce EC -complete parametric families as a sequence model counterpart of the weighted language class EC.
At a high level, a parametric family 𝚯 is
EC-complete if given any ˜ 𝑝 ∈ EC (as a description of a weighted Turing machine), we can construct a parameter vector 𝜽 ∈ 𝚯 ⊆
Q ∗ which deﬁnes ˜ 𝑝 ’s corresponding sequence model.
The model produces an output embedding which we then decide in polytime to be either a ‘halting embedding’ or not.
If it is, we can then extract the weight ˜ 𝑝 ( 𝒙 ) .
(A rigorous exposition is in Appendix C.)
We show that with a modiﬁcation to positional embeddings, the family of arbitrary precision, hard attention Transformers deﬁned in Pérez et al.
(2021) is EC-complete:
Theorem 1.
The class of one-encoder-layer, four-decoder-layer Transformer networks with positional encodings ( 𝑛, 1 / 𝑛 , 1 / 𝑛 2 , 2 𝑛 ) is EC -complete.
Proof sketch.
We extend the construction from Pérez et al.
, adding an additional layer to accumulate the string weight over time steps.
(cid:3)
2.5 Estimates
A main result of this work is that partition functions in an EC-complete family are uncomputable.
Moreover, randomness does not help estimation; and correct asymptotic estimates are not useful.
We deﬁne these estimates here in order to discuss the power of diﬀerent estimates concretely.
Let 𝚯 be a parametric family.
We say function 𝑓 : 𝚯 → Q is an exact estimate if there exists a weighted deterministic Turing machine that takes 𝜽 as input, and, upon halting, outputs 𝑓 ( 𝜽 ) ∈ Q in ﬁnite time.
Many estimation procedures are anytime algorithms : they do not have a predetermined runtime, and they can be stopped at any time before completion to produce a valid output.
Output quality of an anytime algorithm may improve with increased runtime.
Moreover, many of these algorithms have asymptotic guarantees, in that their outputs are optimal in some sense ( e.g. , consistency) in the limit.
We capture these algorithms with asymptotic estimates : a function 𝑓 ( 𝜽 ) is an asymptotic estimate if there exists a weighted Turing machine that takes both 𝜽 and an index 𝑖 as input, then outputs a value 𝑓 ˜ 𝑝,𝑖 ∈ Q in ﬁnite time, such that the outputs converge toward 𝑓 ( 𝜽 ) as 𝑖 increases toward inﬁnity.
(An example is ˆ 𝑍 asym introduced at §5.)
We now extend both exact and asymptotic estimates to the stochastic case, where we compute the estimates using randomized algorithms instead of deterministic ones.
As is conventional for randomized algorithms, we assume the use of probabilistic Turing machines.
These have access to an inﬁnitely long random tape.
The tape describes an inﬁnite binary sequence from tossing a fair two-sided coin inﬁnitely many times.
We deﬁne a randomized exact estimate f as a weighted Turing machine with two input tapes — the random tape 𝜏 ∈ B N distributed according to 𝑝 𝜏 , and an input tape that has 𝜽 — and outputs 𝑓 ( 𝜽 , 𝜏 ) in ﬁnite time.
Likewise, we say f 𝜽 ,𝑖 is a randomized asymptotic estimate if there exists a function 𝑓 ( 𝜽 ) ∈ R and a weighted Turing machine that takes ( 𝜽 , 𝑖 ) , 𝜏 on two input tapes, so that for all random Boolean tapes 𝜏 ∈ B N , we converge with lim 𝑖 →∞ 𝑓 𝜽 ,𝑖,𝜏 = 𝑓 ( 𝜽 ) .
Many Monte Carlo estimates can be seen as randomized asymptotic estimates, including rejection sampling and importance sampling estimates.
Expressiveness and uncomputability: pathological EBMs 3.1 Expressive sequence distributions
To illustrate the uncomputability issues of energy-based models, we construct families of string distributions that are expressive : they require the full power of an EC-complete sequence model family.
We deﬁne ˜
𝑝 𝑀,𝑘 ∈ G 𝑘 to be a weighted language, parametrized by ( 𝑀, 𝑘 ) , where a) 𝑀 is a deterministic
Turing machine that takes an empty string as input (‘input-free’); and b) 𝑘 ∈ Q > 0 .
Let 𝐿 𝑀 ⊆
B ∗ be a preﬁx-free Boolean language of computation sequences of a Turing machine — that is, encodings of the trace of the Turing machine 𝑀 ’s operations.
We deﬁne
[Equation]
and 𝒙 encodes [Equation] valid accepting
[Equation]
The weighted language ˜ 𝑝 𝑀,𝑘 can weight any string 𝒙 of length 𝑛 in time 𝑂 (cid:0) poly ( 𝑛 ) (cid:1) by verifying whether 𝒙 is an accepting execution trace of 𝑀 , from the initial state to an halting state.
That is, ˜ 𝑝 ∈ EC (§2.2).
We also know that for any (deterministic) machine 𝑀 , the language’s partition function 𝑍 ˜ 𝑝 𝑀,𝑘 exists, and it must equal either 1 or 1 + 𝑘 , since there is either one halting trace ( 𝑍 ˜ 𝑝 𝑀,𝑘
= 1 + 𝑘 ), or none ( 𝑍 ˜ 𝑝 𝑀,𝑘
= 1).
Therefore, each ˜ 𝑝 ∈ G 𝑘 deﬁnes a string distribution.
Since for all 𝑘 ∈ Q > 0 , G 𝑘 ⊂ EC, all weighted languages ˜ 𝑝 𝑀,𝑘 have an equivalent parameter vector 𝜽 𝑀,𝑘 in any EC-complete family 𝚯 .
Also, since each G 𝑘 is a bĳection between the set of all input-free Turing machines and a subset of 𝚯 , it follows that there is no exact estimate (§2.5) of the partition function of any
EC-complete family ( e.g. , Transformer EBMs (Theorem 1)), by a reduction from Halt:
Theorem 2.
Let 𝚯 be a parametric EC -complete sequence model family.
And let 𝚯 𝑘 ⊂ 𝚯 be bĳective with G 𝑘 .
There is no 𝑘 ∈ Q > 0 for which there exists an exact estimate ˆ 𝑍 𝑘 that takes as input 𝜽 ∈ 𝚯 𝑘 as input, and computes ˆ 𝑍 𝑘 ( 𝜽 ) =
𝑍 ˜ 𝑝 𝜽 in ﬁnite time.
Proof sketch.
𝚯 𝑘 contains parametric vectors for all input-free Turing machines 𝑀 .
For any of these weighted languages, knowing the exact value of 𝑍 ˜ 𝑝 𝑀,𝑘 ∈ { 1 , 𝑘 + 1 } is enough to decide whether 𝑀 halts ( 𝑀 halts iﬀ 𝑍 ˜ 𝑝 𝑀,𝑗
=
𝑘 + 1).
As Halt is undecidable for input-free Turing machines, ˆ 𝑍 𝑘 cannot exist.
(cid:3)
3.2 An EBM whose partition function is uncomputable
Theorem 2 states there is no estimate that ‘works’ for a subset of parameter vectors.
While every G 𝑘 is much smaller than its superset EC, G 𝑘 is still (countably) inﬁnite.
Here under the assumption that ZFC is consistent, we construct one weighted language ˜ 𝑏 ∈ G 1 (for simplicity; it holds for arbitrary 𝑘 ), where 𝑍 ˜ 𝑏 is uncomputable:
Theorem 3.
Assuming ZFC axioms and that assuming they is consistent, there exists an EC weighted language ˜ 𝑏 ∈ G 1 such that (a) 𝑍 ˜ 𝑏 = 𝑐 ∈ { 1 , 2 } exists, but (b) there is no proof of 𝑍 ˜ 𝑏 = 𝑐 .
Proof sketch.
We construct ˜ 𝑏 such that 𝑍 ˜ 𝑏 = 2 iﬀ ZFC is inconsistent.
If there were a proof 𝑍 ˜ 𝑏 = 1 then we proved ZFC is consistent, which is impossible under Gödel’s 2nd incompleteness theorem (Jech, 1994).
If there were a proof 𝑍 ˜ 𝑏 = 2, then we proved ZFC is not consistent, which violated our assumption.
(cid:3)
The existence of ˜ 𝑏 suggests that if there is an algorithm that approximates 𝑍 ˜ 𝑝 , and produces a witness of its approximation quality, then this algorithm will not work on any set of parameter vectors that can parametrize ˜ 𝑏 — even when this algorithm may work for some subsets of 𝚯 .
This is useful in allowing us to show negative results regarding ﬁnite subsets of 𝚯 .
Appendix F gives one such example.
No randomized algorithm can estimate 𝑍 accurately.
We’ve shown by Theorem 2 that for an EC-complete family, there are no exact estimates than can get 𝑍 perfectly right.
In this section, we show that no randomized exact estimate for this is unbiased.
Further, there isn’t even an estimate whose bias is within some factor 𝜖 , regardless of the variance’s magnitude.
Lemma 1.
Let 𝚯 be an EC -complete parametric family.
There is no multiplicative factor 𝜖 ∈ Q > 1 for which every 𝜽 ∈ 𝚯 can have its partition function approximated with ˆZ 𝜖 ( 𝜽 ) within a factor of 𝜖 — with probability greater than 2 / 3 .
That is, we cannot have
[Equation]
Proof sketch.
Because ˆZ 𝜖 computes an estimate in ﬁnite time, it can only use ﬁnitely many distinct random tape segments.
We therefore derandomize ˆZ 𝜖 by enumerating ﬁnitely many ‘random’ tapes, and can always return correct answer, given the 2 / 3 success probability assumption, and would be able to decide Halt, making use of distributions in G 𝜖 2 .
[Equation]
Taken together, Theorem 2 and Lemma 1 state that no exact estimate ˆ 𝑍 — whether randomized or deterministic — can approximate 𝑍 with good conﬁdence.
In Theorem 4 below, we will make an even stronger claim: regardless of the dispersion magnitude, it is impossible to bound the mean of random exact estimates of 𝑍 to within any (computable) multiplicative factor.
This is because the mean of ˆZ 𝜖 can be computed in ﬁnite time, by derandomizing ˆZ 𝜖 similarly to our proof of Lemma 1:
Theorem 4.
Let 𝚯 be an EC -complete parametric family.
There is no multiplicative bound 𝜖 ∈ Q >such that there exists a randomized exact estimate ˆZ 𝜖 that guarantees 1 / 𝜖 ≤ E [ ˆZ 𝜖 ( 𝜽 )] / 𝑍 ˜ 𝑝 𝜽 ≤ 𝜖 , for every 𝜽 ∈ 𝚯 where ˜ 𝑝 𝜽 is normalizable.
Proof sketch.
We derandomize the expectation of ˆZ 𝜖 .
[Equation]
Common asymptotic estimates do not give useful guarantees
Let’s now recap the progress we’ve made so far.
We’ve shown that no (deterministic) exact estimate can get 𝑍 exactly right in general (Theorem 2), lest it need to solve Halt.
Further, no randomized exact estimate can approximate it within any given relative tolerance, with good conﬁdence (Theorem 4).
But what about asymptotic estimates?
We do know there are correct asymptotic estimates of 𝑍 .
For example, consider the following asymptotic estimate ˆ 𝑍 ( 𝜽 ) backed by a weighted Turing machine that takes 𝜽 ∈ 𝚯 and 𝑖 ∈ N as inputs, and returns 𝑓 𝜽 ,𝑖 (cid:44) (cid:205) 𝒙 : 𝒙 ∈ B ∗ , | 𝒙 |≤ 𝑖 ˜ 𝑝 𝜽 ( 𝒙 ) .
We have lim 𝑖 →∞ ˆ 𝑍 𝜽 ,𝑖 = 𝑍 ˜ 𝑝 𝜽 , so
ˆ 𝑍 is asymptotically correct.
However, ˆ 𝑍 asym does not have a convergence rate guarantee: for any 𝑖 ∈ N , (cid:107) ˆ 𝑍 𝜽 ,𝑖 − 𝑍 ˜ 𝑝 𝜽 (cid:107) is uncomputable.
We also do not know how much can we improve our estimate when we increment 𝑖 .
As Corollary 2 suggests, we likely cannot have such a guarantee.
In this section, we formalize this intuition for two popular asymptotic estimates: rejection and importance sampling methods (with other asymptotic estimates left as future work).
Speciﬁcally, we show that any parametric family that is able to parametrize ˜ 𝑏 from §3.2 cannot have provably useful locally normalized distributions (§2.3) as proposal distributions.
5.1 Rejection sampling estimate of 𝑍 cannot be guaranteed to be possible.
Rejection sampling (Owen, 2013) is a common exact sampling method, applicable even when we cannot sample from an unnormalized distribution ˜ 𝑝 .
We instead sample from an easy-to-sample distribution 𝑞 , then stochastically reject samples, to ensure the probability a sample 𝒙 is kept is proportional to ˜ 𝑝 ( 𝒙 ) .
For rejection sampling to work, the candidate 𝑞 ’s support must contain the target ˜ 𝑝 ’s entire support, so that all true points can be sampled.
We also need some ﬁnite constant 𝑐 so that 𝑐𝑞 envelops ˜ 𝑝 : ∃ 𝑐 ∈ R > 0 such that ∀ 𝒙 ∈ B ∗ , ( ˜ 𝑝 ( 𝒙 )/ 𝑞 ( 𝒙 )) ≤ 𝑐.
We will show that for certain EBMs, one cannot formally guarantee the existence of an eligible 𝑞 .
Theorem 5.
Using ZFC as our axiom set and assuming they are consistent, then there exists a normalizable EC weighted language ˜ 𝑝 , where there does not exist a consistent locally normalized proposal distribution 𝑞 ∈ LN , and 𝑐 𝑞 ∈ Q > 0 , such that it can be proven ∀ 𝒙 ∈ B ∗ , ˜ 𝑝 ( 𝒙 ) / 𝑞 ( 𝒙 ) < 𝑐 𝑞 ≠ ∞ .
Proof sketch.
Let ˜ 𝑝 =
˜ 𝑏 .
If there were such 𝑞 and 𝑐 𝑞 , we could in ﬁnite time prove 𝑍 ˜ 𝑝
= 1 or 𝑍 ˜ 𝑝
= 2, which contradicts Theorem 3.
[Equation]
Theorem 5 implies that there is no way of ensuring rejection sampling works, not only for any EC-complete families, but also for any parametric family that can parametrize ˜ 𝑏 .
5.2 Importance sampling estimate of 𝑍 cannot be guaranteed to be effective.
Similar to the case of rejection sampling, one cannot guarantee an importance-sampling estimate of 𝑍 to be ‘good’ — in this case, we mean that there cannot be a proof that the importance sampling variance is ﬁnite, regardless of the sample size.
We ﬁrst formalize importance sampling estimates of 𝑍 as randomized asymptotic estimates (§2.5).
Let
[Equation]
be an 𝑁 -sample importance sampling estimate of 𝑍 ˜ 𝑝 𝜽 under 𝑞 , so all 𝒙 ( 𝑛 ) are samples from 𝑞 ∈ LN.
We generally want to minimize the variance of ˆZ 𝑞 𝜽 ,𝑁 under 𝑞 : Var 𝑞 (cid:16) ˆZ 𝑞 𝜽 ,𝑁 (cid:17) (Owen & Zhou, 2000).
And we certainly do not want Var 𝑞 (cid:16) ˆZ 𝑞 𝜽 ,𝑁 (cid:17) = ∞ .
Unfortunately, for certain EBMs, we cannot guarantee there is a good locally normalized proposal distribution that has ﬁnite variance:
Theorem 6.
Let 𝚯 be an EC -complete parametric family.
Assuming ZFC axioms and assuming they are consistent, there exists 𝜽 ∈ 𝚯 where there does not exist a consistent locally normalized “proposal” distribution 𝑞 ∈ LN such that it can be proven Var 𝑞 (cid:16) ˆZ 𝑞 𝜽 ,𝑁 (cid:17) < 𝑐 ≠ ∞ , where 𝑐 ∈ Q > 0 .
Proof sketch.
Proven in a manner similar to the proof of Theorem 5.
[Equation]
Uncomputable 𝑍 causes parameter estimation problems
Theorems 2 and 3 state that it is generally impossible to estimate partition functions in an expressive parametric family, such as certain subsets of an EC-complete family.
Here we show how parameter estimation is made diﬃcult as well: parameter identiﬁability is formally undecidable for an EC-complete family.
Model selection is not possible, either, despite attempts to circumvent this (Table 1).
6.1 Parameter identifiability under EC-complete families is undecidable
Consistency of many estimators relies upon the condition of parameter identiﬁability (Lehmann & Casella, 2006) — two diﬀerent parameter vectors should deﬁne diﬀerent string distributions.
But we in general cannot ascertain whether this condition holds, even for ﬁnite subsets of an EC-complete family:
Theorem 7.
Let 𝚯 be an EC -complete family.
There is no algorithm that takes 𝜽 1 ∈ 𝚯 , 𝜽 2 ∈ 𝚯 as input, and decides whether
[Equation] are the same weighted language.
6.2 Model selection is generally impossible for EC-complete sequence model families
Theorem 7 suggests that consistency guarantees of common estimators do not hold, when used to estimate parameter vectors from 𝚯 .
Nonetheless, such ‘oﬀ-label’ use of consistent estimators in an expressive parametric family is quite common — one usually just selects the best model 𝜽 ∗ among ﬁnitely many candidates (say { 𝜽 1 . . .
𝜽 𝑁 } ) that achieves highest held-out likelihood: 𝜽 ∗ =
arg max 𝜽 𝑛 :1 ≤ 𝑛 ≤ 𝑁 (cid:206) 𝒙 ∈D 𝑝 𝜽 𝑛 ( 𝒙 ) , where D is a ﬁnite set of strings.
However, Theorem 8 implies that model selection is generally impossible for EC-complete sequence model families:
Theorem 8.
For any ˜ 𝑝 ∈ EC and for any EC -complete family 𝚯 , there is no algorithm Better ˜ 𝑝 that takes two parameter vectors 𝜽 1 ∈ 𝚯 , 𝜽 2 ∈ 𝚯 , and returns YES if KL ( 𝑝 || 𝑝 𝜽 1 ) ≥ KL ( 𝑝 || 𝑝 𝜽 2 ) , and NO otherwise, where 𝑝, 𝑝 𝜽 1 , 𝑝 𝜽 2 are string distributions deﬁned by ˜ 𝑝, ˜ 𝑝 𝜽 1 , ˜ 𝑝 𝜽 2 respectively.
Palliative alternatives
What is a practitioner to do, given that this class of models is unlearnable in general?
We emphasize that a model family does not have to be EC-complete to suﬀer from model selection problems (§6.2) — for example, model selection is impossible for ﬁxed-size Transformer networks with large enough 𝑑 ’s either, by an extension to Theorem 8 (see Theorem 11 in Appendix G).
In other words, to ensure the problem of uncomputability does not haunt us, the best we can do is to cripple ˜ 𝑝 so severely that uncomputability is impossible.
We identify three palliative choices that restrict the family of EBMs.
Each cripples the model ˜ 𝑝 in its own way, aﬀording computability at the cost of expressiveness.
The choice of triage infuses the model with an inductive bias; we must tailor our models based on our prior beliefs about the problem domain.
Restricting support ( ˜ 𝑝 ) to be ﬁnite.
If ˜ 𝑝 only assigns non-zero weights to ﬁnitely many strings, then 𝑍 ˜ 𝑝 is a ﬁnite sum of rational numbers, and is also rational.
Here, sampling-based inference and estimation methods return to their usefulness.
One way to ensure support ( ˜ 𝑝 ) is ﬁnite is by upper-bounding the maximum string length
( e.g. , Bakhtin et al.
An obvious downside of the ﬁnite-support restriction is that we cannot deal with longer strings.
Moreover, while 𝑍 ˜ 𝑝 is computable when support ( ˜ 𝑝 ) is ﬁnite, this quantity can still be practically inapproximable, assuming that ˜ 𝑝 is expressive ( e.g. , ˜ 𝑝 is an EC weighted language that has ﬁnite support), except for very short strings: assuming NP (cid:42) P / poly, no randomized estimate of 𝑍 ˜ 𝑝 that is a good approximation can have a guaranteed 𝑂 ( poly (| 𝒙 |)) time complexity
(Chandrasekaran et al., 2008).
However, if ˜ 𝑝 has limited expressiveness ( e.g. , when ˜ 𝑝 is an Ising model where a string weight is the sum of pairwise weights), then
FPRAS algorithms for approximating 𝑍 ˜ 𝑝 may exist ˜ describe a low-degree ( ≤ 2) graph (Jerrum &
Sinclair, 1993; Luby & Vigoda, 1999).
However, for high-degree graphs ( ≥ 3) it can be shown no FPRAS algorithm for approximating 𝑍 ˜ 𝑝 exists, assuming RP ≠ NP (Galanis et al., 2016).
Autoregressive parametrization of ˜ 𝑝 .
An alternative choice is to conﬁne ourselves to autoregressive models, i.e. , locally normalized string distributions (§2.3).
Under an autoregressive model 𝑝 , 𝑍 𝑝 = 1 by deﬁnition.
We also note that any (unnormalized) distribution ˜ 𝑝 obtained by removing probability mass from 𝑝 will have a computable partition function, as long as ˜ 𝑝 ∈ EC:
Theorem 9.
Let 𝑝 be any LN weighted language.
Any ˜ 𝑝 ∈ EC where ∀ 𝒙 ∈ 𝑉 ∗ , ˜ 𝑝 ( 𝒙 ) ≤ 𝑝 ( 𝒙 ) has a computable 𝑍 ˜ 𝑝 .
Proof sketch.
We construct an algorithm that approximates 𝑍 ˜ 𝑝 to any arbitrary error level in ﬁnite time, exploiting the consistency property of 𝑝 (Proposition 1).
(cid:3)
Theorem 9 implies that conditionalization operations on 𝑝 , which remove strings from the support of 𝑝 to get weighted language ˜ 𝑝 , result in a computable 𝑍 ˜ 𝑝 (as long as we can decide which strings are removed); and such a ˜ 𝑝 is therefore not subjected to the limitations of Theorem 4.
Unlike the ‘ﬁnite support’ ﬁx, an autoregressively parametrized (or subsequently conditionalized) ˜ 𝑝 can have an inﬁnite support.
A conditionalized ˜ 𝑝 can have an intractable (but computable) partition function, and they are still subject to the expressive limitations imposed on LN languages: namely there is an EC language whose string weight rankings cannot be honored by any such conditionalized ˜ 𝑝 (Lin et al., 2021).
˜ 𝑝 as low-treewidth factor graph grammars.
Finally, we may limit ourselves to weighted languages deﬁned by low-treewidth factor graph grammars (Chiang & Riley, 2020).
Factor graph grammars generalize factor graphs, which cover many classes of graphical sequence models, such as 𝑛 -gram, HMM, and whole-sentence language models (Jelinek, 1980; Kuhn et al., 1994; Rosenfeld et al., 2001), linear CRF models (Laﬀerty et al., 2001), and weighted FSAs in general (Dreyer & Eisner, 2009): a factor graph grammar describes a (possibly inﬁnite) set of factor graphs, generated from a hyperedge replacement graph grammar.
Assuming that an FGG 𝐺 contains at most one 𝑛 -observed-node factor graph for all 𝑛 ∈ N , it then deﬁnes a weighted language ˜ 𝑝 𝐺 ( 𝒙 )
=
(cid:206) 𝜓 ∈ Ψ | 𝒙 | 𝜓 , where factor 𝜓 is a positive function of nodes.
The treewidth of an FGG 𝑊 ( 𝐺 ) is the maximum number of nodes any 𝜓 can be a function of, and Ψ | 𝒙 | is the set of all factors of string length | 𝒙 | .
If 𝑍 ˜ 𝑝 𝐺 ∈ R exists, it can be computed exactly in 𝑂 (cid:0) poly (| 𝐺 | exp ( 𝑊 ( 𝐺 ) + 1 ))
(cid:1) following Chiang & Riley (2020).
Exact computation of 𝑍 ˜ 𝑝 𝐺 may be manageable as long as 𝑊 ( 𝐺 ) is small, which would allow us to exactly compute held-out data likelihood, and also train with a (marginalized) MLE objective function.
However, a small 𝑊 ( 𝐺 ) implies ˜ 𝑝 cannot be expressive: such a grammar cannot capture phenomena like logical coherence (Cook, 1971).
Related work
Turing completeness of formal languages and associated uncomputability issues emerge repeatedly in computer science.
For example, Turing completeness may emerge as an unwanted side eﬀect in programming languages, since it implies undecidability.
One of the best known examples is the Turing completeness of the C++ grammar (Veldhuizen, 2003; Haberman, 2013), which makes both parsing and compiling C++ programs undecidable.
Similar problems exist for Java (Grigore, 2016) and Haskell with (unrestricted) instance declarations (Wansbrough, 1998).
Another example is the (possibly inadvertently introduced) Turing completeness of the page fault handling mechanism on modern computer systems, which raises security concerns in the context of trusted computing (Bangert et al., 2013).
Our work is not the ﬁrst to discuss the consequences of computability in machine learning: assuming we can acquire training data from an oracle, under a supervised learning setting, recognition of an undecidable language is PAC-learnable (Lathrop, 1996).
Agarwal et al.
(2020) extended the deﬁnition of PAC learnability (Valiant, 1984) to computable learners.
By contrast, we are focused on the computability of EBMs for sequences, such as a language model as a component of a larger system for automatic speech recognition.
Designing an appropriate, eﬃcient loss functional is a challenge that several prior works have compared.
With the plethora of learning strategies for EBMs, it is untenable to point out
[Table]
the deﬁciency in each.
Table 1 gives a handful of examples; none share the four properties we desire:
Global normalization (without which the model would be in LN) 2. Support over inﬁnite languages (of ﬁnite strings) 3. Unbounded treewidth in the function assigning weights to strings 4. Estimator consistency ( i
[Table]
e. , asymptotic guarantee to recover the true parameters).
Lin et al.
(2021) noted autoregressive factors of EC languages can be uncomputable (see also Theorem 10).
They also noted that a weighted language can have an uncomputable partition function (presumably resulting from the sum over inﬁnitely many string weights).
But they did not dwell on the question whether such a weighted language could lie within the EC class, much less providing a constructive proof (see also Appendix E).
Instead, they emphasized that under the assumption that oracular access to trained parameter strings is possible, arbitrarily good approximations of the (possibly uncomputable) partition function can be memorized in the (autoregressive) model parameters.
There is an interesting constrast between the stances of Lin et al.
and our work: Lin et al.
(2021) saw the uncomputability of 𝑍 as a trivial issue from the model capacity viewpoint, since good approximations take few bits in the parameters to store.
On the other hand, we see that the uncomputability problem can be a parameter estimation disaster — there will be no guarantee of good approximations can be found in ﬁnite time at all.
Conclusion and future work
Energy-based models are posed as an eﬃcient tool for decision problems, circumventing probabilities and expensive normalization (LeCun et al., 2006).
Extending this vision to generic sequence models, though, can involve complexity/computability problems that are diﬃcult, or even impossible.
We’ve shown that as energy-based sequence models become more powerful, the partition function becomes uncomputable — even when we are restricted to polytime-computable weighted languages.
Exact estimates, even if randomized, cannot have accuracy guarantees.
Popular asymptotic estimates on the other hand are not useful either.
Furthermore, model selection is generally impossible, even if we limit ourselves to ﬁxed-size sequence model families.
This paper continues a discussion started by Lin et al.
(2021), who posture energy-based models as a more powerful alternative to autoregressive sequence models.
Autoregressive sequence models, after all, have wide adoption and empirical successes (Radford et al., 2019; Brown et al., 2020).
By contrast, more general neural energy-based sequence models have not caught on.
Why not?
We give unlearnability — due to uncomputability — as a possible explanation: unless we give up the ability to learn parameters from data, we likely cannot use the full expressiveness aﬀorded by powerful neural networks.
Just like the model capacity problems brought up by Lin et al.
(2021), this result is independent of the amount of training data.
We emphasize that our results do not invalidate the ﬁndings of Lin et al.
(2021): regardless of the actual neural parametrization, autoregressive models can never capture certain distributions that energy-based models can.
Instead, one of our main messages is that we may not be able to ﬁnd those EBM parameters in ﬁnite time, if we do not know what the parameters are .
Of course, if we know the task perfectly well and can in fact manually assign the model parameters, we will not need to learn from data at all.
The middle ground — when we have some prior knowledge about the task, but cannot really design the parameter vectors — is an interesting direction for future work: the three palliative alternatives outlined in §7 do not take task-speciﬁc information into account at all.
Can we do better than that, without suﬀering uncomputability problems?
Acknowledgments
We thank the four reviewers for their comments, especially Reviewer NAcm for shortening the proof of Theorem 4.
Additionally, we thank Alexandra DeLucia, Matthew Francis-Landau, Chin-Fu Liu,
Suzanna Sia, Neha Verma and Chenghao Yang (sorted alphabetically) for discussions that improved the presentation; and Jason Eisner, discussions with whom motivated the original exploration of this work. A Summary of negative results In Table A.1, we give an overview of the negative results in this paper: deﬁciencies of several estimates of the partition function in an energy-based sequence model. B Weighted Turing machines
We extend the deﬁnition of Turing machines into weighted Turing machines as follows.
A weighted
Turing machine 𝑀 is described as a 7-tuple ( 𝑄, Σ , 𝛿, 𝑞 init , 𝐹, update_num , update_denom ) , where the ﬁrst 5 components ( 𝑄, Σ , 𝛿, 𝑞 init , 𝐹 ) have deﬁnitions following that of standard Turing machines.
The last two additions are weight-update functions: both update_num and update_denom have signature 𝑄 → A , where A (cid:44) { same , carry } .
At the end of time step 𝑖 , assuming the current state is 𝑞 , we deﬁne num 𝑖 = (cid:26) num 𝑖 −if update_num ( 𝑞 )
= same 2 𝑖 + num 𝑖 −if update_num ( 𝑞 ) = carry and similarly, denom 𝑖
= (cid:26) denom 𝑖 −if update_denom ( 𝑞 )
= same 2 𝑖 + denom 𝑖 −if update_denom ( 𝑞 ) = carry.
To keep our proof of model family capacity brief, we (arbitrarily) deﬁne that ∀ 𝑞 ∈ 𝐹 ∪ { 𝑞 init } , update_num ( 𝑞 ) =
update_denom ( 𝑞 ) = same.
Finally, upon arriving at a halting state ∈ 𝐹 in 𝑟 time steps, we say num 𝑟 / denom 𝑟 is the weight of an input 𝒙 ∈ B ∗ under 𝑀 , if num 𝑟 / denom 𝑟 is a rational number.
C Definition of sequence model families and EC-complete parametric families
Following Pérez et al. (2021), we say a sequence model family is a set of seq-to-seq models N = { 𝑁 𝜽 : 𝜽 ∈ 𝚯 ⊆
Q ∗ } , where every 𝑁 𝜽 ∈ N recognizes language 𝐿 𝜽 if and only if the seq-to-seq model 𝑁 𝜽 , paired with embedding function 𝑓 : B → Q 𝑑 , initial states 𝒔 ∈ Q 𝑑 , and polytime termination decision function 𝑔 : Q 𝑑 → B , accepts every string 𝒙 ∈ 𝐿 𝜽 . And we deﬁne 𝑁 𝜽 to accept 𝒙 =
[ 𝑥 1 . . .
𝑥 𝑇 ] ∈ B ∗ if and only if there exists 𝑟 ∈ N , such that 𝑁 𝜽 upon input embeddings [ 𝑓 ( 𝑥 1 ) . . .
𝑓 ( 𝑥 𝑇 )] , produces output embeddings 𝒚 𝑟 at the 𝑟 -th time step, where 𝑔 ( 𝒚 𝑟 ) = 1.
A sequence family 𝚯 is EC-complete if given any ˜ 𝑝 ∈ EC (as a description of a weighted Turing machine), we can construct a parameter vector 𝜽 ∈ 𝚯 such that there exist polytime functions 𝑤 𝑝 : Q 𝑑 → N ∪ { 0 } and 𝑤 𝑞 : Q 𝑑 → N ∪ { 0 } , where whenever on input 𝒙 = [ 𝑥 1 . . .
𝑥 𝑇 ] , if 𝑔 ( 𝒚 𝑟 ) = 1 for some output embeddings 𝒚 𝑟 (that is, 𝑁 𝜽 accepts 𝒙 in 𝑟 time steps), 𝑤 𝑝 ( 𝒚 𝑟 ) / 𝑤 𝑞 ( 𝒚 𝑟 ) = ˜ 𝑝 ( 𝒙 ) .
D Proofs of theorems from main text
Theorem 1.
The class of one-encoder-layer, four-decoder-layer Transformer networks with positional encodings ( 𝑛, 1 / 𝑛 , 1 / 𝑛 2 , 2 𝑛 ) is EC -complete.
Proof.
To model all weighted Turing machines deﬁned in §2.2, we extend the Turing-complete oneencoder-layer three-decoder-layer Transformer network construction introduced by Pérez et al.
(2021) with one additional layer.
We also modify the original positional embedding function pos : N → Q 𝑑 to include a new component.
In this proof, we let
[Equation]
instead of pos ( 𝑖 ) =
, 0 , 1 , 𝑖, 1 / 𝑖 , 1 / 𝑖 2 ] as in Pérez et al.
For the sake of clarity, our construction is a ‘stack-on’ construction: the encoder layer and the ﬁrstdecoder layers are largely identical to the design of Pérez et al.
(2021), with the only diﬀerence being necessary changes to accommodate our one additional positional embeddings component.
12 It may be possible to strengthen our results by showing that one-encoder-layer three -decoder-layer Transformer networks with the original positional embeddings — the parametrization family Pérez et al.
(2021) showed to be Turing-complete — are EC-complete as well, with a more involved construction.
We leave such an improvement as future work.
We claim our fourth layer of the decoder has output 𝒚 𝑟 = 𝒚 (cid:48)
𝑟 + 𝒘 𝑟 , where
• 𝒚 (cid:48)
𝑟 is a zero- padded
version of
the original Transformer output embeddings from Pérez et al.
𝒚 (cid:48) 𝑟 is of the form
[Equation]
where (cid:74) 𝑞 𝑟 (cid:75) denotes an one-hot vector of size | 𝑄 | where the 𝑞 𝑟 -th component is 1 (again following the notation of Pérez et al.
And 𝒘 𝑟 is of the form
[Equation]
where num 𝑟 ∈ N ∪ { 0 } and denom 𝑟 ∈ N ∪ { 0 } are deﬁned in §2.2.
Now we describe how 𝒘 𝑟 can be computed from [ 𝒚 (cid:48) 0 . . .
𝒚 (cid:48) 𝑟 ] using the attention mechanism, with the help of our new positional embeddings.
Speciﬁcally, we want to show that we can construct feedforward networks 𝑄 4 , 𝐾 4 , and 𝑉 4 such that
[Equation]
where 𝒚 (cid:48)(cid:48) 𝑖 = 𝒚 (cid:48) 𝑖
+ pos ( 𝑖 ) , Y (cid:48) =
[ 𝒚 (cid:48) 1 . . .
𝒚 (cid:48) 𝑖 ] , and Y (cid:48)(cid:48)
𝑖 =
Y (cid:48)
+ [ pos ( 1 ) , . .
, pos ( 𝑖 )] .
We let 𝑄 4 ( 𝒚 (cid:48)(cid:48) )
=
, 0 , 1 , 0 , 0 , 0 , 0 ] be a constant vector, and 𝐾 4 ( Y (cid:48)(cid:48) 𝑖 )
=
Y (cid:48)(cid:48) 𝑖 .
Finally, we let 𝑉 4 ( Y (cid:48)(cid:48) 𝑖 ) =
[ 0 𝑞 , 0 𝑠 , 0 , I ( update_num ( 𝑞 𝑖 ) = carry ) 2 𝑖 , I ( update_denom ( 𝑞 𝑖 ) = carry ) 2 𝑖 , 0 , . . .
𝑄 4 is a constant function (such that it always attends to the unity component of Y (cid:48)(cid:48) 𝑖 ), so it can be implemented as a single-layer feedforward network.
𝐾 4 is the identity function, which can also be implemented as a single-layer feedforward network.
𝑉 4 on the hand can be implemented as a ﬁxed-size feedforward network, with the piecewise-linear sigmoidal function 𝜎 : in the case of denom 𝑖 , the network would ﬁrst project 𝑞 𝑖 to 𝒂 =
[ I ( update_denom ( 𝑞 𝑖 ) = carry ) , I ( update_denom ( 𝑞 𝑖 ) = same )] (using the one-hot (cid:74)
𝑞 𝑖 (cid:75) segment from 𝒚 (cid:48) 𝑖 ), multiply it by 𝒃 = [ 2 𝑖 , 0 ] (with the help of nonlinearity from 𝜎 ), and put 𝒂 ⊗ 𝒃 at the position of denom 𝑖 in equation (3).
The component at num 𝑖 in equation (3) can be computed likewise.
Given any position 𝑟 ∈ N , we have 𝒚 𝑟 =
Att ( 𝑄 4 ( 𝒚 (cid:48)(cid:48) 𝑟 ) , 𝐾 4 ( Y (cid:48)(cid:48) 𝑟 ) ,𝑉 4 ( Y (cid:48)(cid:48) 𝑟 )) =
[ 0 𝑞 , 0 𝑠 , 0 , 𝑟 𝑟 (cid:213) 𝑖 =
I ( update_num ( 𝑞 𝑖 ) = carry ) 2 𝑖 , 𝑟 𝑟 (cid:213) 𝑖 =
I ( update_denom ( 𝑞 𝑖 ) = carry ) 2 𝑖 , 0 , . . .
Let extract_avg_num ( 𝒚 𝑟 ) be an aﬃne transformation that extracts the (cid:0)
| 𝑄 | + | Σ | + 2 (cid:1) nd component from 𝒚 𝑟 , and extract_avg_denom ( 𝒚 𝑟 ) be an aﬃne transformation that extracts the (cid:0)
| 𝑄 | + | Σ | + 3 (cid:1) rd component from 𝒚 𝑟 , we have extract_avg_num ( 𝒚 𝑟 ) extract_avg_denom ( 𝒚 𝑟
[Equation]
denom
[Equation]
which is the weight of input 𝒙
= [ 𝑥 1 . . .
𝑥
𝑇 ] as we deﬁned in §2.2.
(cid:3)
Theorem 2.
Let 𝚯 be a parametric EC -complete sequence model family.
And let 𝚯 𝑘 ⊂ 𝚯 be bĳective with G 𝑘 .
There is no 𝑘 ∈ Q > 0 for which there exists an exact estimate ˆ 𝑍 𝑘 that takes as input 𝜽 ∈ 𝚯 𝑘 as input, and computes ˆ 𝑍 𝑘 ( 𝜽 ) =
𝑍 ˜ 𝑝 𝜽 in ﬁnite time.
Proof.
We can reduce Halt to computing our partition function.
For the sake of contradiction, let us assume that for some 𝑘 ∈ Q > 0 , the exact estimate ˆ 𝑍 𝑘 exists.
Our reduction from Halt of input-free Turing machines is as follows:
Given any deterministic input-free Turing machine 𝑀 , we build a weighted deterministic Turing machine: 𝑀 (cid:48) (Appendix B).
𝑀 (cid:48) takes as input 𝒙 ∈ B ∗ , and outputs weight 1 / 3 | 𝒙 |+
+ 𝑘 when 𝒙 is an accepting trace of 𝑀 (cid:48) .
Otherwise, 𝑀 (cid:48) outputs weight 1 / 3 | 𝒙 |+ 1 .
𝑀 (cid:48) always returns a weight for 𝒙 in polytime.
By the assumptions of EC-complete families (§2.4), we can build a parameter vector 𝜽 ∈ 𝚯 such that ˜ 𝑝 𝑀 (cid:48) =
˜ 𝑝 𝜽 .
Since from the deﬁnitions of G 𝑘 we know ˜ 𝑝 𝑀 (cid:48) =
˜ 𝑝 𝑀,𝑘 , we have 𝜽 ∈ 𝚯 𝑘 .
We have thus completed our reduction: if ˆ 𝑍 𝑘 existed, we could decide whether any given input-less deterministic Turing machine 𝑀 halts, by ﬁrst constructing the weighted Turing machine 𝑀 (cid:48) , then the corresponding 𝜽 .
By our assumption, ˆ 𝑍 𝑘 ( ˜ 𝑝 𝜽 ) =
𝑘 + 1 if and only if ∃ 𝒙 ∈ B ∗ that is an accepting path of 𝑀 (cid:48) , which is true if and only if 𝑀 halts for some ﬁnite steps.
Since whether 𝑀 halts is undecidable, we have arrived at a contradiction (Turing, 1937; Sipser, 2013).
Therefore for all 𝑘 ∈ Q , the algorithm ˆ 𝑍 𝑘 does not exist.
(cid:3)
Theorem 3.
Assuming ZFC axioms and that assuming they is consistent, there exists an EC weighted language ˜ 𝑏 ∈ G 1 such that (a) 𝑍 ˜ 𝑏 = 𝑐 ∈ { 1 , 2 } exists, but (b) there is no proof of 𝑍 ˜ 𝑏 = 𝑐 .
Proof.
Our proof hinges on Gödel’s second incompleteness theorem: no consistent axiomatic system which includes Peano arithmetic ( e.g. , ZFC) can prove its own consistency.
We construct a Turing machine 𝑀 𝑏 that enumerates all provable propositions under ZF, and halts if only if 𝑀 𝑏 proves the proposition 1 = 0.
One such 𝑀 𝑏 with 7 , 918 states has been built by Yedidia & Aaronson (2016).𝑀 𝑏 is an input-free deterministic Turing machine.
We construct a weighted Turing machine 𝑀 (cid:48) from 𝑀 𝑏 , in the manner of the proof of Theorem 2.
We let 𝑀 (cid:48) return weight 1 + 1 / 3 | 𝒙 |+ 1 , in the case 𝑀 𝑏 halts with trace 𝒙 .
𝑀 (cid:48) returns a weight in polytime, and therefore deﬁnes a weighted language ˜ 𝑏 ∈ G 1 ⊂ EC.
We know from the deﬁnitions of G 1 that 𝑍 ˜ 𝑏 is either 1 or 2.
Assume to the contrary that there exists a proof that 𝑍 ˜ 𝑏
= 1.
Then we know 𝑀 𝑏 did not halt.
And therefore the proof would also imply that ZFC is consistent, which violates Gödel’s second incompleteness theorem.
On the other hand, if there were a proof that 𝑍 ˜ 𝑏 = 2, it would imply 𝑀 𝑏 halted and ZFC is not consistent.
We therefore arrive at a contradiction.
[Equation]
Lemma 1.
Let 𝚯 be an EC -complete parametric family.
There is no multiplicative factor 𝜖 ∈ Q > 1 for which every 𝜽 ∈ 𝚯 can have its partition function approximated with ˆZ 𝜖 ( 𝜽 ) within a factor of 𝜖 — with probability greater than 2 / 3 .
That is, we cannot have
[Equation]
Proof.
In this proof, we make use of the distribution family G 𝜖 2 (§3.1).
We assume to the contrary that a multiplicative bound 𝜖 satisfying equation (2) exists.
Recall that our assumptions state that 𝑃 (cid:0) 1 / 𝜖 ≤ ˆZ 𝜖 ( 𝜽 ) / 𝑍 ˜ 𝑝 𝜽 ≤ 𝜖 (cid:1) > 2 / 3 .
14 Let 𝑀 be an input-free Turing machine.
And let ˜ 𝑝 𝜽 = ˜ 𝑝 𝑀,𝜖 2 ∈ G 𝜖 2 where 𝜽 ∈ 𝚯 .
If the Turing machine 𝑀 halts, 𝑍 ˜ 𝑝 𝑀,𝜖 2 = 1 + 𝜖 2 ; therefore, 𝑃 (cid:16) ( 1 + 𝜖 2 ) 1 / 𝜖 ≤ ˆZ 𝜖 ( 𝜽 ) ≤ ( 1 + 𝜖 2 ) 𝜖 (cid:17) > 2 / 3 .
Similarly, if 𝑀 does not halt, we have 𝑍 ˜ 𝑝 𝑀,𝜖 2 = 1; therefore, 𝑃 (cid:16) 1 / 𝜖 ≤ ˆZ 𝜖 ( 𝜽 ) ≤ 𝜖 (cid:17) > 2 / 3 .
By combining the two conditions, we know that I ( 𝑀 halts ) =
I ( ˆZ 𝜖 ( 𝜽 ) ≥ 𝜖 + 1 / 𝜖 ∧ ˆZ 𝜖 ( 𝜽 ) > 𝜖 )
=
I ( ˆZ 𝜖 ( 𝜽 ) ≥ 𝜖 + 1 / 𝜖 ) .
Therefore, given ( 1 / 𝜖 ) 𝑍 ˜ 𝑝 𝜽 ≤ ˆZ 𝜖 ( 𝜽 ) ≤ 𝜖𝑍 ˜ 𝑝 𝜽 , we can decide if 𝑀 halts by checking if ˆZ 𝜖 ( 𝜽 ) ≥ 𝜖 + 1 / 𝜖 .
Since the condition ( 1 / 𝜖 ) 𝑍 ˜ 𝑝 𝜽 ≤ ˆZ 𝜖 ( 𝜽 ) ≤ 𝜖𝑍 ˜ 𝑝 𝜽 only holds 2 / 3 of all time, we then derandomize the randomized ˆZ 𝜖 to get a deterministic estimate: recall that a randomized exact estimate ﬁnishes computation in ﬁnite time, regardless of the content of the random tape 𝜏 .
There, there are only ﬁnitely many ﬁnitely long possible ‘random’ sequences that ˆ 𝑍 𝜖 will read, which we can enumerate.
More concretely:
[Equation]
where 𝑚 𝜽 ,𝜖 ∈ N is the maximum preﬁx length of the random tape that ˆ 𝑍 𝜖 ( 𝜽 , 𝜏 ) will use on any 𝜏 ∈ B N .
Again 𝑚 𝜽 ,𝜖 is guaranteed to exist because of our assumption ˆ 𝑍 𝜖 ( 𝜽 , 𝜏 ) ends in ﬁnite time.
Since equation (4) is a ﬁnite sum of computable functions, it is also computable.
From the computability of equation (4), it follows that we can derive a deterministic algorithm from ˆZ 𝜖 that decides whether an arbitrary input-free Turing machine halts, which is not possible.
Therefore, there is no 𝜖 ∈ Q > 1 , such that the randomized exact estimate ˆ Z 𝜖 satisﬁes the multiplicative bound 𝑃 ( 1 / 𝜖 ≤ ˆZ 𝜖 ( 𝐺 ) / 𝑍 ≤ 𝜖 ) > 2 / 3 for all 𝜽 ∈ 𝚯 .
[Equation]
Theorem 4.
Let 𝚯 be an EC -complete parametric family.
There is no multiplicative bound 𝜖 ∈ Q >such that there exists a randomized exact estimate ˆZ 𝜖 that guarantees 1 / 𝜖 ≤ E [ ˆZ 𝜖 ( 𝜽 )] / 𝑍 ˜ 𝑝 𝜽 ≤ 𝜖 , for every 𝜽 ∈ 𝚯 where ˜ 𝑝 𝜽 is normalizable.
Proof.
We deﬁne 𝜏 to be distributed according 𝑝 𝜏 .
Therefore the mean E [ ˆZ 𝜖 ( 𝜽 )] can be expanded as
E 𝜏 ∼ 𝑝 𝜏
[ ˆZ 𝜖 ( 𝜽 )]
= (cid:205) 𝜏 ∈ B N 𝑝 𝜏 ( 𝜏 ) ˆ 𝑍 𝜖 ( 𝜽 , 𝜏 ) .
Following the same derandomization technique in the proof of
Lemma 1, we can ﬁnd some 𝑚 ∈ N such that (cid:205)
𝜏 ∈ B N 𝑝 𝜏 ( 𝜏 ) ˆ 𝑍 𝜖 ( 𝜽 , 𝜏 ) = (cid:205) 𝜏 ∈ B 𝑚 1 / 2 𝑚 ˆ 𝑍 𝜖 ( 𝜽 , 𝜏 ) in ﬁnite time.
And subsequently, we can compute E [ ˆZ 𝜖 ( 𝜽 )] exactly in ﬁnite time.
Let the exact estimate be 𝑍 𝜖 .
We then have 𝑍 𝜖 ( 𝜽 ) =
E [ ˆ Z 𝜖 ( 𝜽 )] .
Assuming to the contrary that we could guarantee 1 / 𝜖 ≤ 𝑍 𝜖 ( 𝜽 ) ≤ 𝜖 .
Since 𝑍 𝜖 is a deterministic estimate, we can write 𝑃 ( 1 / 𝜖 ≤ 𝑍 𝜖 ( 𝜽 ) ≤ 𝜖 ) = 1, which contradicts Lemma 1.
Therefore such an estimate 𝑍 𝜖 ( 𝜽 ) does not exist.
(cid:3)
Theorem 5.
Using ZFC as our axiom set and assuming they are consistent, then there exists a normalizable EC weighted language ˜ 𝑝 , where there does not exist a consistent locally normalized proposal distribution 𝑞 ∈ LN , and 𝑐 𝑞 ∈ Q > 0 , such that it can be proven ∀ 𝒙 ∈ B ∗ , ˜ 𝑝 ( 𝒙 ) / 𝑞 ( 𝒙 ) < 𝑐 𝑞 ≠ ∞ .
Proof.
By contradiction; let ˜ 𝑝 =
˜ 𝑏 , where ˜ 𝑏 is ﬁrst introduced in §3.2.
Assume that there exists 𝑞 ∈ LN where it is proven that ∀ 𝒙 ∈ B ∗ , ˜ 𝑝 ( 𝒙 ) / 𝑞 ( 𝒙 ) < 𝑐 < ∞ , where 𝑐 ∈ Q > 0 .
We can either prove ZFC is consistent, or is inconsistent, as follows:
By our assumptions, 𝑞 ∈ LN scores every string 𝒙 as 𝑞 𝜽 ( 𝒙 ) 𝑓 ( 𝜽 ) ∈ Q > 0 .
Also, because 𝑞 is consistent and locally normalized, there exists 𝑛 ∈ N such that we can prove ∀ 𝒙 ∈ X >𝑛 { 𝒙 : 𝒙 ∈ B ∗ , | 𝒙 | > 𝑛 } , 𝑞 ( 𝒙 ) < 1 / 𝑐 (Proposition 1).
We will just prove the existence of 𝑛 constructively by enumerating the strings in B ∗ in increasing length.
Let the complement set X ≤ 𝑛 = B ∗ − X >𝑛 .
The proof then examines the ﬁnitely many strings in X ≤ 𝑛 .
(a) If any of these strings 𝒙 (cid:48) has ˜ 𝑏 ( 𝒙 (cid:48) )
> 1, then we know 𝒙 (cid:48) encodes an accepting trace of 𝑀 𝑏 (§3.2).
Therefore, 𝑀 𝑏 halts, which implies there is a proof of the inconsistency of ZFC.
(b) If none of these strings ∈ X ≤ 𝑛 has ˜ 𝑏 ( 𝒙 ) > 1, then we know there is also no string 𝒙 (cid:48)(cid:48)
∈ X >𝑛 ,
such that ˜ 𝑏 ( 𝒙 (cid:48)(cid:48) ) > 1.
This is because of our assumption 𝑐 ≥ ˜ 𝑏 ( 𝒙 ) / 𝑞 ( 𝒙 ) , which in turn means that ˜ 𝑏 ( 𝒙 ) is less than 1 on these long strings X >𝑛
Therefore, 𝑀 𝑏 does not halt, which implies there is a proof of the consistency of ZFC.
Assuming ZFC is consistent, neither a proof of ZFC, or a disproof of ZFC, is possible.
We have therefore therefore arrived at a contradiction.
Therefore, 𝑞 does not exist.
[Equation]
Theorem 6.
Let 𝚯 be an EC -complete parametric family.
Assuming ZFC axioms and assuming they are consistent, there exists 𝜽 ∈ 𝚯 where there does not exist a consistent locally normalized “proposal” distribution 𝑞 ∈ LN such that it can be proven Var 𝑞 (cid:16) ˆZ 𝑞 𝜽 ,𝑁 (cid:17) < 𝑐 ≠ ∞ , where 𝑐 ∈ Q > 0 .
Proof.
let ˜ 𝑝
= ˜ 𝑏 , where ˜ 𝑏 is ﬁrst introduced in §3.2.
Assume that there exists 𝑞 ∈ LN where it is proven that
[Equation]
We have
[Equation]
𝑘
[Equation]
where 𝑝 𝜽 ( 𝒙 ) (cid:44) ˜ 𝑝 𝜽 ( 𝒙 ) 𝑍 ˜ 𝑝 𝜽 .
After some manipulation, we have (cid:213)
[Equation]
≤ [Equation] +. Since ∀ 𝒙 ∈ B ∗ , 𝑝 2 𝜽 ( 𝒙 ) 𝑞 ( 𝒙 ) ≥ 0, we have
[Equation]
in particular, if 𝒙 (cid:48) encodes a halting trace of 𝑀 𝑏 , from §3.1 we know
[Equation]
Combining equations (5) and (6), we have for any halting trace 𝒙 (cid:48) of 𝑀 𝑏 ,
[Equation]
After some more arrangement,
[Equation]
As in the proof of Theorem 5, the existence of such a 𝑞 allows us to either prove or disprove the consistency of ZFC.
For the sake of completeness, we include a proof sketch below:
By our assumptions, 𝑞 ∈ LN scores every string 𝒙 as 𝑞 𝜽 ( 𝒙 ) 𝑓 ( 𝜽 ) ∈ Q > 0 .
Also, because 𝑞 is consistent and locally normalized, there exists 𝑛 ∈ N such that we can prove ∀ 𝒙 ∈ X >𝑛 { 𝒙 : 𝒙 ∈ B ∗ , | 𝒙 | > 𝑛 } , 𝑞 ( 𝒙 ) < 1 4 ( 𝑁𝑐 + 1 ) (Proposition 1).
We will just prove the existence of 𝑛 constructively by enumerating the strings in B ∗ in increasing length.
Let the complement set X ≤ 𝑛 = B ∗ − X >𝑛 . 2.
The proof then examines the ﬁnitely many strings in X ≤ 𝑛 .
(a) If any of these strings 𝒙 (cid:48) has ˜ 𝑝 𝜽 ( 𝒙 (cid:48) ) > 1, then we know 𝒙 (cid:48) encodes an accepting trace of 𝑀 𝑏 (§3.2).
Therefore, 𝑀 𝑏 halts, which implies there is a proof of the inconsistency of ZFC.
(b) If none of these strings ∈ X ≤ 𝑛 has ˜ 𝑝 𝜽 ( 𝒙 ) > 1, then we know there is also no string 𝒙 (cid:48)(cid:48)
∈ X >𝑛 , such that ˜ 𝑝 𝜽 ( 𝒙 (cid:48)(cid:48) ) > 1, since ˜ 𝑝 𝜽 ( 𝒙 (cid:48) )
> 1 ⇐⇒ ˜ 𝑏
( 𝒙 (cid:48) )
> 1 ⇐⇒ 𝑀
𝑏 halts;
and we have already shown that for any accepting trace 𝒙 (cid:48) of 𝑀 𝑏 , 𝑞 ( 𝒙 (cid:48) ) ≥ 1 / 4 ( 𝑁𝑐 + 1 ) .
Therefore, 𝑀 𝑏 does not halt, which implies there is a proof of the consistency of ZFC.
Assuming ZFC is consistent, neither a proof of ZFC, or a disproof of ZFC, is possible.
We have therefore therefore arrived at a contradiction.
Therefore, such a 𝑞 does not exist.
(cid:3)
Theorem 7.
Let 𝚯 be an EC -complete family.
There is no algorithm that takes 𝜽 1 ∈ 𝚯 , 𝜽 2 ∈ 𝚯 as input, and decides whether ˜ 𝑝 𝜽 1 and ˜ 𝑝 𝜽 2 are the same weighted language.
Proof.
Assuming to the contrary that such an algorithm Distinct exists.
We would be able to reduce
Halt of input-free Turing machines to Distinct: we ﬁrst construct 𝜽 1 ∈ 𝚯 such that ˜ 𝑝 𝜽 1 ( 𝒙 ) = 1 / 3 | 𝒙 |+ 1 .
We then construct 𝜽 2 ∈ 𝚯 such that ˜ 𝑝 𝜽 2 = ˜ 𝑝 𝑀, 1 ∈ G 1 (§3.1).
Distinct ( 𝜽 1 , 𝜽 2 ) is YES if and only if 𝑀 halts.
(cid:3)
Theorem 8.
For any ˜ 𝑝 ∈ EC and for any EC -complete family 𝚯 , there is no algorithm Better ˜ 𝑝 that takes two parameter vectors 𝜽 1 ∈ 𝚯 , 𝜽 2 ∈ 𝚯 , and returns YES if KL ( 𝑝 || 𝑝 𝜽 1 ) ≥ KL ( 𝑝 || 𝑝 𝜽 2 ) , and NO otherwise, where 𝑝, 𝑝 𝜽 1 , 𝑝 𝜽 2 are string distributions deﬁned by ˜ 𝑝, ˜ 𝑝 𝜽 1 , ˜ 𝑝 𝜽 2 respectively.
Proof.
By contradiction; assume that for some ˜ 𝑝 ∈ EC, Better ˜ 𝑝 exists.
We know there exists a weighted
Turing machine (also denoted as ˜ 𝑝 ) that on any string 𝒙 , terminates in 𝑂 ( poly (| 𝒙 |)) and outputs ˜ 𝑝 ( 𝒙 ) (Appendix B).
We show how we can reduce from Halt to Better ˜ 𝑝 .
Given any arbitrary input-less Turing machine 𝑀 , we can deﬁne a new weighted Turing machine ˜ 𝑝 (cid:48)
𝑀 that on input string 𝒙 , ˜ 𝑝 (cid:48) 𝑀 ﬁrst simulates ˜ 𝑝 on it and keeps a record of ˜ 𝑝 ( 𝒙 ) somewhere on the tape.
˜ 𝑝 (cid:48) 𝑀 then veriﬁes whether 𝒙 is an encoded accepting trace of 𝑀 .
If 𝒙 is indeed an encoded accepted trace of 𝑀 , ˜ 𝑝 (cid:48)
𝑀 outputs ˜ 𝑝 ( 𝒙 ) + 1.
Otherwise ˜ 𝑝 (cid:48)
𝑀 outputs ˜ 𝑝 ( 𝒙 ) .
Let 𝜽 1 ∈ 𝚯 parametrize ˜ 𝑝 (cid:48) 𝑀 , and let 𝜽 2 ∈ 𝚯 parametrize ˜ 𝑝 , such that ˜ 𝑝 𝜽 1 ( 𝒙 ) =
˜ 𝑝 (cid:48) 𝑀 ( 𝒙 ) , ˜ 𝑝 𝜽 2 ( 𝒙 ) =
˜ 𝑝 ( 𝒙 ) , ∀ 𝒙 ∈ B ∗ .
We know that KL ( 𝑝 || 𝑝 ) =
for any distribution 𝑝 . better ˜ 𝑝 ( 𝜽 1 , 𝜽 2 ) returns YES if and only if
KL ( 𝑝 || 𝑝 𝜽 1 ) = 0 ≥ KL ( 𝑝 || 𝑝 𝜽 2 ) , which implies that ˜ 𝑝 𝜽 2 and ˜ 𝑝 𝜽 1 deﬁne the same distribution, which in turn implies that ∀ 𝒙 ∈ B ∗ , ˜ 𝑝 ( 𝒙 ) =
˜ 𝑝 (cid:48)
𝑀 ( 𝒙 ) , and that 𝑀 never halts.
Similarly, Better ˜ 𝑝 ( 𝜽 1 , 𝜽 2 ) returns
NO if and only if 𝑀 halts.
We have thus completed our reduction from Halt; and therefore algorithm Better ˜ 𝑝 does not exist for all ˜ 𝑝 ∈ EC.
(cid:3)
Theorem 9.
Let 𝑝 be any LN weighted language.
Any ˜ 𝑝 ∈ EC where ∀ 𝒙 ∈ 𝑉 ∗ , ˜ 𝑝 ( 𝒙 ) ≤ 𝑝 ( 𝒙 ) has a computable 𝑍 ˜ 𝑝 .
Proof.
We prove Theorem 9 by constructing an algorithm ˆ 𝑍 ˜ 𝑝 : Q > 0 → Q ≥ 0 that approximates 𝑍 ˜ 𝑝 within any desired positive rational error 𝜖 : namely | ˆ 𝑍 ˜ 𝑝 ( 𝜖 ) − 𝑍 ˜ 𝑝 | ≤ 𝜖 .
Let ˆ 𝑍 𝑛 = (cid:205) 𝑛ℓ
= 1 (cid:205) 𝒙 : | 𝒙 | = ℓ 𝑝 ( 𝒙 ) .
We ﬁrst observe that lim 𝑛 →∞ 𝑍 𝑛 = 1 by Proposition 1.
In other words, lim 𝑛 →∞ 1 − ˆ 𝑍 𝑛 = 0, or equivalently, given any 𝜖 > 0, there exists 𝑛 ∈ N such that for all 𝑛 (cid:48) > 𝑛, 𝑛 (cid:48) ∈ N , ( 1 − ˆ 𝑍 𝑛 (cid:48) )
< 𝜖 .
Therefore, given any 𝜖 > 0, ∃ 𝑛 ∈ N that divides B ∗ in two sets: X ≥ 𝑛 = { 𝒙 : 𝒙 ∈ B ∗ , | 𝒙 | ≥ 𝑛 } , where (cid:205) 𝒙 ∈X ≥ 𝑛 𝑝 ( 𝒙 ) < 𝜖 , and X <𝑛 = B ∗ − X ≥ 𝑛 .
We are guaranteed to ﬁnd 𝑛 by enumerating all candidates from N .
We can thus implement ˆ 𝑍 ˜ 𝑝 as the following program: given 𝜖 ∈ Q > 0 , we ﬁrst ﬁnd the smallest 𝑛 ∈ N , and partition B ∗ into two sets
X <𝑛 , X ≥ 𝑛 as we described in the previous paragraph.
We then return ˆ 𝑍 ˜ 𝑝 ( 𝜖 ) =
(cid:205) 𝒙 ∈X <𝑛 ˜ 𝑝 ( 𝒙 ) .
We argue that ˆ 𝑍 ˜ 𝑝 is a computable function.
We ﬁrst repeat that since 𝑛 ∈ N exists, we will ﬁnd 𝑛 in ﬁnite time, simply by enumerating.
And since the set X <𝑛 ⊂ B ∗ is ﬁnite, ˆ 𝑍 ˜ 𝑝 ( 𝜖 )
=
(cid:205) 𝒙 ∈X <𝑛 ˜ 𝑝 ( 𝒙 ) can computed in ﬁnite time (under our assumption 𝑝 (cid:48) ∈ EC, ∀ 𝒙 ∈ B ∗ , ˜ 𝑝 ( 𝒙 ) can be computed in time 𝑂
( poly (| 𝒙 |)) ).
Since the program we described above terminates in ﬁnite time, ˆ 𝑍 ˜ 𝑝 is a computable function.
We then argue | ˆ 𝑍 ˜ 𝑝 ( 𝜖 ) − 𝑍 ˜ 𝑝 | ≤ 𝜖 .
Assuming to the contrary that | ˆ 𝑍 ˜ 𝑝 ( 𝜖 ) − 𝑍 ˜ 𝑝 | > 𝜖 , we have 𝑍 ˜ 𝑝 > 𝜖 + ˆ 𝑍 ˜ 𝑝 ( 𝜖 ) , since ˆ 𝑍 ˜ 𝑝 ( 𝜖 ) ≤ 𝑍 ˜ 𝑝 .
The inequality can further be rewritten as 𝑍 ˜ 𝑝 = (cid:205) 𝒙 ∈X ≥ 𝑛 ˜ 𝑝 ( 𝒙 ) + (cid:205) 𝒙 ∈X <𝑛 ˜ 𝑝 ( 𝒙 ) > 𝜖 + ˆ 𝑍 ˜ 𝑝 ( 𝜖 ) .
Since ˆ 𝑍 ˜ 𝑝 ( 𝜖 ) = (cid:205) 𝒙 ∈X <𝑛 ˜ 𝑝 ( 𝒙 ) , we arrive at (cid:205) 𝒙 ∈X ≥ 𝑛 ˜ 𝑝
( 𝒙 ) > 𝜖 .
But we already know 𝜖 > (cid:205) 𝒙 ∈X ≥ 𝑛 𝑝 ( 𝒙 ) ≥ (cid:205) 𝒙 ∈X ≥ 𝑛 𝑝 ( 𝒙 ) ≥ (cid:205) 𝒙 ∈X ≥ 𝑛 ˜ 𝑝 ( 𝒙 ) , arriving at a contradiction.
Therefore | ˆ 𝑍 ˜ 𝑝 ( 𝜖 ) − 𝑍 ˜ 𝑝 | ≤ 𝜖 .
(cid:3)
E Connection between expressive sequence distributions and Lin et al. (2021)’s proof that EC ≠ ELN
Our construction of expressive sequence distributions is similar to Lin et al.
(2021)’s construction of an
EC weighted language that is not in ELN.
Whereas each weighted machine ˜ 𝑝 𝑀,𝑘 ∈ G 𝑘 corresponds to execution traces of a deterministic Turing machine 𝑀 , Lin et al.
(2021) deﬁned a single weighted language ˜ 𝑝 ELN for all Turing machines.
Fore the sake of completeness, we describe the deﬁnition of ˜ 𝑝 ELN from Lin et al.
(2021) below.
Let enc be a preﬁx-free encoding function that maps a deterministic Turing machine 𝑀 to a Boolean string where the function inverse enc − 1 exists.
And let ˜ 𝑝 ELN be a weighted language over Boolean strings, where ˜ 𝑝 ELN ( 𝒙 ) = 1 / 3 | 𝒙 |+ 1 if 𝒙 is of the form 𝒙 ( 1 ) 𝒙 ( 2 ) , where 𝒙 ( 1 ) encodes some Turing machine 𝑀 , and 𝒙 ( 2 ) encodes an accepting execution path of 𝑀 on an empty input.
(Such a path may be represented as a sequence of transitions of 𝑀 that begins with an initial state and ends at an accepting state.)
Lin et al.
(2021) then showed that preﬁx probabilities of ˜ 𝑝 ELN are uncomputable under an ELN, since Halt reduces to computing 𝑍 ( 𝒙 ( 1 ) ) .
Theorem 9 implies that ˜ 𝑝 ELN we have just described above has a computable partition function:
Theorem 10.
Let 𝑍 be the partition function of ˜ 𝑝 ELN .
𝑍 is computable.
Proof.
Let 𝑝 ( 𝒙 ) = 1 / 3 | 𝒙 |+ 1 .
𝑝 ∈ ELN ⊂ LN because 𝑝 (· | ˆ 𝒙 ) = 1 / 3 for all valid preﬁxes ˆ 𝒙 .
Since ˜ 𝑝 ELN ∈ EC by Lin et al.
(2021, Theorem 5) and ∀ 𝒙 ∈ B ∗ , ˜ 𝑝 ELN ( 𝒙 ) ≤ 𝑝 ( 𝒙 ) , we have 𝑍 is computable by
Theorem 9.
(cid:3)
Similarly, the ‘sparse version’ of ˜ 𝑝 ELN introduced in (Lin et al., 2021, Theorem 6) can be shown to have a computable partition function as well.
Since one of our goals is to clearly demonstrate that we can construct a single weighted language ∈ EC that has an uncomputable partition function assuming ZFC ( e.g. , ˜ 𝑏 in §3.2), we deﬁne weighted languages in G 𝑘 to have at most one ‘high’ weight, as opposed to ˜ 𝑝 in Lin et al.
(2021, Theorem 5), where each 𝒙 ( 1 ) that encodes a halting machine has a ‘high weight’ suﬃx 𝒙 ( 2 ) .
In fact, under our construction we can directly show EC ≠ ELN, using the uncomputability of ˜ 𝑏 from
Theorem 3:
Corollary 1 (EC ≠ ELN under ZFC; slightly weaker version of Theorem 5 in Lin et al.
Assuming the axiomatic system of ZFC, EC ≠ ELN .
Proof.
We know there exists a weighted language ˜ 𝑏 ∈ EC (§3.2) that has an uncomputable partition function.
Since ˜ 𝑏 ∈ EC, ˜ 𝑏 ( 𝒙 ) ∈ Q ≥ 0 , ∀ 𝒙 ∈ B ∗ .
Therefore for all strings 𝒙 ∈ B ∗ , 𝑏 ( 𝒙 ) =
˜ 𝑏 ( 𝒙 ) / 𝑍 ˜ 𝑏 is an uncomputable number.
Assuming to the contrary that ˜ 𝑏 ∈ ELN.
By deﬁnition 𝑏 ( 𝒙 ) = (cid:206) 𝑏 ( 𝑥 𝑡 | 𝒙 <𝑡 ) , where each 𝑏 ( 𝑥 𝑡 | 𝒙 <𝑡 ) ∈ Q ≥ (cid:48) and is computable.
Since the set of computable numbers is closed under multiplication, 𝑏 ( 𝒙 ) would also be computable, which contradicts our assumption.
Therefore ˜ 𝑏 ≠ ELN, which implies
EC ≠ ELN.
(cid:3)
F Negative results possible on finite parameter subspaces
In §3.2, we mention that the pathological EBM ˜ 𝑏 can show negative results regarding ﬁnite subsets of 𝚯 .
Here, we provide a concrete example.
Corollary 2.
Assuming ZFC axioms and assuming they are consistent, there exists 𝜽 ∈ 𝚯 such that 𝑍 ˜ 𝑝 𝜽 ∈ Q > 0 exists, but there is no algorithm ˆ 𝑍 proof that takes 𝜽 ∈ 𝚯 as input, and outputs a set of strings { 𝒙 𝑛 : 𝑛 < 𝑁 } , 𝑁 ∈ N where
• there is a proof that (cid:205)
𝑁𝑛 = 1 ˜ 𝑝 𝜽 ( 𝒙 ) ≥ 1 / 2 𝑍 ˜ 𝑝 𝜽 ; or • there is a proof that (cid:205)
𝑁𝑛 = 1 ˜ 𝑝 𝜽 ( 𝒙 ) < 1 / 2 𝑍 ˜ 𝑝 𝜽 .
Proof.
Assuming to the contrary that ˆ 𝑍 proof existed.
We construct 𝜽 ∈ 𝚯 suchthat ∀ 𝒙 ∈ B ∗ , ˜ 𝑝 𝜽 ( 𝒙 ) =
˜ 𝑏 ( 𝒙 ) .
Either proof resulting from ˆ 𝑍 proof ( 𝜽 ) : { 𝒙 𝑛 : 𝑛 < 𝑁 } can be used to prove or disprove the consistency of
ZFC.
(cid:3)
Corollary 2 states that there exists a ‘problematic EBM’ — namely ˜ 𝑏 — where we cannot guarantee to well approximate its partition function, by accumulating ﬁnitely many string weights, regardless of the manner of accumulation ( i.e. , how we choose strings) or the number of strings we enumerate over.
We discuss this in further details in §2.5.
G Impossibility of model selection in fixed-size Transformer EBM families
Theorem 11 is a sibling theorem to Theorem 8.
Just as we show 𝑍 ˜ 𝑏 is uncomputable (§3.2), we can prove that model selection is not only impossible for EC-complete parametric families (where the length of a parameter vector is unbounded), but also impossible for ﬁxed-size Transformer EBMs with a large enough embedding size.
Theorem 11.
Assuming ZFC as our axiomatic system, for any ˜ 𝑝 ∈ EC , there exists 𝑑 0 ∈ N such that for all 𝑑 ≥ 𝑑 0 , ˜ 𝑝 can be captured by one-encoder-layer four-decoder-layer Transformer networks 𝚯 ( 𝑑 ) (Theorem 1) with embedding size 𝑑 , where there is no provably correct algorithm Better ˜
𝑝,𝑑
that takes two parameter vectors 𝜽 1 ∈ 𝚯 ( 𝑑 ) , 𝜽 2 ∈ 𝚯 ( 𝑑 ) , and returns YES if KL ( 𝑝 || 𝑝 𝜽 1 ) ≥ KL ( 𝑝 || 𝑝 𝜽 2 ) , and NO otherwise, where 𝑝, 𝑝 𝜽 1 , 𝑝 𝜽 2 are string distributions deﬁned by ˜ 𝑝, ˜ 𝑝 𝜽 1 , ˜ 𝑝 𝜽 2 respectively.
Proof.
Let 𝑀 𝑏 be the input-free unweighted Turing machine we built in our proof of Theorem 3, whose behavior is independent of ZFC.
Let 𝑀 0 be any weighted Turing machine that deﬁnes the EC weighted language ˜ 𝑝 .
And let 𝑀 1 be a weighted Turing machine that weights 𝒙 as ˜ 𝑝 ( 𝒙 ) + 1 if and only if 𝒙 encodes an accepting trace of 𝑀 𝑏 ; and 𝑀 1 weights 𝒙 as ˜ 𝑝
( 𝒙 ) otherwise.
Since checking whether 𝒙 is a valid trace of 𝑀 𝑏 is in 𝑂 ( poly (| 𝒙 |)) , 𝑀 1 deﬁnes an EC weighted language.
Let 𝑝 𝑀 1 be the string distribution deﬁned by 𝑀 1 .
By an argument similar to our proof of Theorem 8, no algorithm that is provably correct can decide if KL ( 𝑝 || 𝑝 𝑀 1 ) ≥ KL ( 𝑝 || 𝑝 ) .
We note that for any weighted Turing machine 𝑀 with fewer than 𝑛 states, we can build another weighted
Turing machine 𝑀 (cid:48) which has 𝑛 states, such that 𝑀 and 𝑀 (cid:48) deﬁne the same weighted language, simply by having (ﬁnitely many) unreachable states in 𝑀 (cid:48) .
Since any weighted Turing machine with 𝑛 states can be implemented as a 1-encoder-layer-4-decoder-layer Transformer networks with an embedding size ∈ 𝑂 ( 𝑛 ) , it follows that there exists 𝑑 0 ∈ N such that both 𝑀 0 and 𝑀 1 can be encoded as parameter vectors within a ﬁxed-size model family with 𝑑 ≥ 𝑑 0 .
(cid:3)