Paper: The No Free Lunch Theorem, Kolmogorov Complexity, and the Role of Inductive Biases in Machine Learning (ICLR-2024)

Reviewer comment (Theory):
The authors mention that no free lunch theorem holds because when selecting datasets using a uniform distribution, it subtly selects data of high complexity which is not compressible. However, earlier the authors claimed that data of high complexity is exponentially hard to obtain, which means that under uniform distribution over tasks, learners should be able to perform better than random chance because this incompressible data occupies only a small volume. It would be nice if the authors could provide some discussion on this.

Author response:
> Apparent contradiction between incompressible data being uncommon in the world we observe and common in the uniform distribution  
 
When considering the uniform distribution, high complexity sequences are abundant. In the statement “All but exponentially few sequences …”,  $n$ is indeed the sequence length, and we have now added that context to the paragraph. This classical result follows from the fact that there are at most $2^{L+1}$ terminating programs of length $L$, and therefore only at most that many distinct sequences with complexity $\le L$ out of the total $2^n$ sequences. Therefore the fraction of sequences with length $L$ is less than $2^{1+L-n}$. While high complexity sequences are abundant in the uniform distribution, we argue that they are very uncommon in natural data, including via experiments bounding the Kolmogorov complexity of real-world datasets.

Reviewer comment (Theory):
The bound provided in Section 4.1 only requires disproportionate weighting over the hypothesis space based on their complexity. However, it does not take into account the compressible nature of the dataset; and hence it is not clear how the bound is non-trivial for real world datasets when it holds for arbitrary datasets, and in particular could hold for a dataset with high complexity as long as the solution still has low complexity.

Author response:
**Bounds in section 4.1**  
These generalization bounds hold regardless of the dataset. For datasets with high complexity, however, it won’t be possible to find a model which simultaneously achieves low risk and low complexity, and eq 1 is actually a result proving that direction. For real world datasets such as image classification or tabular datasets that have much lower than maximal complexity, it **is** possible to use the bounds to nontrivially constrain the model performance, and we demonstrate this in Figure 1 (right).

Reviewer comment (Novelty):
It is not clear what the authors want to show by training a CNN over tabular data. Yes, it is able to perform better than random chance but this is also expected of any, potentially even mismatched, training architecture on any data with some inherent structure. From such a lens, the finding that CNNs are able to generalize on tabular data is not too interesting, so could the authors clarify on why and how it is an interesting finding?

Author response:
> It is not clear what the authors want to show by training a CNN over tabular data.  

These experiments illustrate why a preference for low Kolmogorov complexity alone is sufficient for a nontrivial degree of generalization, provably, since real-world data labelings tend to have low complexity.  We agree a model’s ability to generalize on a modality it was not designed for is not necessarily surprising, but the generalization bounds we compute only make use of the low Kolmogorov complexity preference, and this preference alone is able to explain almost all of the models’ generalization.  Moreover, many in the community, explicitly motivated by no free lunch theorems, do argue that individual applications require specialized learners [1-6], and our work shows why this notion may be misguided in principle. We have now included additional clarifications in our updated draft.

Reviewer comment (Experiments):
Details about the GPT model are missing. Are the authors using a pre-trained GPT-3, but if so it has been trained on a lot of different kinds of data and the task of binary tree expansion would be quite OoD for the model. Is it that the authors are training GPT-3 on such a dataset? It would be nice if the authors actually also do a small scale training (not necessarily GPT-3) and then use that model to provide trends. In particular, the authors could also look at performing training where on average, input of each complexity is seen roughly the same number of times.

Author response:
**GPT-3 experiment details**  
Curiously, despite being OOD as you point out, GPT-3 does not require finetuning data to produce this preference for the low complexity sequences. After generating sequences according to the expression trees, we simply tokenize the digits separately and then feed the resulting token sequence into the model. The details of this experiment are listed in Appendix F.

Reviewer comment (Theory):
The authors claim that a randomly initialized model could do next token prediction well on low complexity sentences. This is unclear because why would the model be able to do the task at all if it is untrained, irrespective of the complexity of the sentences?

Author response:
> “Randomly initialized models prefer low complexity”  

While considering a single token at a time (independent of the others) at initialization, there exists a symmetry, so that all token probabilities on average would be as likely as all others. In contrast, it is not the case that a uniform initialization distribution would yield equal probabilities for pairs or sequences of tokens. In fact, it is very biased and this can be observed from the self attention layers:  
$Softmax(QK^T)V = Softmax(X^TW_Q^TW_KX/d)V$. For modern LLMs, the $W_Q$ and $W_K$ matrices are shared, and the random initialization is such that $W_Q^TW_K \approx I$. Therefore the attention at initialization becomes $Softmax(X^TX/d)V$. Considering a single self attention layer, this creates correlations between the tokens that favor similarity, for example.

Reviewer comment (Theory):
How do the authors demonstrate or claim that cross validation provably generalizes to millions of models with only thousands of data?

Author response:
> How do the authors demonstrate or claim that cross validation provably generalizes to millions of models with only thousands of data?   

In Section 5.1, we demonstrate this is the case as an application of finite hypothesis generalization bounds, merely by selecting a uniform prior over the millions of models ($P(h) \ge 10^-8$ for 10 million models) and evaluating the bound with n=20000 for a classification problem with $\delta=.01$. Plugging these values into $R(H) - \hat{R}(h) \le \sqrt{\frac{\log 1/P(h)+\log 1/\delta}{2n}}$, we get $R(H) - \hat{R}(h) \le 0.034$, thereby demonstrating the claim.

Reviewer comment (Experiments):
While it might be okay to use large models with a simplicity preference to provide a universal platform for learning in diverse systems, it is quite unclear how and what the right method of providing simplicity performance to architectures is. For example, in the ViT experiments, the authors could change the simplicity bias by considering the same GoogleLeNet architecture with just \( l_2 \) penalty, but that does not have as much of an effect as the simplicity bias chosen by authors in Section 5.2.

Author response:
> Unclear how and what the right method of providing simplicity performance to architectures is.  

We do not claim to have solved the problem of engineering an optimal low-complexity biased framework, but a point we make is that we do not need to compromise on flexibility in order to express a preference for low complexity solutions.  In the GoogLeNet example, we illustrate the importance of also including the flexibility of a large model since some problems, for example ImageNet, require a higher complexity solution.  Simply put, we want a model that prefers low complexity insofar as low complexity is appropriate for the data we observe; in other words, follow Occam’s Razor and choose the simplest explanation for the training set.  For some datasets, the simplest explanation is more complex than for others.  We have additionally updated the draft to make this point clear.

Reviewer comment (Novelty):
By far the biggest weakness of the paper is that its novelty is very limited. Besides the experiments, I would argue that almost all the material can be found in textbooks (which do not cover connections to neural networks) and recent publications (studying inductive biases of neural networks from a complexity perspective). Li & Vitanyi’s ‘An Introduction to Kolmogorov Complexity and Its Applications’ covers a large part of what’s discussed in the paper. I also strongly suggest having a look at Hutter’s ‘Universal Artificial Intelligence’ (which has a brief discussion of NFL vs. Occam’s razor and provides more extensive references on the topic), and Gruenwald’s ‘Minimum Description Length Principle’. To me, the only substantial claim made by the paper that has not been discussed at textbook level before, is that Neural networks (after training, and much more importantly before training) have a bias towards low Kolmogorov complexity patterns. Unfortunately, the paper only scratches the surface in empirically verifying the claim. As a counter-argument consider [1], who investigate neural networks’ ability to learn simple algorithmic patterns across the levels of the Chomsky hierarchy (a hierarchy of computational complexity). Tasks include things like reversing an input string, copying a string, or performing modular arithmetic. The paper finds that standard architectures, in particular transformers fail to generalize on non-regular tasks (i.e. all tasks of higher computational complexity than regular languages). I interpret these results as some low-complexity bias holding on regular-language data, but not on non-regular patterns, e.g. for copying or reversing an input string, very low Kolmogorov complexity programs exist but standard neural networks trained via SGD seem (reliably) not to be able to find them and instead find a solution of higher complexity that does not generalize. These results cannot be reconciled with the broad claim in the paper that neural networks have a general bias for low Kolmogorov complexity. The main claim in the paper is thus either wrong, or somewhat vacuous (in which case it needs further refinement).

Author response:
Thank you for your feedback and for recognizing the importance, timeliness, and simplicity of our work.  We address each of your points below:  

> Its novelty is very limited.  

We indeed cite a number of these works on Kolmogorov complexity, universal induction in the context of no free lunch theorems, or simplicity bias in neural networks.  However, no free lunch theorems are commonly referenced in support of the need for domain-specific learners, and the community lacks an understanding of why machine learning algorithms possess generic generalization capabilities across real-world problems. See [1-6] for example claims that the NFL ensures that we need specialized learners for individual applications of machine learning in practice.  

In our work, we synthesize an argument for shared common structure of real-world data distributions and neural networks, including numerous novel experiments from computing the first cross-domain generalization bounds to measuring the preference of LLMs for low-complexity numerical sequences, and we relate this phenomenon to NFL by deriving a new no free lunch theorem in terms of compressibility.  Our formulation of the no free lunch theorem allows us to tease out the applicability of no free lunch theorems to real-world problems by performing compression of datasets and models.  Our work argues directly against commonly held beliefs, and we synthesize a story that to our knowledge does not exist in its full form in the literature.  Moreover, our experiments lead to conclusions which will likely be surprising to many in the community.  For example, Reviewer VnMw says "given that it is a randomly initialized network, wouldn’t it basically give a uniform distribution over all possible sentences?"  In contrast to the reviewer's intuition, we show that randomly initialized LLMs induce a highly non-uniform distribution over sequences, namely one which puts disproportionate mass on low-complexity sequences.  Due to the controversial nature of our overarching argument, our novel experiments, and formulation of the no free lunch theorem, this work will attract attention and contribute significantly to ICLR 2024.  

> As a counter-argument consider [1], who investigate neural networks’ ability to learn simple algorithmic patterns across the levels of the Chomsky hierarchy (a hierarchy of computational complexity).  

The point of our paper is not that neural networks perfectly mimic Solomonov induction or are near-optimal in any way, from the viewpoint of low-complexity bias. Rather, we argue that the extent of their low-complexity bias can explain their broad generalization capabilities and why general-purpose machine learning algorithms are possible.  We don’t disagree with the point that in practice a given function may be hard to learn for a network even if it can be expressed in the model weights. Furthermore, many model architectures like transformers don’t have the ability to express an algorithm that solves a given problem due to the complexity class and how the compute grows as a function of the context length.  However, we don’t think that the imperfections of neural networks detracts from the arguments we make in our paper.

Reviewer comment (Theory):
The paper seems to suggest that No Free Lunch theorems somehow rule out the possibility of neural networks that can train successfully on a number of datasets and achieve non-trivial prediction or classification performance. I disagree with this interpretation, and think it is a somewhat common misreading of the NFL theorems. As the paper states, the NFL theorems hold under all possible datasets (or similar formulations depending on the theorem). Clearly the theorems do not apply when considering a subset of all possible datasets, in particular if they share some structure (literally meaning they are compressible, which is dual to saying predictors/classifiers can be trained on a subset of the data and they will generalize). Saying that the NFL theorems do not apply when only considering compressible data is thus at best a tautology, but it does not invalidate the theorems. To me, the NFL theorems are unrelated to this paper since they do not apply in the setting considered by the paper - I personally would drop their discussion from the paper.

Author response:
> The paper seems to suggest that No Free Lunch theorems somehow rule out the possibility of neural networks that can train successfully on a number of datasets and achieve non-trivial prediction or classification performance … I disagree with the premise that no-free lunch theorems state that “the world is hostile to inductive reasoning”.  

The No Free Lunch Theorem of Wolpert (1996) proves the non-existence of general purpose learners and the hostility of the world to induction under the very strong assumption that learning problems are uniformly distributed.  This theorem is frequently cited to argue that general purpose learners are impossible in practice, not only in theory.  For example, [1] say “the performance of the different algorithms over the 42 experiments considerably differ. This proves the validity of NFL theorems in our field”,  [2] reference NFL to explain why success on synthetic problems may not transfer to real ones, [3] say “the only way one strategy can outperform another is if it is specialized to the specific problem under consideration”, and [4] reference the NFL in stating that “the business of developing algorithms is one of building special-purpose methods to solve application-specific problems”.  See [5,6] for examples of prominent community members leveraging the NFL to argue for specialized learners.  

Our work fights back against this pervasive idea, arguing that real-world data shares a tremendous amount of structure in common with neural networks.  We formalize the no free lunch problem in terms of compression, delineating between the compressible regime and incompressible regime where learning is impossible.  We then examine the extent to which neural networks can compress data.  Our experiments include results which fly in the face of arguments for specialized learners, for example we demonstrate the ability of convolutional networks to compress tabular data (with features written as pixels) which lacks any spatial structure and consequently prove highly non-vacuous cross-domain PAC-Bayes generalization bounds.  In summary, while we agree that real-world learning problems are clearly non-uniform, there is a controversy surrounding the extent to which learning problems share common structure and the extent to which general purpose learners are possible.  Our work synthesizes an explanation for why general purpose learners are possible and presents numerous novel experiments, including ones which show how established ideas, for example from information theory or learning theory, can explain the abilities of modern neural networks including LLMs.

Reviewer comment (Experiments):
The assertion that a universal high-degree polynomial can be effectively applied across diverse sample sizes, given a bias towards simplicity, lacks comparative analysis with contemporary leading methods.

Author response:
Our polynomial experiment serves as an illustration of a broader principle, namely that the low complexity of datasets allows a single model to perform well on many problems (in this illustration, sample sizes) simultaneously as long as the model too encodes a preference for low complexity.  This illustration is not intended to advocate for polynomials in particular as a competitive method for machine learning problems  In fact, other expressive models likely encode an even better approximation of the Solomonov prior, for example, on relevant problems.

Reviewer comment (Evaluation):
The commentary on the combined effectiveness of GoogLeNet and ViT, underpinned by a simplicity preference, does not evidently align with the primary findings of the study.

Author response:
This experiment shows that problems which seemingly benefit from different learners (e.g. small datasets vs large datasets) can be solved by a single learner with a simplicity bias.  This point aligns with the broad message of our paper, namely that no free lunch theorems do not limit machine learning since real data tends to share common structure, enabling general purpose learners in principle.  On the other hand, this experiment also verifies that ViTs do not possess as strong a simplicity bias as GoogLeNet, highlighting that different neural networks exhibit varying degrees of simplicity bias, and this bias can affect performance on various tasks.  Thank you for pointing this out, and we have updated our draft with a corresponding discussion.

Reviewer comment (Theory):
The authors propose a heuristic for addressing the non-computability of Kolmogorov complexity by employing a simplified, non-universal computational language. However, the adequacy of this approximation, both theoretically and empirically, remains unexplained.

Author response:
We adopt three different setups for complexity: (a) we bound Kolmogorov complexity directly via compression.  In an ideal world, we could use Kolmogorov complexity across the boards, but comparing Kolmogorov complexity of different sequences is infeasible because we can only compare upper bounds; (b) we employ expression trees in Section 4.3. This setup allows us to measure complexity exactly for small sequences, and it allows us to bound Kolmogorov complexity formally as we discuss, but the language has the drawback of being simpler than a universal language; (c) we generate periodic sequences in 4.4.  This language is even less rich than the expression trees, but it enables us to measure the complexity of very long sequences which are useful for experiments with randomly initialized models.  Nonetheless, this third setup also is conveniently related to an upper bound on Kolmogorov complexity (e.g. in Python) given by the length of say print(n*x), where n denotes the number of repetitions, and x denotes the repeated subsequence.

Reviewer comment (Evaluation):
While the findings of the paper are intriguing, many of the experimental results seem tangential or conceptually disconnected from the core conclusions drawn in Theorem 1.

Author response:
Theorem 1 delineates the cases where data is highly structured and those where data is incompressible and learning is impossible.  We then spend the remainder of the paper showing that both real-world data and ML models share a common notion of structure and hence why learning is possible in practice.  Proving a no free lunch theorem in terms of compressibility allows us to explore the extent to which that theorem is limiting in the real world by applying compression to datasets and models.

Reviewer comment (Theory):
In addressing the non-computability of Kolmogorov complexity, the authors choose a simplified language as a heuristic. Could they elaborate on the justification for this choice and its validity as a proxy, both theoretically and empirically?

Author response:
We adopt three different setups for complexity: (a) we bound Kolmogorov complexity directly via compression.  In an ideal world, we could use Kolmogorov complexity across the boards, but comparing Kolmogorov complexity of different sequences is infeasible because we can only compare upper bounds; (b) we employ expression trees in Section 4.3. This setup allows us to measure complexity exactly for small sequences, and it allows us to bound Kolmogorov complexity formally as we discuss, but the language has the drawback of being simpler than a universal language; (c) we generate periodic sequences in 4.4.  This language is even less rich than the expression trees, but it enables us to measure the complexity of very long sequences which are useful for experiments with randomly initialized models.  Nonetheless, this third setup also is conveniently related to an upper bound on Kolmogorov complexity (e.g. in Python) given by the length of say print(n*x), where n denotes the number of repetitions, and x denotes the repeated subsequence.