Relaxing the Kolmogorov Structure Function for Realistic Computational Constraints
Introduction
Given a high-dimensional and noisy data source, quantifying the complexity of datasets or individual datapoints is a central challenge in machine learning, with applications including data collection, active learning, ﬁnding hidden subpopulations, detecting outliers or mislabeled data, compression, and generalization bounds.
Shannon entropy (Shannon, 1948) and Kolmogorov complexity ( ?
Kolmogorov, 1965) quantify the complexity of an object through its shortest possible compression.
While such measures provide a rigorous foundation for understanding the true irreducible complexity of a dataset, their assumption of a computationally unconstrained compressor makes them infeasible to compute in practice.
Recent works (Xu et al., 2020; Ethayarajh et al., 2022) incorporate computational constraints into measures of information by assuming that the compressor belongs to a ﬁxed family of predictive functions.
While appealing, a single ﬁxed family of functions may not show the complete story about the information inside data because different aspects of the data become decodable at different scales of computation.
One way of viewing the process of learning is as a cascading sequence of rules and exceptions, in which one ﬁrst learns simple and broad predictive rules, then the commonalities among exceptions to those rules, the exceptions to the exceptions, and so on.
This viewpoint suggests a natural ordering of datapoints within any dataset, ranging from the most typical and representative examples to the hardest edge cases.
By ﬁtting models with different capacities to a given dataset, we can observe the amount and content of information corresponding to each level of the rule-exception hierarchy.
The
Kolmogorov Structure Function (KSF) (Kolmogorov, 1974) encodes this intuition by measuring, for each integer k , the ﬁt of the best k -length description of a given string x .
However, the KSF is infeasible to compute in practice, because the set of all k -bit descriptions is an infeasibly large search space for even moderately high values of k .
We propose the Constrained Structure Function (CSF): a practical and scalably computable function of any dataset that inherits many of the desirable properties of the KSF.
Given a sequence of increasingly large hypothesis classes, the CSF measures the change in ﬁt quality with increasing capacity.
The CSF has an additional property not present in the KSF: that is useful for understanding the structure of datasets consisting of interchangeable datapoints.
Under a mild factorization assumption about hypothesis classes that is satisﬁed by all standard neural networks for supervised learning, the CSF of a dataset can be expressed as the sum of datapoint-wise functions.
The contribution of one datapoint to the dataset’s CSF can show the properties of that datapoint in the context of the dataset.
Kolmogorov Structure Function
The Kolmogorov complexity (KC) (Kolmogorov, 1965) of a datapoint x is deﬁned as the length of the shortest binary program that outputs x and halts.
This quantity can be seen as encoding all irreducible bits of information inside x .
As a running example, consider a binary string x of length
N = n 2 that encodes an n (cid:2) n black-and-white image.
The image has a white background represented by zeros and a gray circle covering m pixels, where in order to represent gray using binary pixels, we sample each of the m pixels according to an independent coin ﬂip ( (cid:24) Bern (0 . 5) ).
Even though x is a ﬁxed string, the values of its gray pixels can be considered random in the sense that these pixel values cannot be compressed with high probability.
The expected Kolmogorov complexity of this string is m + O (1) , where we use a constant number of bits to describe the background and circle, and m bits to describe the speciﬁc pixel values inside the circle.
Note how in this example, the underlying structure inside the image informs the ﬂow of the shortest program that describes it, and thus its description length.
The Kolmogorov structure function (KSF) is a generalization of the Kolmogorov complexity which reveals the internal structure of an object through the change in its description length at multiple scales.
The only written record of this concept by Kolmogorov himself is the following abstract (Kolmogorov, 1974; Vereshchagin & Vitányi, 2004), translated from Russian by L. A. Levin: “To each constructive object corresponds a function Φ x ( k ) of a natural number k —the log of minimal cardinality of x -containing sets that allow deﬁnitions of complexity at most k .
If the element x itself allows a simple deﬁnition, then the function Φ drops to 1 even for small k .
Lacking such deﬁnition, the element is “random” in a negative sense.
But it is “probabilistically random” only when function Φ having taken the value Φ 0 at a relatively small k = k 0 , then changes approximately as Φ( k ) =
Φ 0 (cid:0) ( k (cid:0) k 0 ) .”
By deﬁnition, the KSF of any object x is a decreasing function of the compression capacity k : Φ x ( k ) <
Φ x ( k + 1) .
The rate at which the KSF decreases, i.e. the value of Φ x ( k ) (cid:0) Φ x ( k + 1) , is the rate at which specifying one more bit of information improves the ﬁt to x .
Returning to the image example, the strategy of ﬁrst describing the background and circle and then the “gray” pixel values maps onto the KSF as follows.
We ﬁrst provide a short description of the background, which narrows down the number of possible binary strings from 2 N to 2 m .
Therefore, the KSF of the image steeply drops from N to m given a small constant number of bits.
Compressing further corresponds to specifying the index of x among a lexicographical ordering of the 2 m possible images.
The KSF from this point on has a moderate slope of (cid:0) 1 because each additional bit of information can only halve the number of possible images.
A core property of the KSF is that it shows internal structure: all of the regularity inside x lies in the fact that it is an element of the larger set of size 2 m , and the slope of the KSF reﬂects this.
Note that a “vertical slice”, i.e., a single function value of the KSF at some complexity level k , will not reveal such structure, and considering the entire function’s shape can show useful properties of the data.
We formally deﬁne the KSF below, with notation following Vereshchagin & Vitányi (2004).
Deﬁnition 1 (Kolmogorov structure function, KSF) .
Let K ( (cid:1) ) be the Kolmogorov complexity: the length of the shortest program that outputs the given object and halts.
Let x be data and α 2 N + an integer which bounds the complexity of distributions to consider.
The structure function of x maps the complexity α to the best-ﬁt distribution P as:
[Equation]
Note that we consider distributions P ( x ) instead of discrete sets that contain x .
This generalization allows us to consider continuous input spaces; the set-based deﬁnition is a special case where the P is restricted to uniform categorical distributions.
In fact, for discrete input spaces, the two deﬁnitions are equivalent up to a constant factor; see Appendix A.2 for more discussion.
The Constrained Structure Function 3.1 Relaxing the KSF via a Restricted Predictive Family
We now formally deﬁne the Constrained Structure Function (CSF), a relaxation of the KSF that allows for any increasing sequence of predictive families.
The core idea is to replace the minimization in Deﬁnition 1 with an optimization over a restricted family of models.
We consider datasets D = f
( x 1 , y 1 ) , . . .
, ( x n , y n ) g that consist of inputs x i 2 X
and labels y
i 2 Y .
We are interested in measuring the complexity of the entire dataset D , where there is often redundant information among the many input-label pairs.
We adapt the notion of predictive families from Xu et al.
(2020) as follows:
Deﬁnition 2 (Predictive family, V ) .
Let Ω =
f f : X !
P ( Y ) g , where P ( Y ) is the set of probability measures over the Borel algebra on Y .
We say that V (cid:18) Ω is a predictive family if it satisﬁes
[Equation]
Given a predictive family and a dataset, we deﬁne the V -complexity as
Deﬁnition 3 ( V -complexity) .
Let D be a dataset, and V a predictive family.
The V -complexity of D is the lowest achievable negative log likelihood of D under V :
[Equation]
Taking motivation from this overall structure, we consider arbitrary nested sequences of predictive families.
This predictive family is much more amenable to the minimization in (3), and has the property that V i (cid:26) V i +1 for each i .
We ﬁrst show that such nested sequences satisfy the same decreasing property as the Kolmogorov structure function.
Deﬁnition 4 (Constrained Structure Function, CSF) .
Let D be a dataset and V 1 (cid:26) V 2 (cid:26) . . .
(cid:26) V N a nested sequence of predictive families.
3.2 Properties of the CSF
Basic properties.
We show that based on only the constraint that V i (cid:26) V i +1 for each i , the Constrained Structure Function (CSF) shares many desirable properties with the Kolmogorov Structure
Function (KSF).
Proposition 1.
Let D be a dataset and V 1 (cid:26) V 2 (cid:26) . . .
(cid:26) V N be deﬁned as in Deﬁnition 4.
We have
Decreasing in N : h ( D ; V 1 ) (cid:21) h
( D ; V 2 ) (cid:21) . . .
(cid:21) h
( D ; V N ) .
Nonnegative: h ( D ; V ) (cid:21)
for all V . 3.
Zero limit: For any ϵ > 0 , there exists V such that h ( D ; V ) <
ϵ . 4.
Monotonicity in D :
If D (cid:18) D + , then h ( D ; V ) (cid:20) h
( D + ; V ) for all V .
These properties demonstrate the validity of the CSF as a measure of the complexity of a dataset.
Additive decomposition.
Recall that we consider datasets rather than general binary strings.
As datasets consists of unordered pairs of inputs and labels, a standard choice of model class for datasets is a factorized model.
In such a model, the joint distribution over the dataset is the product of the conditional distributions over each input-label pair
[Equation]
This mild factorization assumption is satisﬁed by standard discriminative models including neural networks.
For such V 1 (cid:26) . . .
, V N , we can compute a datapoint-wise analogue of the CSF, which shows the learnability of the datapoint ( x, y ) in the context of the entire dataset D .
We formally deﬁne the datapoint-speciﬁc CSF as follows:
Deﬁnition 5.
Let D be a dataset, V a predictive family, and ( x, y ) 2 D .
We deﬁne the datapointspeciﬁc CSF as
[Equation]
Intuitively, datapoints that are more “typical” will have a more rapidly decreasing datapoint-speciﬁc
CSF, since even the predictive models with lower capacity will ﬁt them.
An important property is that the sum of all datapoint-wise CSFs is equal to the CSF of the full dataset D .
Proposition 2.
The CSF of a dataset D is the sum of the datapoint-speciﬁc CSFs of its elements:
[Equation]
5 Polynomial Degree
M S E L o ss y
=2 x 3 2 x y =2 x 2y = x    8 Polynomial Degree
M S E L o ss 10.05.01.
Experiments 4.1 Didactic Example: The CSF in Polynomial Regression
We demonstrate the CSF on a simple regression setting to build intuition for what properties of a dataset are captured by this function.
We consider two types of regression tasks with varying complexity: polynomials of degree (1 , 2 , 3) and sinewaves with frequencies (0 . 1 , 1 . 0 , 5 . 0 , 10 . 0) .
We construct the training dataset by uniformly sampling n = 100 points from [0 , 1] .
For the sinewaves only, we add Gaussian noise with standard deviation σ = 0 .
01 to the targets.
These functions are visualized in Figure 2.
As predictive families for the CSF, we consider the class of polynomials of degree d = {+
+} up to d = 8 .
We show the CSFs for these tasks in Figure 3; recall that the CSF will always converge to zero as d ! and the CSFs of different datasets differ only in lower degrees ( d < n ).
First, note that for the datasets generated from polynomial functions, the CSF sharply decreases at the degree of the underlying function.
For example, the CSF of the cubic function decreases most at degree 3 , indicating that most of the structure of this function is captured by a cubic polynomial.
The CSF decreases more smoothly for the datasets generated from sinewaves because these functions do not precisely ﬁt any low-degree polynomial.
Still, the CSF decreases at rates consistent with the functions’ shapes, with higher frequencies ﬁtting at a slower rate.
This experiment demonstrates that the overall shape of the CSF can be a good indicator of the complexity of a dataset.
4.2 Analyzing the Mandelbrot Set with Pointwise CSF
We consider a binary classiﬁcation problem where the input is a two-dimensional point ( x, y ) and the target is 1 if the point belongs to the Mandelbrot set (Brooks & Matelski, 1981).
The boundary of the Mandelbrot set is known to have a fractal structure with broad patterns at each scale but with exceptions to these patterns at smaller scales.
As predictive families, we consider two-layer ReLU networks with width 2 f 8 , 16 , 32 , 64 , 128 , 256 , 512 g .
The pointwise CSF for these functions at some representative points in Figure 4 shows that the CSF reﬂects the extent to which each point is typical in the overall shape of the function.
The gray and pink points are densely surrounded by points with label 0 and 1 , respectively, and this fact is reﬂected in their rapidly decreasing CSF.
The red point is part of a smaller “pocket” of points with label 1 , and its CSF decreases at a slow but steady rate.
Finally, the blue point is quite close to the decision boundary with many ﬂuctuations, and its CSF stays at a high value due to the difﬁculty in predicting its label.
In Figure 5, we further visualize pointwise CSFs through a heat map of (1) the smallest function that achieves a particular loss and (2) the x-intercept of linear regression to each CSF.
Both graphs reﬂect how much each point’s output is typical within the dataset, with slightly differing tendencies: the ﬁrst graph has highest values on the less common label in each region, while the second graph “overshoots” and has the highest value on the more common label in each region.
We note that both are properties of the CSF, and we can extract different information by considering
A Appendix A.1 Important Special Cases of the CSF
We now review the relationship between the CSF and a few important existing measures of complexity.
Many existing notions of complexity can be viewed as special cases of the CSF with a particular choice of predictive family V
The comparative advantage of the CSF is that by using predictive families such as neural networks, we can capture the complexity of datasets that are not amenable to existing measures.
Kolmogorov strucure function.
We now show that the Kolmogorov structure function (Deﬁnition 1) itself is a special case of V -complexity for a speciﬁc choice of predictive family V .
Recall that for integer α , the value of the structure function h x ( α ) is deﬁned as the maximum log likelihood of x among all distributions with Kolmogorov complexity at most α .
Therefore, h x ( α ) is equal to the V -complexity with respect to the predictive family V α , which consists of all functions that map to uniform distributions over sets with description length at most α .
Note that such predictive families have the property V α =1 (cid:26) V α =2 (cid:26) . . .
(cid:18) Ω and h ( D ; V α =1 ) (cid:21) h
( D ; V α =2 ) (cid:21) . . .
(cid:21) 0 .
While this is an appealing connection, the difﬁculty in direct evaluation remains: we cannot efﬁciently search over the predictive family V α for any value of α .
Predictive V -entropy and V -information.
The V -complexity is closely related to predictive Ventropy
(Xu et al., 2020), a measure of usable information under computational constraints.
For V = V , the predictive V -entropy is deﬁned as
[Equation]
where p ( X, Y ) is a data distribution.
Equation (3) replaces the expectation with respect to p ( X, Y ) with an average over the given dataset D .
The two notions differ in that V -complexity is a property of the dataset D itself, whereas V -entropy is a property of an underlying data distribution, which we may not have access to.
The V -information of two random variables X, Y is an analogue of the Shannon mutual information
I ( X ; Y ) , but with respect to a predictive family V .
It is formally deﬁned as
[Equation]
Pointwise V -information
Our notion of the datapoint-speciﬁc CSF is related to the pointwise Vinformation (Ethayarajh et al., 2022), a measure of usable information within a single datapoint in the context of the entire dataset.
For individual datapoint ( x, y ) , PVI is deﬁned as
[Equation]
where g, g ′ 2 V are two functions that achieve maximum likelihood conditioned on ∅ and x , respectively.
A key property is that the predictive V -information is the expected value of the PVI under the data distribution:
[Equation]
This is analogous to how the CSF is the sum of the datapoint-speciﬁc CSFs.
set -based and distribution
-based
deﬁnitions of KSF
Deﬁnition 6 (Kolmogorov structure function, set-based) .
Let K ( (cid:1) ) be the Kolmogorov complexity: the length of the shortest program that outputs the given object and halts.
Let x be data and α 2 N + an integer bounding the complexity of sets to consider.
The structure function of x maps the complexity α to a minimal cardinality of x -containing sets as:
[Equation]
The use of sets S can be seen as an instance of a probabilistic model which puts weight 1 | S | on each member of S .
This model class is known to be equivalent, up to a logarthmic additive term, to the model class of computable probaility distributions, as shown in the following lemma.
Lemma 1.
(Shen, 1983) Let S be the set of strings of length n , and let P be a computable probability distribution on S .
We have
[Equation] [Equation]