Original: Formatting Instructions For InfoCog@NeurIPS  September 30,
Revised: Successive Refinement and Coarsening of the Information Bottleneck

Original: Abstract
Revised: Abstract

Original: We discuss two models for two central issues appearing in conjunction with cognitive processing.
Revised: We discuss two models for two central issues appearing in conjunction with cognitive processing.

Original: One is the ability to incorporate fresh information to already learnt models; the other one is moving to a better understanding of what happens when information “trickles” through many layers of a cognitive processing pipeline.
Revised: One is the ability to incorporate fresh information to already learnt models; the other one is moving to a better understanding of what happens when information “trickles” through many layers of a cognitive processing pipeline.

Original: We do so by investigating formal properties of the Information Bottleneck method: namely how it relates to successive refinement, and successive coarsening of information.
Revised: We do so by investigating formal properties of the Information Bottleneck method: namely how it relates to successive refinement, and successive coarsening of information.

Original:  Introduction
Revised:  Introduction

Original: We discuss two models for two central issues appearing in conjunction with cognitive processing.
Revised: We discuss two models for two central issues appearing in conjunction with cognitive processing.

Original: One is the ability to incorporate fresh information to already learnt models; the other one is moving to a better understanding of what happens when information “trickles” through many layers of a cognitive processing pipeline such as a deep network, or any cognitive system with a sensory interface and a rich inner information processing.
Revised: One is the ability to incorporate fresh information to already learnt models; the other one is moving to a better understanding of what happens when information “trickles” through many layers of a cognitive processing pipeline such as a deep network, or any cognitive system with a sensory interface and a rich inner information processing.

Original: Both can be treated using two — in a way — “dual” perspectives of the successive refinement and coarsening of the Information Bottleneck (IB) method [5, 2].
Revised: Both can be treated using two — in a way — “dual” perspectives of the successive refinement and coarsening of the Information Bottleneck (IB) method [5, 2].

Original: This method describes fundamental limits on the optimal relevancy/complexity trade-offs that one can achieve while extracting (in a single stage of information processing) relevant features within a given system.
Revised: This method describes fundamental limits on the optimal relevancy/complexity trade-offs that one can achieve while extracting (in a single stage of information processing) relevant features within a given system.

Original: Similarly, one can use an analogous framework to understand optimal limits on several stages information processing, and compare them to the single-stage case.
Revised: Similarly, one can use an analogous framework to understand optimal limits on several stages information processing, and compare them to the single-stage case.

Original: Formally, the IB method considers a given joint probability p ( X, Y ) , where X is the source of information (e.g., environmental variables), and Y the variable of interest (e.g., to the agent).
Revised: Formally, the IB method considers a given joint probability p ( X, Y ) , where X is the source of information (e.g., environmental variables), and Y the variable of interest (e.g., to the agent).

Original: The IB method produces a “bottleneck” T (e.g., the agent’s representation), which maximises the information about the relevancy Y under constraints on the information extracted from X : i.e., T solves
Revised: The IB method produces a “bottleneck” T (e.g., the agent’s representation), which maximises the information about the relevancy Y under constraints on the information extracted from X : i.e., T solves

Original: [Equation]
Revised: [Equation]

Original: where λ ≥ 0 controls the trade-off between relevancy and complexity.
Revised: where λ ≥ 0 controls the trade-off between relevancy and complexity.

Original: In the next sections, we study first refinement, and then coarsening of information.
Revised: In the next sections, we study first refinement, and then coarsening of information

Original:  Successive refinement
Revised:  Successive refinement

Original: The IB framework, like classical rate-distortion problems before it [1, 4], has recently been adapted to the case of several-stages information processing [3].
Revised: The IB framework, like classical rate-distortion problems before it [1, 4], has recently been adapted to the case of several-stages information processing [3].

Original: In learning-theoretic terms 1 , a two-stage bottleneck with parameters ( λ 1 , λ 2 ) , where λ 2 > λ 1 , can be described as the successive computation of a solution Tto the classic IB
Revised: In learning-theoretic terms 1 , a two-stage bottleneck with parameters ( λ 1 , λ 2 ) , where λ 2 > λ 1 , can be described as the successive computation of a solution T 1 to the classic IB

Original: [Equation]
Revised: [Equation]

Original: and then a solution S 2 to
Revised: and then a solution S 2 to

Original: [Equation]
Revised: [Equation]

Original: The second, conditional bottleneck S 2 optimally “supplements” the information already optimally acquired in T 1 , yielding a final representation T =
Revised: The second, conditional bottleneck S 2 optimally “supplements” the information already optimally acquired in T 1 , yielding a final representation T =

Original: ( T 1 , S 2 ) .
Revised: ( T 1 , S 2 ) .

Original: The notion of successive refinement [1, 4, 3] describes if and when this two-stage IB (2)-(3) differs from the one-stage IB (1):
Revised: The notion of successive refinement [1, 4, 3] describes if and when this two-stage IB (2)-(3) differs from the one-stage IB (1):

Original: Definition 1.
Revised: Definition 1.

Original: There is successive refinement from parameters λ 1 to λ 2 > λ 1 , if there is a two-stage bottleneck T =
Revised: There is successive refinement from parameters λ 1 to λ 2 > λ 1 , if there is a two-stage bottleneck T =

Original: ( T 1 , S 2 ) with parameters ( λ 1 , λ 2 ) , such that T is also a solution to the one-stage IB with parameter λ = λ 2 .
Revised: ( T 1 , S 2 ) with parameters ( λ 1 , λ 2 ) , such that T is also a solution to the one-stage IB with parameter λ = λ 2 .

Original: We then have simple characterisations:
Revised: We then have simple characterisations:

Original: Proposition 2.
Revised: Proposition 2.

Original: There is successive refinement from λ 1 to λ 2 > λ 1 if
Revised: There is successive refinement from λ 1 to λ 2 > λ 1 if

Original: and only if there exists distinct one-stage bottlenecks T 1 and T 2 , corresponding respectively to λ 1 and λ 2 , such that, equivalently
Revised: and only if there exists distinct one-stage bottlenecks T 1 and T 2 , corresponding respectively to λ 1 and λ 2 , such that, equivalently

Original: (i)
Revised: (i)

Original: We have the Markov chain T 1 − T 2 − X − Y ; (ii) Or for all t 1 ∈ T 1 , we have p ( X | t 1 )
Revised: We have the Markov chain T 1 − T 2 − X − Y ; (ii) Or for all t 1 ∈ T 1 , we have p ( X | t 1 )

Original: ∈
Revised: ∈

Original: Hull { (cid:8) p ( X | t 2 ) , t 2 ∈ T 2 } , where Hull ( E ) denotes the convex hull of a set E .
Revised: Hull { (cid:8) p ( X | t 2 ) , t 2 ∈ T 2 } , where Hull ( E ) denotes the convex hull of a set E .

Original: To investigate successive refinement, we computed bottlenecks on synthetic examples, with randomly chosen p ( X, Y ) .
Revised: To investigate successive refinement, we computed bottlenecks on synthetic examples, with randomly chosen p ( X, Y ) .

Original: We used the Lagrangian formulation of the IB [5] - and thus conventionally denote, from now on, β the trade-off parameter - and the corresponding Blahut-Arimoto algorithm [5], combined with reverse deterministic annealing (as in [6]), starting at β = ∞ from the (generically) optimal distribution T = 
Revised: We used the Lagrangian formulation of the IB [5] — and thus conventionally denote, from now on, β the trade-off parameter — and the corresponding Blahut-Arimoto algorithm [5], combined with reverse deterministic annealing (as in [6]), starting at β = ∞ from the (generically) optimal distribution T = X .

Original: X . We test our geometric characterisation by focusing on very low-dimensional distributions: |X| = |Y| = 3 .
Revised: We test our geometric characterisation by focusing on very lowdimensional distributions: |X| = |Y| = 3 .

Original: We can then plot the trajectories of q β ( X | T ) , on the simplex ∆ X .
Revised: We can then plot the trajectories of q β ( X | T ) , on the simplex ∆ X .

Original: Figure 1 provides two representative examples.
Revised: Figure 1 provides two representative examples.

Original: On these plots, we can visualise the bifurcations described in [6], which correspond to points where some q β ( Y | t ) , and equivalently some q β ( X | t ) , splits into two symbols.
Revised: On these plots, we can visualise the bifurcations described in [6], which correspond to points where some q β ( Y | t ) , and equivalently some q β ( X | t ) , splits into two symbols.

Original: Moreover, between each bifurcation, the bottleneck “trajectories” seem to target specific points, which are the p ([ x ]) for some merges [ x ] of source symbol probabilities.
Revised: Moreover, between each bifurcation, the bottleneck “trajectories” seem to target specific points, which are the p ([ x ]) for some merges [ x ] of source symbol probabilities.

Original: More importantly for us here, we observe that the conditional probabilities q β ( X | t ) do have a general tendency to contain the q ( X | t β ′ ) for β ′ > β .
Revised: More importantly for us here, we observe that the conditional probabilities q β ( X | t ) do have a general tendency to contain the q ( X | t β ′ ) for β ′ > β .

Original: For instance, this is usually the case after the split into 3 distinct symbols.
Revised: For instance, this is usually the case after the split into 3 distinct symbols.

Original: But the condition can fail: for instance if the condition held, the first curve segments where we have only two symbols (bluer families of points in Figure 1) should be linear segments.
Revised: But the condition can fail: for instance if the condition held, the first curve segments where we have only two symbols (bluer families of points in Figure 1) should be linear segments.

Original: But it is clearly not the case for the right plot: on the latter, the first trajectory segment is non-negligibly curved - a relatively typical behavior.
Revised: But it is clearly not the case for the right plot: on the latter, the first trajectory segment is non-negligibly curved - a relatively typical behavior.

Original: Thus, even though there are many pairs of β and β ′ <
Revised: Thus, even though there are many pairs of β and β ′ <

Original: β for which the successive refinement condition holds, these numerical results suggest that this is not generally the case.
Revised: β for which the successive refinement condition holds, these numerical results suggest that this is not generally the case.

Original:  Dense information coarsening
Revised:  Dense information coarsening

Original: In the case of sequential information processing, if X is the original source, and T 1 , T 2 are two representations at successive stages, we must have the Markov chain X − T 1 − T 2 - and the question is rather: how much optimality do we loose as compared to a one stage processing?
Revised: In the case of sequential information processing, if X is the original source, and T 1 , T 2 are two representations at successive stages, we must have the Markov chain X − T 1 − T 2 - and the question is rather: how much optimality do we lose as compared to a one stage processing?

Original: Moreover it is natural to assume that each stage of the information processing is a bottleneck - where the relevancy Y keeps being the same.
Revised: Moreover it is natural to assume that each stage of the information processing is a bottleneck - where the relevancy Y keeps being the same.

Original: In the case of cognitive systems, the number of processing stages stemming from the sensory interfaces might be extremely large, and thus it is natural to consider the limit of an infinite number of processing stages.
Revised: In the case of cognitive systems, the number of processing stages stemming from the sensory interfaces might be extremely large, and thus it is natural to consider the limit of an infinite number of processing stages.

Original: We chose to focus on the case where the processing is highly distributed among stages: in other words, each stage performs only a small amount of compression.
Revised: We chose to focus on the case where the processing is highly distributed among stages: in other words, each stage performs only a small amount of compression.

Original: We formalised this situation by considering an original source T 0 =
Revised: We formalised this situation by considering an original source T 0 =

Original: X , a fixed relevancy variable Y , and a sequence of bottlenecks ( T k ) k ∈ N ∗ , where T k is bottleneck with source T k − 1 and relevancy Y .
Revised: X , a fixed relevancy variable Y , and a sequence of bottlenecks ( T k ) k ∈ N ∗ , where T k is bottleneck with source T k − 1 and relevancy Y .

Original: We chose the sequence ( β k ) k ∈ N ∗ with small increments, such that I ( X ; T 1 ) =
Revised: We chose the sequence ( β k ) k ∈ N ∗ with small increments, such that I ( X ; T 1 ) =

Original: I ( X ; Y ) and I ( X ; T k ) → 0 when k → + ∞ .
Revised: I ( X ; Y ) and I ( X ; T k ) → 0 when k → + ∞ .

Original: We will refer to the described method as the “dense information  (DIC for short).
Revised: We will refer to the described method as the “dense information coarsening” (DIC for short).

Original: We then plotted, in Figure 2, the q β k ( Y | T k ) trajectories, on the ∆ Y simplex, for a representative example, comparing them to the trajectories of a regular IB with source X and relevancy Y .
Revised: We then plotted, in Figure 2, the q β k ( Y | T k ) trajectories, on the ∆ Y simplex, for a representative example, comparing them to the trajectories of a regular IB with source X and relevancy Y .

Original: We see that the two trajectories present similarities: they seem to “target”, before reaching a bifurcation point, the same.
Revised: We see that the two trajectories present similarities: they seem to “target”, before reaching a bifurcation point, the same.

Original: probabilities on the simplex, which are some convex combinations of the p ( Y | x ) with weights given by the corresponding p ( x ) - this corresponds to the similar phenomenon seen on the ∆ X simplex in Figure 1.
Revised: probabilities on the simplex, which are some convex combinations of the p ( Y | x ) with weights given by the corresponding p ( x ) — this corresponds to the similar phenomenon seen on the ∆ X simplex in Figure 1.

Original: The main difference here is that the DIC trajectories move strictly along linear segments.
Revised: The main difference here is that the DIC trajectories move strictly along linear segments.

Original: But more crucial are the differences exhibited by the information curves (Figure 3).
Revised: But more crucial are the differences exhibited by the information curves (Figure 3).

Original: Differently from the IB information curve, the DIC information curve is not concave.
Revised: Differently from the IB information curve, the DIC information curve is not concave.

Original: At the contrary, it clearly displays cusps - which correspond to the bifurcation one can observe on the simplex.
Revised: At the contrary, it clearly displays cusps — which correspond to the bifurcation one can observe on the simplex.

Original: These bifurcations are points at which the DIC information curve is the closest to the IB information curve.
Revised: These bifurcations are points at which the DIC information curve is the closest to the IB information curve.

Original: The overall feature which is interesting to us here, and was present in all the synthetic examples we observed, is that along the flow of the DIC of information, there seems to be crucial points at which the non-optimality of the trade-off, resulting from the fact that the information has successively degraded along the various stages of processing, is minimal.
Revised: The overall feature which is interesting to us here, and was present in all the synthetic examples we observed, is that along the flow of the DIC, there seems to be crucial points at which the non-optimality of the trade-off, resulting from the fact that the information has successively degraded along the various stages of processing, is minimal.

Original: In particular, it seems that at least in some cases, there is virtually no loss of optimality at these crucial, bifurcation points.
Revised: In particular, it seems that at least in some cases, there is virtually no loss of optimality at these crucial, bifurcation points.

Original: Thus these points are advantageous to sit on within the information processing of any cognitive system, and it seems reasonable to hypothesise that living organisms have evolved towards organising their information-processing around those critical points.
Revised: Thus these points are advantageous to sit on within the information processing of any cognitive system, and it seems reasonable to hypothesise that living organisms have evolved towards organising their information-processing around those critical points.