Original: Successive Refinement and Coarsening of the Information Bottleneck
Revised: Successive Refinement and Coarsening of the Information Bottleneck

Original: Abstract
Revised: Abstract

Original: We study two central aspects of information processing in cognitive systems: one is the ability to incorporate fresh information to already learnt models; the other one is the “trickling” of information through the many layers of a cognitive processing pipeline.
Revised: We study two central aspects of information processing in cognitive systems: one is the ability to incorporate fresh information to already learnt models; the other one is the “trickling” of information through the many layers of a cognitive processing pipeline.

Original: We do so by investigating formal properties of the Information Bottleneck method: namely, how it relates to successive refinement, and successive coarsening of information.
Revised: We do so by investigating formal properties of the Information Bottleneck method: namely, how it relates to successive refinement, and successive coarsening of information.

Original:  Introduction
Revised:  Introduction

Original: One of the most crucial tensions characterising the activity of embodied agents is that between pursuing one’s own interests, thus behaving in a purposeful and relevant manner, and doing so without violating the constraints on cognitive processing specific to each agent and situation.
Revised: One of the most crucial tensions characterising the activity of embodied agents is that between pursuing one’s own interests, thus behaving in a purposeful and relevant manner, and doing so without violating the constraints on cognitive processing specific to each agent and situation.

Original: The framework of information-theoretic bounded rationality operationalises this tension as the fundamental trade-off resulting from the optimization of the agent’s preference, while keeping its information processing cost below a given bound.
Revised: The framework of information-theoretic bounded rationality operationalises this tension as the fundamental trade-off resulting from the optimization of the agent’s preference, while keeping its information processing cost below a given bound.

Original: More precisely, on the one hand this information processing cost is measured with the mutual information I ( X ; T ) between an environmental variable X and an agent’s representation T .
Revised: More precisely, on the one hand this information processing cost is measured with the mutual information I ( X ; T ) between an environmental variable X and an agent’s representation T .

Original: On the other hand, behavioral relevancy is most often quantified with an explicit utility function, but it can also be measured implicitly through the mutual information I ( Y ; T ) that the representation T extracts about a “relevancy” variable Y .
Revised: On the other hand, behavioral relevancy is most often quantified with an explicit utility function, but it can also be measured implicitly through the mutual information I ( Y ; T ) that the representation T extracts about a “relevancy” variable Y .

Original: Assuming the information about Y can be only extracted from the environmental variable X , i.e., that the Markov chain T − X − Y holds, the representation T then solves the so-called Information Bottleneck (IB) problem [18, 8]
Revised: Assuming the information about Y can be only extracted from the environmental variable X , i.e., that the Markov chain T − X − Y holds, the representation T then solves the so-called Information Bottleneck (IB) problem [18, 8]

Original: [Equation]
Revised: [Equation]

Original: where λ > 0 controls the complexity-relevancy trade-off — so that the solutions to (1) for varying λ draw the optimal limits on the feasible trade-offs.
Revised: where λ > 0 controls the complexity-relevancy trade-off — so that the solutions to (1) for varying λ draw the optimal limits on the feasible trade-offs.

Original: Note that for the actual computation of solutions to (1), due to the non-linearity of the constraint, one usually rather maximises the Lagrangian
Revised: Note that for the actual computation of solutions to (1), due to the non-linearity of the constraint, one usually rather maximises the Lagrangian

Original: I ( Y ; T ) − βI ( X ; T ) , where the trade-off is now parametrised by the multiplier β .
Revised: I ( Y ; T ) − βI ( X ; T ) , where the trade-off is now parametrised by the multiplier β .

Original: A richer description of real-world biological and artificial agents demands taking into the picture another key characteristic of their information-processing dynamics: namely, that informationprocessing operates along hierarchies [11, 10, 5, 9], producing sequences of representations, instead of single ones.
Revised: A richer description of real-world biological and artificial agents demands taking into the picture another key characteristic of their information-processing dynamics: namely, that informationprocessing operates along hierarchies [11, 10, 5, 9], producing sequences of representations, instead of single ones.

Original: Hierarchical and sequential generalisations of the IB problem (1), or related problems, have been proposed: in particular concerning multiple-stage decision-making [7, 13] and the analysis of deep networks’ learning dynamics [17, 20, 19].
Revised: Hierarchical and sequential generalisations of the IB problem (1), or related problems, have been proposed: in particular concerning multiple-stage decision-making [7, 13] and the analysis of deep networks’ learning dynamics [17, 20, 19].

Original: In this paper, we are interested in the following question: how are the information-processing limits drawn by solutions to (1) — which are not constrained to satisfy any specific structure — affected when one does impose hierarchical sequential structures?
Revised: In this paper, we are interested in the following question: how are the information-processing limits drawn by solutions to (1) — which are not constrained to satisfy any specific structure — affected when one does impose hierarchical sequential structures?

Original: We will take a look at two cases of sequential processing.
Revised: We will take a look at two cases of sequential processing.

Original: The first one investigates to which extent the optimal information bounds on (1) can be implemented through incremental incorporation of fresh information into already learned models [12].
Revised: The first one investigates to which extent the optimal information bounds on (1) can be implemented through incremental incorporation of fresh information into already learned models [12].

Original: In other words, in a developmental language, can the optimal bounds on the information trade-off (1) be achieved by a system which first passes through a given stage of development, and then builds on it to acquire further information about the environment [15, 16]?
Revised: In other words, in a developmental language, can the optimal bounds on the information trade-off (1) be achieved by a system which first passes through a given stage of development, and then builds on it to acquire further information about the environment [15, 16]?

Original: The second one considers the loss of optimality, in terms of discrepancy from the bound (1), induced by information being feed-forwardly processed through many layers of a cognitive processing pipeline.
Revised: The second one considers the loss of optimality, in terms of discrepancy from the bound (1), induced by information being feed-forwardly processed through many layers of a cognitive processing pipeline.

Original: We will investigate these questions from a formal point of view, with mathematical characterisations and numerical experiments on low-dimensional examples.
Revised: We will investigate these questions from a formal point of view, with mathematical characterisations and numerical experiments on low-dimensional examples.

Original:  Successive refinement
Revised:  Successive refinement

Original: Consider an incremental learning process where each step incorporates new information so as to refine the previously acquired representations.
Revised: Consider an incremental learning process where each step incorporates new information so as to refine the previously acquired representations.

Original: In a similar spirit to the classic IB, one can describe the optimal informational bounds on this multiple-stage processing: this is precisely what the notion of successive refinement does.
Revised: In a similar spirit to the classic IB, one can describe the optimal informational bounds on this multiple-stage processing: this is precisely what the notion of successive refinement does.

Original: This notion has initially been formulated for classic rate-distortion problems [6, 14], but inspired by recent work [13, 7], we adapt it to the IB framework.
Revised: This notion has initially been formulated for classic rate-distortion problems [6, 14], but inspired by recent work [13, 7], we adapt it to the IB framework.

Original: Consider the following two-stage bottleneck problem 1 : for λ 2 > λ 1 ≥ 0 , we first design a bottleneck T
Revised: Consider the following two-stage bottleneck problem 1 : for λ 2 > λ 1 ≥ 0 , we first design a bottleneck T

Original: = T 1 that solves (1) with λ = λ 1 ; and then design a solution S 2 to
Revised: = T 1 that solves (1) with λ = λ 1 ; and then design a solution S 2 to

Original: [Equation]
Revised: [Equation]

Original: The second, conditional bottleneck S 2 optimally “supplements” the information already optimally acquired in T 1 , yielding a final “two-stage bottleneck” T 2 :=
Revised: The second, conditional bottleneck S 2 optimally “supplements” the information already optimally acquired in T 1 , yielding a final “two-stage bottleneck” T 2 :=

Original: ( T 1 , S 2 ) .
Revised: ( T 1 , S 2 ) .

Original: Definition 1.
Revised: Definition 1.

Original: There is successive refinement from parameters λ 1 to λ 2 > λ 1 , if there is a two-stage bottleneck T 2 :=
Revised: There is successive refinement from parameters λ 1 to λ 2 > λ 1 , if there is a two-stage bottleneck T 2 :=

Original: ( T 1 , S 2 ) with parameters ( λ 1 , λ 2 ) , such that T 2 is also a one-stage bottleneck with parameter λ 2 , i.e., solves the IB problem (1) with T
Revised: ( T 1 , S 2 ) with parameters ( λ 1 , λ 2 ) , such that T 2 is also a one-stage bottleneck with parameter λ 2 , i.e., solves the IB problem (1) with T

Original: =
Revised: =

Original: T 2 and λ = λ 2 .
Revised: T 2 and λ = λ 2 .

Original: Using the chain rule for mutual information [4] and the fact that the inequality constraint must be saturated for the IB problem [1], one can easily prove that there is successive refinement from λto λ 2 if and only if there exists one-stage bottlenecks T 1 and T 2 , with respective parameters λ 1 and λ 2 , such that the Markov chain T 1 − T 2 − X holds.
Revised: Using the chain rule for mutual information [4] and the fact that the inequality constraint must be saturated for the IB problem [1], one can easily prove that there is successive refinement from λto λ 2 if and only if there exists one-stage bottlenecks T 1 and T 2 , with respective parameters λ 1 and λ 2 , such that the Markov chain T 1 − T 2 − X holds.

Original: To determine whether or not these equivalent properties hold true, we propose a geometric approach.
Revised: To determine whether or not these equivalent properties hold true, we propose a geometric approach.

Original: Denote by Hull ( E ) the convex hull of a set
Revised: Denote by Hull ( E ) the convex hull of a set

Original: E , and consider the condition
Revised: E , and consider the condition

Original: [Equation]
Revised: [Equation]

Original: where t 1 , t 2 , resp.
Revised: where t 1 , t 2 , resp.

Original: T 1 , T 2 , denote the symbols, resp.
Revised: T 1 , T 2 , denote the symbols, resp.

Original: alphabets, of distinct one-stage bottlenecks Tand T 2 , with respective parameters λ 1 , λ 2 , where λ 1 < λ 2 .
Revised: alphabets, of distinct one-stage bottlenecks Tand T 2 , with respective parameters λ 1 , λ 2 , where λ 1 < λ 2 .

Original: Proposition 2.
Revised: Proposition 2.

Original: Successive refinement from λ 1 to λ 2 implies (3) , and (3) implies successive refinement if the transition matrix defined by q ( X | T 2 ) is injective.
Revised: Successive refinement from λ 1 to λ 2 implies (3) , and (3) implies successive refinement if the transition matrix defined by q ( X | T 2 ) is injective.

Original: Moreover, if the bottleneck problem (1) has strictly concave information curve 2 , then for any bottleneck
Revised: Moreover, if the bottleneck problem (1) has strictly concave information curve 2 , then for any bottleneck

Original: T , the transition matrix defined by q ( X | T ) can always be chosen injective.
Revised: T , the transition matrix defined by q ( X | T ) can always be chosen injective.

Original: In short, Proposition 2 says that there is successive refinement if the convex hull of the q ( X | t 2 ) still contains the q ( X | t 1 ) .
Revised: In short, Proposition 2 says that there is successive refinement if the convex hull of the q ( X | t 2 ) still contains the q ( X | t 1 ) .

Original: Equipped with this new characterisation, we computed bottlenecks on 1 Our formulation is inspired by, and equivalent to, the asymptotic formulation in [13].
Revised: Equipped with this new characterisation, we computed bottlenecks on 1 Our formulation is inspired by, and equivalent to, the asymptotic formulation in [13].

Original: It is also related to so-called “parallel information-processing hierarchy” in [7].
Revised: It is also related to so-called “parallel information-processing hierarchy” in [7].

Original: The information curve is the curve drawn by points ( I ( X ; T ) , I ( Y ; T )) , for bottlenecks T corresponding to every possible λ ≥ 0 .
Revised: The information curve is the curve drawn by points ( I ( X ; T ) , I ( Y ; T )) , for bottlenecks T corresponding to every possible λ ≥ 0 .

Original: It is always concave, and generically strictly concave if p ( Y | X ) is not deterministic.
Revised: It is always concave, and generically strictly concave if p ( Y | X ) is not deterministic.

Original: The proof of this proposition will be presented in a publication under preparation.
Revised: The proof of this proposition will be presented in a publication under preparation.

Original: synthetic examples, with example distributions p ( X, Y ) sampled uniformly on the simplex.
Revised: synthetic examples, with example distributions p ( X, Y ) sampled uniformly on the simplex.

Original: We used the Lagrangian formulation of the IB (see Section 1) — where β is the trade-off parameter — and optimised it with the modified Blahut-Arimoto algorithm [18] combined with reverse deterministic annealing [21], starting at β = ∞ from the generic optimum T = X .
Revised: We used the Lagrangian formulation of the IB (see Section 1) — where β is the trade-off parameter — and optimised it with the modified Blahut-Arimoto algorithm [18] combined with reverse deterministic annealing [21], starting at β = ∞ from the generic optimum T = X .

Original: We choose |X| =
Revised: We choose |X| =

Original: |Y| = 3 , which allows to plot the trajectories q β ( X | T ) on the simplex ∆ X .
Revised: |Y| = 3 , which allows to plot the trajectories q β ( X | T ) on the simplex ∆ X .

Original: We only focused on examples with strictly concave information injective
Revised: We only focused on examples with strictly concave information injective

Original:  Figure 1 shows two representative examples 4 .
Revised: | Figure 1 shows two representative examples 4 .

Original: On both plots 1-(a) and 1-(b), we can visualise the bifurcations at which some q β ( X | t ) splits into two distinct distributions as β grows, yielding an increase of the effective bottleneck cardinality [21].
Revised: On both plots 1-(a) and 1-(b), we can visualise the bifurcations at which some q β ( X | t ) splits into two distinct distributions as β grows, yielding an increase of the effective bottleneck cardinality [21].

Original: More importantly for us here, we observe that
Revised: More importantly for us here, we observe that

Original: Hull { q β  ( X | t ) , t ∈ T } seems to have a general tendency to contain the q β ′ ( X | t ) for β ′ < β  .
Revised: Hull { q β 2 ( X | t ) , t ∈ T } seems to have a general tendency to contain the q β 1 ( X | t ) for β 1 < β 2 .

Original: For instance, this is the case after the split into 3 distinct symbols.
Revised: For instance, this is the case after the split into 3 distinct symbols.

Original: But the condition can fail: for instance if the condition held, the first curve segments where we have only two symbols (bluer families of points in Figures 1-(a) and 1-(b)) should be linear segments.
Revised: But the condition can fail: for instance if the condition held, the first curve segments where we have only two symbols (bluer families of points in Figures 1-(a) and 1-(b)) should be linear segments.

Original: But this is not the case for Figure 1-(b): there the first trajectory segment is non-negligibly curved - a relatively typical behavior.
Revised: But this is not the case for Figure 1-(b): there the first trajectory segment is non-negligibly curved - a relatively typical behavior.

Original: Thus, even though there are many pairs of β  and β ′ < β  for which condition (3) holds, these numerical results suggest that the q β ′ ( X | t ) , for β ′ < β  , are not always encompassed by Hull { q β  ( X | t ) , t ∈ T } .
Revised: Thus, even though there are many pairs of β 1 and β 2 > β 1 for which condition (3) holds, these numerical results suggest that the q β 1 ( X | t ) , for β 1 < β 2 , are not always encompassed by Hull { q β 2 ( X | t ) , t ∈ T } .

Original: We propose to quantify this “spreading out” from the convex hull with the measure of unique information proposed in [3], using the algorithm 5 designed in [2]:
Revised: We propose to quantify this “spreading out” from the convex hull with the measure of unique information proposed in [3], using the algorithm 5 designed in [2]:

Original: [Equation]
Revised: [Equation]

Original: where ∆ q is the set of probabilities r ( X, T β ′ , T β ) consistent with the bottleneck probabilities 6 , i.e., such that r ( X, T β ′ )
Revised: where ∆ q is the set of probabilities r ( X, T 1 , T 2 ) consistent with the bottleneck probabilities 6 , i.e., such that r ( X, T 1 )

Original: =
Revised: =

Original: q ( X, T β ′ ) and r ( X, T β ) =
Revised: q ( X, T 1 ) and r ( X, T 2 ) =

Original: q ( X, T β ) .
Revised: q ( X, T 2 ) .

Original: This quantity vanishes if and only if there is successive refinement from T β to T β ′ .
Revised: This quantity vanishes if and only if there is successive refinement from T 1 to T 2 .

Original: In the case ofthe left trajectory in Figure 1, where the convex hull condition (3) seems visually always satisfied, the maximum of UI ( X : T β ′ \ T β ) over all  β ′ < β  is approx.
Revised: In the case of Figure 1-(a), where the convex hull condition (3) seems visually always satisfied, the maximum of UI ( X : T 1 \ T 2 ) over all T 2 finer than T 1 (i.e., with β 2 > β 1 ) is approx.

Original: 4 . 0 · 10 − 4 , whereas for the right trajectory, where (3) is clearly broken, it is approx.
Revised: 4 . 0 · 10 − 5 , whereas for the right trajectory, where (3) is clearly broken, it is approx.

Original: Importantly, even if UI ( X : T β ′ \ T β ) is nonzero in the second case, it is still noticeably small.
Revised: Importantly, even if UI ( X : T 1 \ T 2 ) is nonzero in the second case, it is still noticeably small.

Original: Among all the examples we studied, it never exceeded 5 .
Revised: Among all the examples we studied, it never exceeded 5 .

Original: 4% of I ( X ; T β , T β ′ ) .
Revised: 4% of I ( X ; T 1 , T 2 ) .

Original: Overall, these numerical results suggest that imposing two-stage processing, with incorporation of new information at the second stage, does not essentially degrade the optimal informational limits drawn by the classic IB — and the same statement can be made for n -stage processing, by direct iterations of the definitions, arguments and computations in this section.
Revised: Overall, these numerical results suggest that imposing two-stage processing, with incorporation of new information at the second stage, does not essentially degrade the optimal informational limits drawn by the classic IB — and the same statement can be made for n -stage processing, by direct iterations of the definitions, arguments and computations in this section.

Original: This is encouraging for the feasibility of the optimal IB trade-offs (1) in real-world cognitive processing systems, as the representations they possess about their environment are most often the result of a sequential – and in particular, developmental — process.
Revised: This is encouraging for the feasibility of the optimal IB trade-offs (1) in real-world cognitive processing systems, as the representations they possess about their environment are most often the result of a sequential – and in particular, developmental — process.

Original: However, as the modified BA algorithm is only guaranteed to find local optima [18], one should be wary that the observed patterns do not necessarily reflect the behavior of actual, global solutions to (1).Thanks to J
Revised: However, as the modified BA algorithm is only guaranteed to find local optima [18], one should be wary that the observed patterns do not necessarily reflect the behavior of actual, global solutions to (1).

Original:  Dense information coarsening
Revised:  Dense information coarsening

Original: Here we consider the converse situation as that of Section 2, where along a feed-forward cognitive processing pipeline, the information is coarsened rather than refined, without additional external input between layers.
Revised: Here we consider the converse situation of that of Section 2, where along a feed-forward cognitive processing pipeline, the information is coarsened rather than refined, without additional external input between layers.

Original: In other words, we consider a hierarchy of representations T 1 , . . .
Revised: In other words, we consider a hierarchy of representations T 1 , . . .

Original: , T n , that, by assumption, satisfies the Markov chain X − T 1 − · · · − T n , where X is an environmental variable, and T k has only direct access to the previous layer T k − 1 — deep neural networks being an example of this setting (note that the indices are in reverse order as those of the previous section, because we stick to an indexing describing the order in which representations are produced).
Revised: , T n , that, by assumption, satisfies the Markov chain X − T 1 − · · · − T n , where X is an environmental variable, and T k has only direct access to the previous layer T k − 1 — deep neural networks being an example of this setting [17]. Note that the indices are in reverse order as those of the previous section, because we stick to an indexing describing the order in which representations are produced.

Original: Thus if each layer extracts information about a fixed relevancy variable Y , the optimal information trade-off at each layer is drawn by bottlenecks T k with relevancy Y , but where the source X in (1) is replaced by
Revised: Thus if each layer extracts information about a fixed relevancy variable Y , the optimal information trade-off at each layer is drawn by bottlenecks T k with relevancy Y , but where the source X in (1) is replaced by

Original: T k − 1 .
Revised: T k − 1 .

Original: The question is then: given these hard constraints on the cognitive processing pipeline, how much information optimality do we lose as compared to a one-stage processing as described by (1)?
Revised: The question is then: given these hard constraints on the cognitive processing pipeline, how much information optimality do we lose as compared to a one-stage processing as described by (1)?

Original: We focus on the limit of an infinite number of processing stages, upper-bounding the number of processing stages stemming from the sensory interfaces in cognitive systems.
Revised: We focus on the limit of an infinite number of processing stages, which models cognitive systems where the number of processing stages stemming from the sensory interfaces is large.

Original: Moreover we consider the case where the processingis highly distributed among stages: in other words, each stage performs only a small amount of compression, resulting in a “trickling down” of information through dense information processing layers.
Revised: Moreover we consider a processing highly distributed among stages: in other words, each stage performs only a small amount of compression, resulting in a “trickling down” of information through dense information-processing layers.

Original: We thus consider a sequence of bottlenecks ( T k ) k ∈ N ∗ , where T k is a bottleneck with source T k − 1 and relevancy Y .
Revised: We thus consider a sequence of bottlenecks ( T k ) k ∈ N ∗ , where T k is a bottleneck with source T k − 1 and relevancy Y .

Original: We choose the sequence ( β k ) k ∈ N ∗ with small increments, such that I ( X ; T 1 )
Revised: We choose the sequence ( β k ) k ∈ N ∗ with small increments, such that I ( X ; T 1 )

Original: ≈ I ( X ; Y )
Revised: ≈ I ( X ; Y )

Original: and I ( X ; T k ) → 0 when k → + ∞ .
Revised: and I ( X ; T k ) → 0 when k → + ∞ .

Original: We will refer here to this process as the “dense information coarsening” (DIC for short).
Revised: We will refer here to this process as the “dense information coarsening” (DIC for short).

Original: We compared, for distributions p ( X, Y ) sampled uniformly on the simplex, the information curve of the regular IB (1) with fixed source X , with the DIC information curve — i.e., the curve drawn by points ( I ( X ; T k ) , I ( Y ; T k )) .
Revised: We compared, for distributions p ( X, Y ) sampled uniformly on the simplex, the information curve of the regular IB (1) with fixed source X , with the DIC information curve — i.e., the curve drawn by points ( I ( X ; T k ) , I ( Y ; T k )) .

Original: Figure 2 (left) shows a representative example 7 with |X| =
Revised: Figure 2 (left) shows a representative example 7 with |X| =

Original: |Y| = 3 .
Revised: |Y| = 3 .

Original: Here, like other examples explored (not shown), the DIC curve tracks rather neatly the regular IB curve.
Revised: Here, like other examples explored (not shown), the DIC curve tracks rather neatly the regular IB curve.

Original: However, the discrepancy between the curves is non-negligible and the DIC curve displays a distinctive structure: as opposed to the IB curve which is always concave, the DIC curve is only piece-wise concave, resulting in slight cusps at the junctions — in Figure 2 (left), there are two such pieces of concave curve and one cusp.
Revised: However, the discrepancy between the curves is non-negligible and the DIC curve displays a distinctive structure: as opposed to the IB curve which is always concave, the DIC curve is only piece-wise concave, resulting in slight cusps at the junctions — in Figure 2 (left), there are two such pieces of concave curve and one cusp.

Original: These cusps correspond to symbol splitting points of the
Revised: These cusps correspond to symbol splitting points of the

Original: DIC trajectories on the simplex ∆ X (see Appendix A for further details).
Revised: DIC trajectories on the simplex ∆ X (see Appendix A for further details).

Original: In Figure 2 (right), we superimposed these graphs with the trajectories of points ( I ( T k − 1 ; T k ) , I ( Y ; T k )) which suggests an hypothesis on the reason for the discrepancy between the DIC and IB curve.
Revised: In Figure 2 (right), we superimposed these graphs with the trajectories of points ( I ( T k − 1 ; T k ) , I ( Y ; T k )) which suggests an hypothesis on the reason for the discrepancy between the DIC and IB curve.

Original: Indeed, the curves show that the increase in discrepancy between IB and DIC curves occur concomitantly with a brutal decrease of the information
Revised: Indeed, the curves show that the increase in discrepancy between IB and DIC curves occur concomitantly with a brutal decrease of the information

Original: I ( T k − 1 ; T k ) between consecutive layers, suggesting that this abrupt decrease in cross-layer information induces a divergence from the optimal informational trade-off.
Revised: I ( T k − 1 ; T k ) between consecutive layers, suggesting that this abrupt decrease in cross-layer information induces a divergence from the optimal informational trade-off.

Original: Whether this hypothesis is satisfied in real-world cognitive systems or not, these numerical results suggest a phenomenon of divergence, at critical bifurcation points, from the optimal trade-off drawn by the regular IB (1), for information-processing pipelines  resembling very deep neural networks.
Revised: Whether this hypothesis is satisfied in real-world cognitive systems or not, these numerical results suggest a phenomenon of divergence, at critical bifurcation points, from the optimal trade-off drawn by the regular IB (1), for information-processing pipelines with multiple layers resembling very deep neural networks [17].

Original: This phenomenon will be better clarified in future work.
Revised: This phenomenon will be better clarified in future work.

Original: Appendices A IB and DIC trajectories on the simplex
Revised: Appendices A IB and DIC trajectories on the simplex

Original: One can compare the simplex trajectories of the IB and DIC methods.
Revised: One can compare the simplex trajectories of the IB and DIC methods.

Original: Even though a deeper understanding of the observed patterns (in particular how it relates to the main content in Section 3) is left to future work, we still describe them here.
Revised: Even though a deeper understanding of the observed patterns (in particular how it relates to the main content in Section 3) is left to future work, we still describe them here.

Original: In Figure 3 (left), we plot the IB trajectory on the simplex ∆ X of the q β ( X | T ) distributions corresponding to the IB solutions drawing the optimal IB curve in Figure 2 (blue curve).
Revised: In Figure 3 (left), we plot the IB trajectory on the simplex ∆ X of the q β ( X | T ) distributions corresponding to the IB solutions drawing the optimal IB curve in Figure 2 (blue curve).

Original: In Figure 3 (right), we compare this IB trajectory to the DIC trajectory, still on ∆ X , of the q ( X | T k ) distributions that draw the DIC curve in Figure 2 (red curve).
Revised: In Figure 3 (right), we compare this IB trajectory to the DIC trajectory, still on ∆ X , of the q ( X | T k ) distributions that draw the DIC curve in Figure 2 (red curve).

Original: See resp.
Revised: See resp.

Original: Sections 2 and 3 for notations.
Revised: Sections 2 and 3 for notations.

Original: The two kinds of trajectories are to a large extent similar.
Revised: The two kinds of trajectories are to a large extent similar.

Original: Crucially, the DIC trajectory displays symbol merges as k increases, just like the IB trajectory displays symbol merges when β increases — note that the symbol merge in Figure 3 (right) corresponds exactly to the cusp which can be observed on the DIC information curve in Figure 2.
Revised: Crucially, the DIC trajectory displays symbol merges as k increases, just like the IB trajectory displays symbol merges when β increases — note that the symbol merge in Figure 3 (right) corresponds exactly to the cusp which can be observed on the DIC information curve in Figure 2.

Original: However we notice two key differences: first, contrarily to the IB trajectories which have in general non-zero curvatures, the DIC trajectories are always composed of straight lines, between two symbol merges.
Revised: However we notice two key differences: first, contrarily to the IB trajectories which have in general non-zero curvatures, the DIC trajectories are always composed of straight lines, between two symbol merges.

Original: Second, the symbol merge, which can occur in the interior of the simplex for the IB case, happens on the border of the simplex for the DIC trajectories — a feature which was always satisfied for DIC trajectories.
Revised: Second, the symbol merge, which can occur in the interior of the simplex for the IB case, happens on the border of the simplex for the DIC trajectories — a feature which was always satisfied for DIC trajectories.

Original: These numerical results hint towards possible ways of understanding the discrepancy between the
Revised: These numerical results hint towards possible ways of understanding the discrepancy between the

Original: IB and DIC and information curve: it might be partly explained by the discrepancy between the distributions q β ( X | T ) , resp.
Revised: IB and DIC and information curve: it might be partly explained by the discrepancy between the distributions q β ( X | T ) , resp.

Original: q
Revised: q

Original: ( X | T k ) , at which symbol merge occurs in resp.
Revised: ( X | T k ) , at which symbol merge occurs in resp.

Original: the IB and DIC case; and it might also be related to a different nature of the trajectories on the simplex.
Revised: the IB and DIC case; and it might also be related to a different nature of the trajectories on the simplex.

Original: B Values of sample distributions
Revised: B Values of sample distributions

Original: The distributions p ( X, Y ) used for our numerical experiments were sampled from the uniform distribution on the probability simplex.
Revised: The distributions p ( X, Y ) used for our numerical experiments were sampled from the uniform distribution on the probability simplex.

Original: In this paper, we presented only features which were qualitatively always satisfied among the examples we studied.
Revised: In this paper, we presented only features which were qualitatively always satisfied among the examples we studied.

Original: The sample p ( X, Y ) was defined from
Revised: The sample p ( X, Y ) was defined from

Original: [Equation]
Revised: [Equation]

Original: for Figure 1-(a), from p ( X ) =
Revised: for Figure 1-(a), from p ( X ) =

Original: (cid:32) 0 .0 .0 .(cid:33) ,
Revised: (cid:32) 0 .0 .0 .(cid:33) ,

Original: p ( Y | X ) =
Revised: p ( Y | X ) =

Original: (cid:32) 0 .
Revised: (cid:32) 0 .

Original: 562 0 .(cid:33) .
Revised: 562 0 .(cid:33) .

Original: for Figure p ( X ) =
Revised: for Figure p ( X ) =

Original: (cid:32) 0 .0 .0 .(cid:33) ,
Revised: (cid:32) 0 .0 .0 .(cid:33) ,

Original: p ( Y | X ) =
Revised: p ( Y | X ) =

Original: (cid:32) 0 .
Revised: (cid:32) 0 .

Original: 550 0 .(cid:33) , for Figure 2.
Revised: 550 0 .(cid:33) , for Figure 2.

Original: The conditional distributions p ( Y | X ) are represented on the Y simplex in Figure 4.
Revised: The conditional distributions p ( Y | X ) are represented on the Y simplex in Figure 4.