Original: Successive Refinement and Coarsening of the Information Bottleneck
Revised: Successive Refinement and Coarsening of the Information Bottleneck

Original: Abstract
Revised: Abstract

Original: We study two central aspects of information processing in cognitive systems: one is the ability to incorporate fresh information to already learnt models; the other is the “trickling” of information through the many layers of a cognitive processing pipeline.
Revised: We study two central aspects of information processing in cognitive systems: one is the ability to incorporate fresh information to already learnt models; the other is the “trickling” of information through the many layers of a cognitive processing pipeline.

Original: We investigate the extent to which these specific structures of cognitive processing impact their informational optimal limits.
Revised: We investigate the extent to which these specific structures of cognitive processing impact their informational optimal limits.

Original: To do so, we present mathematical characterisations and low-dimensional numerical examples, which explore formal properties of the Information Bottleneck method: namely, how it relates to successive refinement, and successive coarsening of information
Revised: To do so, we present mathematical characterisations and low-dimensional numerical examples, which explore formal properties of the Information Bottleneck method: namely, how it relates to successive refinement, and successive coarsening of information

Original:  Introduction
Revised:  Introduction

Original: One of the most crucial tensions characterising the activity of embodied agents, is that between pursuing one’s own interests, thus behaving in a purposeful and relevant manner, and doing so without violating the constraints on cognitive processing specific to each agent and situation.
Revised: One of the most crucial tensions characterising the activity of embodied agents, is that between pursuing one’s own interests, thus behaving in a purposeful and relevant manner, and doing so without violating the constraints on cognitive processing specific to each agent and situation.

Original: The framework of information-theoretic bounded rationality [14, 7, 18] operationalises this tension as the fundamental trade-off resulting from the optimization of the agent’s preference, while keeping its informationprocessing cost below a given bound.
Revised: The framework of information-theoretic bounded rationality [14, 7, 18] operationalises this tension as the fundamental trade-off resulting from the optimization of the agent’s preference, while keeping its informationprocessing cost below a given bound.

Original: More precisely, on the one hand this information-processing cost is measured with the mutual information I ( X ; T ) between an environmental variable X and an agent’s representation T .
Revised: More precisely, on the one hand this information-processing cost is measured with the mutual information I ( X ; T ) between an environmental variable X and an agent’s representation T .

Original: On the other hand, behavioral relevancy is most often quantified with an explicit utility function, but it can also be measured implicitly through the mutual information I ( Y ; T ) that the representation T extracts about a “relevancy” variable Y .
Revised: On the other hand, behavioral relevancy is most often quantified with an explicit utility function, but it can also be measured implicitly through the mutual information I ( Y ; T ) that the representation T extracts about a “relevancy” variable Y .

Original: Assuming the information about Y can be only extracted from the environmental variable X , i.e., that the Markov chain T − X − Y holds, the representation T then solves the so-called Information Bottleneck (IB) problem [20, 8] arg
Revised: Assuming the information about Y can be only extracted from the environmental variable X , i.e., that the Markov chain T − X − Y holds, the representation T then solves the so-called Information Bottleneck (IB) problem [20, 8] arg

Original: [Equation]
Revised: [Equation]

Original: ( Y
Revised: ( Y

Original: [Equation]
Revised: [Equation]

Original: ) , (1) where λ > 0 controls the complexity-relevancy trade-off — so that the solutions to (1) for varying λ draw the optimal limits on the feasible trade-offs.
Revised: ) , (1) where λ > 0 controls the complexity-relevancy trade-off — so that the solutions to (1) for varying λ draw the optimal limits on the feasible trade-offs.

Original: Note that for the actual computation of solutions to (1), due to the non-linearity of the constraint, one usually rather maximises the Lagrangian
Revised: Note that for the actual computation of solutions to (1), due to the non-linearity of the constraint, one usually rather maximises the Lagrangian

Original: I ( Y ; T ) − βI ( X ; T ) , where the trade-off is now parametrised by the multiplier β .
Revised: I ( Y ; T ) − βI ( X ; T ) , where the trade-off is now parametrised by the multiplier β .

Original: A richer description of real-world biological and artificial agents demands taking into the picture another key characteristic of their information-processing dynamics: namely, that information- processing operates along hierarchies [11, 10, 5, 9], producing sequences of representations, instead of single ones.
Revised: A richer description of real-world biological and artificial agents demands taking into the picture another key characteristic of their information-processing dynamics: namely, that information- processing operates along hierarchies [11, 10, 5, 9], producing sequences of representations, instead of single ones.

Original: Hierarchical and sequential generalisations of the IB problem (1), or related problems, have been proposed: in particular concerning multiple-stage decision-making [7, 13] and the analysis of deep networks’ learning dynamics [19, 22, 21].
Revised: Hierarchical and sequential generalisations of the IB problem (1), or related problems, have been proposed: in particular concerning multiple-stage decision-making [7, 13] and the analysis of deep networks’ learning dynamics [19, 22, 21].

Original: In this paper, we are interested in the following question: how are the information-processing limits drawn by solutions to (1) — which are not constrained to satisfy any specific structure — affected when one does impose hierarchical sequential structures?
Revised: In this paper, we are interested in the following question: how are the information-processing limits drawn by solutions to (1) — which are not constrained to satisfy any specific structure — affected when one does impose hierarchical sequential structures?

Original: We will take a look at two cases of sequential processing.
Revised: We will take a look at two cases of sequential processing.

Original: The first one investigates to which extent the optimal information bounds on (1) can be implemented through incremental incorporation of fresh information into already learned models [12].
Revised: The first one investigates to which extent the optimal information bounds on (1) can be implemented through incremental incorporation of fresh information into already learned models [12].

Original: In other words, in a developmental language, can the optimal bounds on the information trade-off (1) be achieved by a system which first passes through a given stage of development, and then builds on it to acquire further information about the environment [16, 17]?
Revised: In other words, in a developmental language, can the optimal bounds on the information trade-off (1) be achieved by a system which first passes through a given stage of development, and then builds on it to acquire further information about the environment [16, 17]?

Original: The second one considers the loss of optimality, in terms of discrepancy from the bound (1), induced by information being feed-forwardly processed through many layers of a cognitive processing pipeline.
Revised: The second one considers the loss of optimality, in terms of discrepancy from the bound (1), induced by information being feed-forwardly processed through many layers of a cognitive processing pipeline.

Original: We will investigate these questions from a formal point of view, with mathematical characterisations and numerical experiments on low-dimensional examples.
Revised: We will investigate these questions from a formal point of view, with mathematical characterisations and numerical experiments on low-dimensional examples.

Original:  Successive refinement
Revised:  Successive refinement

Original: Consider an incremental learning process where each step incorporates new information so as to refine the previously acquired representations.
Revised: Consider an incremental learning process where each step incorporates new information so as to refine the previously acquired representations.

Original: In a similar spirit to the classic IB, one can describe the optimal informational bounds on this multiple-stage processing: this is precisely what the notion of successive refinement does.
Revised: In a similar spirit to the classic IB, one can describe the optimal informational bounds on this multiple-stage processing: this is precisely what the notion of successive refinement does.

Original: This notion has initially been formulated for classic rate-distortion problems [6, 15], but inspired by recent work [13, 7], we adapt it to the IB framework.
Revised: This notion has initially been formulated for classic rate-distortion problems [6, 15], but inspired by recent work [13, 7], we adapt it to the IB framework.

Original: Consider the following two-stage bottleneck problem 1 : for λ 2 > λ 1 ≥ 0 , we first design a bottleneck T
Revised: Consider the following two-stage bottleneck problem 1 : for λ 2 > λ 1 ≥ 0 , we first design a bottleneck T

Original: = T 1 that solves (1) with λ = λ 1 ; and then design a solution S 2 to
Revised: = T 1 that solves (1) with λ = λ 1 ; and then design a solution S 2 to

Original: [Equation]
Revised: [Equation]

Original: The second, conditional bottleneck S 2 optimally “supplements” the information already optimally acquired in T 1 , yielding a final “two-stage bottleneck” T 2 :=
Revised: The second, conditional bottleneck S 2 optimally “supplements” the information already optimally acquired in T 1 , yielding a final “two-stage bottleneck” T 2 :=

Original: ( T 1 , S 2 ) .
Revised: ( T 1 , S 2 ) .

Original: Definition 1.
Revised: Definition 1.

Original: There is successive refinement from parameters λ 1 to λ 2 > λ 1 , if there is a two-stage bottleneck T 2 :=
Revised: There is successive refinement from parameters λ 1 to λ 2 > λ 1 , if there is a two-stage bottleneck T 2 :=

Original: ( T 1 , S 2 ) with parameters ( λ 1 , λ 2 ) , such that T 2 is also a one-stage bottleneck with parameter λ 2 , i.e., solves the IB problem (1) with T
Revised: ( T 1 , S 2 ) with parameters ( λ 1 , λ 2 ) , such that T 2 is also a one-stage bottleneck with parameter λ 2 , i.e., solves the IB problem (1) with T

Original: =
Revised: =

Original: T 2 and λ = λ 2 .
Revised: T 2 and λ = λ 2 .

Original: Using the chain rule for mutual information [4] and the fact that the inequality constraint must be saturated for the IB problem [1], one can easily prove that there is successive refinement from λto λ 2 if and only if there exists one-stage bottlenecks T 1 and T 2 , with respective parameters λ 1 and λ 2 , such that the Markov chain T 1 − T 2 − X holds.
Revised: Using the chain rule for mutual information [4] and the fact that the inequality constraint must be saturated for the IB problem [1], one can easily prove that there is successive refinement from λto λ 2 if and only if there exists one-stage bottlenecks T 1 and T 2 , with respective parameters λ 1 and λ 2 , such that the Markov chain T 1 − T 2 − X holds.

Original: To determine whether or not these equivalent properties hold true, we propose a geometric approach.
Revised: To determine whether or not these equivalent properties hold true, we propose a geometric approach.

Original: Denote by Hull ( E ) the convex hull of a set
Revised: Denote by Hull ( E ) the convex hull of a set

Original: E , and consider the condition
Revised: E , and consider the condition

Original: [Equation]
Revised: [Equation]

Original: where t 1 , t 2 , resp.
Revised: where t 1 , t 2 , resp.

Original: T 1 , T 2 , denote the symbols, resp.
Revised: T 1 , T 2 , denote the symbols, resp.

Original: alphabets, of distinct one-stage bottlenecks Tand T 2 , with respective parameters λ 1 , λ 2 , where λ 1 < λ 2 .
Revised: alphabets, of distinct one-stage bottlenecks Tand T 2 , with respective parameters λ 1 , λ 2 , where λ 1 < λ 2 .

Original: Proposition 2.
Revised: Proposition 2.

Original: We use the notations defined above.
Revised: We use the notations defined above.

Original: Successive refinement from λ 1 to λ 2 implies (3) , and (3) implies successive refinement if the transition matrix defined by q ( X | T 2 ) is injective.
Revised: Successive refinement from λ 1 to λ 2 implies (3) , and (3) implies successive refinement if the transition matrix defined by q ( X | T 2 ) is injective.

Original: Moreover, if the bottleneck problem (1) has strictly concave information curve 2 , then for any bottleneck
Revised: Moreover, if the bottleneck problem (1) has strictly concave information curve 2 , then for any bottleneck

Original: T , the transition matrix defined by q ( X | T ) can always be chosen injective.
Revised: T , the transition matrix defined by q ( X | T ) can always be chosen injective.

Original: In short, Proposition 2 says that under conditions that are easily satisfied, there is successive refinement if the convex hull of the q ( X | t 2 ) still contains the q ( X | t 1 ) .
Revised: In short, Proposition 2 says that under conditions that are easily satisfied, there is successive refinement if the convex hull of the q ( X | t 2 ) still contains the q ( X | t 1 ) .

Original: Equipped with this new characterisation, we computed bottlenecks on synthetic examples, with example distributions p ( X, Y ) sampled uniformly on the simplex.
Revised: Equipped with this new characterisation, we computed bottlenecks on synthetic examples, with example distributions p ( X, Y ) sampled uniformly on the simplex.

Original: We used the Lagrangian formulation of the IB (see Section 1) — where β is the trade-off parameter — and optimised it with the modified Blahut-Arimoto algorithm [20] combined with reverse deterministic annealing [23], starting at β = ∞ from the generic optimum
Revised: We used the Lagrangian formulation of the IB (see Section 1) — where β is the trade-off parameter — and optimised it with the modified Blahut-Arimoto algorithm [20] combined with reverse deterministic annealing [23], starting at β = ∞ from the generic optimum

Original: T = X .
Revised: T = X .

Original: We chose |X| =
Revised: We chose |X| =

Original: |Y| = 3 , which allows to plot the trajectories of q β ( X | T ) on the simplex ∆ X .
Revised: |Y| = 3 , which allows to plot the trajectories of q β ( X | T ) on the simplex ∆ X .

Original: We only focused on examples with strictly concave information curve and injective q ( X | T 2 ) , so condition (3)
Revised: We only focused on examples with strictly concave information curve and injective q ( X | T 2 ) , so condition (3)

Original: Figure 1 shows two representative examples 4 .
Revised: Figure 1 shows two representative examples 4 .

Original: On both plots 1-(a) and 1-(b), we can visualise the bifurcations at which some q β ( X | t ) splits into two distinct distributions as β grows, yielding an increase of the effective bottleneck cardinality [23].
Revised: On both plots 1-(a) and 1-(b), we can visualise the bifurcations at which some q β ( X | t ) splits into two distinct distributions as β grows, yielding an increase of the effective bottleneck cardinality [23].

Original: More importantly for us here, we observe that
Revised: More importantly for us here, we observe that

Original: Hull { q β 2 ( X | t ) , t ∈ T } seems to have a general tendency to contain the q β 1 ( X | t ) for β 1 < β 2 .
Revised: Hull { q β 2 ( X | t ) , t ∈ T } seems to have a general tendency to contain the q β 1 ( X | t ) for β 1 < β 2 .

Original: For instance, this is the case after the split into 3 distinct symbols.
Revised: For instance, this is the case after the split into 3 distinct symbols.

Original: But the condition can fail: for instance if the condition held, the first curve segments where we have only two symbols (bluer families of points in Figures 1-(a) and 1-(b)) should be linear segments.
Revised: But the condition can fail: for instance if the condition held, the first curve segments where we have only two symbols (bluer families of points in Figures 1-(a) and 1-(b)) should be linear segments.

Original: But this is not the case for Figure 1-(b): there the first trajectory segment is non-negligibly curved — a relatively typical behavior.
Revised: But this is not the case for Figure 1-(b): there the first trajectory segment is non-negligibly curved — a relatively typical behavior.

Original: Thus, even though there are many pairs of β 1 and β 2 > β 1 for which condition (3) holds, these numerical results suggest that the q β 1 ( X | t ) , for β 1 < β 2 , are not always encompassed by Hull { q β 2 ( X | t ) , t ∈ T } .
Revised: Thus, even though there are many pairs of β 1 and β 2 > β 1 for which condition (3) holds, these numerical results suggest that the q β 1 ( X | t ) , for β 1 < β 2 , are not always encompassed by Hull { q β 2 ( X | t ) , t ∈ T } .

Original: We propose to quantify this “spreading out” from the convex hull with the measure of unique information proposed in [3], using the algorithm 5 designed in [2]:
Revised: We propose to quantify this “spreading out” from the convex hull with the measure of unique information proposed in [3], using the algorithm 5 designed in [2]:

Original: [Equation]
Revised: [Equation]

Original: where ∆ q is the set of probabilities r ( X, T 1 , T 2 ) consistent with the bottleneck probabilities 6 , i.e., such that r ( X, T 1 )
Revised: where ∆ q is the set of probabilities r ( X, T 1 , T 2 ) consistent with the bottleneck probabilities 6 , i.e., such that r ( X, T 1 )

Original: =
Revised: =

Original: q ( X, T 1 ) and r ( X, T 2 )
Revised: q ( X, T 1 ) and r ( X, T 2 )

Original: =
Revised: =

Original: q ( X, T 2 ) .
Revised: q ( X, T 2 ) .

Original: This quantity vanishes if and only if there is successive refinement from T 1 to T 2 .
Revised: This quantity vanishes if and only if there is successive refinement from T 1 to T 2 .

Original: In the case of Figure 1-(a), where the convex hull condition (3) seems visually always satisfied, the maximum of UI ( X : T 1 \ T 2 ) over all T 2 finer than T 1 (i.e., with β 2 > β 1 ) is approx.
Revised: In the case of Figure 1-(a), where the convex hull condition (3) seems visually always satisfied, the maximum of UI ( X : T 1 \ T 2 ) over all T 2 finer than T 1 (i.e., with β 2 > β 1 ) is approx.

Original: 4 . 0 · 10 − 5 , whereas for the right trajectory, where (3) is clearly broken, it is approx.
Revised: 4 . 0 · 10 − 5 ; whereas in Figure 1-(b), where (3) is clearly broken, it is approx.

Original: Importantly, even if UI ( X : T 1 \ T 2 ) is nonzero in the second case, it is still noticeably small.
Revised: Importantly, even if UI ( X : T 1 \ T 2 ) is nonzero in the second case, it is still noticeably small.

Original: Among all the examples we studied, it never exceeded 5 .
Revised: Among all the examples we studied, it never exceeded 5 .

Original: 4% of I ( X ; T 1 , T 2 ) .
Revised: 4% of I ( X ; T 1 , T 2 ) .

Original: Overall, these numerical results suggest that imposing two-stage processing, with incorporation of new information at the second stage, does not essentially degrade the optimal informational limits drawn by the classic IB — and the same statement can be made for n -stage processing, by direct iterations of the definitions, arguments and computations in this section.
Revised: Overall, these numerical results suggest that imposing two-stage processing, with incorporation of new information at the second stage, does not essentially degrade the optimal informational limits drawn by the classic IB — and the same statement can be made for n -stage processing, by direct iterations of the definitions, arguments and computations in this section.

Original: This is encouraging for the feasibility of the optimal IB trade-offs (1) in real-world cognitive processing systems, as the representations they possess about their environment are most often the result of a sequential – and in particular, developmental — process.
Revised: This is encouraging for the feasibility of the optimal IB trade-offs (1) in real-world cognitive processing systems, as the representations they possess about their environment are most often the result of a sequential – and in particular, developmental — process.

Original: However, as the modified BA algorithm is only guaranteed to find local optima [20], one should be wary that the observed patterns do not necessarily reflect the behavior of actual, global solutions to (1).
Revised: However, as the modified BA algorithm is only guaranteed to find local optima [20], one should be wary that the observed patterns do not necessarily reflect the behavior of actual, global solutions to (1).

Original:  Dense information coarsening
Revised:  Dense information coarsening

Original: Here we consider the converse situation of that of Section 2, where along a feed-forward cognitive processing pipeline, the information is coarsened rather than refined, without additional external input between layers.
Revised: Here we consider the converse situation of that of Section 2, where along a feed-forward cognitive processing pipeline, the information is coarsened rather than refined, without additional external input between layers.

Original: In other words, we consider a hierarchy of representations T 1 , . . .
Revised: In other words, we consider a hierarchy of representations T 1 , . . .

Original: , T n , that, by assumption, satisfies the Markov chain X − T 1 − · · · − T n , where X is an environmental variable, and T k has only direct access to the previous layer T k − 1 — deep neural networks being an example of this setting [19].
Revised: , T n , that, by assumption, satisfies the Markov chain X − T 1 − · · · − T n , where X is an environmental variable, and T k has only direct access to the previous layer T k − 1 — deep neural networks being an example of this setting [19].

Original: Note that the indices are in reverse order as those of the previous section, because we stick to an indexing describing the order in which representations are produced.
Revised: Note that the indices are in reverse order as those of the previous section, because we stick to an indexing describing the order in which representations are produced.

Original: If each layer extracts information about a fixed relevancy variable Y , then the optimal information trade-off at each layer is drawn by bottlenecks T k with relevancy Y , but where the source X in (1) is replaced by
Revised: If each layer extracts information about a fixed relevancy variable Y , then the optimal information trade-off at each layer is drawn by bottlenecks T k with relevancy Y , but where the source X in (1) is replaced by

Original: T k − 1 .
Revised: T k − 1 .

Original: The question is then: given these hard constraints on the cognitive processing pipeline, how much information optimality do we lose as compared to a one-stage processing as described by (1)?
Revised: The question is then: given these hard constraints on the cognitive processing pipeline, how much information optimality do we lose as compared to a one-stage processing as described by (1)?

Original: We focus on the limit of an infinite number of processing stages, which models cognitive systems where the number of information-processing stages stemming from the sensory interfaces is large.
Revised: We focus on the limit of an infinite number of processing stages, which models cognitive systems where the number of information-processing stages stemming from the sensory interfaces is large.

Original: Moreover we consider a processing highly distributed among stages: in other words, each stage performs only a small amount of compression, resulting in a “trickling down” of information through dense information-processing layers.
Revised: Moreover we consider a processing highly distributed among stages: in other words, each stage performs only a small amount of compression, resulting in a “trickling down” of information through dense information-processing layers.

Original: We thus consider a sequence of bottlenecks ( T k ) k ∈ N ∗ , where
Revised: We thus consider a sequence of bottlenecks ( T k ) k ∈ N ∗ , where

Original: T k is a bottleneck with source T k − 1 and relevancy Y .
Revised: T k is a bottleneck with source T k − 1 and relevancy Y .

Original: We choose the sequence ( β k ) k ∈ N ∗ with small increments, such that I ( X ; T 1 )
Revised: We choose the sequence ( β k ) k ∈ N ∗ with small increments, such that I ( X ; T 1 )

Original: ≈ I ( X ; Y )
Revised: ≈ I ( X ; Y )

Original: and I ( X ; T k ) → 0 when k → + ∞ .
Revised: and I ( X ; T k ) → 0 when k → + ∞ .

Original: We will refer here to this process as the “dense information coarsening” (DIC for short).
Revised: We will refer here to this process as the “dense information coarsening” (DIC for short).

Original: We compared, for distributions p ( X, Y ) sampled uniformly on the simplex, the information curve of the regular IB (1) with fixed source X , with the DIC information curve — i.e., the curve drawn by points ( I ( X ; T k ) , I ( Y ; T k )) .
Revised: We compared, for distributions p ( X, Y ) sampled uniformly on the simplex, the information curve of the regular IB (1) with fixed source X , with the DIC information curve — i.e., the curve drawn by points ( I ( X ; T k ) , I ( Y ; T k )) .

Original: Figure 2 (left) shows a representative example 7 with |X| =
Revised: Figure 2 (left) shows a representative example 7 with |X| =

Original: |Y| = 3 .
Revised: |Y| = 3 .

Original: Here, like other examples explored (not shown), the DIC curve tracks rather neatly the regular IB curve.
Revised: Here, and in the other examples explored (not shown), the DIC curve tracks rather neatly the regular IB curve.

Original: However, the discrepancy between the curves is non-negligible and the DIC curve displays a distinctive structure: as opposed to the IB curve which is always concave, the DIC curve is only piece-wise concave, resulting in slight cusps at the junctions — in Figure 2 (left), there are two such pieces of concave curve and one cusp.
Revised: However, the discrepancy between the curves is non-negligible and the DIC curve displays a distinctive structure: as opposed to the IB curve which is always concave, the DIC curve is only piece-wise concave, resulting in slight cusps at the junctions — in Figure 2 (left), there are two such pieces of concave curve and one cusp.

Original: These cusps correspond to symbol splitting points of the
Revised: These cusps correspond to symbol splitting points of the

Original: DIC trajectories on the simplex ∆ X (see Appendix A for further details).
Revised: DIC trajectories on the simplex ∆ X (see Appendix A for further details).

Original: In Figure 2 (right), we superimposed these graphs with the trajectories of points ( I ( T k − 1 ; T k ) , I ( Y ; T k )) which suggests a hypothesis on the reason for the discrepancy between the DIC and IB curve.
Revised: In Figure 2 (right), we superimposed these graphs with the trajectories of points ( I ( T k − 1 ; T k ) , I ( Y ; T k )) which suggests a hypothesis on the reason for the discrepancy between the DIC and IB curve.

Original: Indeed, the curves show that the increase in discrepancy between IB and DIC curves occur concomitantly with a brutal decrease of the information
Revised: Indeed, the curves show that the increase in discrepancy between IB and DIC curves occur concomitantly with a brutal decrease of the information

Original: I ( T k − 1 ; T k ) between consecutive layers, suggesting that this abrupt decrease in cross-layer information induces a divergence from the optimal informational trade-off.
Revised: I ( T k − 1 ; T k ) between consecutive layers, suggesting that this abrupt decrease in cross-layer information induces a divergence from the optimal informational trade-off.

Original: Whether this hypothesis is satisfied in real-world cognitive systems or not, these numerical results suggest a phenomenon of divergence, at critical bifurcation points, from the optimal trade-off drawn by the regular IB (1), for information-processing pipelines with multiple layers resembling very deep neural networks [19].
Revised: Whether this hypothesis is satisfied in real-world cognitive systems or not, these numerical results suggest a phenomenon of divergence, at critical bifurcation points, from the optimal trade-off drawn by the regular IB (1), for information-processing pipelines with multiple layers resembling very deep neural networks [19].

Original: This phenomenon will be better clarified in future work.
Revised: This phenomenon will be better clarified in future work.

Original: Appendices A IB and DIC trajectories on the simplex
Revised: Appendices A IB and DIC trajectories on the simplex

Original: One can compare the simplex trajectories of the IB and DIC methods.
Revised: One can compare the simplex trajectories of the IB and DIC methods.

Original: Even though a deeper understanding of the observed patterns (in particular how they relate to the main content in Section 3) is left to future work, we still describe them here.
Revised: Even though a deeper understanding of the observed patterns (in particular how they relate to the main content in Section 3) is left to future work, we still describe them here.

Original: In Figure 3 (left), we plot the IB trajectory, on the simplex ∆ X , of the q β ( X | T ) distributions corresponding to the IB solutions drawing the optimal
Revised: In Figure 3 (left), we plot the IB trajectory, on the simplex ∆ X , of the q β ( X | T ) distributions corresponding to the IB solutions drawing the optimal

Original: IB curve in Figure 2 (blue curve).
Revised: IB curve in Figure 2 (blue curve).

Original: This IB trajectory is compared, in Figure 3 (right), with the DIC trajectory, still on ∆ X , of the q ( X | T k ) distributions that draw the DIC curve in Figure 2 (red curve).
Revised: This IB trajectory is compared, in Figure 3 (right), with the DIC trajectory, still on ∆ X , of the q ( X | T k ) distributions that draw the DIC curve in Figure 2 (red curve).

Original: See resp.
Revised: See resp.

Original: Sections 2 and 3 for notations.
Revised: Sections 2 and 3 for notations.

Original: The two kinds of trajectories are to a large extent similar.
Revised: The two kinds of trajectories are to a large extent similar.

Original: Crucially, the DIC trajectory displays symbol merges as k increases, just like the IB trajectory displays symbol merges when β decreases.
Revised: Crucially, the DIC trajectory displays symbol merges as k increases, just like the IB trajectory displays symbol merges when β decreases.

Original: Moreover the symbol merge in Figure 3 (right) corresponds exactly to the cusp which can be observed on the DIC information curve in Figure 2.
Revised: Moreover the symbol merge in Figure 3 (right) corresponds exactly to the cusp which can be observed on the DIC information curve in Figure 2.

Original: However we notice two key differences: first, contrarily to the IB trajectories which have in general non-zero curvatures, the DIC trajectories are always composed of straight lines between two symbol merges.
Revised: However we notice two key differences: first, contrarily to the IB trajectories which have in general non-zero curvatures, the DIC trajectories are always composed of straight lines between two symbol merges.

Original: Second, the symbol merge, which can occur in the interior of the simplex for the IB case, happens on the border of the simplex for the DIC trajectories — a feature which was always satisfied for  explored examples of DIC trajectories.
Revised: Second, the symbol merge, which can occur in the interior of the simplex for the IB case, happens on the border of the simplex for the DIC trajectories — a feature which was always satisfied for other explored examples of DIC trajectories.

Original: These numerical results thus hint towards possible ways of understanding the discrepancy between the IB and DIC and information curve: it might be partly explained by the discrepancy between the distributions q β ( X | T ) , resp.
Revised: These numerical results thus hint towards possible ways of understanding the discrepancy between the IB and DIC and information curve: it might be partly explained by the discrepancy between the distributions q β ( X | T ) , resp.

Original: q
Revised: q

Original: ( X | T k ) , at which symbol merge occurs in resp.
Revised: ( X | T k ) , at which symbol merge occurs in resp.

Original: the IB and DIC case; and it might also be related to a different nature of the trajectories on the simplex.
Revised: the IB and DIC case; and it might also be related to a different nature of the trajectories on the simplex.

Original: B Values of sample distributions
Revised: B Values of sample distributions

Original: The distributions p ( X, Y ) used for our numerical experiments were sampled from the uniform distribution on the probability simplex.
Revised: The distributions p ( X, Y ) used for our numerical experiments were sampled from the uniform distribution on the probability simplex.

Original: In this paper, we presented only features which were qualitatively always satisfied among the examples we studied.
Revised: In this paper, we presented only features which were qualitatively always satisfied among the examples we studied.

Original: ForFigure 1-(a)m the sample p ( X, Y ) was defined from
Revised: For Figure 1-(a), the sample p ( X, Y ) was defined from

Original: [Equation]
Revised: [Equation]

Original: for Figure 1-(b), from p ( X )
Revised: for Figure 1-(b), from p ( X )

Original: = (cid:32) 0 .0 .0 .(cid:33)
Revised: = (cid:32) 0 .0 .0 .(cid:33)

Original: , p ( Y | X ) =
Revised: , p ( Y | X ) =

Original: (cid:32) 0 .
Revised: (cid:32) 0 .

Original: 562 0 .(cid:33) ; and for Figure2, from p ( X )=
Revised: 562 0 .(cid:33) ; and for Figure from p ( X )

Original:  (cid:32) 0 .0 .0 .(cid:33),
Revised: = (cid:32) 0 .0 .0 .(cid:33)

Original:  p ( Y | X ) =
Revised: , p ( Y | X ) =

Original: (cid:32) 0 .
Revised: (cid:32) 0 .

Original: 550 0 .(cid:33) .
Revised: 550 0 .(cid:33) .

Original: The respective conditional distributions p ( Y | X ) are represented on the Y simplex in Figure 4.
Revised: The respective conditional distributions p ( Y | X ) are represented on the Y simplex in Figure 4.