Original: L OSSY C OMPRESSION F OR L OSSLESS P REDICTION
Revised: L OSSY C OMPRESSION F OR L OSSLESS P REDICTION

Original: A BSTRACT
Revised: A BSTRACT

Original: Most data is automatically collected and only ever “seen” by algorithms.
Revised: Most data is automatically collected and only ever “seen” by algorithms.

Original: Yet, data compressors preserve perceptual ﬁdelity rather than just the information needed by algorithms performing downstream tasks.
Revised: Yet, data compressors preserve perceptual ﬁdelity rather than just the information needed by algorithms performing downstream tasks.

Original: In this paper, we characterize the minimum bit-rate required to ensure high performance on all predictive tasks that are invariant under a set of transformations, such as data augmentations.
Revised: In this paper, we characterize the minimum bit-rate required to ensure high performance on all predictive tasks that are invariant under a set of transformations, such as data augmentations.

Original: Based on our theory, we design unsupervised objectives for training neural compressors.
Revised: Based on our theory, we design unsupervised objectives for training neural compressors.

Original: Using these objectives, we achieve rate savings of around 60% on standard datasets,like MNIST, without decreasing  classiﬁcation performance.
Revised: Using these objectives, we train a generic compressor that achieves substantial rate savings (more than 1000 × on Imagenet) compared to JPEG on 8 datasets, without decreasing downstream classiﬁcation performance.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: [Figure]
Revised: [Figure]

Original:  Zetabytes ( 10 21 ) of data are collected every year (Reinsel et al., 2017), which is too much for humans to process.
Revised: Standard rec [Figure] Zetabytes ( 10 21 ) of data are collected every year (Reinsel et al., 2017), which is too much for humans to process.

Original: As a result, most of this data will only be processed by algorithms performing a task.
Revised: As a result, most of this data will only be processed by algorithms performing a task.

Original: So, there is a growing need for compression methods that retain only the information necessary to ensure high performance on downstream tasks.
Revised: So, there is a growing need for compression methods that retain only the information necessary to ensure high performance on downstream tasks.

Original: Existing compression methods assume either that all information is important or that the goal is perceptual ﬁdelity.
Revised: Existing compression methods assume either that all information is important or that the goal is perceptual ﬁdelity.

Original: However, much of that information is not useful for downstream tasks.
Revised: However, much of that information is not useful for downstream tasks.

Original: For example, image classiﬁcation is often invariant under small rescalings or rotations, but a standard compressor will faithfully reconstruct this information.
Revised: For example, image classiﬁcation is often invariant under small rescalings or rotations, but a standard compressor will faithfully reconstruct this information.

Original: If we care only about predictive performance, we should be able to improve compression by discarding such information, as seen in Fig.
Revised: If we care only about predictive performance, we should be able to improve compression by discarding such information, as seen in Fig.

Original: Our goal is to quantify the bit-rate gains that come from removing this unnecessary information and to learn compressors that discards it.
Revised: Our goal is to quantify the bit-rate gains that come from removing this unnecessary information and to learn compressors that discards it.

Original: The minimum bit-rate required for high performance on a supervised task corresponds to compressing the labels.
Revised: The minimum bit-rate required for high performance on a supervised task corresponds to compressing the labels.

Original: Achieving this rate requires access to the labels, and caring only about a single task.
Revised: Achieving this rate requires access to the labels, and caring only about a single task.

Original: Instead, we want a compressed representation that ensures good performance on any future tasks of interest.
Revised: Instead, we want a compressed representation that ensures good performance on any future tasks of interest.

Original: Importantly, this set will rarely be known at compression time or might be too large to even enumerate.
Revised: Importantly, this set will rarely be known at compression time or might be too large to even enumerate.

Original: We overcome these challenges by focusing on tasks that are invariant under user-deﬁned transformations.
Revised: We overcome these challenges by focusing on tasks that are invariant under user-deﬁned transformations.

Original: In this paper, we characterize the minimum bit-rate needed to ensure predictability of all invariant tasks.
Revised: In this paper, we characterize the minimum bit-rate needed to ensure predictability of all invariant tasks.

Original: The key is that we construct a worst-case task, which bounds your performance on any invariant tasks.
Revised: The key is that we construct a worst-case task, which bounds your performance on any invariant tasks.

Original: As a result, the bitrate required to perform well on all invariant tasks is exactly the rate required to compress the labels of the worst-case task.
Revised: As a result, the bit- rate required to perform well on all invariant tasks is exactly the rate required to compress the labels of the worst-case task.

Original: Intuitively, this task is to recognize which examples are transformed version of one another, and the rate savings come from discarding that information, as seen in Fig.
Revised: Intuitively, this task is to recognize which examples are transformed version of one another, and the rate savings come from discarding that information, as seen in Fig.

Original: In addition to establishing optimal rates, we provide unsupervised neural compression methods that approximate these rates.
Revised: In addition to establishing optimal rates, we provide unsupervised neural compression methods that approximate these rates.

Original: In particular, we design two training objectives.
Revised: In particular, we design two training objectives.

Original: One is a modiﬁed variational autoencoder (VAE) (Kingma & Welling, 2014) that reconstructs prototypical examples, as shown in
Revised: One is a modiﬁed variational autoencoder (VAE) (Kingma & Welling, 2014) that reconstructs prototypical examples, as shown in

Original: Fig.
Revised: Fig.

Original: The second is a simple modiﬁcation of InfoNCE (Oord et al., 2019), used in self-supervised learning (SSL), which allows us to convert pre-trained SSL models into powerful compressors.
Revised: The second is a simple modiﬁcation of InfoNCE (Oord et al., 2019), used in self-supervised learning (SSL), which allows us to convert pre-trained SSL models into powerful compressors.

Original:  M INIMUM B IT -R ATES FOR H IGH P REDICTIVE P ERFORMANCE
Revised:  M INIMUM B IT -R ATES FOR H IGH P REDICTIVE P ERFORMANCE

Original: The goal of lossy compression is to ﬁnd the number of bits (the bit-rate ) required to store a r.v X so that it can be reconstructed to within a certain tolerance.
Revised: The goal of lossy compression is to ﬁnd the number of bits (the bit-rate ) required to store a r.v X so that it can be reconstructed to within a certain tolerance.

Original: Speciﬁcally, given a distribution p ( Z | X ) mapping X to a r.v. Z (the representation ) and a distortion measure D[ X, Z ] , Shannon’s (1959) rate-distortion (RD) theory, characterizes the minimal achievable rate for a distortion threshold δ by
Revised: Speciﬁcally, given a distribution p ( Z | X ) mapping X to a r.v. Z (the representation ) and a distortion measure D[ X, Z ] , Shannon’s (1959) rate-distortion (RD) theory, characterizes the minimal achievable rate for a distortion threshold δ by

Original: [Equation]
Revised: [Equation]

Original: Our aim is different: to ﬁnd the minimal bit-rate required to store X so that we can still achieve high performance on a set of downstream tasks.
Revised: Our aim is different: to ﬁnd the minimal bit-rate required to store X so that we can still achieve high performance on a set of downstream tasks.

Original: To do this, we deﬁne a distortion that ensures that predicting from the compressed Z is approximately as good as from X .
Revised: To do this, we deﬁne a distortion that ensures that predicting from the compressed Z is approximately as good as from X .

Original: All proofs are in Appx.
Revised: All proofs are in Appx.

Original: C.
Revised: C.

Original: Let us represent a downstream tasks by a set of r.v.s T
Revised: Let us represent a downstream tasks by a set of r.v.s T

Original: =
Revised: =

Original: { Y 1 , Y 2 , . . .
Revised: { Y 1 , Y 2 , . . .

Original: } , which are jointly distributed with X and represent all variables that we may be interested in predicting.
Revised: } , which are jointly distributed with X and represent all variables that we may be interested in predicting.

Original: For example, Y 1 ∈ T might indicate whether X is an image of a dog, while Y 2 ∈ T might indicate whether X is hand-drawn.
Revised: For example, Y 1 ∈ T might indicate whether X is an image of a dog, while Y 2 ∈ T might indicate whether X is hand-drawn.

Original: Let R[ Y | X ] = inf q E p ( X,Y )
Revised: Let R[ Y | X ] = inf q E p ( X,Y )

Original: [ − log q ( Y | X )] be the best log-loss risk when predicting Y from X .
Revised: [ − log q ( Y | X )] be the best log-loss risk when predicting Y from X .

Original: We would like to ﬁnd a compressed representation Z such that the increase in risk due to predicting
Revised: We would like to ﬁnd a compressed representation Z such that the increase in risk due to predicting

Original: Y ∈ T from Z (as opposed to X ) is bounded, i.e., R[ Y | Z ] − R[ Y | X ] ≤ δ , for all tasks.
Revised: Y ∈ T from Z (as opposed to X ) is bounded, i.e., R[ Y | Z ] − R[ Y | X ] ≤ δ , for all tasks.

Original: This suggests using the following as a distortion measure for our rate-distortion theory,
Revised: This suggests using the following as a distortion measure for our rate-distortion theory,

Original: [Equation]
Revised: [Equation]

Original: Unfortunately, working with Eq.
Revised: Unfortunately, working with Eq.

Original: (2) assumes access to all downstream tasks of interest T during compression and the ability to optimize over them, which is unrealistic in practice.
Revised: (2) assumes access to all downstream tasks of interest T during compression and the ability to optimize over them, which is unrealistic in practice.

Original: However, sets of tasks that we care about are oriented to human goals, which suggests that there may be exploitable structure in realistic task sets.
Revised: However, sets of tasks that we care about are oriented to human goals, which suggests that there may be exploitable structure in realistic task sets.

Original: For example, image classiﬁcation often relates to the concepts present in the image, rather than ﬁne-grain details about brightness and object positions.
Revised: For example, image classiﬁcation often relates to the concepts present in the image, rather than ﬁne-grain details about brightness and object positions.

Original: These tasks are thus invariant to mild augmentation such as brightness changes and translations.
Revised: These tasks are thus invariant to mild augmentation such as brightness changes and translations.

Original: In such case, the maximization in Eq.
Revised: In such case, the maximization in Eq.

Original: (2) is achieved by certain “hardest” invariant tasks.
Revised: (2) is achieved by certain “hardest” invariant tasks.

Original: Intuitively, these hard tasks consist of retaining only the information which you should not be invariant to.
Revised: Intuitively, these hard tasks consist of retaining only the information which you should not be invariant to.

Original: This can essentially be done by predicting a prototypical version of the input X , denoted as M ( X ) .
Revised: This can essentially be done by predicting a prototypical version of the input X , denoted as M ( X ) .

Original: Formally, we assume that the conditional distribution of each Y ∈ T is invariant to an equivalence relation ∼ on X ’s sample space .
Revised: Formally, we assume that the conditional distribution of each Y ∈ T is invariant to an equivalence relation ∼ on X ’s sample space .

Original: 1 That is, x ∼ x (cid:48) =
Revised: 1 That is, x ∼ x (cid:48) =

Original: ⇒ p
Revised: ⇒ p

Original: ( Y | x ) =
Revised: ( Y | x ) =

Original: p ( Y | x [Equation]
Revised: p ( Y | x (cid:48) ) .

Original: Under weak regularity conditions, we prove that there exists a maximal invariant 2 task M ( X ) ∈ T that simpliﬁes Eq.[Equation]
Revised: Under weak regularity conditions, we prove that there exists a maximal invariant 2 task M ( X ) ∈ T that simpliﬁes Eq.

Original: ) to
Revised: (2) to

Original: [Equation]
Revised: [Equation]

Original: Our “Rate-Invariance” theorem uses this simpliﬁcation to characterize the minimal bit-rate required to ensure high predictive performance on invariant tasks T .
Revised: Our “Rate-Invariance” theorem uses this simpliﬁcation to characterize the minimal bit-rate required to ensure high predictive performance on invariant tasks T .

Original: Let H[ · ] denote the (discrete or differential) entropy.
Revised: Let H[ · ] denote the (discrete or differential) entropy.

Original: Intuitively, Eq.
Revised: Intuitively, Eq.

Original: (3) suggests that the minimal bit-rate is related to compressing M ( X ) , which requires H[ M ( X )] bits.
Revised: (3) suggests that the minimal bit-rate is related to compressing M ( X ) , which requires H[ M ( X )] bits.

Original: We formalize this by incorporating our distortion in RD theory.
Revised: We formalize this by incorporating our distortion in RD theory.

Original: Theorem 1 (Rate Invariance) .
Revised: Theorem 1 (Rate Invariance) .

Original: For δ ≥ 0 , let Rate ( δ ) denote the minimum achievable bit-rate for transmitting Z such that for any invariant Y ∈ T we have R[ Y | Z ] − R[ Y | X ] ≤ δ .
Revised: For δ ≥ 0 , let Rate ( δ ) denote the minimum achievable bit-rate for transmitting Z such that for any invariant Y ∈ T we have R[ Y | Z ] − R[ Y | X ] ≤ δ .

Original: Then Rate ( δ ) is 

Revised: Then Rate ( δ ) is

Original: if δ ≥ H[ M ( X )]
Revised: if δ ≥ H[ M ( X )]

Original: and otherwise it is ﬁnite and given by
Revised: and otherwise it is ﬁnite and given by

Original: [Equation]
Revised: [Equation]

Original: Theorem 1 relates compression and statistical learning theory by showing that allowing a δ decrease in log-loss performance can save exactly δ bits during compression.
Revised: Theorem 1 relates compression and statistical learning theory by showing that allowing a δ decrease in log-loss performance can save exactly δ bits during compression.

Original: To illustrate Thm. 1, consider a scenario where we require no loss in predictive performance, i.e., Rate (0)
Revised: To illustrate Thm. 1, consider a scenario where we require no loss in predictive performance, i.e., Rate (0)

Original: =
Revised: =

Original: H[ X ] − H
Revised: H[ X ] − H

Original: [ X | M ( X )] , and contrast this rate to the standard lossless compression rate, H[ X ] .
Revised: [ X | M ( X )] , and contrast this rate to the standard lossless compression rate, H[ X ] .

Original: That is, we can remove any information in X that is not in M ( X ) while ensuring lossless prediction of all invariant tasks.
Revised: That is, we can remove any information in X that is not in M ( X ) while ensuring lossless prediction of all invariant tasks.

Original: Consider compressing a sequence of n coin ﬂips.
Revised: Consider compressing a sequence of n coin ﬂips.

Original: If one is only interested in predicting labels that are permutation invariant, then instead of compressing the entire sequence, one could simply compress the number of heads.
Revised: If one is only interested in predicting labels that are permutation invariant, then instead of compressing the entire sequence, one could simply compress the number of heads.

Original: The number of heads is a maximal invariant for permutation invariance.
Revised: The number of heads is a maximal invariant for permutation invariance.

Original: In this case, the sequence can be compressed to a bit-rate that grows as O (log n ) , as opposed to O ( n ) for the lossless compression case.
Revised: In this case, the sequence can be compressed to a bit-rate that grows as O (log n ) , as opposed to O ( n ) for the lossless compression case.

Original: In we show how our result recovers other previous results (i) lossless compression; (ii) unlabeled graph compression (Rashevsky, 1955); (iii) multiset compression (Varshney & Goyal, 2007); (iv) the information bottleneck (Tishby et al., 2000).
Revised: In we show how our result recovers other previous results (i) lossless compression; (ii) unlabeled graph compression (Rashevsky, 1955); (iii) multiset compression (Varshney & Goyal, 2007); (iv) the information bottleneck (Tishby et al., 2000).

Original:  L EARNING I NVARIANT C OMPRESSION USING D ATA A UGMENTATIONS
Revised:  L EARNING I NVARIANT C OMPRESSION USING D ATA A UGMENTATIONS

Original: In this section, we design practical loss functions for training neural compressors that approximate optimal rates.
Revised: In this section, we design practical loss functions for training neural compressors that approximate optimal rates.

Original: We do so by optimizing an unconstrained (equivalent 3 ) formulation of Eq.
Revised: We do so by optimizing an unconstrained (equivalent 3 ) formulation of Eq.

Original: (1), i.e.,
Revised: (1), i.e.,

Original: [Equation]
Revised: [Equation]

Original: Both terms in the minimization of Eq.
Revised: Both terms in the minimization of Eq.

Original: (5) are challenging to compute.
Revised: (5) are challenging to compute.

Original: Here we address how to approximate them.
Revised: Here we address how to approximate them.

Original: See Appx.
Revised: See Appx.

Original: D for derivations and the resulting training algorithms.
Revised: D for derivations and the resulting training algorithms.

Original: For I[ X ; Z ] , we rely on Ballé et al.’s (2017)
Revised: For I[ X ; Z ] , we rely on Ballé et al.’s (2017)

Original: Variational Compressors (VC), which uses a speciﬁc entropy model q θ ( Z ) and the bound I
Revised: Variational Compressors (VC), which uses a speciﬁc entropy model q θ ( Z ) and the bound I

Original: [ Z ; X ] ≤ min θ E p ( X ) p ϕ
Revised: [ Z ; X ] ≤ min θ E p ( X ) p ϕ

Original: ( Z | X ) [ − log q θ ( Z )] .
Revised: ( Z | X ) [ − log q θ ( Z )] .

Original: The challenge with our term R[ M ( X ) | Z ] is that maximal invariants are typically inaccessible.
Revised: The challenge with our term R[ M ( X ) | Z ] is that maximal invariants are typically inaccessible.

Original: To overcome this we make two assumptions, both of which are implicit in SSL.
Revised: To overcome this we make two assumptions, both of which are implicit in SSL.

Original: First, we assume that we can sample data augmentations A to which our tasks should be invariant, e.g., image shearing.
Revised: First, we assume that we can sample data augmentations A to which our tasks should be invariant, e.g., image shearing.

Original: This enables sampling of equivalent points x , i.e., x ∼ X .
Revised: This enables sampling of equivalent points x , i.e., x ∼ X .

Original: Second, we assume that no two observations X, X (cid:48)
Revised: Second, we assume that no two observations X, X (cid:48)

Original: ∈ D in our dataset are equivalent X (cid:54)∼ X (cid:48) , i.e., they are not augmented versions of one another.
Revised: ∈ D in our dataset are equivalent X (cid:54)∼ X (cid:48) , i.e., they are not augmented versions of one another.

Original: This assumption lets us use each sample in D as the maximal invariant of its own equivalence class, i.e., M ( x ) =
Revised: This assumption lets us use each sample in D as the maximal invariant of its own equivalence class, i.e., M ( x ) =

Original: X for all x ∼ X .
Revised: X for all x ∼ X .

Original: The following two variational upper bounds on our invariance distortion encourage networks to solve the task (explicitly or implicitly) of mapping A ( X ) back to X .
Revised: The following two variational upper bounds on our invariance distortion encourage networks to solve the task (explicitly or implicitly) of mapping A ( X ) back to X .

Original: Variational Invariant Compressor (VIC).
Revised: Variational Invariant Compressor (VIC).

Original: Our ﬁrst loss is closely related to the VC and the VAE.
Revised: Our ﬁrst loss is closely related to the VC and the VAE.

Original: The model has an encoder p ϕ
Revised: The model has an encoder p ϕ

Original: ( Z | X ) , an entropy model q θ ( Z ) , and a decoder q φ ( X | Z ) .
Revised: ( Z | X ) , an entropy model q θ ( Z ) , and a decoder q φ ( X | Z ) .

Original: Given a data point X , we apply an augmentation A ( X ) , pass it through the encoder to get a representation Z .
Revised: Given a data point X , we apply an augmentation A ( X ) , pass it through the encoder to get a representation Z .

Original: The decoder then attempts to reconstruct the unaugmented X from A ( X ) .
Revised: The decoder then attempts to reconstruct the unaugmented X from A ( X ) .

Original: This leads to the objective,
Revised: This leads to the objective,

Original: [Equation]
Revised: [Equation]

Original: Bottleneck InfoNCE (BINCE).
Revised: Bottleneck InfoNCE (BINCE).

Original: Our second loss is based on InfoNCE (Oord et al., 2019).
Revised: Our second loss is based on InfoNCE (Oord et al., 2019).

Original: For every
Revised: For every

Original: X , we sample a sequence of random points X
Revised: X , we sample a sequence of random points X

Original: =
Revised: =

Original: ( X + , X 1 , . . .
Revised: ( X + , X 1 , . . .

Original: , X n ) ,
Revised: , X n ) ,

Original: where X + = A ( X ) is an augmentation of X (“positive”) and each X i are non equivalent examples
Revised: where X + = A ( X ) is an augmentation of X (“positive”) and each X i are non equivalent examples

Original: X i (cid:54)∼ X (“negatives”).
Revised: X i (cid:54)∼ X (“negatives”).

Original: Let Z =
Revised: Let Z =

Original: ( Z + , Z 1 , . . .
Revised: ( Z + , Z 1 , . . .

Original: , Z n ) , be the corresponding representations given by p ϕ
Revised: , Z n ) , be the corresponding representations given by p ϕ

Original: ( Z | X ) .
Revised: ( Z | X ) .

Original: Let f ψ be a discriminator that is optimized to score the equivalence of two representation.
Revised: Let f ψ be a discriminator that is optimized to score the equivalence of two representation.

Original: The ﬁnal loss is:
Revised: The ﬁnal loss is:

Original: [Equation]
Revised: [Equation]

Original: Eq.
Revised: Eq.

Original: (7) is the standard SSL loss with an additional entropy bottleneck.
Revised: (7) is the standard SSL loss with an additional entropy bottleneck.

Original: 4 Instead of directly predicting
Revised: 4 Instead of directly predicting

Original: X as in VIC, BINCE retains information about X by classifying (as seen by the softmax) which
Revised: X as in VIC, BINCE retains information about X by classifying (as seen by the softmax) which

Original: Z is associated with an equivalent example x ∼ X .
Revised: Z is associated with an equivalent example x ∼ X .

Original: This has the advantage of not requiring a high dimensional decoder but can require many negative examples n .
Revised: This has the advantage of not requiring a high dimensional decoder but can require many negative examples n .

Original: Both VIC and BINCE give rise to efﬁcient compressors by passing X through p ϕ
Revised: Both VIC and BINCE give rise to efﬁcient compressors by passing X through p ϕ

Original: ( Z | X ) and entropy coding using q θ ( Z ) .
Revised: ( Z | X ) and entropy coding using q θ ( Z ) .

Original: In theory they can recover the δ = 0 optimal bit-rate, i.e., H[ M ( X )] , in the limit of inﬁnite samples ( |D| , n ) and unconstrained variational families.
Revised: In theory they can recover the δ = 0 optimal bit-rate, i.e., H[ M ( X )] , in the limit of inﬁnite samples ( |D| , n ) and unconstrained variational families.

Original:  E XPERIMENTS AND D ISCUSSION
Revised:  E XPERIMENTS AND D ISCUSSION

Original: Let us evaluate our learned compressors.
Revised: Let us evaluate our learned compressors.

Original: Refer to Appx.
Revised: See Appx.

Original: F for experimental details. [Figure] Fig
Revised: F for experimental details and Appx.

Original: Curves are generated by sweeping β averaged
Revised: G for additional results.

Original: Banana.
Revised: Banana.

Original: To illustrate our method, we compress samples of the Banana distribution (Ballé et al., 2020) assumingthat downstream tasks are rotation invariant w.r.t.
Revised: We compress samples of the Banana distribution (Ballé et al., 2020) assuming downstream tasks are rotation invariant w.r.t.

Original: the origin.
Revised: the origin.

Original: As a result, we can disregard angular information of samples x , i.e. we learn to retain only information about the norm of x, M ( x ) = (cid:107) x (cid:107) 2 . Our method (Fig. 1, right) hence learns disk-shaped quantization bins, with reconstruction points retaining only radial information.
Revised: Our method (Fig. 1, right) learns diskshaped quantization bins to retain only information about the norm of x, M ( x ) = (cid:107) x (cid:107) 2 , and disregard angular information.

Original: As a result, the bitrate is 2 .
Revised: As a result, the bitrate is 2 .

Original: 5bits per sample as opposed to 5 .
Revised: 5 as opposed to 5 .

Original: 4 for a compressor with standard distortion (Fig. 1, left).
Revised: 4 for a compressor with standard distortion (Fig. 1, left).

Original: In Appx.

Original: G.1 we provide examples for other invariances.

Original: MNIST.
Revised: MNIST.

Original: Next, we evaluate the effect of standard geometrical augmentations on MNIST compression performance.
Revised: Next, we evaluate the effect of standard geometrical augmentations on MNIST compression performance.

Original: We contrast our VIC with a standard VC.
Revised: We contrast our VIC with a standard VC.

Original: We present two scenarios, (i) when we know the downstream tasks to be invariant to the set of augmentations, which we enforce by augmenting the test set; however (ii) more commonly we do not know exactly what transformations the tasks are invariant to, which we evaluate by leaving the test set unaugmented. When the invariances are known, we achieve a bitrate of 48 bits compared to 130 for the
Revised: We assume that we know the downstream tasks to be invariant to the set of augmentations, which we enforce by augmenting the test set.

Original:  VC (Rate-Error curve in Fig. 8b), and reconstructions from both models can be seen in Fig. In the more realistic scenarios where we do not know the exact augmentations, VIC requires 54 bits compared to 74 for VC. In all cases compression levels are chosen to ensure that a ResNet can achieve 99% downstreamtest accuracy from the reconstructions.
Revised: To reach at least 99% downstream accuracy our compressor requires onlybits compared to 130 for the VC (reconstructions from both models can be seen in Fig. 2). Similar gains can be seen for any level of downstream accuracy as seen in Fig.

Original: For additional
Revised: [Table]

Original: MNIST experiments see Appx.
Revised: Our MLP probe (Acc.)

Original: G.2.
Revised: [Table]

Original: ImageNet.
Revised: Zero-shot compressor using SSL.

Original: Our BINCE objective suggest that pre-trained SSL methodsthat optimize InfoNCE, such as SimCLR (Chen et al., 2020), can give-rise to strong (invariant) compressors.As SimCLR’s authors showed that it learned representations Z that enables high downstream performance (even outperforming predictions from X ), we focus here on compression.
Revised: As previously pointed out, BINCE consists in a standard SSL loss with an additional entropy bottleneck. Given the availability and impressive results from pretrained SSL models, a natural question is whether we can take advantage of pre-trained SSL methods to give rise to powerful (invariant) compressors.

Original: An interesting question is whether the representations from a pre-trained
Revised: To investigate this question, we add and train an entropy bottleneck on top of a SSL model. Speciﬁcally, we ﬁrst download a SOTA SSL model, CLIP (Radford et al., 2021), and freeze it. Then we add an entropy bottleneck and train it on the CLIP’s output on a small dataset, MSCOCO (Lin et al., 2015). Finally, we evaluate our resulting compressor on various datasets (different tasks and shapes) which were never seen during training.

Original: SimCLR can already increase compression rates compared to the raw images. With this simple idea, we show that we can decrease the bit-rate by a 8 × factor (from 191 · 10 3 to 22 · 10bits) compared to standard JPEG compression of ImageNet.
Revised: Table 1 shows that this simple method gives rise to a powerful and generic compressor, which achieves more than 1000 × bit-rate gains compared to JPEG. The bit-rate gains (1 st row) are signiﬁcant across all datasets, 6 even on PCAM (Veeling et al., 2018) which consists of biological tissues.

Original: This result is very encouraging as it shows that off-the-shelves
Revised: Importantly, these gains do not come at the cost of removing information needed for desired tasks.

Original: SSL methods are already strong invariant compressors. For an extended version of the paper, we are investigating whether we can improve the bit-rate by ﬁnetuning SimCLR with an entropy bottleneck, i.e., by using our BINCE objective.
Revised: The second row shows the accuracy of a simple multi-layer perceptron trained on the output of our compressed representation.

Original: 5 End-to-end training of compressed features was empirically shown to be effective for the case of transfer learning rather than SSL (Singh et al., 2020).
Revised: The last row shows Radford et al.’s (2021) baselines from a near SOTA fully supervised model trained on the uncompressed images.

Original: Discussion.
Revised: Discussion.

Original: Given the exponentially increasing amount of collected data and the prevalence of task-speciﬁc algorithms that analyze that data, it is urgent to rethink our current task-agnostic compression paradigm.
Revised: Given the exponentially increasing amount of collected data and the prevalence of taskspeciﬁc algorithms that analyze that data, it is urgent to rethink our current task-agnostic compression paradigm.

Original: To the best our knowledge we derive the ﬁrst theoretical framework for task-centric compression (see Appx.
Revised: To the best our knowledge we derive the ﬁrst theoretical framework for task-centric compression (see Appx.

Original: E for related work).
Revised: E for related work).

Original: Furthermore, our theory andpreliminary experimental results suggests that the recent advancements and open-sourcing of pretrained selfsupervised networks can be exploited to decreases current compression rate by orders of magnitude.
Revised: Furthermore, our theory and experimental results shows that the recent advancements and open-sourcing of pretrained self-supervised networks can be exploited to decreases current compression rate by orders of magnitude.

Original:  from 
Revised: The large variance in bit-rate gains come from the variance in shapes of the raw images.

Original: 8% to 
Revised: For example, CIFAR10 has 32 × 32 bits so there is less information to gain compared to JPEG.

Original: A CKNOWLEDGMENTS
Revised: A CKNOWLEDGMENTS

Original: We would like to thank Emile Mathieu for helpful feedback.
Revised: We would like to thank Emile Mathieu for helpful feedback.

Original: Resources used in preparing this research were provided, in part, by the Province of Ontario, the Government of Canada through CIFAR
Revised: Resources used in preparing this research were provided, in part, by the Province of Ontario, the Government of Canada through CIFAR

Original: A N OTATION AND A SSUMPTIONS A. N OTATION
Revised: A N OTATION AND A SSUMPTIONS A. N OTATION

Original: Letters that are upper-case X , calligraphic X , and lower-case x , represent, respectively, a random variable (r.v.), its associated sample space, and a realization of it.
Revised: Letters that are upper-case X , calligraphic X , and lower-case x , represent, respectively, a random variable (r.v.), its associated sample space, and a realization of it.

Original: We denote the probability density function as p ( X ) , which we suppose always exist.
Revised: We denote the probability density function as p ( X ) , which we suppose always exist.

Original: { x i } i.i.d.
Revised: { x i } i.i.d.

Original: ∼ p ( X ) denotes independent and identical samples from p ( X ) .
Revised: ∼ p ( X ) denotes independent and identical samples from p ( X ) .

Original: X d ∼ N (0 , 1) denotes that X has a certain distribution (here Gaussian).
Revised: X d ∼ N (0 , 1) denotes that X has a certain distribution (here Gaussian).

Original: The composition of a function f with a r.v.
Revised: The composition of a function f with a r.v.

Original: X is denoted f ( X ) .
Revised: X is denoted f ( X ) .

Original: Expectations are written as: E p ( X ) [ X ] := (cid:82) xp
Revised: Expectations are written as: E p ( X ) [ X ] := (cid:82) xp

Original: ( x ) d x , while their Monte Carlo approximations have a hat ˆE p(X)
Revised: ( x ) d x , while their Monte Carlo approximations have a hat ˆE p(X)

Original: [ X ] = 1 |{ x i }| (cid:80) i x i for { x i } i.i.d.
Revised: [ X ] = 1 |{ x i }| (cid:80) i x i for { x i } i.i.d.

Original: ∼ p ( X ) .
Revised: ∼ p ( X ) .

Original: The KL divergence is denoted as D KL [ p ( X ) (cid:107) q ( X )] := (cid:82) log p ( X ) p ( X ) q ( X ) d x .
Revised: The KL divergence is denoted as D KL [ p ( X ) (cid:107) q ( X )] := (cid:82) log p ( X ) p ( X ) q ( X ) d x .

Original: The mutual information I[ X ; Z ] := D KL [ p ( X, Z ) (cid:107) p ( X ) p
Revised: The mutual information I[ X ; Z ] := D KL [ p ( X, Z ) (cid:107) p ( X ) p

Original: Z )] .
Revised: Z )] .

Original: The (differential or discrete) entropy of a r.v. is H
Revised: The (differential or discrete) entropy of a r.v. is H

Original: [ X ] := E p ( X ) [ − log p ( X )] , while the conditional (differential) entropy is H
Revised: [ X ] := E p ( X ) [ − log p ( X )] , while the conditional (differential) entropy is H

Original: [ X | Z ] :=
Revised: [ X | Z ] :=

Original: E p ( X,Z ) [ − log p ( X | Z )] .
Revised: E p ( X,Z ) [ − log p ( X | Z )] .

Original: Independence between two r.v.s is denoted with · ⊥ · .
Revised: Independence between two r.v.s is denoted with · ⊥ · .

Original: The cardinality of a set is denoted by | · | .
Revised: The cardinality of a set is denoted by | · | .

Original: · ◦ · denotes a composition of two functions.
Revised: · ◦ · denotes a composition of two functions.

Original: x ∼ x (cid:48) denotes that x and x (cid:48) are equivalent w.r.t.
Revised: x ∼ x (cid:48) denotes that x and x (cid:48) are equivalent w.r.t.

Original: an equivalence relation on X (the exact relation being implicit).
Revised: an equivalence relation on X (the exact relation being implicit).

Original: The equivalence class of x under ∼ consist of all elements that are equivalent to x , i.e.
Revised: The equivalence class of x under ∼ consist of all elements that are equivalent to x , i.e.

Original: [ x ] := { x (cid:48) ∈ X | x (cid:48) ∼ x } .
Revised: [ x ] := { x (cid:48) ∈ X | x (cid:48) ∼ x } .

Original: We will often use variational optimization over probability distribution, when the variational family is not made explicit it means that the optimiztion is over all possible densities, e.g. min q ( Y | X ) means that that the optimization is done over the collection of all conditional probability densities on Y given x ∈ X .
Revised: We will often use variational optimization over probability distribution, when the variational family is not made explicit it means that the optimiztion is over all possible densities, e.g. min q ( Y | X ) means that that the optimization is done over the collection of all conditional probability densities on Y given x ∈ X .

Original: Letters X , Z , Y respectively refer to the input, representation and target of a predictive task.
Revised: Letters X , Z , Y respectively refer to the input, representation and target of a predictive task.

Original: A. A SSUMPTIONS
Revised: A. A SSUMPTIONS

Original: Our results require some assumptions and we make additional mild assumptions for clarity, we discuss those in this section.
Revised: Our results require some assumptions and we make additional mild assumptions for clarity, we discuss those in this section.

Original: Extending our framework to different losses (dropping Assumption 1) should be investigated in future work.
Revised: Extending our framework to different losses (dropping Assumption 1) should be investigated in future work.

Original: All other assumptions should hold in most practical scenarios.
Revised: All other assumptions should hold in most practical scenarios.

Original: Assumption 1 (Log Loss) .
Revised: Assumption 1 (Log Loss) .

Original: We restrict ourselves to the Bayes risk w.r.t.
Revised: We restrict ourselves to the Bayes risk w.r.t.

Original: log loss.
Revised: log loss.

Original: Using the log loss makes the link between information theory and Bayes risk more natural as seen in Lemma 3.
Revised: Using the log loss makes the link between information theory and Bayes risk more natural as seen in Lemma 3.

Original: This is the standard loss to train neural classiﬁers.
Revised: This is the standard loss to train neural classiﬁers.

Original: Although our theory also holds for regression tasks (continuous Y ) it is more common to work with the squared error in that case.
Revised: Although our theory also holds for regression tasks (continuous Y ) it is more common to work with the squared error in that case.

Original: This assumption is necessary in the current version of Thm. 1, but we hope to generalize the result to other losses in future work.
Revised: This assumption is necessary in the current version of Thm. 1, but we hope to generalize the result to other losses in future work.

Original: Assumption 2 (Existence of Densities) .
Revised: Assumption 2 (Existence of Densities) .

Original: We restrict ourselves to cases where the probability mass/density function exist, i.e., to probability distributions that are absolutely continuous w.r.t.
Revised: We restrict ourselves to cases where the probability mass/density function exist, i.e., to probability distributions that are absolutely continuous w.r.t.

Original: to the underlying measure.
Revised: to the underlying measure.

Original: This is not a necessary assumption but it simpliﬁes the notation, and ensures that the differential entropy of r.v.s is well deﬁned.
Revised: This is not a necessary assumption but it simpliﬁes the notation, and ensures that the differential entropy of r.v.s is well deﬁned.

Original: Assumption 3 (Bounded Bayes Risk) .
Revised: Assumption 3 (Bounded Bayes Risk) .

Original: We restrict ourselves to tasks Y s.t.
Revised: We restrict ourselves to tasks Y s.t.

Original: the Bayes risk is always bounded, i.e. ∀ Z we have | R[ Y | Z ] | < ∞ .
Revised: the Bayes risk is always bounded, i.e. ∀ Z we have | R[ Y | Z ] | < ∞ .

Original: This ensures that taking differences of Bayes risks as in
Revised: This ensures that taking differences of Bayes risks as in

Original: Def. 3 is well deﬁned.
Revised: Def. 3 is well deﬁned.

Original: We could also directly assume that the latter difference is well deﬁned, but that would require dealing with limits which would unnecessarily complicate the proofs.
Revised: We could also directly assume that the latter difference is well deﬁned, but that would require dealing with limits which would unnecessarily complicate the proofs.

Original: For the case of discrete Y this is for example the case when Var[ Y ] < ∞ .
Revised: For the case of discrete Y this is for example the case when Var[ Y ] < ∞ .

Original: 6 For continuous Y the same is true (as H[ Y | Z ] ≤ H[ Y ] ≤ 12 log(2 πe (Var[ Y ])) < ∞ ) as long as the the (differential) entropy is not −∞ which essentially happens if there are no “singularities”.
Revised: For continuous Y the same is true (as H[ Y | Z ] ≤ H[ Y ] ≤ 12 log(2 πe (Var[ Y ])) < ∞ ) as long as the the (differential) entropy is not −∞ which essentially happens if there are no “singularities”.

Original: Assumption 4 (Measurability of Functions) .
Revised: Assumption 4 (Measurability of Functions) .

Original: We assume that all functions are measurable.
Revised: We assume that all functions are measurable.

Original: We particularly require (i) the measurability of M ( · ) which implies that M ( X ) is a r.v.. (ii) the measurability of the projection π : X → X / ∼ which implies that there always exists a maximal invariant as π is one of them.
Revised: We particularly require (i) the measurability of M ( · ) which implies that M ( X ) is a r.v.. (ii) the measurability 7 For the case where Y = Z we have H[ Y | Z ] ≤ H[ Y ] ≤ 12 log(2 πe (Var[ Y ]) + 112 ) < ∞ where the second inequality comes from (Massey, 1988) and the last inequality comes from our assumption of ﬁnite Var[ Y ] .

Original: This assumption essentially holds for all practical purposes.
Revised: This can be generalized to any countable Y by realizing that Y can always be rewritten as a bijection of Y (cid:48) where Y (cid:48) =

Revised: Z which has ﬁnite entropy due to the previous proof, and so Y also does.

Revised: of the projection π : X → X / ∼ which implies that there always exists a maximal invariant as π is one of them.

Revised: This assumption essentially holds for all practical purposes.

Original: Assumption 5 (Existence of regular conditional probabilities) .
Revised: Assumption 5 (Existence of regular conditional probabilities) .

Original: We restrict probability spaces (e.g.
Revised: We restrict probability spaces (e.g.

Original: Radon spaces) that satisfy the regular conditional probability property, so that all considered random variables admit a regular conditional probability.
Revised: Radon spaces) that satisfy the regular conditional probability property, so that all considered random variables admit a regular conditional probability.

Original: This is necessary to ensure the existence of probability kernel in Lemma 2.
Revised: This is necessary to ensure the existence of probability kernel in Lemma 2.

Original: This assumption essentially holds for all practical purposes.
Revised: This assumption essentially holds for all practical purposes.

Original: Assumption 6 (Countably Many Equivalence Classes) .
Revised: Assumption 6 (Countably Many Equivalence Classes) .

Original: We restrict our discussion to equivalences ∼ on X s.t.
Revised: We restrict our discussion to equivalences ∼ on X s.t.

Original: the quotient set X / ∼ :=
Revised: the quotient set X / ∼ :=

Original: { [ x ] | x ∈ X} is countable.
Revised: { [ x ] | x ∈ X} is countable.

Original: This ensures that M ( X ) is a discrete r.v. thereby ensuring that our invariance distortion D ∼ [ X, Z ] is independent of the choice of maximal invariant M as the conditional entropy is invariant to bijections.
Revised: This ensures that M ( X ) is a discrete r.v. thereby ensuring that our invariance distortion D ∼ [ X, Z ] is independent of the choice of maximal invariant M as the conditional entropy is invariant to bijections.

Original: As currently written our results (not only the proofs) do not hold without that assumption, but they can probably be extended.
Revised: As currently written our results (not only the proofs) do not hold without that assumption, but they can probably be extended.

Original: Note that this assumption holds when X is countable which always happens in practice due to ﬂoating point arithmetics, i.e. every real number has to be rounded to the closest 64 bits number.
Revised: Note that this assumption holds when X is countable which always happens in practice due to ﬂoating point arithmetics, i.e. every real number has to be rounded to the closest 64 bits number.

Original: Another perspective is to say that X is actually uncountable, but that all tasks we care about are always invariant to rounding to the nearest 64 bits number due to ﬂoating point arithmetics.
Revised: Another perspective is to say that X is actually uncountable, but that all tasks we care about are always invariant to rounding to the nearest 64 bits number due to ﬂoating point arithmetics.

Original: As a result, the maximal invariant is the usual maximal invariant rounded to the closest ﬂoating point.
Revised: As a result, the maximal invariant is the usual maximal invariant rounded to the closest ﬂoating point.

Original: For example, if
Revised: For example, if

Original: X is a 2D Gaussian we cannot work directly with translations on the y-axis (which gives uncountably many [ x ] , one for each real number on the x-axis), but can work with y-axis invariance combined with invariance to rounding on the x-axis (e.g. closest 64 bits number).
Revised: X is a 2D Gaussian we cannot work directly with translations on the y-axis (which gives uncountably many [ x ] , one for each real number on the x-axis), but can work with y-axis invariance combined with invariance to rounding on the x-axis (e.g. closest 64 bits number).

Original: Assumption 7 (Finite H[ M ( X )] ) .
Revised: Assumption 7 (Finite H[ M ( X )] ) .

Original: We restrict our discussion to X and equivalences on ( X , ∼ ) s.t.
Revised: We restrict our discussion to X and equivalences on ( X , ∼ ) s.t.

Original: if there exists a maximal invariant M then at least one has ﬁnite discrete entropy H[ M ( X )] < ∞ .
Revised: if there exists a maximal invariant M then at least one has ﬁnite discrete entropy H[ M ( X )] < ∞ .

Original: This is a necessary condition for M ( X ) to be in the tasks T of interest as we restricted our tasks of interests to the ones with bounded Bayes risk (Assumption 3).
Revised: This is a necessary condition for M ( X ) to be in the tasks T of interest as we restricted our tasks of interests to the ones with bounded Bayes risk (Assumption 3).

Original: We require this in the proof of Prop.to ensure that D ∼ [ X, Z ]
Revised: We require this in the proof of Prop.to ensure that D ∼ [ X, Z ]

Original: = R[ M ( X ) | Z ] is achievable.
Revised: = R[ M ( X ) | Z ] is achievable.

Original: As discussed in Assumption 3, it is sufﬁcient (but not necessary) to assume that M ( X ) has ﬁnite variance which is implied from X having ﬁnite variance.
Revised: As discussed in Assumption 3, it is sufﬁcient (but not necessary) to assume that M ( X ) has ﬁnite variance which is implied from X having ﬁnite variance.

Original: as is often the case.
Revised: as is often the case.

Original: B F ORMAL D EFINITIONS
Revised: B F ORMAL D EFINITIONS

Original: In the main paper we were relatively informal in our deﬁnitions, here we restate our main deﬁnitions more formally.
Revised: In the main paper we were relatively informal in our deﬁnitions, here we restate our main deﬁnitions more formally.

Original: First let us deﬁne the notion of valid distortion, which is necessary for applying the original rate distortion theorem (Shannon, 1959; Cover & Thomas, 2006).
Revised: First let us deﬁne the notion of valid distortion, which is necessary for applying the original rate distortion theorem (Shannon, 1959; Cover & Thomas, 2006).

Original: Deﬁnition 1 (Valid Distortion) .
Revised: Deﬁnition 1 (Valid Distortion) .

Original: Let X and Z be two r.v.s that respectively t.v.i.
Revised: Let X and Z be two r.v.s that respectively t.v.i.

Original: X and Z .
Revised: X and Z .

Original: Then an (expected) distortion D is said valid w.r.t.
Revised: Then an (expected) distortion D is said valid w.r.t.

Original: X, Z if there exists a point wise distortion d : X ×Z → R + s.t.
Revised: X, Z if there exists a point wise distortion d : X ×Z → R + s.t.

Original: for any x, z s.t. p ( x, z ) > 0 we have 8 d ( x, z ) ≤ d max < ∞ and
Revised: for any x, z s.t. p ( x, z ) > 0 we have 9 d ( x, z ) ≤ d max < ∞ and

Original: [Equation]
Revised: [Equation]

Original: Let us deﬁne the notion of representation Z which arises by encoding X using p ( Z | X ) independently of any task Y .
Revised: Let us deﬁne the notion of representation Z which arises by encoding X using p ( Z | X ) independently of any task Y .

Original: Deﬁnition 2 (Representation) .
Revised: Deﬁnition 2 (Representation) .

Original: Let X, Z be two r.v.s and T be a set of r.v.s.
Revised: Let X, Z be two r.v.s and T be a set of r.v.s.

Original: Z is a representation of
Revised: Z is a representation of

Original: X for T iff ∀ Y ∈ T we have the pairwise conditional independence Y ⊥ Z | X .
Revised: X for T iff ∀ Y ∈ T we have the pairwise conditional independence Y ⊥ Z | X .

Original: Let us now deﬁne the tasks of interest.
Revised: Let us now deﬁne the tasks of interest.

Original: Deﬁnition 3 (Invariant Tasks of Interest) .
Revised: Deﬁnition 3 (Invariant Tasks of Interest) .

Original: Let X and Y be two r.v.s that respectively t.v.i.
Revised: Let X and Y be two r.v.s that respectively t.v.i.

Original: X and Y .
Revised: X and Y .

Original: Let ∼ denote an equivalence relation on X satisfying Assumption 6 and s.t.
Revised: Let ∼ denote an equivalence relation on X satisfying Assumption 6 and s.t.

Original: X satisﬁes Assumption 7.
Revised: X satisﬁes Assumption 7.

Original: We say that Y is an invariant task of interest w.r.t.
Revised: We say that Y is an invariant task of interest w.r.t.

Original: ( X , ∼ ) , iff:
Revised: ( X , ∼ ) , iff:

Original: • for any r.v.
Revised: • for any r.v.

Original: Z the Bayes risk is bounded R
Revised: Z the Bayes risk is bounded R

Original: [ Y | Z ] < ∞ (Assumption 3).
Revised: [ Y | Z ] < ∞ (Assumption 3).

Original: • the function x
Revised: • the function x

Original: (cid:55)→ p ( Y | x ) is invariant w.r.t.
Revised: (cid:55)→ p ( Y | x ) is invariant w.r.t.

Original: ∼ , i.e. ∀ x, x (cid:48) ∈
Revised: ∼ , i.e. ∀ x, x (cid:48) ∈

Original: X we have
Revised: X we have

Original: [Equation]
Revised: [Equation]

Original: Furthermore, we denote as T all such invariant tasks of interest ( X and ∼ are implicit in the notation).
Revised: Furthermore, we denote as T all such invariant tasks of interest ( X and ∼ are implicit in the notation).

Original: Let us now recall our desired distortion D T .
Revised: Let us now recall our desired distortion D T .

Original: Deﬁnition 4 (Invariance Distortion) .
Revised: Deﬁnition 4 (Invariance Distortion) .

Original: Let X and Z be two r.v.s.
Revised: Let X and Z be two r.v.s.

Original: Let T be the invariant tasks of interest as in Def. 3.
Revised: Let T be the invariant tasks of interest as in Def. 3.

Original: The invariance distortion D T is deﬁned as:
Revised: The invariance distortion D T is deﬁned as:

Original: [Equation]
Revised: [Equation]

Original: Finally, let us deﬁne the notion of maximal invariant.
Revised: Finally, let us deﬁne the notion of maximal invariant.

Original: Deﬁnition 5 (Maximal Invariant) .
Revised: Deﬁnition 5 (Maximal Invariant) .

Original: Let ∼ denote an equivalence relation on X .
Revised: Let ∼ denote an equivalence relation on X .

Original: We say that a measurable (Assumption 4) function M : X → M is a maximal invariant w.r.t.
Revised: We say that a measurable (Assumption 4) function M : X → M is a maximal invariant w.r.t.

Original: ( X , ∼ ) iff
Revised: ( X , ∼ ) iff

Original: [Equation]
Revised: [Equation]

Original: Note that our notion of maximal invariants generalizes the notion of maximal invariants in probabilistic group theory (Eaton, 1989).
Revised: Note that our notion of maximal invariants generalizes the notion of maximal invariants in probabilistic group theory (Eaton, 1989).

Original: Refer to Lehmann & Romano (2005) for many more examples in the group case.
Revised: Refer to Lehmann & Romano (2005) for many more examples in the group case.

Original: Important examples of maximal invariants can be summarized informally as follows:
Revised: Important examples of maximal invariants can be summarized informally as follows:

Original: Translations If X = R 2 and the examples are invariant to translations of the second coordinate: x (cid:55)→ x + [0 , t ] T ; then a maximal invariant is the ﬁrst coordinate M : x (cid:55)→ x 1 .
Revised: Translations If X = R 2 and the examples are invariant to translations of the second coordinate: x (cid:55)→ x + [0 , t ] T ; then a maximal invariant is the ﬁrst coordinate M : x (cid:55)→ x 1 .

Original: Scaling If X = R n and the examples are invariant to scalar scaling : x (cid:55)→ c · x ; then a maximal invariant is given by rescaling by the last coordinate M : x (cid:55)→ x/x n .
Revised: Scaling If X = R n and the examples are invariant to scalar scaling : x (cid:55)→ c · x ; then a maximal invariant is given by rescaling by the last coordinate M : x (cid:55)→ x/x n .

Original: Rotations If X = R n and the examples are invariant to rotations : x (cid:55)→ Rot( θ ) · x ; then a maximal invariant is given by the Euclidean norm M : x (cid:55)→ || x
Revised: Rotations If X = R n and the examples are invariant to rotations : x (cid:55)→ Rot( θ ) · x ; then a maximal invariant is given by the Euclidean norm M : x (cid:55)→ || x

Original: || 2 .
Revised: || 2 .

Original: Permutations If X = R n and the examples are invariant to permutations of the coordinates : x (cid:55)→ ( x π (1) , . . .
Revised: Permutations If X = R n and the examples are invariant to permutations of the coordinates : x (cid:55)→ ( x π (1) , . . .

Original: , x π ( n ) ) ; then a maximal invariant is the empirical measure (the type class, or histogram).
Revised: , x π ( n ) ) ; then a maximal invariant is the empirical measure (the type class, or histogram).

Original: Graph Isomorphisms If X is the set of all graphs, and the examples are invariant to graph isomorphisms; then a maximal invariant is the graph canonization.
Revised: Graph Isomorphisms If X is the set of all graphs, and the examples are invariant to graph isomorphisms; then a maximal invariant is the graph canonization.

Original: C P ROOFS : O PTIMAL B IT -R ATE
Revised: C P ROOFS : O PTIMAL B IT -R ATE

Original: In this section we prove all our claims in Sec. 2.
Revised: In this section we prove all our claims in Sec. 2.

Original: C. R EFORMULATING AND V ALIDATING D EF .
Revised: C. R EFORMULATING AND V ALIDATING D EF .

Original: In this section we prove the equivalence between Def. 4 and the nicer H[ M ( X ) | Z ] which is the core of our work.
Revised: In this section we prove the equivalence between Def. 4 and the nicer H[ M ( X ) | Z ] which is the core of our work.

Original: The main steps in the proof are the following:
Revised: The main steps in the proof are the following:

Original: We show that if Y is an invariant tasks then Y − M ( X ) − X forms a Markov chain.
Revised: We show that if Y is an invariant tasks then Y − M ( X ) − X forms a Markov chain.

Original: Using the strict properness of the log loss, we relate the Bayes risk to the differential entropy:
Revised: Using the strict properness of the log loss, we relate the Bayes risk to the differential entropy:

Original: [Equation]
Revised: [Equation]

Original: Using (1) and (2), the chain rule and the data processing inequality we show that the supremum is achieved by M ( X ) :
Revised: Using (1) and (2), the chain rule and the data processing inequality we show that the supremum is achieved by M ( X ) :

Original: [Equation]
Revised: [Equation]

Original: As M is a (deterministic) function and M ( X ) is discrete we have
Revised: As M is a (deterministic) function and M ( X ) is discrete we have

Original: [Equation]
Revised: [Equation]

Original: The ﬁrst step, consists in showing that for invariant tasks we have Y ⊥⊥ X | M ( X ) .
Revised: The ﬁrst step, consists in showing that for invariant tasks we have Y ⊥⊥ X | M ( X ) .

Original: Speciﬁcally, we prove that that any conditionally invariant r.v. can be decomposed as a function of a maximal invariant and independent noise.
Revised: Speciﬁcally, we prove that that any conditionally invariant r.v. can be decomposed as a function of a maximal invariant and independent noise.

Original: This can be seen as a probabilistic extension of the theorem on projections (Theorem 19 and its corollary in Lane & Birkhoff (1999)) .
Revised: This can be seen as a probabilistic extension of the theorem on projections (Theorem 19 and its corollary in Lane & Birkhoff (1999)) .

Original: This can also be seen as a generalization of an important probabilistic group theoretical results (Theorem 4.4 in Eaton (1989), Theorem 7 in
Revised: This can also be seen as a generalization of an important probabilistic group theoretical results (Theorem 4.4 in Eaton (1989), Theorem 7 in

Original: Bloem-Reddy & Teh (2020)), to any equivalences (rather than only groups) and without making the assumption of (marginal) invariance of p ( X ) to ∼ .
Revised: Bloem-Reddy & Teh (2020)), to any equivalences (rather than only groups) and without making the assumption of (marginal) invariance of p ( X ) to ∼ .

Original: The following intermediate result is needed.
Revised: The following intermediate result is needed.

Original: Lemma 1.
Revised: Lemma 1.

Original: A measurable function f : X → F is invariant with respect to ( X , ∼ ) if and only if there exists a measurable function g : M → F such that f ( x ) =
Revised: A measurable function f : X → F is invariant with respect to ( X , ∼ ) if and only if there exists a measurable function g : M → F such that f ( x ) =

Original: ( g ◦ M )( x ) for all x ∈ X .
Revised: ( g ◦ M )( x ) for all x ∈ X .

Original: Proof.
Revised: Proof.

Original: Clearly, if f ( x ) =
Revised: Clearly, if f ( x ) =

Original: ( g ◦ M )( x ) =
Revised: ( g ◦ M )( x ) =

Original: g ( M ( x )) then f is ( X , ∼ ) -invariant because M is.
Revised: g ( M ( x )) then f is ( X , ∼ ) -invariant because M is.

Original: Conversely, assume that f is ( X , ∼ ) -invariant.
Revised: Conversely, assume that f is ( X , ∼ ) -invariant.

Original: Let F 0 ∈ F denote an arbitrary ﬁxed element.
Revised: Let F 0 ∈ F denote an arbitrary ﬁxed element.

Original: Then we can construct g as
Revised: Then we can construct g as

Original: [Equation]
Revised: [Equation]

Original: Lemma 2.
Revised: Lemma 2.

Original: Let X
Revised: Let X

Original: and Y be 2 r.v., and M : X → M be a maximal invariant w.r.t.
Revised: and Y be 2 r.v., and M : X → M be a maximal invariant w.r.t.

Original: ( X , ∼ ) as in
Revised: ( X , ∼ ) as in

Original: Def. 5.
Revised: Def. 5.

Original: Then Y is (conditionally) invariant w.r.t.
Revised: Then Y is (conditionally) invariant w.r.t.

Original: ( X , ∼ ) as in Eq.
Revised: ( X , ∼ ) as in Eq.

Original: (9) if and only if Y ⊥⊥ X | M ( X ) .
Revised: (9) if and only if Y ⊥⊥ X | M ( X ) .

Original: Proof.
Revised: Proof.

Original: Assume that Y is (conditionally) invariant w.r.t.
Revised: Assume that Y is (conditionally) invariant w.r.t.

Original: ( X , ∼ ) as in Eq.
Revised: ( X , ∼ ) as in Eq.

Original: By standard results, there exists a probability kernel K ( A, x ) such that for all measurable sets
Revised: By standard results, there exists a probability kernel K ( A, x ) such that for all measurable sets

Original: A , x (cid:55)→ K ( A, x ) is a measurable function mapping X
Revised: A , x (cid:55)→ K ( A, x ) is a measurable function mapping X

Original: → R + .
Revised: → R + .

Original: Conditional invariance means that x ∼ x (cid:48) ⇒ K
Revised: Conditional invariance means that x ∼ x (cid:48) ⇒ K

Original: ( A, x ) =
Revised: ( A, x ) =

Original: K ( A, x (cid:48) )
Revised: K ( A, x (cid:48) )

Original: for all x, x (cid:48) .
Revised: for all x, x (cid:48) .

Original: That is, as a function of x , K ( A, • ) is invariant w.r.t.
Revised: That is, as a function of x , K ( A, • ) is invariant w.r.t.

Original: ( X , ∼ ) .
Revised: ( X , ∼ ) .

Original: By Lemma 1, x (cid:55)→ K ( A, x ) can be expressed as a measurable function of M , i.e., K ( A, x ) =
Revised: By Lemma 1, x (cid:55)→ K ( A, x ) can be expressed as a measurable function of M , i.e., K ( A, x ) =

Original: K (cid:48)
Revised: K (cid:48)

Original: ( A, M ( x )) , for another probability kernel K (cid:48) .
Revised: ( A, M ( x )) , for another probability kernel K (cid:48) .

Original: This implies that P ( Y | X ) =
Revised: This implies that P ( Y | X ) =

Original: P ( Y | M ( X )) almost surely.
Revised: P ( Y | M ( X )) almost surely.

Original: Because M ( X ) is a function of X , that further implies that P ( Y | X, M ( X ))
Revised: Because M ( X ) is a function of X , that further implies that P ( Y | X, M ( X ))

Original: =
Revised: =

Original: P ( Y | M ( X )) , i.e., Y ⊥⊥ X | M ( X ) .
Revised: P ( Y | M ( X )) , i.e., Y ⊥⊥ X | M ( X ) .

Original: We now relate the Bayes risk and conditional entropy.
Revised: We now relate the Bayes risk and conditional entropy.

Original: This is a simple lemma that directly comes from the fact that the conditional distribution p ( Y | Z ) is the Bayes predictor.
Revised: This is a simple lemma that directly comes from the fact that the conditional distribution p ( Y | Z ) is the Bayes predictor.

Original: Lemma 3.
Revised: Lemma 3.

Original: Let Y, X be r.v.s with bounded R
Revised: Let Y, X be r.v.s with bounded R

Original: [ Y | X ] then the log loss Bayes irsk is equal to the conditional (differential) entropy:
Revised: [ Y | X ] then the log loss Bayes irsk is equal to the conditional (differential) entropy:

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: [Equation]
Revised: [Equation]

Original: Deﬁnition
Revised: Deﬁnition

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Strict Proper.
Revised: Strict Proper.

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Deﬁnition
Revised: Deﬁnition

Original: [Equation]
Revised: [Equation]

Original: Where Eq.
Revised: Where Eq.

Original: (17) uses the strict properness of the logarithmic scoring function rule (Gneiting & Raftery, 2007).
Revised: (17) uses the strict properness of the logarithmic scoring function rule (Gneiting & Raftery, 2007).

Original: In the rest of this section we will often be working with the entropy H[ M ( X )] and conditional entropies such as H[ M ( X ) | Z ] .
Revised: In the rest of this section we will often be working with the entropy H[ M ( X )] and conditional entropies such as H[ M ( X ) | Z ] .

Original: Importantly, we would like our results to be independent of the choice of maximal invariant M .
Revised: Importantly, we would like our results to be independent of the choice of maximal invariant M .

Original: We now prove that this will indeed be the case as all these (conditional) entropy terms are independent of the choice of M .
Revised: We now prove that this will indeed be the case as all these (conditional) entropy terms are independent of the choice of M .

Original: We only prove it for the marginal entropy
Revised: We only prove it for the marginal entropy

Original: H[ M ( X )] but the same proof holds for conditional entropies.
Revised: H[ M ( X )] but the same proof holds for conditional entropies.

Original: Lemma 4.
Revised: Lemma 4.

Original: Let X be a r.v. that t.v.i X .
Revised: Let X be a r.v. that t.v.i X .

Original: Let ∼ denote an equivalence relation on X satisfying
Revised: Let ∼ denote an equivalence relation on X satisfying

Original: Assumption 6.
Revised: Assumption 6.

Original: Let M and M (cid:48)
Revised: Let M and M (cid:48)

Original: two different maximal invariants w.r.t.
Revised: two different maximal invariants w.r.t.

Original: ( X , ∼ ) as in Def. 5, then
Revised: ( X , ∼ ) as in Def. 5, then

Original: H[ M ( X )]
Revised: H[ M ( X )]

Original: =
Revised: =

Original: H[ M (cid:48)
Revised: H[ M (cid:48)

Original: ( X )] .
Revised: ( X )] .

Original: Proof.
Revised: Proof.

Original: First notice that if M and M (cid:48) are both maximal invariant then 9 there exists a bijective function f : M → M s.t.
Revised: First notice that if M and M (cid:48) are both maximal invariant then 10 there exists a bijective function f : M → M s.t.

Original: M (cid:48)
Revised: M (cid:48)

Original: =
Revised: =

Original: f ◦ M .
Revised: f ◦ M .

Original: Indeed, from the projection theorem (Theorem 19 in Lane & Birkhoff (1999))
Revised: Indeed, from the projection theorem (Theorem 19 in Lane & Birkhoff (1999))

Original: we know that M is a maximal invariant if and only if there is a bijective function g : X / ∼→ M s.t.
Revised: we know that M is a maximal invariant if and only if there is a bijective function g : X / ∼→ M s.t.

Original: the maximal invariant is the composition of g and the projection onto equivalence classes,
Revised: the maximal invariant is the composition of g and the projection onto equivalence classes,

Original: i.e. ∀ x ∈ X
Revised: i.e. ∀ x ∈ X

Original: we have M ( x ) =
Revised: we have M ( x ) =

Original: g ([ x ]) .
Revised: g ([ x ]) .

Original: Let g (cid:48) be the corresponding bijection for M (cid:48) .
Revised: Let g (cid:48) be the corresponding bijection for M (cid:48) .

Original: Then we have M (cid:48)
Revised: Then we have M (cid:48)

Original: =
Revised: =

Original: f ◦ M with f := g (cid:48) ◦ g − 1 which is indeed bijective f − 1 :=
Revised: f ◦ M with f := g (cid:48) ◦ g − 1 which is indeed bijective f − 1 :=

Original: g ◦ g (cid:48)− 1 .
Revised: g ◦ g (cid:48)− 1 .

Original: Due to Assumption 6, M ( X ) is a discrete r.v.
Revised: Due to Assumption 6, M ( X ) is a discrete r.v.

Original: and so H[ M ( X )] is the discrete entropy, which is invariant to bijective functions indeed H[ M ( X )]
Revised: and so H[ M ( X )] is the discrete entropy, which is invariant to bijective functions indeed H[ M ( X )]

Original: = I[ M ( X ); M ( X )]
Revised: = I[ M ( X ); M ( X )]

Original: =
Revised: =

Original: I[ f ( M ( X )) , f ( M ( X ))]
Revised: I[ f ( M ( X )) , f ( M ( X ))]

Original: =
Revised: =

Original: H[ f ( M ( X ))] where we used the invariance of mutual information to bijections (Kraskov et al., 2004).
Revised: H[ f ( M ( X ))] where we used the invariance of mutual information to bijections (Kraskov et al., 2004).

Original: We thus conclude that H[ M ( X )]
Revised: We thus conclude that H[ M ( X )]

Original: = H[ f ( M ( X ))]
Revised: = H[ f ( M ( X ))]

Original: =
Revised: =

Original: H[ M (cid:48)
Revised: H[ M (cid:48)

Original: ( X )] as desired.
Revised: ( X )] as desired.

Original: We now prove that all M ( X ) are always in the set of downstream tasks T .
Revised: We now prove that all M ( X ) are always in the set of downstream tasks T .

Original: Lemma 5.
Revised: Lemma 5.

Original: Let X be a r.v.
Revised: Let X be a r.v.

Original: Let T be the invariant tasks of interest as in Def. 3.
Revised: Let T be the invariant tasks of interest as in Def. 3.

Original: Then any maximal invariant (Def. 5)
Revised: Then any maximal invariant (Def. 5)

Original: M is such that M ( X ) ∈ T and there exists at least one maximal invariant.
Revised: M is such that M ( X ) ∈ T and there exists at least one maximal invariant.

Original: Proof.
Revised: Proof.

Original: First, we have to prove that a maximal invariant always exists.
Revised: First, we have to prove that a maximal invariant always exists.

Original: We do so by construction.
Revised: We do so by construction.

Original: By deﬁnition equivalent elements have the same equivalence class and so x ∼ x (cid:48)
Revised: By deﬁnition equivalent elements have the same equivalence class and so x ∼ x (cid:48)

Original: ⇐⇒ [ x ] =
Revised: ⇐⇒ [ x ] =

Original: [ x (cid:48) ] .
Revised: [ x (cid:48) ] .

Original: We thus have that the projection map π : x (cid:55)→ [ x ] satisﬁes Eq.
Revised: We thus have that the projection map π : x (cid:55)→ [ x ] satisﬁes Eq.

Original: Due to Assumption 4 the projection map is measurable and so it is a maximal invariant.
Revised: Due to Assumption 4 the projection map is measurable and so it is a maximal invariant.

Original: As there exists (at least) one maximal invariant we have that there exists M ( X ) s.t.
Revised: As there exists (at least) one maximal invariant we have that there exists M ( X ) s.t.

Original: ∀
Revised: ∀

Original: Z we have
Revised: Z we have

Original: R[ M ( X ) | Z ] =
Revised: R[ M ( X ) | Z ] =

Original: H[ M ( X ) | Z ] ≤ H[ M ( X )] < ∞ , where the ﬁrst equality comes from Lemma 3, and second inequality from the fact that conditioning decreases entropy, and the ﬁnal inequality comes from Assumption 7.
Revised: H[ M ( X ) | Z ] ≤ H[ M ( X )] < ∞ , where the ﬁrst equality comes from Lemma 3, and second inequality from the fact that conditioning decreases entropy, and the ﬁnal inequality comes from Assumption 7.

Original: Due to Lemma 4 we further have that all maximal invariants
Revised: Due to Lemma 4 we further have that all maximal invariants

Original: M i have a bounded R
Revised: M i have a bounded R

Original: [ M i ( X ) | Z ] which is necessary for all M i ( X ) to be in T .
Revised: [ M i ( X ) | Z ] which is necessary for all M i ( X ) to be in T .

Original: As all maximal invariants satisfy by deﬁnition the invariance in Eq.
Revised: As all maximal invariants satisfy by deﬁnition the invariance in Eq.

Original: (9), we conclude that they are all in T as desired.
Revised: (9), we conclude that they are all in T as desired.

Original: We are now ready to prove the desired proposition.
Revised: We are now ready to prove the desired proposition.

Original: Proposition 1 (Nicer D T ) .
Revised: Proposition 1 (Nicer D T ) .

Original: Let X be a r.v.
Revised: Let X be a r.v.

Original: Let T be the invariant tasks of interest as in Def. 3, M be any maximal invariant as in Def. 5, and Z be a representation of X as in Def.
Revised: Let T be the invariant tasks of interest as in Def. 3, M be any maximal invariant as in Def. 5, and Z be a representation of X as in Def.

Original: Let D T be as in
Revised: Let D T be as in

Original: Def. 4.
Revised: Def. 4.

Original: Then D T is a valid distortion(Def. 1) and
Revised: Then D T is a valid distortion(Def. 1) and

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: First let us prove that D ∼ [ X, Z ]
Revised: First let us prove that D ∼ [ X, Z ]

Original: = H[ M ( X ) | Z ]
Revised: = H[ M ( X ) | Z ]

Original: [Equation]
Revised: [Equation]

Original: H[ Y | X ]
Revised: H[ Y | X ]

Original: Lemma(21)
Revised: Lemma(21)

Original: =
Revised: =

Original: sup Y ∈T H[ Y | Z ] − H
Revised: sup Y ∈T H[ Y | Z ] − H

Original: [ Y | X, M ( X )]
Revised: [ Y | X, M ( X )]

Original: Y ⊥ M ( X ) | X
Revised: Y ⊥ M ( X ) | X

Original: (22) =
Revised: (22) =

Original: sup Y ∈T H[ Y | Z ] − H
Revised: sup Y ∈T H[ Y | Z ] − H

Original: [ Y | M ( X )]
Revised: [ Y | M ( X )]

Original: Lemma 2: Y ⊥ X | M ( X ) (23) =
Revised: Lemma 2: Y ⊥ X | M ( X ) (23) =

Original: sup Y ∈T H[ Y | Z ] − H
Revised: sup Y ∈T H[ Y | Z ] − H

Original: [ Y | M ( X ) , Z ]
Revised: [ Y | M ( X ) , Z ]

Original: Def. 2: Y ⊥ Z | M ( X ) (24) =
Revised: Def. 2: Y ⊥ Z | M ( X ) (24) =

Original: sup Y ∈T I
Revised: sup Y ∈T I

Original: [ Y ; M ( X ) | Z ]
Revised: [ Y ; M ( X ) | Z ]

Original: Def.
Revised: Def.

Original: (25) =
Revised: (25) =

Original: sup Y ∈T I[ M ( X ); Y | Z ]
Revised: sup Y ∈T I[ M ( X ); Y | Z ]

Original: Symmetry (26) =
Revised: Symmetry (26) =

Original: sup Y ∈T H[ M ( X ) | Z ] − H[ M ( X ) | Y, Z ]
Revised: sup Y ∈T H[ M ( X ) | Z ] − H[ M ( X ) | Y, Z ]

Original: Def.
Revised: Def.

Original: [Equation]
Revised: [Equation]

Original: Discrete entropy
Revised: Discrete entropy

Original: [Equation]
Revised: [Equation]

Original: To go from Eq.
Revised: To go from Eq.

Original: (24) to Eq.
Revised: (24) to Eq.

Original: (27) we use the the symmetry and deﬁnition of conditional mutual information, which essentially corresponds to using the chain rule.
Revised: (27) we use the the symmetry and deﬁnition of conditional mutual information, which essentially corresponds to using the chain rule.

Original: Eq.
Revised: Eq.

Original: (30) uses the fact that
Revised: (30) uses the fact that

Original: H[ M ( X ) | M ( X )] ≤ inf Y ∈T H
Revised: H[ M ( X ) | M ( X )] ≤ inf Y ∈T H

Original: [ M ( X ) | Y, Z ] because of the data processing inequality and the trivial fact that M ( X ) − M ( X ) − ( Z, Y ) forms a Markov Chain.
Revised: [ M ( X ) | Y, Z ] because of the data processing inequality and the trivial fact that M ( X ) − M ( X ) − ( Z, Y ) forms a Markov Chain.

Original: As M ( X ) ∈ T (Lemma 5) we also have the other inequality inf Y ∈T H
Revised: As M ( X ) ∈ T (Lemma 5) we also have the other inequality inf Y ∈T H

Original: [ M ( X ) | Y, Z ]
Revised: [ M ( X ) | Y, Z ]

Original: ≤ H
Revised: ≤ H

Original: [ M ( X )
Revised: [ M ( X )

Original: | M ( X ) , Z ] ≤ H[ M ( X ) | M ( X )] , and so we conclude that the equality H[ M ( X ) | M ( X )]
Revised: | M ( X ) , Z ] ≤ H[ M ( X ) | M ( X )] , and so we conclude that the equality H[ M ( X ) | M ( X )]

Original: =
Revised: =

Original: inf Y ∈T H[ M ( X ) | Y, Z ] holds.
Revised: inf Y ∈T H[ M ( X ) | Y, Z ] holds.

Original: Finally,
Revised: Finally,

Original: Eq.
Revised: Eq.

Original: (31) uses the fact that M ( X ) is a discrete r.v. due to Assumption 6 and
Revised: (31) uses the fact that M ( X ) is a discrete r.v. due to Assumption 6 and

Original: so the discrete conditional entropy H[ M ( X ) | M ( X )]
Revised: so the discrete conditional entropy H[ M ( X ) | M ( X )]

Original: = 0 .
Revised: = 0 .

Original: It is now easy to see that D T is valid as D ∼ [ X, Z ]
Revised: It is now easy to see that D T is valid as D ∼ [ X, Z ]

Original: = H[ M ( X ) | Z ] =
Revised: = H[ M ( X ) | Z ] =

Original: E p ( X,Z )
Revised: E p ( X,Z )

Original: [ d ( X, Z )] with d ( x, z ) :
Revised: [ d ( X, Z )] with d ( x, z ) :

Original: = − log p ( M ( x ) | z ) which due to the discreteness of M ( X ) (Assumption 6) is a function whose codomain is R + as desired.
Revised: = − log p ( M ( x ) | z ) which due to the discreteness of M ( X ) (Assumption 6) is a function whose codomain is R + as desired.

Original: As conditioning decreases entropy we also have H[ M ( X ) | Z ] ≤
Revised: As conditioning decreases entropy we also have H[ M ( X ) | Z ] ≤

Original: H[ M ( X )] < ∞ where the last inequality comes from Assumption 7.
Revised: H[ M ( X )] < ∞ where the last inequality comes from Assumption 7.

Original: As M ( X ) is discrete we conclude that ∀ x, z s.t. p ( x, z ) > 0 there exists d max s.t.
Revised: As M ( X ) is discrete we conclude that ∀ x, z s.t. p ( x, z ) > 0 there exists d max s.t.

Original: d ( x, z ) ≤ d max < ∞ and so D T is valid.
Revised: d ( x, z ) ≤ d max < ∞ and so D T is valid.

Original: C. P ROOFS FOR T HEOREM
Revised: C. P ROOFS FOR T HEOREM

Original: Our main theoretical contribution is to characterize the minimal achievable rate to bound the Bayes risk of any invariant task.
Revised: Our main theoretical contribution is to characterize the minimal achievable rate to bound the Bayes risk of any invariant task.

Original: The result follows from Shannon’s (1959) rate distortion theorem, the deﬁnition of D T in terms of worst Bayes risk, the fact that D T is valid, and our characterization of
Revised: The result follows from Shannon’s (1959) rate distortion theorem, the deﬁnition of D T in terms of worst Bayes risk, the fact that D T is valid, and our characterization of

Original: D T in terms of entropy (Prop. 1).
Revised: D T in terms of entropy (Prop. 1).

Original: First let us restate the well known rate distortion theorem.
Revised: First let us restate the well known rate distortion theorem.

Original: Here we use the statement as given in
Revised: Here we use the statement as given in

Original: Cover & Thomas (2006).
Revised: Cover & Thomas (2006).

Original: Note that achievability is usually deﬁned for deterministic encoders (e.g. on 306 of Cover & Thomas (2006) ) but the proof holds for stochastic encoders (as noted on p.316 of
Revised: Note that achievability is usually deﬁned for deterministic encoders (e.g. on 306 of Cover & Thomas (2006) ) but the proof holds for stochastic encoders (as noted on p.316 of

Original: Cover & Thomas (2006)) which we use in our work.
Revised: Cover & Thomas (2006)) which we use in our work.

Original: Lemma 6.
Revised: Lemma 6.

Original: (Theorem 10.2.1 in Cover & Thomas (2006))
Revised: (Theorem 10.2.1 in Cover & Thomas (2006))

Original: Let D[ X ; Z ] be a valid distortion as in
Revised: Let D[ X ; Z ] be a valid distortion as in

Original: Def. 1.
Revised: Def. 1.

Original: The minimum achievable bit-rate for transmitting an i.i.d.
Revised: The minimum achievable bit-rate for transmitting an i.i.d.

Original: source X with expected distortion less than δ ≥ 0 is given by the rate-distortion function:
Revised: source X with expected distortion less than δ ≥ 0 is given by the rate-distortion function:

Original: [Equation]
Revised: [Equation]

Original: We can now state our rate invariance theorem.
Revised: We can now state our rate invariance theorem.

Original: Theorem 1 (Rate Invariance) .
Revised: Theorem 1 (Rate Invariance) .

Original: Let X be a r.v. and δ ≥ 0 .
Revised: Let X be a r.v. and δ ≥ 0 .

Original: Let T be the invariant tasks of interest as in Def. 3, M be any maximal invariant as in Def. 5, and Z be a representation of X as in Def. 2.
Revised: Let T be the invariant tasks of interest as in Def. 3, M be any maximal invariant as in Def. 5, and Z be a representation of X as in Def. 2.

Original: Let
Revised: Let

Original: Rate ( δ ) denote the minimum achievable bit-rate for transmitting an i.i.d.
Revised: Rate ( δ ) denote the minimum achievable bit-rate for transmitting an i.i.d.

Original: source of Z s.t.
Revised: source of Z s.t.

Original: for any
Revised: for any

Original: Y ∈ T we have R[ Y | Z ] ≤ δ + R[ Y | X ] .
Revised: Y ∈ T we have R[ Y | Z ] ≤ δ + R[ Y | X ] .

Original: Then Rate ( δ ) is ﬁnite and given by
Revised: Then Rate ( δ ) is ﬁnite and given by

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: In the following we ﬁrst prove that Rate ( δ ) ≤ H[ M ( X )] − δ .
Revised: In the following we ﬁrst prove that Rate ( δ ) ≤ H[ M ( X )] − δ .

Original: We then prove that the rate max(0 , H[ M ( X )] − δ ) is achievable from which we conclude that Rate ( δ )
Revised: We then prove that the rate max(0 , H[ M ( X )] − δ ) is achievable from which we conclude that Rate ( δ )

Original: = max(0 ,
Revised: = max(0 ,

Original: H[ M ( X )] − δ ) .
Revised: H[ M ( X )] − δ ) .

Original: Finally, we conclude by proving H
Revised: Finally, we conclude by proving H

Original: [ M ( X )]
Revised: [ M ( X )]

Original: =
Revised: =

Original: H[ X ] − H
Revised: H[ X ] − H

Original: [ X | M ( X )] to get our result.
Revised: [ X | M ( X )] to get our result.

Original: We want to transmit Z s.t.
Revised: We want to transmit Z s.t.

Original: ∀
Revised: ∀

Original: Y ∈ T we have R[ Y | Z ] ≤ δ + R[ Y | X ] , in other words we would like sup Y ∈T
Revised: Y ∈ T we have R[ Y | Z ] ≤ δ + R[ Y | X ] , in other words we would like sup Y ∈T

Original: R[ Y | Z ] − R
Revised: R[ Y | Z ] − R

Original: [ Y | X ]
Revised: [ Y | X ]

Original: = :
Revised: = :

Original: D ∼ [ X, Z ] < δ .
Revised: D ∼ [ X, Z ] < δ .

Original: We thus need to compute the minimal achievable bit-rate for transmitting an i.i.d.
Revised: We thus need to compute the minimal achievable bit-rate for transmitting an i.i.d.

Original: source of Z s.t.
Revised: source of Z s.t.

Original: D ∼ [ X, Z ] ≤ δ .
Revised: D ∼ [ X, Z ] ≤ δ .

Original: As D T is valid (Prop. 1) we can directly apply the rate distortion theorem (Lemma 6):
Revised: As D T is valid (Prop. 1) we can directly apply the rate distortion theorem (Lemma 6):

Original: [Equation]
Revised: [Equation]

Original: Lemma 6 and Prop
Revised: Lemma 6 and Prop

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Bijection
Revised: Bijection

Original: [Equation]
Revised: [Equation]

Original: Chain Rule
Revised: Chain Rule

Original: [Equation]
Revised: [Equation]

Original: Positivity
Revised: Positivity

Original: [Equation]
Revised: [Equation]

Original: Prop
Revised: Prop

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Z
Revised: Z

Original: [Equation]
Revised: [Equation]

Original: Where Eq.
Revised: Where Eq.

Original: (37) uses the invariance of mutual information to bijections which here is X (cid:55)→ X, M ( X ) .
Revised: (37) uses the invariance of mutual information to bijections which here is X (cid:55)→ X, M ( X ) .

Original: As the rate is always non-negative we have Rate ( δ ) ≥ max(0 , H
Revised: As the rate is always non-negative we have Rate ( δ ) ≥ max(0 , H

Original: [ M ( X )] − δ ) .
Revised: [ M ( X )] − δ ) .

Original: We now prove that max(0 , H[ M ( X )] − δ ) is attainable and so Rate ( δ )
Revised: We now prove that max(0 , H[ M ( X )] − δ ) is attainable and so Rate ( δ )

Original: = max(0 ,
Revised: = max(0 ,

Original: H[ M ( X )] − δ ) .
Revised: H[ M ( X )] − δ ) .

Original: Speciﬁcally we need to ﬁnd a representation Z of X s.t.
Revised: Speciﬁcally we need to ﬁnd a representation Z of X s.t.

Original: [Equation]
Revised: [Equation]

Original: The ﬁrst case is trivial: set Z to be independent of M ( X ) and X , e.g. a constant.
Revised: The ﬁrst case is trivial: set Z to be independent of M ( X ) and X , e.g. a constant.

Original: Then, D ∼ [ X, Z ] =
Revised: Then, D ∼ [ X, Z ] =

Original: H[ M ( X ) | Z ] =
Revised: H[ M ( X ) | Z ] =

Original: H[ M ( X )] ≤ δ and Rate ( δ ) =
Revised: H[ M ( X )] ≤ δ and Rate ( δ ) =

Original: I[ Z ; X ] = 0 .
Revised: I[ Z ; X ] = 0 .

Original: For the second case we need Rate ( δ ) ≥ H
Revised: For the second case we need Rate ( δ ) ≥ H

Original: [ M ( X )] − δ to be an equality when δ > H
Revised: [ M ( X )] − δ to be an equality when δ > H

Original: [ M ( X )] .
Revised: [ M ( X )] .

Original: This happens iff there exists a Z s.t.
Revised: This happens iff there exists a Z s.t.

Original: inequalities Eq.
Revised: inequalities Eq.

Original: (39) and Eq.
Revised: (39) and Eq.

Original: (42) are equalities, i.e. iff
Revised: (42) are equalities, i.e. iff

Original: I[ X ; Z | M ( X )]
Revised: I[ X ; Z | M ( X )]

Original: = 

Revised: =

Original: and D ∼ [ X, Z ] = δ .
Revised: and D ∼ [ X, Z ] = δ .

Original: To get that we start from Z = M ( X ) (which satisﬁes
Revised: To get that we start from Z = M ( X ) (which satisﬁes

Original: I[ X ; Z | M ( X )] = 0 ) and “modify” it s.t.
Revised: I[ X ; Z | M ( X )] = 0 ) and “modify” it s.t.

Original: D ∼ [ X, Z ] = δ .
Revised: D ∼ [ X, Z ] = δ .

Original: We can do so by “erasing” a fraction of bits by mapping all m → (cid:15) with a constant probability α , similarly to binary erasure channels.
Revised: We can do so by “erasing” a fraction of bits by mapping all m → (cid:15) with a constant probability α , similarly to binary erasure channels.

Original: Speciﬁcally, let Z :=
Revised: Speciﬁcally, let Z :=

Original: M∪{ (cid:15) } and Z be a r.v.
Revised: M∪{ (cid:15) } and Z be a r.v.

Original: that t.v.i.
Revised: that t.v.i.

Original: Z and whose conditional density parametrized by α ∈ [0 , 1[ is:
Revised: Z and whose conditional density parametrized by α ∈ [0 , 1[ is:

Original: [Equation]
Revised: [Equation]

Original: A simple computation then gives D ∼ [ X, Z ]
Revised: A simple computation then gives D ∼ [ X, Z ]

Original: = H[ M ( X ) | Z ] =
Revised: = H[ M ( X ) | Z ] =

Original: (1 − α )
Revised: (1 − α )

Original: H[ M ( X ) | Z = M ( X )]
Revised: H[ M ( X ) | Z = M ( X )]

Original: + α H[ M ( X )
Revised: + α H[ M ( X )

Original: | Z
Revised: | Z

Original: =
Revised: =

Original: (cid:15) ]
Revised: (cid:15) ]

Original: = α H[ M ( X )] .
Revised: = α H[ M ( X )] .

Original: To have D ∼ [ X, Z ]
Revised: To have D ∼ [ X, Z ]

Original: = δ we thus need to set α = δ H
Revised: = δ we thus need to set α = δ H

Original: [ M ( X )] .
Revised: [ M ( X )] .

Original: Note that we do not divide by zero as if H[ M ( X )]
Revised: Note that we do not divide by zero as if H[ M ( X )]

Original: = 0 would be in the ﬁrst case of Eq.
Revised: = 0 would be in the ﬁrst case of Eq.

Original: We thus proved that max(0 , H[ M ( X )] − δ ) is obtainable and that Rate ( δ ) ≥ max(0 ,
Revised: We thus proved that max(0 , H[ M ( X )] − δ ) is obtainable and that Rate ( δ ) ≥ max(0 ,

Original: H[ M ( X )] − δ ) .
Revised: H[ M ( X )] − δ ) .

Original: From which we conclude that the best achievable bit-rate is Rate ( δ )
Revised: From which we conclude that the best achievable bit-rate is Rate ( δ )

Original: = max(0 ,
Revised: = max(0 ,

Original: H[ M ( X )] − δ ) .
Revised: H[ M ( X )] − δ ) .

Original: Eq.
Revised: Eq.

Original: (35), follows from H[ M ( X )]
Revised: (35), follows from H[ M ( X )]

Original: = I[ M ( X ); X ]
Revised: = I[ M ( X ); X ]

Original: = H[ X ] − H
Revised: = H[ X ] − H

Original: [ X | M ( X )] .
Revised: [ X | M ( X )] .

Original: The ﬁniteness of Rate ( δ ) comes from the fact that Rate ( δ ) ≤ H[ M ( X )] < ∞ due to Assumption 7.
Revised: The ﬁniteness of Rate ( δ ) comes from the fact that Rate ( δ ) ≤ H[ M ( X )] < ∞ due to Assumption 7.

Original: By setting δ = 

Revised: By setting δ =

Original: we directly get the best achievable rate for the lossless prediction but lossy compression setting.
Revised: we directly get the best achievable rate for the lossless prediction but lossy compression setting.

Original: Corollary 1 (Invariant Source Coding) .
Revised: Corollary 1 (Invariant Source Coding) .

Original: Let X be a r.v. and δ ≥ 0 .
Revised: Let X be a r.v. and δ ≥ 0 .

Original: Let T be the invariant tasks of interest as in Def. 3, M be any maximal invariant as in Def. 5, and Z be a representation of X as in
Revised: Let T be the invariant tasks of interest as in Def. 3, M be any maximal invariant as in Def. 5, and Z be a representation of X as in

Original: Def. 2.
Revised: Def. 2.

Original: Let Rate ( δ ) denote the minimum achievable bit-rate for transmitting an i.i.d.
Revised: Let Rate ( δ ) denote the minimum achievable bit-rate for transmitting an i.i.d.

Original: source of Z s.t.
Revised: source of Z s.t.

Original: for any Y ∈ T we have R[ Y | Z ]
Revised: for any Y ∈ T we have R[ Y | Z ]

Original: =
Revised: =

Original: R[ Y | X ] .
Revised: R[ Y | X ] .

Original: Then Rate ( δ ) is ﬁnite and given by
Revised: Then Rate ( δ ) is ﬁnite and given by

Original: [Equation]
Revised: [Equation]

Original: C. R ECOVERING S UBCASES
Revised: C. R ECOVERING S UBCASES

Original: Corollary 1 recovers many previous results in the literature:
Revised: Corollary 1 recovers many previous results in the literature:

Original: Unlabeled Graphs Let us consider the task of compressing unlabeled graphs, here we consider tasks that are invariant to graph isomorphisms.
Revised: Unlabeled Graphs Let us consider the task of compressing unlabeled graphs, here we consider tasks that are invariant to graph isomorphisms.

Original: A possible maximal invariant is the graph canonization and H[ M ( X )] becomes the well known structural entropy (Rashevsky, 1955;
Revised: A possible maximal invariant is the graph canonization and H[ M ( X )] becomes the well known structural entropy (Rashevsky, 1955;

Original: Yongwook Choi & Szpankowski, 2009).
Revised: Yongwook Choi & Szpankowski, 2009).

Original: 10 If all isomorphic graphs are permissible and equiprobable, Yongwook Choi & Szpankowski (2009) show that the structural entropy is
Revised: 11 If all isomorphic graphs are permissible and equiprobable, Yongwook Choi & Szpankowski (2009) show that the structural entropy is

Original: [Equation]
Revised: [Equation]

Original: ( x )
Revised: ( x )

Original: [Equation]
Revised: [Equation]

Original: (cid:105) .
Revised: (cid:105) .

Original: This is Eq.
Revised: This is Eq.

Original: (47), with the second term corresponds to H
Revised: (47), with the second term corresponds to H

Original: [ X | M ( X )] with a uniform distribution on isomorphic graphs.
Revised: [ X | M ( X )] with a uniform distribution on isomorphic graphs.

Original: Multisets Let us derive the best achievable bit-rate for compressing multisets.
Revised: Multisets Let us derive the best achievable bit-rate for compressing multisets.

Original: Let X be any sequence and T be invariant to permutations of that sequence.
Revised: Let X be any sequence and T be invariant to permutations of that sequence.

Original: One possible maximal invariant in that case is the empirical measure (also called type), i.e., the counts K 1 , . .
Revised: One possible maximal invariant in that case is the empirical measure (also called type), i.e., the counts K 1 , . .

Original: , K n of each of the n elements that are present in the sequence X .
Revised: , K n of each of the n elements that are present in the sequence X .

Original: Lossless compression of multisets thus requires
Revised: Lossless compression of multisets thus requires

Original: H[ M ( X )]
Revised: H[ M ( X )]

Original: =
Revised: =

Original: H[ K 1 , . .
Revised: H[ K 1 , . .

Original: . , K n ] , as discussed in (Varshney & Goyal, 2007).
Revised: . , K n ] , as discussed in (Varshney & Goyal, 2007).

Original: Using Eq.
Revised: Using Eq.

Original: (47) we can also characterize the bits gains that you obtain by considering the invariance, namely,
Revised: (47) we can also characterize the bits gains that you obtain by considering the invariance, namely,

Original: H
Revised: H

Original: [ X | M ( X )] .
Revised: [ X | M ( X )] .

Original: This recovers theorem 1 of (Varshney & Goyal, 2007), where H[ X | M ( X )] is called the “order entropy”.
Revised: This recovers theorem 1 of (Varshney & Goyal, 2007), where H[ X | M ( X )] is called the “order entropy”.

Original: Note that similarly to our example in the text about i.i.d.
Revised: Note that similarly to our example in the text about i.i.d.

Original: coin ﬂips, the amount of bits needed to losslessly compress the multiset grows as θ (log n ) (Varshney & Goyal, 2007).
Revised: coin ﬂips, the amount of bits needed to losslessly compress the multiset grows as θ (log n ) (Varshney & Goyal, 2007).

Original: Information Bottleneck (IB) Suppose you are interested in predicting a single task Y
Revised: Information Bottleneck (IB) Suppose you are interested in predicting a single task Y

Original: = t ( X ) , where t is a (deterministic) “target function”.
Revised: = t ( X ) , where t is a (deterministic) “target function”.

Original: The task is invariant to the labeling so the maximal invariant is t ( · ) and the distortion becomes H[ T ( X ) | Z ]
Revised: The task is invariant to the labeling so the maximal invariant is t ( · ) and the distortion becomes H[ T ( X ) | Z ]

Original: = H[ Y | Z ] .
Revised: = H[ Y | Z ] .

Original: Then Eq.
Revised: Then Eq.

Original: (1) becomes the information bottleneck (IB) (Tishby et al., 2000).
Revised: (1) becomes the information bottleneck (IB) (Tishby et al., 2000).

Original: Using Corollary 1 we see that for lossless predictions the optimal rate is Rate (0) =
Revised: Using Corollary 1 we see that for lossless predictions the optimal rate is Rate (0) =

Original: H[ Y ] =
Revised: H[ Y ] =

Original: H[ X ] − H
Revised: H[ X ] − H

Original: [ X | Y ] =
Revised: [ X | Y ] =

Original: I[ X ; Y ] as shown in (Wu et al., 2019; Fischer, 2020).
Revised: I[ X ; Y ] as shown in (Wu et al., 2019; Fischer, 2020).

Original: From a compression stand point this is nevertheless not very useful as Rate (0) =
Revised: From a compression stand point this is nevertheless not very useful as Rate (0) =

Original: H[ Y ] , so IB for deterministic labels tells you to entropy code Y .
Revised: H[ Y ] , so IB for deterministic labels tells you to entropy code Y .

Original: Lossless Let X be discrete.
Revised: Lossless Let X be discrete.

Original: Every task will always be invariant to equality “ = “.
Revised: Every task will always be invariant to equality “ = “.

Original: In this case the maximal invariant is the identity function, and we recover Shannon’s source coding theorem
Revised: In this case the maximal invariant is the identity function, and we recover Shannon’s source coding theorem

Original: Rate (0) =
Revised: Rate (0) =

Original: H[ M ( X )]
Revised: H[ M ( X )]

Original: = H
Revised: = H

Original: [ X ] .
Revised: [ X ] .

Original: D V ARIATIONAL O BJECTIVES
Revised: D V ARIATIONAL O BJECTIVES

Original: In this section we will derive the variational bounds for estimating the rate and the distortion.
Revised: In this section we will derive the variational bounds for estimating the rate and the distortion.

Original: Recall that the optimal bit-rate is simply the Rate Distortion function wusing our invariance distortion (
Revised: Recall that the optimal bit-rate is simply the Rate Distortion function wusing our invariance distortion (

Original: Eq.
Revised: Eq.

Original: (36) ), so pareto optimal encoder (for delta ) can be obtained by using the following arg minimum:
Revised: (36) ), so pareto optimal encoder (for delta ) can be obtained by using the following arg minimum:

Original: [Equation]
Revised: [Equation]

Original: As optimization in machine learning is typically unconstrained, we can use the Lagragian relaxation instead
Revised: As optimization in machine learning is typically unconstrained, we can use the Lagragian relaxation instead

Original: [Equation]
Revised: [Equation]

Original: Both terms I[ X ; Z ] and R[ M ( X ) | Z ] are hard to estimate from samples, so the rest of the section is devoted to deriving variational upper bounds on them.
Revised: Both terms I[ X ; Z ] and R[ M ( X ) | Z ] are hard to estimate from samples, so the rest of the section is devoted to deriving variational upper bounds on them.

Original: D. V ARIATIONAL R ATE I[ X ; Z ]
Revised: D. V ARIATIONAL R ATE I[ X ; Z ]

Original: Let us discuss how to approximate the rate term
Revised: Let us discuss how to approximate the rate term

Original: I[ X ; Z ] .
Revised: I[ X ; Z ] .

Original: The mutual information is well known to be hard to estimate from samples (Paninski,2003; McAllester & Stratos, 2020), but fortunaltely many 
Revised: The mutual information is well known to be hard to estimate from samples (Paninski, McAllester & Stratos, 2020), but fortunaltely many of

Original: Mutual Information Bottleneck.
Revised: Mutual Information Bottleneck.

Original: The ﬁrst bound that we consider is the standard upper bound on
Revised: The ﬁrst bound that we consider is the standard upper bound on

Original: I[ X ; Z ] , e.g. in VAE or VIB.
Revised: I[ X ; Z ] , e.g. in VAE or VIB.

Original: Speciﬁcally:
Revised: Speciﬁcally:

Original: [Equation]
Revised: [Equation]

Original: We call this the variational mutual information bottleneck bound.
Revised: We call this the variational mutual information bottleneck bound.

Original: The approximation gap is then min q ∈Q D KL [ p ( Z ) (cid:107) q ( Z )] .
Revised: The approximation gap is then min q ∈Q D KL [ p ( Z ) (cid:107) q ( Z )] .

Original: This bound has the advantage that if p ( Z ) ∈ Q then bound is tight, which is of course the case when the variational family Q is unconstrained.
Revised: This bound has the advantage that if p ( Z ) ∈ Q then bound is tight, which is of course the case when the variational family Q is unconstrained.

Original: The major issue with the mutual information bottleneck, is that no efﬁcient compressors can in general achieve the rate given by it (Agustsson & Theis, 2020).
Revised: The major issue with the mutual information bottleneck, is that no efﬁcient compressors can in general achieve the rate given by it (Agustsson & Theis, 2020).

Original: Entropy Bottleneck.
Revised: Entropy Bottleneck.

Original: To have efﬁcient compressors we would like a bound such that the given rate can be achieved by an entropy coder.
Revised: To have efﬁcient compressors we would like a bound such that the given rate can be achieved by an entropy coder.

Original: Indeed, entropy coders have been developped for years and are now very efﬁcient (Rissanen, 1976; Duda, 2009).
Revised: Indeed, entropy coders have been developped for years and are now very efﬁcient (Rissanen, 1976; Duda, 2009).

Original: To dervie such bounds it sufﬁces to realize that the mutual information of two r.v. is upperbounded by the entropy of each of those r.v.s., speciﬁcally,
Revised: To dervie such bounds it sufﬁces to realize that the mutual information of two r.v. is upperbounded by the entropy of each of those r.v.s., speciﬁcally,

Original: I[ Z ; X ] =
Revised: I[ Z ; X ] =

Original: H[ Z ] − H
Revised: H[ Z ] − H

Original: [ Z | X ] ≤ HZ and the bound is tight for Z that are deterministic transformations of X .
Revised: [ Z | X ] ≤ HZ and the bound is tight for Z that are deterministic transformations of X .

Original: So Eq.
Revised: So Eq.

Original: (55) becomes
Revised: (55) becomes

Original: [Equation]
Revised: [Equation]

Original: This is the standard bound used in neural compressors (Ballé et al., 2017; Theis et al., 2017).
Revised: This is the standard bound used in neural compressors (Ballé et al., 2017; Theis et al., 2017).

Original: We call this the variational entropy bottleneck.
Revised: We call this the variational entropy bottleneck.

Original: Note that achieving the rate can be done efﬁciently if Z is discrete by entropy entropy coding using the trained q ( Z ) .
Revised: Note that achieving the rate can be done efﬁciently if Z is discrete by entropy entropy coding using the trained q ( Z ) .

Original: This advantage comes with two main downsides of Eq.
Revised: This advantage comes with two main downsides of Eq.

Original: • It is generally not true that any (for any δ ) optimal rate can be achieved by a discrete and deterministic Z .
Revised: • It is generally not true that any (for any δ ) optimal rate can be achieved by a discrete and deterministic Z .

Original: For teh speciﬁc case of δ = 

Revised: For teh speciﬁc case of δ =

Original: it is the case, as we can simply set Z = M ( X ) .
Revised: it is the case, as we can simply set Z = M ( X ) .

Original: • Eq.
Revised: • Eq.

Original: (56) is unfortunately not suitable for gradient based optimization w.r.t.
Revised: (56) is unfortunately not suitable for gradient based optimization w.r.t.

Original: to the encoder (due to the discreteness of Z ) so we typically have to add noise during training (Ballé et al., 2017) which can cause a mismatch between training and testing (Agustsson & Theis, 2020).
Revised: to the encoder (due to the discreteness of Z ) so we typically have to add noise during training (Ballé et al., 2017) which can cause a mismatch between training and testing (Agustsson & Theis, 2020).

Original: Despite these issues we will mostly use the entropy bottleneck bound in experiments as we want our method to give rise to practical compressors.
Revised: Despite these issues we will mostly use the entropy bottleneck bound in experiments as we want our method to give rise to practical compressors.

Original: D. V ARIATIONAL D ISTORTION R[ M ( X ) | Z ]
Revised: D. V ARIATIONAL D ISTORTION R[ M ( X ) | Z ]

Original: Let us now consider variational upper-bounds on the distortion R[ M ( X ) | Z ] .
Revised: Let us now consider variational upper-bounds on the distortion R[ M ( X ) | Z ] .

Original: Direct Distortion.
Revised: Direct Distortion.

Original: The obvious variational bound on the conditional entropy is the standard cross entropy loss as used for the distortion of VAE, VIB, and standard VC.
Revised: The obvious variational bound on the conditional entropy is the standard cross entropy loss as used for the distortion of VAE, VIB, and standard VC.

Original: Let Q (cid:48) denote a family of regular conditional distributions (decoders), then:
Revised: Let Q (cid:48) denote a family of regular conditional distributions (decoders), then:

Original: [Equation]
Revised: [Equation]

Original: Lemma
Revised: Lemma

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation]
Revised: [Equation]

Original: We call this the variational direct distortion, as we directly try to predict / reconstruct M ( X ) using a decoder q (cid:48)
Revised: We call this the variational direct distortion, as we directly try to predict / reconstruct M ( X ) using a decoder q (cid:48)

Original: ( M ( X ) | Z ) .
Revised: ( M ( X ) | Z ) .

Original: The approximation gap here is min q (cid:48) ∈Q (cid:48)
Revised: The approximation gap here is min q (cid:48) ∈Q (cid:48)

Original: D KL [ p ( M ( X ) , Z ) (cid:107) q (cid:48)
Revised: D KL [ p ( M ( X ) , Z ) (cid:107) q (cid:48)

Original: ( M ( X ) | Z ) p ( Z )] .
Revised: ( M ( X ) | Z ) p ( Z )] .

Original: This direct distortion is simple and well understood but requires M which is usually not known and will thus have to be approximated.
Revised: This direct distortion is simple and well understood but requires M which is usually not known and will thus have to be approximated.

Original: Furthermore, if M is in high dimension (such as when reconstructing unaugmented images) the trained decoder q (cid:48) ( M ( X ) | Z ) can often underﬁt.
Revised: Furthermore, if M is in high dimension (such as when reconstructing unaugmented images) the trained decoder q (cid:48) ( M ( X ) | Z ) can often underﬁt.

Original: Contrastive Distortion.
Revised: Contrastive Distortion.

Original: We now consider a bound that does not directly require M , by considering a contrastive estimator.
Revised: We now consider a bound that does not directly require M , by considering a contrastive estimator.

Original: Suppose that for every X we can sample a sequence X :=
Revised: Suppose that for every X we can sample a sequence X :=

Original: ( X + , X − 1 , . . .
Revised: ( X + , X − 1 , . . .

Original: , X − n ) s.t.
Revised: , X − n ) s.t.

Original: X + ∼ x
Revised: X + ∼ x

Original: (it is “postive”) and each X − i are not ( X − i (cid:54)∼ x , they are “negatives”).
Revised: (it is “postive”) and each X − i are not ( X − i (cid:54)∼ x , they are “negatives”).

Original: Then, we can use the InfoNCE (Oord et al., 2019) bound on information, that is standard in self-supervised learning.
Revised: Then, we can use the InfoNCE (Oord et al., 2019) bound on information, that is standard in self-supervised learning.

Original: Let Z , be the sequence of representations that are sampled by passing x through p ( Z | X ) .
Revised: Let Z , be the sequence of representations that are sampled by passing x through p ( Z | X ) .

Original: Let
Revised: Let

Original: F :=
Revised: F :=

Original: { f : Z × Z → R } be a family of discriminators that scores how equivalent two representations are, then:
Revised: { f : Z × Z → R } be a family of discriminators that scores how equivalent two representations are, then:

Original: [Equation]
Revised: [Equation]

Original: InfoNCE
Revised: InfoNCE

Original: [Equation]
Revised: [Equation]

Original: Eq.
Revised: Eq.

Original: (64) uses the fact that InfoNCE is a valid lower bound on mutual information (Poole et al., 2019).
Revised: (64) uses the fact that InfoNCE is a valid lower bound on mutual information (Poole et al., 2019).

Original: The last equation removes constant w.r.t.
Revised: The last equation removes constant w.r.t.

Original: Z and shows that we are only left with a log softmax term which essentially tries to classify which of all the sampled representations is positive.
Revised: Z and shows that we are only left with a log softmax term which essentially tries to classify which of all the sampled representations is positive.

Original: Note that the contrastive distortion has the advantage of not having to reconstruct high dimensional data (e.g. for images), but it suffers from bias in the case where the number of negatives n is small (Poole et al., 2019).
Revised: Note that the contrastive distortion has the advantage of not having to reconstruct high dimensional data (e.g. for images), but it suffers from bias in the case where the number of negatives n is small (Poole et al., 2019).

Original: D. P RACTICAL L OSSES
Revised: D. P RACTICAL L OSSES

Original: In practice we will use parametrized neural network for the different variational families ( q ϕ ( X | Z ) , q θ ( Z ) , q φ ( X | Z ) ) as well as for the encoder ( p ϕ ( X | Z ) ).
Revised: In practice we will use parametrized neural network for the different variational families ( q ϕ ( X | Z ) , q θ ( Z ) , q φ ( X | Z ) ) as well as for the encoder ( p ϕ ( X | Z ) ).

Original: As discussed in the main text we usually assume access to samples from M ( X ) and to a random generator of augmentation A .
Revised: As discussed in the main text we usually assume access to samples from M ( X ) and to a random generator of augmentation A .

Original: The four previous bounds can be mixed and matched to provide four different practical losses to learn invariant compressors.
Revised: The four previous bounds can be mixed and matched to provide four different practical losses to learn invariant compressors.

Original: As the mutual information bounds do not give rise to practical compressors, we focus below on the entropy bottleneck bound for the rate, which gives rise to the VIC and BINCE loss discussed in the main text.
Revised: As the mutual information bounds do not give rise to practical compressors, we focus below on the entropy bottleneck bound for the rate, which gives rise to the VIC and BINCE loss discussed in the main text.

Original: • The VIC loss is recovered using the variational entropy bottleneck and the variational direct distortion bounds.
Revised: • The VIC loss is recovered using the variational entropy bottleneck and the variational direct distortion bounds.

Original: The ﬁnal criterion for training a neural network is summarized in Algorithm 1.
Revised: The ﬁnal criterion for training a neural network is summarized in Algorithm 1.

Original: As previously discussed the encoder is stochastic during training to enable backpropagation but deterministic at test time, for details see (Ballé et al., 2017).
Revised: As previously discussed the encoder is stochastic during training to enable backpropagation but deterministic at test time, for details see (Ballé et al., 2017).

Original: • The BINCE loss is recovered using the variational entropy bottleneck and the variational contrative distortion bound.
Revised: • The BINCE loss is recovered using the variational entropy bottleneck and the variational contrative distortion bound.

Original: The ﬁnal criterion for training a neural network is summarized in Algorithm 2.
Revised: The ﬁnal criterion for training a neural network is summarized in Algorithm 2.

Original: Similarly to VIC the encoder is stochastic during training but deterministic at test time.
Revised: Similarly to VIC the encoder is stochastic during training but deterministic at test time.

Original: Require:
Revised: Require:

Original: Encoder p ϕ
Revised: Encoder p ϕ

Original: ( Z | X ) , Entropy Model q θ ( Z ) , Decoder q φ ( X | Z )
Revised: ( Z | X ) , Entropy Model q θ ( Z ) , Decoder q φ ( X | Z )

Original: Require: Dataset D , Random augmentation generator A , β 1: x ← select( D ) (cid:46) Sample 2: ˜ x
Revised: Require: Dataset D , Random augmentation generator A , β 1: x ← select( D ) (cid:46) Sample 2: ˜ x

Original: ← A ( x ) (cid:46) Augment 3: z ← sample( p ϕ ( Z | ˜ x ))
Revised: ← A ( x ) (cid:46) Augment 3: z ← sample( p ϕ ( Z | ˜ x ))

Original: (cid:46) Encode 4: rate _ loss ← − log q θ ( z ) (cid:46)
Revised: (cid:46) Encode 4: rate _ loss ← − log q θ ( z ) (cid:46)

Original: Entropy Bottleneck 5: distortion _ loss ← − log q φ ( x | z ) (cid:46) Direct Distortion 6: Return rate _ loss + β · distortion _ loss
Revised: Entropy Bottleneck 5: distortion _ loss ← − log q φ ( x | z ) (cid:46) Direct Distortion 6: Return rate _ loss + β · distortion _ loss

Original: Require:
Revised: Require:

Original: Encoder p ϕ
Revised: Encoder p ϕ

Original: ( Z | X ) , Entropy Model q θ ( Z ) , discriminator f ψ ,
Revised: ( Z | X ) , Entropy Model q θ ( Z ) , discriminator f ψ ,

Original: Require: Dataset D , Random augmentation generator A , β , number of negatives n 1: x ← select( D ) (cid:46) Sample 2: ˜ x
Revised: Require: Dataset D , Random augmentation generator A , β , number of negatives n 1: x ← select( D ) (cid:46) Sample 2: ˜ x

Original: ← A ( x ) (cid:46) Augment 3: z ← sample( p ϕ ( Z | ˜ x ))
Revised: ← A ( x ) (cid:46) Augment 3: z ← sample( p ϕ ( Z | ˜ x ))

Original: (cid:46) Encode 4: rate _ loss ← − log q θ ( z ) (cid:46)
Revised: (cid:46) Encode 4: rate _ loss ← − log q θ ( z ) (cid:46)

Original: Entropy Bottleneck 5: x + ← A ( x ) (cid:46)
Revised: Entropy Bottleneck 5: x + ← A ( x ) (cid:46)

Original: Sample Positive 6: { x − i } ni
Revised: Sample Positive 6: { x − i } ni

Original: =1 ← select( D ) n times (cid:46)
Revised: =1 ← select( D ) n times (cid:46)

Original: Sample Negatives 7: x ← [ x + , x − 1 , . . .
Revised: Sample Negatives 7: x ← [ x + , x − 1 , . . .

Original: , x − n ] (cid:46) Concatenate 8: z ← sample( p ϕ ( Z | x ))
Revised: , x − n ] (cid:46) Concatenate 8: z ← sample( p ϕ ( Z | x ))

Original: (cid:46)
Revised: (cid:46)

Original: Encode
Revised: Encode

Original: each x 9: softmax ← exp f
Revised: each x 9: softmax ← exp f

Original: ( z + ,z ) (cid:80) z (cid:48)∈ z exp f ( z (cid:48) ,z ) 10: distortion _ loss ← − log(softmax) (cid:46) Contrastive Distortion 11: return rate _ loss + β ·
Revised: ( z + ,z ) (cid:80) z (cid:48)∈ z exp f ( z (cid:48) ,z ) 10: distortion _ loss ← − log(softmax) (cid:46) Contrastive Distortion 11: return rate _ loss + β ·

Original: distortion _ loss
Revised: distortion _ loss

Original: Although the variational mutual information bound does not give rise to efﬁcient coders, they can still be of interest for other tasks such as for representation learning.
Revised: Although the variational mutual information bound does not give rise to efﬁcient coders, they can still be of interest for other tasks such as for representation learning.

Original: For example, VIC with a variational mutual information bound becomes a standard VAE where the input is augmented, but the target reconstructions M ( X ) are not.
Revised: For example, VIC with a variational mutual information bound becomes a standard VAE where the input is augmented, but the target reconstructions M ( X ) are not.

Original: This is also equivalent to a VIB where the task is to predict the maximal invariant M ( X ) , which to the best of our knowledge is the ﬁrst self-supervised formulation of the information bottleneck.
Revised: This is also equivalent to a VIB where the task is to predict the maximal invariant M ( X ) , which to the best of our knowledge is the ﬁrst self-supervised formulation of the information bottleneck.

Original: To train VIC or BINCE using the variational mutual information bound instead of the entropy bound it sufﬁces to use stochastic encoders (typically neural networks predicting a mean and covariance of a mutlivariate Gaussian) and replacing E p ( X ) p ( Z | X ) [ − log q ( Z )] by E p ( X ) [D KL [ p ( Z | X )] q ( Z )] in the “entropy bottleneck” step of both algorithms.
Revised: To train VIC or BINCE using the variational mutual information bound instead of the entropy bound it sufﬁces to use stochastic encoders (typically neural networks predicting a mean and covariance of a mutlivariate Gaussian) and replacing E p ( X ) p ( Z | X ) [ − log q ( Z )] by E p ( X ) [D KL [ p ( Z | X )] q ( Z )] in the “entropy bottleneck” step of both algorithms.

Original: E E XTENDED P REVIOUS W ORK Lossy image compression
Revised: E E XTENDED P REVIOUS W ORK Lossy image compression

Original: The connection between the rate-distortion objective and variational inference is well known (MacKay, 1992; ? ).
Revised: The connection between the rate-distortion objective and variational inference is well known (MacKay, 1992; ? ).

Original: Ballé et al.
Revised: Ballé et al.

Original: (2017) and Theis et al.
Revised: (2017) and Theis et al.

Original: ﬁrst implemented large scale neural image compression based on Variational Autoencoders (VAEs).
Revised: ﬁrst implemented large scale neural image compression based on Variational Autoencoders (VAEs).

Original: Later approaches improved upon the original model by adding a hierarchical layers and autoregressive components (Ballé et al., 2018; Minnen et al., 2018; Lee et al., 2019; Johnston et al., 2018).
Revised: Later approaches improved upon the original model by adding a hierarchical layers and autoregressive components (Ballé et al., 2018; Minnen et al., 2018; Lee et al., 2019; Johnston et al., 2018).

Original: Further recent studies improve the compressor by improving the inference procedure of these models (Yang et al., 2020).
Revised: Further recent studies improve the compressor by improving the inference procedure of these models (Yang et al., 2020).

Original: Another aspect, is computational complexity (Johnston et al., 2019).
Revised: Another aspect, is computational complexity (Johnston et al., 2019).

Original: Task-Centric Compression
Revised: Task-Centric Compression

Original: At the core of our work is the idea that compression should be taskcentric, i.e., optimized for a set of tasks of interest.
Revised: At the core of our work is the idea that compression should be taskcentric, i.e., optimized for a set of tasks of interest.

Original: To the best of our knowledge, the only theoretical framework that considers this question is Tishby et al.’s (2000) Information Bottleneck (IB), which was derived using rate distortion theory for the speciﬁc distortion H[ Y | Z ] .
Revised: To the best of our knowledge, the only theoretical framework that considers this question is Tishby et al.’s (2000) Information Bottleneck (IB), which was derived using rate distortion theory for the speciﬁc distortion H[ Y | Z ] .

Original: IB was nevertheless never (to our knowledge) used for compression/
Revised: IB was nevertheless never (to our knowledge) used for compression/

Original: The lack of use of IB in compression probably comes from its requirement of (1) a single task; (2) knowing the labels at compression time.
Revised: The lack of use of IB in compression probably comes from its requirement of (1) a single task; (2) knowing the labels at compression time.

Original: The ﬁrst problem could easily be overcome by using a “multi-task” distortion H[ T | Z ] .
Revised: The ﬁrst problem could easily be overcome by using a “multi-task” distortion H[ T | Z ] .

Original: The second point is the main drawback of IB for compression.
Revised: The second point is the main drawback of IB for compression.

Original: Indeed, it would be easier to directly compress the labels if we had access to them.
Revised: Indeed, it would be easier to directly compress the labels if we had access to them.

Original: Our paper can be seen as a self-supervised extension of IB that is useful for compression as it does not require the labels at compression time.
Revised: Our paper can be seen as a self-supervised extension of IB that is useful for compression as it does not require the labels at compression time.

Original: From a practical perspective, there have recent task-centric compression methods.
Revised: From a practical perspective, there have recent task-centric compression methods.

Original: Mentzer et al.
Revised: Mentzer et al.

Original: (2020) shows that they can signiﬁcantly decrease the bit-rates by using a generative adversarial network (Goodfellow et al., 2014).
Revised: (2020) shows that they can signiﬁcantly decrease the bit-rates by using a generative adversarial network (Goodfellow et al., 2014).

Original: This can be seen as a task-speciﬁc compression where the task is to ensure that a discriminator cannot discriminate between a highly compressed reconstruction and the original image.
Revised: This can be seen as a task-speciﬁc compression where the task is to ensure that a discriminator cannot discriminate between a highly compressed reconstruction and the original image.

Original: More related is Singh et al.’s
Revised: More related is Singh et al.’s

Original: (2020) work on end-to-end compression of pre-trained features for transfer learning.
Revised: (2020) work on end-to-end compression of pre-trained features for transfer learning.

Original: From a very practical perspective, their method is similar to our idea of compressing pretrained self-supervised features.
Revised: From a very practical perspective, their method is similar to our idea of compressing pretrained self-supervised features.

Original: Their work does not provide any theory justiﬁcation, and constrained to cases where downstream tasks to similar the task on which the featurizer was pre-trained.
Revised: Their work does not provide any theory justiﬁcation, and constrained to cases where downstream tasks to similar the task on which the featurizer was pre-trained.

Original: Nevertheless the intuitive idea is similar and shows that there is interest in taking advantage of recent advancements and open sourcing of pretrained models to perform task speciﬁc compression.
Revised: Nevertheless the intuitive idea is similar and shows that there is interest in taking advantage of recent advancements and open sourcing of pretrained models to perform task speciﬁc compression.

Original: F R EPRODUCIBILITY
Revised: F R EPRODUCIBILITY

Original: In this section we provide further details of the hyperparameters chosen for the various experiments in the main text.
Revised: In this section we provide further details of the hyperparameters chosen for the various experiments in the main text.

Original: Unless stated otherwise, all the models are trained for 200 epochs, using Adam ( ? )
Revised: Unless stated otherwise, all the models are trained for 200 epochs, using Adam ( ? )

Original: as the optimizer, a learning rate of 1 e − 3 , a batch-size of 128.
Revised: as the optimizer, a learning rate of 1 e − 3 , a batch-size of 128.

Original: We checkpoint and use the model which achieves the smallest validation loss for evaluation.
Revised: We checkpoint and use the model which achieves the smallest validation loss for evaluation.

Original: Results are averaged overrandom seeds, and standard errors are reported.
Revised: Results are averaged overrandom seeds, and standard errors are reported.

Original: For all convolutional layers we use Kaiming normal initialization(He et al., 2015), for all linear layers we use Kaiming uniform initialization(He et al., 2015), while all biases are always initialized at 0.
Revised: For all convolutional layers we use Kaiming normal initialization(He et al., 2015), for all linear layers we use Kaiming uniform initialization(He et al., 2015), while all biases are always initialized at 0.

Original: Activation functions are ReLUs while other unspeciﬁed parameters are PyTorch (Paszke et al., 2019) defaults.
Revised: Activation functions are ReLUs while other unspeciﬁed parameters are PyTorch (Paszke et al., 2019) defaults.

Original: Throughout this section, instead of optimizing I[ Z ; X ] + β D ∼ [ X, Z ] we optimize λ I
Revised: Throughout this section, instead of optimizing I[ Z ; X ] + β D ∼ [ X, Z ] we optimize λ I

Original: [ Z ; X ] + D ∼ [ X, Z ] , which is a more standard formulation for VIB, VAE, VC.
Revised: [ Z ; X ] + D ∼ [ X, Z ] , which is a more standard formulation for VIB, VAE, VC.

Original: F. B ANANA E XPERIMENTS
Revised: F. B ANANA E XPERIMENTS

Original: For the Banana dataset most of the arguments were selected so as to replicate Fig.1.B. from (Ballé et al., 2020).
Revised: For the Banana dataset most of the arguments were selected so as to replicate Fig.1.B. from (Ballé et al., 2020).

Original: The data distribution is obtained by starting from a bivariate Gaussian X ∼ N ( X ; 0 , diag([3 , 0 . 5])) .
Revised: The data distribution is obtained by starting from a bivariate Gaussian X ∼ N ( X ; 0 , diag([3 , 0 . 5])) .

Original: It is then transformed to a banana distribution using the following transformation: x 2 = x 2 +0 .
Revised: It is then transformed to a banana distribution using the following transformation: x 2 = x 2 +0 .

Original: 1 x 21 − 9 .
Revised: 1 x 21 − 9 .

Original: We then rotate it and shift it: X =
Revised: We then rotate it and shift it: X =

Original: (Rot( − 40) · X ) + [ − 3 , − 4] T .
Revised: (Rot( − 40) · X ) + [ − 3 , − 4] T .

Original: For every epoch we resample 1024000 new points, i.e., examples are never seen twice during training).
Revised: For every epoch we resample 1024000 new points, i.e., examples are never seen twice during training).

Original: For all Banana experiments we use a 2 dimensional representation Z ∈ R 2 , and a batch size of 8192 .
Revised: For all Banana experiments we use a 2 dimensional representation Z ∈ R 2 , and a batch size of 8192 .

Original: The encoders (and decoders if there is one) is always a 2-hidden layer MLP with 1024 hidden neurons, batch norm (Ioffe & Szegedy, 2015), and softplus activation.
Revised: The encoders (and decoders if there is one) is always a 2-hidden layer MLP with 1024 hidden neurons, batch norm (Ioffe & Szegedy, 2015), and softplus activation.

Original: The learning rate scheduling consists in decreasing the learning rate by a factor 10 at epochs 50,75,87, and 120.
Revised: The learning rate scheduling consists in decreasing the learning rate by a factor 10 at epochs 50,75,87, and 120.

Original: We train both a standard variational compressor (VC) and our variational invariant compressor (VIC), in both cases we use the factorized prior entropy model from (Ballé et al., 2018).
Revised: We train both a standard variational compressor (VC) and our variational invariant compressor (VIC), in both cases we use the factorized prior entropy model from (Ballé et al., 2018).

Original: To obtain RD curves we sweep over λ = 0 .
Revised: To obtain RD curves we sweep over λ = 0 .

Original: For Fig.
Revised: For Fig.

Original: 2 we use λ = 0 .
Revised: 2 we use λ = 0 .

Original: 1 for both plots.
Revised: 1 for both plots.

Original: For the classical compression we use a variational upper bound on H[ X | Z ] , namely min q (cid:48) ∈Q (cid:48)
Revised: For the classical compression we use a variational upper bound on H[ X | Z ] , namely min q (cid:48) ∈Q (cid:48)

Original: E p ( Z,X ) [ − log q (cid:48)
Revised: E p ( Z,X ) [ − log q (cid:48)

Original: ( X | Z )] .
Revised: ( X | Z )] .

Original: Essentially we are trying to reconstruct the input X using an MLP Q (cid:48) .
Revised: Essentially we are trying to reconstruct the input X using an MLP Q (cid:48) .

Original: For our compression we use the variational distortion, namely min q (cid:48) ∈Q (cid:48)
Revised: For our compression we use the variational distortion, namely min q (cid:48) ∈Q (cid:48)

Original: E p ( Z,X ) [ − log q (cid:48)
Revised: E p ( Z,X ) [ − log q (cid:48)

Original: ( M ( X ) | Z )] .
Revised: ( M ( X ) | Z )] .

Original: Where M ( X ) is a representative of each equivalence class, which we select to be M : x (cid:55)→ (cid:107) x (cid:107) 2 · [ − 0 .
Revised: Where M ( X ) is a representative of each equivalence class, which we select to be M : x (cid:55)→ (cid:107) x (cid:107) 2 · [ − 0 .

Original: 7071 , − 0 .
Revised: 7071 , − 0 .

Original: T , i.e., the point with the same radius but at 225 degrees.
Revised: T , i.e., the point with the same radius but at 225 degrees.

Original: Essentially we are trying to reconstruct a representative of the equivalence clas of X using the same MLP Q (cid:48) .
Revised: Essentially we are trying to reconstruct a representative of the equivalence clas of X using the same MLP Q (cid:48) .

Original: F. MNIST E XPERIMENTS
Revised: F. MNIST E XPERIMENTS

Original: For our MNIST (LeCun et al., 1998) experiments we compare again our VIC (as described in
Revised: For our MNIST (LeCun et al., 1998) experiments we compare again our VIC (as described in

Original: Algorithm 1) against a standard VC criterion (same as in Algorithm 1 but the direct distortion is evaluated at ˜ x instead of x ).
Revised: Algorithm 1) against a standard VC criterion (same as in Algorithm 1 but the direct distortion is evaluated at ˜ x instead of x ).

Original: Each image ﬁrst passes through a ResNet18 (He et al., 2016) encoder, which maps it to adimensional representation Z ∈ R 128 .
Revised: Each image ﬁrst passes through a ResNet18 (He et al., 2016) encoder, which maps it to adimensional representation Z ∈ R 128 .

Original: We then pass Z through an entropy bottleneck with a scaled hyperprior entropy model from (Ballé et al., 2018) which gives us the reconstruction ˆ Z .
Revised: We then pass Z through an entropy bottleneck with a scaled hyperprior entropy model from (Ballé et al., 2018) which gives us the reconstruction ˆ Z .

Original: The reconstructed ˆ Z is ﬁnally passed through a 5-layer transposed CNN decoder, which reconstructs the augmented input (in the standard case) or the non-augmented input (in the invariant case).
Revised: The reconstructed ˆ Z is ﬁnally passed through a 5-layer transposed CNN decoder, which reconstructs the augmented input (in the standard case) or the non-augmented input (in the invariant case).

Original: Once the compressor is trained we freeze it, apply it to the dataset and train a new ResNet18 to classify the digits using the reconstructions.
Revised: Once the compressor is trained we freeze it, apply it to the dataset and train a new ResNet18 to classify the digits using the reconstructions.

Original: This thus simulated how well you could perform downstream tasks, by considering one possible task, namely, classifying the digits.
Revised: This thus simulated how well you could perform downstream tasks, by considering one possible task, namely, classifying the digits.

Original: This classiﬁer is trained using
Revised: This classiﬁer is trained using

Original: SGD for 100 epochs, an initial learning rate 0.1, and a scheduler that decreases the learning rate every 20 epochs.
Revised: SGD for 100 epochs, an initial learning rate 0.1, and a scheduler that decreases the learning rate every 20 epochs.

Original: To obtain RD curves we sweep over λ = 0 .
Revised: To obtain RD curves we sweep over λ = 0 .

Original: We consider three settings of data augmentations.
Revised: We consider three settings of data augmentations.

Original: Highly augmented test and train
Revised: Highly augmented test and train

Original: First we consider a highly augmented MNIST datasets, which is augmented both at train and test time.
Revised: First we consider a highly augmented MNIST datasets, which is augmented both at train and test time.

Original: Speciﬁcally, we apply random rotations sampled from [ − 45 , 45] degrees, random translations between [0 , 14 ] percentage of pixels, random shearing between [0 , 25] degrees, and random scaling by a factor in [0 . 6 , 1 . 4] .
Revised: Speciﬁcally, we apply random rotations sampled from [ − 45 , 45] degrees, random translations between [0 , 14 ] percentage of pixels, random shearing between [0 , 25] degrees, and random scaling by a factor in [0 . 6 , 1 . 4] .

Original: Mildly augmented test and train We consider more mild data augmentations applied both at train and test time.
Revised: Mildly augmented test and train We consider more mild data augmentations applied both at train and test time.

Original: Speciﬁcally, we apply random rotations sampled from [ − 30 , 30] degrees, random translations between [0 , 110 ] percentage of pixels, random shearing between [0 , 10] degrees, and random scaling by a factor in [0 . 8 , 1 . 2] .
Revised: Speciﬁcally, we apply random rotations sampled from [ − 30 , 30] degrees, random translations between [0 , 110 ] percentage of pixels, random shearing between [0 , 10] degrees, and random scaling by a factor in [0 . 8 , 1 . 2] .

Original: Mildly augmented train Finally we consider the same mildly augmented data described above but only applied at training time
Revised: Mildly augmented train Finally we consider the same mildly augmented data described above but only applied at training time

Original: F. I MAGE N ET E XPERIMENTS
Revised: F. I MAGE N ET E XPERIMENTS

Original: For ImageNet (Deng et al., 2009) experiments, we downloaded a pretrained SimCLR model with a
Revised: For ImageNet (Deng et al., 2009) experiments, we downloaded a pretrained SimCLR model with a

Original: ResNet50 architecture, with a 2048 representations Z = R 2048 .
Revised: ResNet50 architecture, with a 2048 representations Z = R 2048 .

Original: We used this SimCLR to encode a 256 × 256 ImagNet, and zipped the resulting features to get the average bit-rate.
Revised: We used this SimCLR to encode a 256 × 256 ImagNet, and zipped the resulting features to get the average bit-rate.

Original: For our preliminary result on BINCE, we start with the pretrained SimCLR, and add an entropy bottleneck with Ballé et al.’s
Revised: For our preliminary result on BINCE, we start with the pretrained SimCLR, and add an entropy bottleneck with Ballé et al.’s

Original: (2018) factorized prior model.
Revised: (2018) factorized prior model.

Original: We then ﬁnetune both losses in an end-toend fashion for 3 epochs with Adam optimizer and a learning rate of 1e-6.
Revised: We then ﬁnetune both losses in an end-toend fashion for 3 epochs with Adam optimizer and a learning rate of 1e-6.

Original: During ﬁnetuning we also train a single layer (1024 hidden neurons) MLP to classify ImageNet labels from the reconstrcuted representations.
Revised: During ﬁnetuning we also train a single layer (1024 hidden neurons) MLP to classify ImageNet labels from the reconstrcuted representations.

Original: G A DDITIONAL R ESULTS G. B ANANA G.1. D IFFERENT I NVARIANCES
Revised: G A DDITIONAL R ESULTS G. B ANANA G.1. D IFFERENT I NVARIANCES

Original: Fig.
Revised: Fig.

Original: 1. compares a classical compressor to our invariant distortion in the case of rotation invariant tasks.
Revised: 2 compares a classical compressor to our invariant distortion in the case of rotation invariant tasks.

Original: The standard compressor achieves a rate 5 .
Revised: The standard compressor achieves a rate 5 .

Original: 42 ± 0 . 

Revised: 42 ± 0 .

Original: bits for an invariance distortion of
Revised: bits for an invariance distortion of

Original: D ∼ [ X, Z ] = 7 . 04 e-
Revised: D ∼ [ X, Z ] = 7 . 04 e-

Original: 2 ± 0 .
Revised: 2 ± 0 .

Original: 19 e- 2 .
Revised: 19 e- 2 .

Original: While our compressor achieves a rate 2 .
Revised: While our compressor achieves a rate 2 .

Original: 54 ± 0 . 00 bits for an invariance distortion of D ∼ [ X, Z ] = 5 .
Revised: 54 ± 0 . 00 bits for an invariance distortion of D ∼ [ X, Z ] = 5 .

Original: e- 2 ± 0 .
Revised: e- 2 ± 0 .

Original: 08 e-
Revised: 08 e-

Original: 2 .Here
Revised: 2 .Here

Original: we show the same experiment for other invariances.
Revised: we show the same experiment for other invariances.

Original: Fig.
Revised: Fig.

Original: 4 considers the case where downstream tasks are invariant to translations on the x axis.
Revised: 4 considers the case where downstream tasks are invariant to translations on the x axis.

Original: The maximal invariant used during training is chosen to be M : x (cid:55)→ [0 , x 2 ] T .
Revised: The maximal invariant used during training is chosen to be M : x (cid:55)→ [0 , x 2 ] T .

Original: We only ran a single run for visualization.
Revised: We only ran a single run for visualization.

Original: We see that our model can essentially perform as well on all downstream tasks (similar invariant distortion) for only 60% of the bit-rate.
Revised: We see that our model can essentially perform as well on all downstream tasks (similar invariant distortion) for only 60% of the bit-rate.

Original: Unsurprisingly we see that the codebook is
Revised: Unsurprisingly we see that the codebook is

Original: [Figure]
Revised: [Figure]

Original: in shape of horizontal stripes as these can cover the entire distribution with a few codes (small bit rate) while incurring a small invariance distortion (which only depends on the y value).
Revised: in shape of horizontal stripes as these can cover the entire distribution with a few codes (small bit rate) while incurring a small invariance distortion (which only depends on the y value).

Original: [Figure]
Revised: [Figure]

Original: G.1. R ATE -D ISTORTION C URVES
Revised: G.1. R ATE -D ISTORTION C URVES

Original: By sweeping over the hyperparameter β we can can move along the rate distortion curve, thus balancing the respective rate and distortion term.
Revised: By sweeping over the hyperparameter β we can can move along the rate distortion curve, thus balancing the respective rate and distortion term.

Original: Speciﬁcally, by increasing β in Eq.
Revised: Speciﬁcally, by increasing β in Eq.

Original: (5) we give more importance to the distortion term and thus will have to increase the size of the codebook.
Revised: (5) we give more importance to the distortion term and thus will have to increase the size of the codebook.

Original: This is exactly what can be seen in Fig.
Revised: This is exactly what can be seen in Fig.

Original: To get a better sense of the gains in rates that can be achieved we show in Fig.
Revised: To get a better sense of the gains in rates that can be achieved we show in Fig.

Original: 7 the (average) rate-distortion curve for our invariant neural compressor and the non-invariant compressor.
Revised: 7 the (average) rate-distortion curve for our invariant neural compressor and the non-invariant compressor.

Original: We see that our invariant compressor signiﬁcantly outperforms the non invariant compressor.
Revised: We see that our invariant compressor signiﬁcantly outperforms the non invariant compressor.

Original: Indeed, the area-under-the RD curve for the invariant model is 35 .
Revised: Indeed, the area-under-the RD curve for the invariant model is 35 .

Original: 8 ± 4 .
Revised: 8 ± 4 .

Original: 2 bits while it is 48 .
Revised: 2 bits while it is 48 .

Original: 1 ± 0 .
Revised: 1 ± 0 .

Original: 3 bits for the the non invariant case, this means that, if you are interested in rotation invariant tasks, you can increase your compression rates by an average of 12 .
Revised: 3 bits for the the non invariant case, this means that, if you are interested in rotation invariant tasks, you can increase your compression rates by an average of 12 .

Original: 3 bits without hindering your downstream performance.
Revised: 3 bits without hindering your downstream performance.

Original: Note that our Thm. 1 shows that the rate and the distortion have a linear relationship rate ( δ )
Revised: Note that our Thm. 1 shows that the rate and the distortion have a linear relationship rate ( δ )

Original: = ( const ) − δ , yet the rate distortion curve in Fig.
Revised: = ( const ) − δ , yet the rate distortion curve in Fig.

Original: 7 shows an approximately logarithmic relationship (notice the log-scale in the x axis).
Revised: 7 shows an approximately logarithmic relationship (notice the log-scale in the x axis).

Original: This happens because we are estimating the invariant distortion using the mean squared error (MSE) or conditional variance, and the conditional entropy is proportional to the logarithm of the conditional variance H[ M ( X ) | Z ] ∝ log(Var[ M ( X ) | Z ]) + ( const ) .
Revised: This happens because we are estimating the invariant distortion using the mean squared error (MSE) or conditional variance, and the conditional entropy is proportional to the logarithm of the conditional variance H[ M ( X ) | Z ] ∝ log(Var[ M ( X ) | Z ]) + ( const ) .

Original: [Figure]
Revised: [Figure]

Original: G. MNIST
Revised: G. MNIST

Original: Highly Augmented Test and Train.
Revised: Highly Augmented Test and Train.

Original: As discussed in the main paper, we are interested in knowing the gains in bit-rate that can be achieved by our practical loss VIC (compared to standard VC) in the case where you have access to some augmentations w.r.t.
Revised: As discussed in the main paper, we are interested in knowing the gains in bit-rate that can be achieved by our practical loss VIC (compared to standard VC) in the case where you have access to some augmentations w.r.t.

Original: which your tasks of interest are invariant to.
Revised: which your tasks of interest are invariant to.

Original: To simulate this we apply large augmentations at training and test time.
Revised: To simulate this we apply large augmentations at training and test time.

Original: The results are summarized in Fig.
Revised: The results are summarized in Fig.

Original: 8b.
Revised: 8b.

Original: Mildly Augmented Test and Train.
Revised: Mildly Augmented Test and Train.

Original: We then consider the same experiment but with less data augmentation.
Revised: We then consider the same experiment but with less data augmentation.

Original: The results are summarized in Fig. 9b.
Revised: The results are summarized in Fig. 9b.

Original: Unsurprisingly, the gains in bit rate decrease compared to the highly augmented case.
Revised: Unsurprisingly, the gains in bit rate decrease compared to the highly augmented case.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: Mildly Augmented Test and Train.
Revised: Mildly Augmented Test and Train.

Original: Finally, we consider the case where the augmentations are not known and so we apply augmentations at training time w.r.t.
Revised: Finally, we consider the case where the augmentations are not known and so we apply augmentations at training time w.r.t.

Original: which you might not be invariant to.
Revised: which you might not be invariant to.

Original: We thus do not apply the same augmentations at test time, and can only hope that MNIST has similar inherent invariances.
Revised: We thus do not apply the same augmentations at test time, and can only hope that MNIST has similar inherent invariances.

Original: The results are summarized in Fig. 10b.
Revised: The results are summarized in Fig. 10b.

Original: In this case the gains in bit-rate is much smaller but still signiﬁcant.
Revised: In this case the gains in bit-rate is much smaller but still signiﬁcant.

Original: Importantly we see that the rate for VC is highly dependent on whether the images are augmented at test time or not, while the it is not for VIC.
Revised: Importantly we see that the rate for VC is highly dependent on whether the images are augmented at test time or not, while the it is not for VIC.

Original: This makes sense as
Revised: This makes sense as

Original: VIC essentially learned to compress the maximal invariant M ( X ) which is the same regardless as to whether the input is augmented or not.
Revised: VIC essentially learned to compress the maximal invariant M ( X ) which is the same regardless as to whether the input is augmented or not.

Original: [Figure]
Revised: [Figure]