Drop-Bottleneck : L EARNING D ISCRETE C OMPRESSED R EPRESENTATION FOR N OISE -R OBUST E XPLORATION
A BSTRACT
We propose a novel information bottleneck (IB) method named Drop-Bottleneck , which discretely drops features that are irrelevant to the target variable.
DropBottleneck not only enjoys a simple and tractable compression objective but also additionally provides a deterministic compressed representation of the input variable, which is useful for inference tasks that require consistent representation.
Moreover, it can jointly learn a feature extractor and select features considering each feature dimension’s relevance to the target task, which is unattainable by most neural network-based IB methods.
We propose an exploration method based on Drop-Bottleneck for reinforcement learning tasks.
In a multitude of noisy and reward sparse maze navigation tasks in VizDoom (Kempka et al., 2016) and DMLab (Beattie et al., 2016), our exploration method achieves state-of-the-art performance.
As a new IB framework, we demonstrate that Drop-Bottleneck outperforms Variational Information Bottleneck (VIB) (Alemi et al., 2017) in multiple aspects including adversarial robustness and dimensionality reduction.
I NTRODUCTION
Data with noise or task-irrelevant information easily harm the training of a model; for instance, the noisy-TV problem (Burda et al., 2019a) is one of well-known such phenomena in reinforcement learning.
If observations from the environment are modiﬁed to contain a TV screen, which changes its channel randomly based on the agent’s actions, the performance of curiosity-based exploration methods dramatically degrades (Burda et al., 2019a;b; Kim et al., 2019;
Savinov et al., 2019).
The information bottleneck (IB) theory (Tishby et al., 2000; Tishby & Zaslavsky, 2015) provides a framework for dealing with such task-irrelevant information, and has been actively adopted to exploration in reinforcement learning (Kim et al., 2019; Igl et al., 2019).
For an input variable X and a target variable Y , the IB theory introduces another variable Z , which is a compressed representation of X .
The IB objective trains Z to contain less information about X but more information about Y as possible, where the two are quantiﬁed by mutual information terms of I ( Z ; X ) and I ( Z ; Y ) , respectively.
IB methods such as Variational Information Bottleneck (VIB) (Alemi et al., 2017; Chalk et al., 2016) and Information Dropout (Achille & Soatto, 2018) show that the compression of the input variable X can be done by neural networks.
In this work, we propose a novel information bottleneck method named Drop-Bottleneck that compresses the input variable by discretely dropping a subset of its input features that are irrelevant to the target variable.
Drop-Bottleneck provides some nice properties as follows:
• The compression term of Drop-Bottleneck’s objective is simple and is optimized as a tractable solution.
• Drop-Bottleneck provides a deterministic compressed representation that still maintains majority of the learned indistinguishability i.e . compression.
It is useful for inference tasks that require the input representation to be consistent and stable.
• Drop-Bottleneck jointly trains a feature extractor and performs feature selection, as it learns the feature-wise drop probability taking into account each feature dimension’s relevance to the target task.
Hence, unlike the compression provided by most neural network-based IB methods, our deterministic representation reduces the feature dimensionality, which makes the following inference better efﬁcient with less amount of data.
• Compared to VIB, both of Drop-Bottleneck’s original (stochastic) and deterministic compressed representations can greatly improve the robustness to adversarial examples.
Based on the newly proposed Drop-Bottleneck, we design an exploration method that is robust against noisy observations in very sparse reward environments for reinforcement learning.
Our exploration maintains an episodic memory and generates intrinsic rewards based on the predictability of new observations from the compressed representations of the ones in the memory.
As a result, our method achieves state-of-the-art performance on multiple environments of VizDoom (Kempka et al., 2016) and DMLab (Beattie et al., 2016).
We also show that combining our exploration method with VIB instead of Drop-Bottleneck degrades the performance by meaningful margins.
Additionally, we empirically compare with VIB to show Drop-Bottleneck’s superior robustness to adversarial examples and ability to reduce feature dimensionality for inference with ImageNet dataset (Russakovsky et al., 2015).
We also demonstrate that Drop-Bottleneck’s deterministic representation can be a reasonable replacement for its original representation in terms of the learned indistinguishability, with Occluded CIFAR dataset (Achille & Soatto, 2018).
R ELATED W ORK 2. I NFORMATION B OTTLENECK M ETHODS
There have been a number of IB methods that are approximations or special forms of the original IB objective.
Variational Information Bottleneck (VIB) (Alemi et al., 2017) approximates the original IB objective by establishing variational bounds on the compression and prediction terms.
Chalk et al.
(2016) propose the same variational bound on the IB objective in the context of sparse coding tasks.
Conditional Entropy Bottleneck (CEB) and Variational Conditional Entropy Bottleneck (VCEB) (Fischer, 2020; Fischer & Alemi, 2020) use an alternative form of the original IB objective derived under the Minimum Necessary Information (MNI) criterion to preserve only a necessary amount of information.
The IB theory (Tishby et al., 2000) has been used for various problems that require restriction of information or dealing with task-irrelevant information.
Information Dropout (Achille & Soatto, 2018) relates the IB principle to multiple practices in deep learning, including Dropout, disentanglement and variational autoencoding.
Moyer et al.
(2018) obtain representations invariant to speciﬁc factors under the variational autoencoder (VAE) (Kingma & Welling, 2013) and VIB frameworks.
Amjad & Geiger (2019) discuss the use of IB theory for classiﬁcation tasks from a theoretical point of view.
Dai et al.
employ IB theory for compressing neural networks by pruning neurons in networks.
Schulz et al.
(2020) propose an attribution method that determines each input feature’s importance by enforcing compression of the input variable via the IB framework.
Similar to our goal, some previous research has proposed variants of the original IB objective.
Deterministic information bottleneck (DIB) (Strouse & Schwab, 2017) replaces the compression term with an entropy term and solves the new objective using a deterministic encoder.
Nonlinear information bottleneck (NIB) (Kolchinsky et al., 2019) modiﬁes the IB objective by squaring the compression term and uses a non-parametric upper bound on the compression term.
While DIB is always in the deterministic form, we can ﬂexibly choose the stochastic one for training and the deterministic one for test.
Compared to NIB, which is more computationally demanding than VIB due to its non-parametric upper bound, our method is faster.
2. R EINFORCEMENT L EARNING WITH I NFORMATION B OTTLENECK M ETHODS
The IB theory has been applied to several reinforcement learning (RL) tasks.
Variational discriminator bottleneck (Peng et al., 2019) regulates the discriminator’s accuracy using the IB objective to improve adversarial training, and use it for imitation learning.
Information Bottleneck Actor Critic (Igl et al., 2019) employs VIB to make the features generalize better and encourage the compression of states as input to the actor-critic algorithm.
Curiosity-Bottleneck (Kim et al., 2019) employs the VIB framework to train a compressor that compresses the representation of states, which is still informative about the value function, and uses the compressiveness as exploration signals.
InfoBot (Goyal et al., 2019) proposes a conditional version of VIB to improve the transferability of a goal-conditioned policy by minimizing the policy’s dependence on the goal.
Variational bandwidth bottleneck (Goyal et al., 2020) uses a modiﬁed, conditional version of VIB, and solves RL tasks with privileged inputs ( i.e . valuable information that comes with a cost).
Our exploration method differs from these methods in two aspects.
First, we propose a new information bottleneck method that is not limited to exploration in RL but generally applicable to the problems for which the IB theory is used.
Second, our method generates exploration signals based on the noise-robust predictability i.e .
the predictability between noise-robust representations of observations.
D ROP -B OTTLENECK 3. P RELIMINARIES OF I NFORMATION B OTTLENECK
Given an input random variable X , the information bottleneck (IB) framework (Tishby et al., 2000;
Tishby & Zaslavsky, 2015) formalizes a problem of obtaining X ’s compressed representation Z , which still and only preserves information relevant to the target variable Y .
Its objective function is
[Equation]
where β is a Lagrangian multiplier.
The ﬁrst and second terms are prediction and compression terms, respectively.
The prediction term I ( Z ; Y ) encourages Z to preserve task-relevant information while the compression term I ( Z ; X ) compresses the input information as much as possible.
As reviewed in the previous section, there have been several IB methods (Alemi et al., 2017; Chalk et al., 2016; Achille & Soatto, 2018; Strouse & Schwab, 2017; Kolchinsky et al., 2019), among which the ones derived using variational inference such as Variational Information Bottleneck (VIB) (Alemi et al., 2017) have become dominant due to its applicability to general problems.
3. D ROP -B OTTLENECK
We propose a novel information bottleneck method called Drop-Bottleneck (DB), where the input information is compressed by discretely dropping a subset of input features.
Thus, its compression objective is simple and easy to optimize.
Moreover, its representation is easily convertible to a deterministic version for inference tasks (Section 3.3), and it allows joint training with a feature extractor (Section 3.4).
While discrete dropping of features has been explored by prior works including Dropout (Srivastava et al., 2014), DB differs in that its goal is to assign different drop probabilities to feature variables based on their relevance to the target variable.
For an input variable X
=
[ X 1 , . .
X d ] and a drop probability p
=
[ p 1 , . .
. , p d ] ∈ [0 , 1] d , we deﬁne its compressed representation as Z = C p ( X ) =
[ c ( X 1 , p 1 ) , . . . ,
c ( X d , p d )] such that
[Equation]
, where
[Equation]
for i = 1 , . . .
, d .
That is, the compression procedure drops the i -th input feature ( i.e . replaced by zero) with probability p i .
Since the drop probability is to be learned, we use a scaling factor b to keep the scale of Z constant.
We use a single scaling factor for all feature dimensions in order to preserve the relative scales between the features.
With the deﬁnition in Equation (2), we derive the compression term of DB to minimize as
I ( Z ; X ) =
[Equation]
[Equation]
[Equation]
i
[Equation]
) (5) using that Z i ⊥⊥ X 1 , . .
, X i − 1 , X i +1 , . . .
, X d | Z 1 , . . .
, Z i − 1 , X i and Z
i ⊥⊥ Z 1 , . .
, Z i − 1 |
X i .
Note that
ˆ
I
( Z ; X ) − I ( Z ; X ) =
(cid:16)(cid:80) di =1 H ( Z i ) (cid:17) − H
( Z 1 , . . .
, Z d ) =
TC ( Z ) where TC ( Z ) is the total correlation of Z and H ( · ) denotes the entropy, and ˆ I ( Z ; X )
=
I ( Z ; X ) if X 1 , . . .
, X d are independent.
To provide a rough view on the gap, due to the joint optimization with the compression term ˆ I ( Z ; X ) and the prediction term I
( Z ; Y ) , Z becomes likely to preserve less redundant and less correlated features, and TC ( Z ) could decrease as the optimization progresses.
Finally, DB’s new compression term, ˆ I ( Z ; X ) , is computed as
[Equation]
From Equation (7) to Equation (8), we use the two ideas: (i) H
( X i | Z i = 0)
=
H ( X i ) because
Z i = 0 means it contains no information about X i , and
(ii) H ( X i | Z i = bX i )
=
because Z i = bX i means Z i preserves the feature ( i.e .
Z i fully identiﬁes X i ) and thus their conditional entropy becomes zero.
Importantly, DB’s compression term is computed as the simple tractable expression in Equation (8).
As the goal of the compression term is to penalize I ( Z ; X ) not H ( X ) , the drop probability p is the only parameter optimized with our compression term.
Thus, each H ( X i ) can be computed with any entropy estimation method such as the binning-based estimation, which involves quantization for continuous X i , since the computation has no need to be differentiable.
However, one issue of Equation (8) is that Z is not differentiable with respect to p as Bernoulli distributions are not differentiable.
We thus use the Concrete relaxation (Maddison et al., 2017;
Jang et al., 2016) of the Bernoulli distribution to update p via gradients from Z :
[Equation]
where u ∼ Uniform (0 , 1) and λ is a temperature for the Concrete distribution.
Intuitively, p is trained to assign a high drop probability to the feature that is irrelevant to or redundant for predicting the target variable Y .
3. D ETERMINISTIC C OMPRESSED R EPRESENTATION
With Drop-Bottleneck, one can simply obtain the deterministic version of the compressed representation as
[Equation]
)] for
[Equation]
¯ b is deﬁned similarly to b with a minor exception that the scaling is done based on the actual, deterministic number of the dropped features.
The deterministic compressed representation ¯ Z is useful for inference tasks that require stability or consistency of the representation as well as reducing the feature dimensionality for inference, as we demonstrate in Section 5.4.
3. T RAINING WITH D ROP -B OTTLENECK
We present how Drop-Bottleneck (DB) is trained with the full IB objective allowing joint training with a feature extractor.
Since DB proposes only a new compression term, any existing method for maximizing the prediction term I ( Z ; Y ) can be adopted.
We below discuss an example with Deep Infomax (Hjelm et al., 2019) since our exploration method uses this framework (Section 4).
Deep Infomax, instead of I ( Z ; Y ) , maximizes its Jensen-Shannon mutual information estimator
[Equation]
where T ψ is a discriminator network with parameter ψ and ζ ( · ) is the softplus function.
Finally, the IB objective with Drop-Bottleneck becomes
[Equation]
which can be optimized via gradient descent.
To make p more freely trainable, we suggest simple element-wise parameterization of p as p i
=
σ ( p (cid:48) i ) and initializing p (cid:48)
i ∼ Uniform ( a, b ) .
We obtain the input variable X from a feature extractor, whose parameters are trained via the prediction term, jointly with p and ψ .
In next section, we will discuss its application to hard exploration problems for reinforcement learning.
R OBUST E XPLORATION WITH D ROP -B OTTLENECK
Based on DB, we propose an exploration method that is robust against highly noisy observations in a very sparse reward environment for reinforcement learning tasks.
We ﬁrst deﬁne a parametric feature extractor f φ that maps a state to X .
For transitions ( S, A, S (cid:48) ) where S, A, and S (cid:48) are current states, actions and next states, respectively, we use the DB objective with
[Equation]
For every transition ( S, A, S (cid:48) ) , the compression term I ( Z ; X ) =
I ( C p ( f φ ( S (cid:48) )); f φ ( S (cid:48) )) encourages
C p to drop unnecessary features of the next state embedding f φ ( S (cid:48) ) as possible.
The prediction term I ( Z ; Y ) =
I ( C p ( f φ ( S (cid:48) )); C p ( f φ ( S ))) makes the compressed representations of the current and next state embeddings, C p ( f φ ( S )) and C p ( f φ ( S (cid:48) )) , informative about each other.
Applying Equation (13) to Equation (12), the Drop-Bottleneck objective for exploration becomes
[Equation]
While f φ , p , and T ψ are being trained online, we use them for the agent’s exploration with the help of episodic memory inspired by Savinov et al.
Starting from an empty episodic memory M , we add the learned feature of the observation at each step.
For example, at time step t , the episodic memory is M = { ¯ C p ( f φ ( s 1 )) , ¯ C p ( f φ ( s 2 )) , . . .
, ¯ C p ( f φ ( s t − 1 )) } where s 1 , . . .
, s t − 1 are the earlier observations from that episode.
We then quantify the degree of novelty of a new observation as an intrinsic reward.
Speciﬁcally, the intrinsic reward for s t is computed utilizing the Deep Infomax discriminator T ψ , which is trained to output a large value for joint (or likely) input and a small value for marginal (or arbitrary) input:
[Equation]
where g ( s t , s j ) and g ( s j , s t ) denote the unlikeliness of s t being the next and the previous state of s j , respectively.
Thus, intuitively, for s t that is close to a region covered by the earlier observations in the state space, r i M,t ( s t ) becomes low, and vice versa.
It results in a solid exploration method capable of handling noisy environments with very sparse rewards.
For computing the intrinsic reward, we use the DB’s deterministic compressed representation of states to provide stable exploration signals to the policy optimization.
E XPERIMENTS
We carry out three types of experiments to evaluate Drop-Bottleneck (DB) in multiple aspects.
First, we apply DB exploration to multiple VizDoom (Kempka et al., 2016) and DMLab (Beattie et al., 2016) environments with three hard noise settings, where we compare with state-of-the-art methods as well as its VIB variant.
Second, we empirically show that DB is superior to VIB for adversarial robustness and feature dimensionality reduction in ImageNet classiﬁcation (Russakovsky et al., 2015), and we juxtapose DB with VCEB, which employs a different form of the IB object, for the same adversarial robustness test, in Appendix A. Finally, in Appendix B, we make another comparison with VIB in terms of removal of task-irrelevant information and the validity of the deterministic compressed representation, where Appendix C provides the visualization of the task-irrelevant information removal on the same task.
5. E XPERIMENTAL S ETUP FOR E XPLORATION T ASKS
For the exploration tasks on VizDoom (Kempka et al., 2016) and DMLab (Beattie et al., 2016), we use the proximal policy optimization (PPO) algorithm (Schulman et al., 2017) as the base RL as baseline exploration methods.
ICM learns a dynamics model of the environment and uses the prediction errors as exploration signals for the agent.
EC and ECO are curiosity methods that use episodic memory to produce novelty bonuses according to the reachability of new observations, and show the state-of-the-art exploration performance on VizDoom and DMLab navigation tasks.
In summary, we compare with four baseline methods: PPO, PPO + ICM, PPO + EC, and PPO + ECO.
Additionally, we report the performance of our method combined with VIB instead of DB.
We conduct experiments with three versions of noisy-TV settings named “Image Action”, “Noise” and “Noise Action”, as proposed in Savinov et al.
We present more details of noise and their observation examples in Appendix E.1.
Following Savinov et al.
(2019), we resize observations as 160 × 120 only for the episodic curiosity module while as 84 × 84 for the PPO policy and exploration methods.
We use the ofﬁcial source code 1 of Savinov et al.
(2019) to implement and conﬁgure the baselines (ICM, EC and ECO)
and the three noise settings.
For the feature extractor f φ , we use the same CNN with the policy network of PPO from Mnih et al.
The only modiﬁcation is to use d = 128 i.e . 128 -dimensional features instead of 512 to make features lightweight enough to be stored in the episodic memory.
The Deep Infomax discriminator T ψ consists of three FC hidden layers with 64 , 32 , 16 units each, followed by a ﬁnal
FC layer for prediction.
We initialize the drop probability p with p i =
σ ( p (cid:48) i ) and p (cid:48)
i ∼ Uniform ( a, b ) where a = − 2 , b = 1 .
We collect samples and update p , T ψ , f φ with Equation (14) every 10 . 8 K and 21 .
6 K time steps in VizDoom and DMLab, respectively, with a batch size of 512 .
We duplicate the compressed representation 50 times with differently sampled drop masks, which help better training of the feature extractor, the drop probability and the discriminator by forming diverse subsets of features.
Please refer to Appendix E for more details of our experimental setup.
5. E XPLORATION IN N OISY S TATIC M AZE E NVIRONMENTS
VizDoom (Kempka et al., 2016) provides a static 3D maze environment.
We experiment on the MyWayHome task with nine different settings by combining three reward conditions (“Dense”, “Sparse” and “Very Sparse”) in Pathak et al.
(2017) and three noise settings in the previous section.
In the “Dense”, “Sparse” and “Very Sparse” cases, the agent is randomly spawned in a near, medium and very distant room, respectively.
Thus, “Dense” is a relatively easy task for the agent to reach the goal even with a short random walk, while “Sparse” and “Very Sparse” require the agent to perform a series of directed actions, which make the goal-oriented navigation difﬁcult.
Table 1 and Figure 1 compare the DB exploration with three baselines, PPO, PPO + ICM, and PPO + ECO on the VizDoom tasks, in terms of the ﬁnal performance and how quickly they learn.
The results suggest that even in the static maze environments, the three noise settings can degrade the performance of the exploration by large margins.
On the other hand, our method with DB shows robustness to such noise or task-irrelevant information, and outperforms the baselines in all the nine challenging tasks, whereas our method combined with VIB does not exhibit competitive results.
5. E XPLORATION IN N OISY AND R ANDOMLY G ENERATED M AZE E NVIRONMENTS
As a more challenging exploration task, we employ DMLab (Savinov et al., 2019), which are general and randomly generated maze environments where at the beginning of every episode, each maze is procedurally generated with its random goal location.
Thanks to the random map generator, each method is evaluated on test mazes that are independent of training mazes.
As done in Savinov et al.
(2019), we test with six settings according to two reward conditions (“Sparse” and “Very Sparse”) and the three noise settings.
In the “Sparse” scenario, the agent is (re-)spawned at a random location when each episode begins or every time it reaches the goal i.e .
the sparse reward; the agent should reach the goal as many times as possible within the ﬁxed episode length.
The “Very Sparse” is its harder version: the agent does not get (re-)spawned near or in the same room with the goal.
Table 1 compares between different exploration methods on the DMLab tasks.
The results demonstrate that our DB exploration method achieves state-of-the-art performance with signiﬁcant margins from the baselines on all the 6 tasks, and performs better than our method equipped with VIB as well.
The plots too suggest that our method provides stable exploration signals to the agent under different environmental and noise settings.
As mentioned in Section 5.1, our method can achieve better per-
[Table]
[Figure]
PPO + Grid Oracle
[Figure] [Figure]
formance even using much lower resolution observations of 84 × 84 than 160 × 120 of EC and ECO.
Also, excluding the policy network, our method maintains 0 .
5 M parameters, which is signiﬁcantly smaller compared to ECO with 13 M parameters.
Please refer to Appendix D for an ablation study,
Figure 2 shows evolution examples of the drop probability distribution over training time steps.
It overviews the role of drop probability p in DB.
As the joint training of the feature extractor f φ with p progresses, p gets separated into high- and low-value groups, where the former drops task-irrelevant or redundant features and the latter preserves task-relevant features.
This suggests that in the DMLab environments, the DB objective of Equation (14) successfully encourages dropping the features unrelated to transition between observations and also the deterministic compressed representation becomes stable as the training progresses.
[Table]
[Figure]
DB (determ.) quickly drops many feature dimensions with increased β , while VIB retains them at 1024 regardless of β .
[-∞ 99 .
-]
5. C OMPARISON WITH VIB: A DVERSARIAL R OBUSTNESS & D IMENSION R EDUCTION
We experiment with image classiﬁcation on ImageNet (Russakovsky et al., 2015) to compare DropBottleneck (DB) with Variational Information Bottleneck (VIB)
(Alemi et al., 2017), the most widely-used IB framework, regarding the robustness to adversarial attacks and the reduction of feature dimensionality.
We follow the experimental setup from Alemi et al. (2017) with some exceptions. We use β 1 = 0 .
9 and no learning rate decay for DB’s Adam optimizer (Kingma & Ba, 2015).
For prediction, we use one Monte Carlo sample of each stochastic representation.
Additionally, we provide a similar comparison with the mutual information-based feature selection method.
Robustness to adversarial attacks .
Following Alemi et al.
, we employ the targeted (cid:96) 2 and (cid:96) ∞ adversarial attacks from Carlini & Wagner (2017).
For each method, we determine the ﬁrst 200 validation images on ImageNet that are classiﬁed correctly, and apply the attacks to them by selecting uniformly random attack target classes.
Please refer to Appendix E.2 for further details.
Table 2 shows the results.
For the targeted (cid:96) 2 attacks, choosing the value of β from [0 . 003162 , 0 . 1] provides the improved robustness of DB with the maximum at β = 0 .
On the other hand, VIB has no improved robustness in all ranges of β .
For the targeted (cid:96) ∞ attacks, DB can reduce the attack success rate even near to 0% ( e.g . β = 0 .
003162 or 0 . 01 ).
Although VIB decreases the attack success rate to 15 .
5% at β = 0 .
1 , VIB already suffers from the performance degradation at β = 0 .compared to DB (Figure 3), and increasing β accelerates VIB’s degradation even further.
Note that the validation accuracies of both VIB and DB are close to zero at β = 0 . 3162 .
Dimensionality reduction .
Figure 3 compares the accuracy of DB and VIB by varying β on the
ImageNet validation set.
Overall, their accuracies develop similarly with respect to β ; while VIB is slightly better in the lower range of β , DB produces better accuracy in the higher range of β .
Note that DB (determ.)
shows the almost identical accuracy plot with DB .
Importantly, DB (determ.)
still achieves a reasonable validation accuracy ( ≥ 75% ) using only a few feature dimensions ( e.g . 8 ) out of the original 1024 dimensions.
This suggests that DB’s deterministic compressed representation can greatly reduce the feature dimensionality for inference with only a small trade-off with the performance.
It is useful for improving the efﬁciency of the model after the training is complete.
On the other hand, VIB has no such capability.
Finally, as Figure 3 shows, the trade-off between the dimensionality reduction and the performance can be controlled by the value of β .
Comparison with feature selection .
As the deterministic representation of DB, DB (determ.) , provides the dimensionality reduction, we also empirically compare DB with the univariate mutual information-based feature selection method for obtaining the feature space with a reduced dimensionality.
In the experiments, the same features provided to DB and VIB are given as input to the feature selection method as well, and for a more straightforward comparison, we let the feature se- [Table] lection method preserve the same number of features as DB (determ.) .
Refer to Appendix E.3 for further details of the feature selection procedure.
Figure 4 shows the classiﬁcation accuracy of the two methods for the same numbers of features.
The results show that while the mutual informationbased feature selection method could provide a marginal performance beneﬁt when a larger subset of the pre-trained features is preserved, DB is signiﬁcantly better at retaining the accuracy with a small number of feature dimensions.
For instance, DB achieves the accuracy over 71% even with 4 features, but the accuracy of feature selection method drops from ≈ 68% to ≈ 10% when the number of features is < 2 6 .
Also, we make a comparison of the adversarial robustness; Tablesuggests that the features preserved with the feature selection method show almost no robustness to the targeted (cid:96) 2 and (cid:96) ∞ attacks, where every attack success rate is ≥ 97% .
On the other hand, DB (determ.) can reduce the success rate to 20% for the (cid:96) 2 and to 2% for the (cid:96) ∞ attacks with 8 features.
C ONCLUSION
We presented Drop-Bottleneck as a novel information bottleneck method where compression is done by discretely dropping input features, taking into account each input feature’s relevance to the target variable and allowing its joint training with a feature extractor.
We then proposed an exploration method based on Drop-Bottleneck, and it showed state-of-the-art performance on multiple noisy and reward-sparse navigation environments from VizDoom and DMLab.
The results showed the robustness of Drop-Bottleneck’s compressed representation against noise or task-irrelevant information.
With experiments on ImageNet, we also showed that Drop-Bottleneck achieves better adversarial robustness compared to VIB and can reduce the feature dimension for inference.
In the exploration experiments, we directly fed the noisy observations to the policy, which can be one source of performance degradation in noisy environments.
Therefore, applying Drop-Bottleneck to the policy network can improve its generalization further, which will be one interesting future research.
A CKNOWLEDGMENTS
We thank Myeongjang Pyeon, Hyunwoo Kim, Byeongchang Kim and the anonymous reviewers for the helpful comments. This work was supported by Samsung Advanced Institute of Technology, Basic Science Research Program through the National Research Foundation of Korea (NRF) (2020R1A2B5B03095585) and Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No.2019-0-01082, SW StarLab). Jaekyeom Kim was supported by Hyundai Motor Chung Mong-Koo Foundation.
Gunhee Kim is the corresponding author.
A C OMPARISON WITH VCEB: A DVERSARIAL R OBUSTNESS
In this section, we compare Drop-Bottleneck (DB) with Variational Conditional Entropy Bottleneck (VCEB)
(Fischer, 2020; Fischer & Alemi, 2020) on the same ImageNet (Russakovsky et al., 2015) tasks for the adversarial robustness as in Section 5.4.
VCEB variationally approximates the CEB objective, which is deﬁned as
[Equation]
Note that Equation (16) is an alternative form of the original IB objective, Equation (1), as
I ( Z ; X, Y ) =
[Equation]
( Z
[Equation]
Y ) +
[Equation]
( Z ; X | Y )
=
I ( Z ; X ) + I ( Z ; Y | X {+) and I ( Z ; Y | X ) = 
+}
( ∵ Z ⊥⊥ Y | X ) .
As in Section 5.4, we employ the experimental setup from VIB (Alemi et al., 2017) with small modiﬁcations to the hyperparameters for the Adam optimizer (Kingma & Ba, 2015) that β 1 = 0 .
9 and no learning rate decay is used.
Additionally for VCEB, we apply the conﬁgurations suggested by
Fischer & Alemi (2020): 1) at test time, use the mean of the Gaussian encoding instead of sampling from the distribution, and 2) reparameterize γ = exp( − ρ ) and anneal the value of ρ from ρ =to the ﬁnal ρ during training.
For our experiments, the annealing is performed via the ﬁrsttraining steps, where each epoch consists of 6405 steps.
Employing the experimental setup from Alemi et al.
(2017), we adopt the targeted (cid:96) 2 and (cid:96) ∞ adversarial attacks from Carlini & Wagner (2017).
We determine the ﬁrst 200 correctly classiﬁed validation images on ImageNet for each setting and perform the attacks where the attack target classes are chosen randomly and uniformly.
Further details are described in Appendix E.2.
[Figure]
[Table] [Figure] Figure 5 visualizes the classiﬁcation accuracy for each corresponding ﬁnal value of ρ , and the adversarial robustness results are shown in Table 4.
Overall, both
VCEB and DB provide meaningful robustness to the targeted (cid:96) 2 and (cid:96) ∞ attacks.
For the targeted (cid:96) 2 attacks, although VCEB could achieve the higher average perturbation distance over successful attacks, DB and its deterministic form show better robustness compared to VCEB in terms of the attack success rates: 18 .
5% ( DB at β = 0 .
01 ) and 20 .
% ( DB (determ.)
at β = 0 . 01 ) versus 45 . 0% (VCEB at the ﬁnal ρ = 3 . 454 ).
For the targeted (cid:96) ∞ attacks, DB and its deterministic version again achieve the lower attack success rates than VCEB: 1 . 5% and 2 .
% ( DB and DB (determ.)
at β ∈ { 0 . 003162 , 0 . 01 } ))
versus 12 .
5% (VCEB at the ﬁ- nal ρ = 3 . 454 ).
B R EMOVAL OF T ASK - IRRELEVANT I NFORMATION AND V ALIDITY OF D ETERMINISTIC C OMPRESSED R EPRESENTATION
We experiment Drop-Bottleneck (DB) and Variational Information Bottleneck (VIB)
(Alemi et al., 2017) on occluded image classiﬁcation tasks to show the following:
• DB can control the degree of compression ( i.e . degree of removal of task-irrelevant information) in the same way with VIB as the popular IB method.
• DB’s deterministic compressed representation works as a reasonable replacement for its stochastic compressed representation and it maintains the learned indistinguishability better than the attempt of VIB’s deterministic compressed representation.
We employ the Occluded CIFAR dataset using the experimental settings from Achille & Soatto (2018).
The Occluded CIFAR dataset is created by occluding CIFAR-10 (Krizhevsky, 2009) images with MNIST (LeCun et al., 2010) images as shown in Figure 6a, and each image has two labels of
CIFAR and MNIST.
We use a modiﬁed version of All-CNN-32 (Achille & Soatto, 2018) equipped with an IB method (either of DB or VIB) for the feature extractor whose output dimension is d . Each [Table] [Table]
[Table]
[Table]
[Table] [Table] [Table] [Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table] [Table] run consists of two phases. In the ﬁrst phase, we train the feature extractor with a logistic classiﬁer using both the classiﬁcation loss for CIFAR and the compression objective of the IB method.
Fixing the trained feature extractor, we train a logistic classiﬁer for MNIST in the second phase.
We train two different versions of classiﬁers for each of VIB and DB using stochastic or deterministic compressed representation from the feature extractor.
For the deterministic representation of VIB, we use the mode of the output Gaussian distribution.
Figures 6b–6d contain the experimental results with d = 32 , d = 64 , and d = 128 .
In the ﬁrst phase,
DB retains only a subset of features that concentrate more on the CIFAR part of the images.
Thus, the trained feature extractor preserves less information about the MNIST parts, and the errors of the
MNIST classiﬁcation are high.
The ﬁrst observation is that for both DB and VIB with the original stochastic compressed representation, nuisance plots show that increasing β from the minimum value to ∼ 0 .
1 /d barely changes the primary CIFAR errors but grows the nuisance MNIST errors up to ∼ 90% ( i.e . the maximum error for the 10 -way classiﬁcation).
With even larger β , enforcing stronger compression results in the increase of the primary errors too, as shown in primary plots.
This suggests that both DB and VIB provide ﬁne controllability over the removal of task-irrelevant information.
Secondly, if we move our focus to the nuisance (deterministic) plots in Figures 6b–6d, which show the test errors on the nuisance MNIST classiﬁcation with the feature extractor’s deterministic representation, the results become different between DB and VIB.
In DB, the nuisance (deterministic) plots follow the nuisance plots in the range of β where the compression takes effect ( i.e . where the nuisance errors increase).
Moreover, the two plots get closer as β increases.
It means that
Drop-Bottleneck’s deterministic compressed representations maintain the majority of the indistinguishability for the task-irrelevant information learned during the ﬁrst phase, especially when β is large enough to enforce some degree of the compression.
On the other hand, VIB’s nuisance (deterministic) plots are largely different from the nuisance plots; even the primary errors rise before the nuisance (deterministic) errors reach their maximum values.
This shows that employing the mode of VIB’s output distribution as its deterministic representation results in loss of the learned indistinguishability. In summary, we conﬁrm that DB provides controllability over the degree of compression in a similar way as VIB. On the other hand, DB’s deterministic representation can be a reasonable replacement
[Table]
/ryOuk26l6zfvvQqLUuijjKcAbncAUeXEML7qENHSAQwTO8wpsjnBfn3flYtpacYuYU/sD5/AEw1I5C</latexit> (a
[Figure] [Figure]
[Table]
/3KgvNolDDOqx9gEjKfAKNohe8oz6rZGvu2YBokxXAP0rTxeFCrNxvN+QkvgtYS1MnydAY7FeZ7oYVAaSgHrXutZm76FpRhlKOrRoVG3+XYN9TzUIJA3bfzmbhw1zPDMMmU/6QJ5+y/ERaE1jMRe6cAM9LrWklepvUKkzzvWybzwqCki0JJwUOTheWAwyFTSA2feQBUMd9rSEeggBr/DCtVYnFeQOKEZkKAHNooFlNX3nbq3LoyWSiTUjlAPxaFb3yGdzkqMJl6YiNQqWDSzUH0MCrx/6wwPbd6fKnV+ncQzpbXJp1B6vMYnBo7xxtr+n6YYJ/R2TO00QrTU+spWh3HcMydPRgs674+dOvz4pnWrkxkRhS4PXQrS2M1mnLNfVq/nK31VbwIjtqN1tNG+327vv9iuabb5AF5RPZIizwj [Figure] + [Figure]
kz8BD74EX4NvC2tlaxlzn6yc4Ptf1HlW9Q==
[Figure]
/latexit> [Figure] Nuisance (agg.) means the maps on the nuisance task aggregated over all the logits ( i.e . 10 logits). (a) indicates that the feature extractor without DB trained on the primary task still outputs much information about the nuisance tasks, and thus the nuisance classiﬁer could depend on the features extracted from the nuisance (MNIST) regions.
In contrast, (b) suggests that the feature extractor with DB could learn to discard the nuisance features, so that the nuisance classiﬁer mostly fails to learn due to the lack of nuisance-relevant features.
for its original stochastic representation in terms of preserving the learned indistinguishability, which is not exhibited by VIB.
C V ISUALIZATION OF T ASK - IRRELEVANT I NFORMATION R EMOVAL
In this section, we visualize the removal of task-irrelevant information with Drop-Bottleneck (DB).
To this end, we employ the Occluded CIFAR dataset (Achille & Soatto, 2018) with the same experimental setup as in Appendix B. Each image of Occluded CIFAR dataset is one of the CIFAR-
(Krizhevsky, 2009) images occluded by MNIST (LeCun et al., 2010) digit images.
In the ﬁrst phase of the experiments, the feature extractor and the classiﬁer are trained on the primary (occluded CIFAR classiﬁcation) task in a normal way.
During the second phase, the learned feature extractor is ﬁxed, and only a new classiﬁer is trained on the nuisance (MNIST classiﬁcation) task.
In Appendix B, we quantitatively showed that the feature extractor with DB could focus more on the occluded CIFAR images and preserve less information about the MNIST occlusions.
We take a qualitative approach in this section and visualize the phenomenon using Grad-CAM (Selvaraju et al., 2017).
Grad-CAM is popular for providing visual explanation given convolutional neural networks with their target values ( e.g . target logits in classiﬁcation tasks).
We ﬁrst sample multiple test images from the Occluded CIFAR dataset, and load full, trained models, which include the feature extractor, primary classiﬁer and nuisance classiﬁer.
We then obtain the activation maps for the last convolutional layer of the feature extractor on the primary and nuisance tasks.
On the primary task, we compute the activation maps simply targeting the logits for the sample labels.
However, on the nuisance task, we get the activation maps of all the logits and aggregate them by taking the maximum of the maps at each pixel location.
Therefore, the aggregated maps on the nuisance task visualize the activation related to not only the logits for the true class labels but also the other logits, capturing most of the feature usage induced during the training on the nuisance task.
Figure 7 compares two trained models: the d = 128 model without DB, and the d = 128 model with DB (deterministic).
We use the DB model with β = 5 .
623 /d for the visualization, as the value is sufﬁciently large enough to enforce strong compression while it still keeps the primary error not high.
Figure 7a shows that regarding the logits for the nuisance (MNIST) task, a large portion of each image including the MNIST digit is activated in most cases, and thus it indicates that the feature extractor trained without DB preserves much of the nuisance features.
On the other hand, Figure 7b visualizes that the feature extractor with DB outputs notably less of the nuisance features, preventing the nuisance classiﬁer from learning correctly.
To sum up, we provide the visual demonstration that on the classiﬁcation task with the Occluded
CIFAR dataset, the feature extractor equipped with DB trained on the primary task could discard majority of the nuisance i.e . task-irrelevant information given a value of β that is strong enough.
D A BLATION S TUDY : E XPLORATION WITHOUT D ROP -B OTTLENECK
We perform an ablation study to show Drop-Bottleneck (DB)’s ability of dealing with task-irrelevant input information.
We examine the performance of the same exploration method as described in Section 4 but without DB; that is, the feature vectors are fully used with no dropping.
In order to emphasize the effectiveness of DB for noisy and task-irrelevant information, we conduct experiments with both noisy and original ( i.e . without explicit noise injection) settings.
[Table]
Table 5 shows the results on “Very Sparse” DMLab environments with “Image Action”, “Noise”, “Noise Action” and “Original” settings.
Compared to “Original” setting where observations contain only implicit, inherent noisy information irrelevant to state transitions, DB brings much more signiﬁcant improvement to exploration methods in “Image Action”, “Noise” [Table] and “Noise Action” settings, which inject explicit, severe transition-irrelevant information to observations.
These results suggest that DB plays an important role handling noisy or task-irrelevant input information
[Table]
E D ETAILS OF E XPERIMENTS
We describe additional details of the experiments in Section 5.
For all the experiments with DB, each entropy H ( X i ) is computed with the binning-based estimation using 32 bins, and the drop probability p is initialized with p
i = σ ( p (cid:48) i ) and p (cid:48)
i ∼ Uniform ( a, b ) for a = − 2 , b = 1 . [Table] [Table]
E. D ETAILS OF E XPLORATION E XPERIMENTS
To supplement Section 5. [Table] we describe additional details of the VizDoom (Kempka et al., 2016) and DMLab (Beattie et al., 2016) exploration experiments.
We collect training samples in a buffer and update p , T ψ , f φ with Equation (14) periodically.
We use Adam optimizer (Kingma & Ba, 2015) with a learning rate of 0 . 0001 and a batch size of 512 .
In each optimization epoch, the training samples from the buffer are re-shufﬂed.
For each mini-batch, we optimize the Deep Infomax (Hjelm et al., 2019) discriminator T ψ with 8 extra epochs with the same samples, to make the Jensen-Shannon mutual information bound tighter.
This way of training T ψ only runs forward and backward passes on T ψ for the ﬁxed output of the feature extractor f φ , and thus can be done with low computational cost.
β is the hyperparameter that determines the relative scales of the compression term and the Deep Infomax Jensen-Shannon mutual information estimator.
It is tuned to β = 0 .
001 / 128 for DB and β = 0 .
0005 / 128 for VIB.
To make experiments simpler, we normalize our intrinsic rewards with the running mean and standard deviation.
Table 6 and Table 7 report the hyperparameters of the methods for VizDoom and DMLab experiments, respectively.
We tune the hyperparameters based on the ones provided by Savinov et al.
Unless speciﬁed, we use the same hyperparameters with Savinov et al.
Under the three noise settings suggested in Savinov et al.
(2019), the lower right quadrant of every observation is occupied by a TV screen as follows
• “Image Action”: Every time the agent performs a speciﬁc action, it changes the channel of the TV randomly to one of the 30 predeﬁned animal images.
• “Noise”:
At every observation, a new noise pattern is sampled and shown on the TV screen (independently from the agent’s actions).
• “Noise Action”:
Same as “Noise”, but the noise pattern only changes when the agent does a speciﬁc action.
Figure 8 shows some observation examples from VizDoom and DMLab environments with “Image
Action” and “Noise” settings.
[Figure]
E. D ETAILS OF A DVERSARIAL R OBUSTNESS E XPERIMENTS
For the experiments, each input image is sized as 299 × 299 × 3 and each pixel value is ranged [ − 1 , 1] .
We perform our adversarial robustness experiments based on the ofﬁcial source code 2 of
Carlini & Wagner (2017).
For the targeted (cid:96) 2 attack we increase the number of binary search steps to 20 and use a batch size of 25 .
Other than the two, the default hyperparameters are used.
E. D ETAILS OF R UNNING F EATURE S ELECTION M ETHOD
The input of the feature selection method is the same as DB’s and VIB’s: the input features are obtained with a pre-trained model of Inception-ResNet-v2 on ImageNet (Russakovsky et al., 2015).
We randomly pick 100 k samples out of the total 1281167 training samples of ImageNet, and compute the relevance scores for features using those samples by estimating the mutual information between each feature and its label using the entropy estimation with the k -nearest neighbors (Kraskov et al., 2004; Ross, 2014).
Given the computed scores, we perform the feature selection by preserving the features with the highest scores.
E. D ETAILS OF O CCLUDED I MAGE C LASSIFICATION E XPERIMENTS
We use a modiﬁed version of All-CNN-32 (Achille & Soatto, 2018).
The model architecture for feature dimension d is described in Table 8.
Batch normalization (Ioffe & Szegedy, 2015) is applied to Conv layers, and the ReLU (Nair & Hinton, 2010; Glorot et al., 2011) activation is used at every hidden layer.
We use the Adam optimizer (Kingma & Ba, 2015) with a learning rate of 0 . 001 .
To ensure the convergence, in each training, the model is trained for 200 epochs with a batch size of 100 .
[Table]