Original: T RIANGULAR D ROPOUT : V ARIABLE N ETWORK W IDTH WITHOUT R ETRAINING
Revised: T RIANGULAR D ROPOUT : V ARIABLE N ETWORK W IDTH WITHOUT R ETRAINING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: One of the most fundamental design choices in neural networks is layer width: it affects the capacity of what a network can learn and determines the complexity of the solution.
Revised: One of the most fundamental design choices in neural networks is layer width: it affects the capacity of what a network can learn and determines the complexity of the solution.

Original: This latter property is often exploited when introducing information bottlenecks, forcing a network to learn compressed representations.
Revised: This latter property is often exploited when introducing information bottlenecks, forcing a network to learn compressed representations.

Original: However, such an architecture decision is typically immutable once training begins; switching to a more compressed architecture requires retraining.
Revised: However, such an architecture decision is typically immutable once training begins; switching to a more compressed architecture requires retraining.

Original: In this paper we present a new layer design, called Triangular Dropout, which does not have this limitation.
Revised: In this paper we present a new layer design, called Triangular Dropout, which does not have this limitation.

Original: After training, the layer can be arbitrarily reduced in width to exchange performance for narrowness.
Revised: After training, the layer can be arbitrarily reduced in width to exchange performance for narrowness.

Original: We demonstrate the construction and potential use cases of such a mechanism in three areas.
Revised: We demonstrate the construction and potential use cases of such a mechanism in three areas.

Original: Firstly, we describe the formulation of Triangular Dropout in autoencoders, creating an MNIST autoencoder with selectable compression after training.
Revised: Firstly, we describe the formulation of Triangular Dropout in autoencoders, creating models with selectable compression after training.

Original: Secondly, we add Triangular Dropout to VGG19 on ImageNet, creating a powerful network which, without retraining, can be significantly reduced in parameters.
Revised: Secondly, we add Triangular Dropout to VGG19 on ImageNet, creating a powerful network which, without retraining, can be significantly reduced in parameters.

Original: Lastly, we explore the application of Triangular Dropout to reinforcement learning (RL) policies on selected control problems.
Revised: Lastly, we explore the application of Triangular Dropout to reinforcement learning (RL) policies on selected control problems.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: In this paper we describe Triangular Dropout, a novel dropout (Srivastava et al., 2014) technique that imparts a powerful property to fully-connected layers: the layer can be reduced in width after training to select a desired size of representation.
Revised: In this paper we describe Triangular Dropout, a novel dropout (Srivastava et al., 2014) technique that imparts a powerful property to fully-connected layers: the layer can be reduced in width after training to select a desired size of representation.

Original: This enables structured pruning in fully-connected layers such that the pruned network is densely connected.
Revised: This enables structured pruning in fully-connected layers such that the pruned network is densely connected.

Original: This exposes a trade-off between performance and network width that can be varied without retraining the network.
Revised: This exposes a trade-off between performance and network width that can be varied without retraining the network.

Original: Additionally, constructing and applying Triangular Dropout is straightforward.
Revised: Additionally, constructing and applying Triangular Dropout is straightforward.

Original: Triangular Dropout amounts to masking minibatches during training with a binary lower triangular matrix with zeros above the diagonal.
Revised: Triangular Dropout amounts to masking minibatches during training with a binary lower triangular matrix with zeros above the diagonal.

Original: After training, the relevant weights can be copied into a physically smaller dense network.
Revised: After training, the relevant weights can be copied into a physically smaller dense network.

Original: Architecturally, Triangular Dropout represents a shift towards models that can be densely downsized to fit the needs of a particular application.
Revised: Architecturally, Triangular Dropout represents a shift towards models that can be densely downsized to fit the needs of a particular application.

Original: For example, a wide model could be trained once on a computer cluster, and narrower versions would be immediately available for use on personal machines, mobile devices, IoT devices, or other embedded platforms.
Revised: For example, a wide model could be trained once on a computer cluster, and narrower versions would be immediately available for use on personal machines, mobile devices, IoT devices, or other embedded platforms.

Original: An image processing pipeline could have its frame-rate or feature descriptor length altered as needed, perhaps to process more samples in exchange for lower performance per sample.
Revised: An image processing pipeline could have its frame-rate or feature descriptor length altered as needed, perhaps to process more samples in exchange for lower performance per sample.

Original: Triangular Dropout is tangent to many areas of research, and includes ideas of compression, structured network pruning, and variable computation.
Revised: Triangular Dropout is tangent to many areas of research, and includes ideas of compression, structured network pruning, and variable computation.

Original: Our objective is not to outperform in any specific domain, but rather to demonstrate a technique that we find has useful properties from several areas and is very easy to implement.
Revised: Our objective is not to outperform in any specific domain, but rather to demonstrate a technique that we find has useful properties from several areas and is very easy to implement.

Original: We discuss related works to Triangular Dropout in Section 2, and formally define our mechanism in Section 3.
Revised: We discuss related works to Triangular Dropout in Section 2, and formally define our mechanism in Section 3.

Original: We then investigate some applications of Triangular Dropout in unsupervised learning of latent encodings (Section 4), supervised learning in larger models (Section 5), and reinforcement learning for locomotion control problems (Section 6).
Revised: We then investigate some applications of Triangular Dropout in unsupervised learning of latent encodings (Section 4), supervised learning in larger models (Section 5), and reinforcement learning for locomotion control problems (Section 6).

Original: Sections 7 and 8 further discuss the characteristics of Triangular Dropout and suggest potential future research directions.
Revised: Sections 7 and 8 further discuss the characteristics of Triangular Dropout and suggest potential future research directions.

Original:  B ACKGROUND
Revised:  B ACKGROUND

Original: The width of a specific network layer is perhaps most critical in the domain of autoencoding (Rumelhart et al., 1986), in which the width of the encoding layer (along with its activation) describes the amount of information retained in an encoded sample.
Revised: The width of a specific network layer is perhaps most critical in the domain of autoencoding (Rumelhart et al., 1986), in which the width of the encoding layer (along with its activation) describes the amount of information retained in an encoded sample.

Original: The smaller the latent space (narrower the encoding layer), the more compressed the learned representation, and therefore, the more lossy the compression performed by the network.
Revised: The smaller the latent space (narrower the encoding layer), the more compressed the learned representation, and therefore, the more lossy the compression performed by the network.

Original: There are countless variations of autoencoders which seek to improve on this compression ratio (Theis et al., 2017) or introduce additional properties to the latent variables (Vincent et al., 2008; Rifai et al., 2011).
Revised: There are countless variations of autoencoders which seek to improve on this compression ratio (Theis et al., 2017) or introduce additional properties to the latent variables (Vincent et al., 2008; Rifai et al., 2011).

Original: The degree to which the latent variables are independent is the subject of disentanglement research, which introduces further properties into the latent variables.
Revised: The degree to which the latent variables are independent is the subject of disentanglement research, which introduces further properties into the latent variables.

Original: β -VAE (Higgins et al., 2017) demonstrates that the gaussian distributions in a VAE (Kingma & Welling, 2014) can be made increasingly orthogonal in exchange for reconstruction accuracy.
Revised: β -VAE (Higgins et al., 2017) demonstrates that the gaussian distributions in a VAE (Kingma & Welling, 2014) can be made increasingly orthogonal in exchange for reconstruction accuracy.

Original: The independence and ordering of learned features can also be enforced manually by methods such as PCAAE (Pham et al., 2020), which learns one latent variable at a time, progressively widening the latent layer to learn additional features via additional training.
Revised: The independence and ordering of learned features can also be enforced manually by methods such as PCAAE (Pham et al., 2020), which learns one latent variable at a time, progressively widening the latent layer to learn additional features via additional training.

Original: A comparison to PCAAE is discussed in detail in Section 4.
Revised: A comparison to PCAAE is discussed in detail in Section 4.

Original: Autoencoding makes use of a well-known tradeoff between network size and capability to encourage compression, a relationship that holds in other domains (Amodei & Hernandez, 2019; Kaplan et al., 2020; McCandlish et al., 2018).
Revised: Autoencoding makes use of a well-known tradeoff between network size and capability to encourage compression, a relationship that holds in other domains (Amodei & Hernandez, 2019; Kaplan et al., 2020; McCandlish et al., 2018).

Original: This is perhaps most well known with regards to ImageNet (Deng et al., 2009), in which historically a clear correlation is seen between network size and performance (summarized nicely in Figure 1 of Tan & Le (2020)).
Revised: This is perhaps most well known with regards to ImageNet (Deng et al., 2009), in which historically a clear correlation is seen between network size and performance (summarized nicely in Figure 1 of Tan & Le (2020)).

Original: However, such large models can also be very inefficient in their use of parameters, often learning unnecessary weights or redundant representations.
Revised: However, such large models can also be very inefficient in their use of parameters, often learning unnecessary weights or redundant representations.

Original: Network pruning is a technique to reduce an over-parameterized network down to essential parameters, recovering a sub-network that is similar or even superior in performance (Blalock et al., 2020; Frankle & Carbin, 2018).
Revised: Network pruning is a technique to reduce an over-parameterized network down to essential parameters, recovering a sub-network that is similar or even superior in performance (Blalock et al., 2020; Frankle & Carbin, 2018).

Original: In unstructured pruning, the network is sparse, whereas in structured pruning, the sub-network is dense.
Revised: In unstructured pruning, the network is sparse, whereas in structured pruning, the sub-network is dense.

Original: Triangular
Revised: Triangular

Original: Dropout falls into the latter category, but examines fully-connected layers instead of the typical pruning of convolutional elements, as in Anwar et al.
Revised: Dropout falls into the latter category, but examines fully-connected layers instead of the typical pruning of convolutional elements, as in Anwar et al.

Original: (2015) (channels and kernels), Li et al.
Revised: (2015) (channels and kernels), Li et al.

Original: (2016) (filters), and He et al.
Revised: (2016) (filters), and He et al.

Original: (2017) (channels).
Revised: (2017) (channels).

Original: Triangular Dropout additionally does not require finetuning and is not iterative, as in the above works and many others (Molchanov et al., 2016;
Revised: Triangular Dropout additionally does not require finetuning and is not iterative, as in the above works and many others (Molchanov et al., 2016;

Original: Luo et al., 2019), although fine-tuning could of course be applied if desired.
Revised: Luo et al., 2019), although fine-tuning could of course be applied if desired.

Original: In Section 5, we explore dense prunings recovered by Triangular Dropout on the classification head of VGG19, an early ImageNet solution that is well-known to be overparameterized as evidenced by subsequent works with fewer parameters (Iandola et al., 2016; Tan & Le, 2020).
Revised: In Section 5, we explore dense prunings recovered by Triangular Dropout on the classification head of VGG19, an early ImageNet solution that is well-known to be overparameterized as evidenced by subsequent works with fewer parameters (Iandola et al., 2016; Tan & Le, 2020).

Original: This overparameterization and history of use makes it popular for pruning studies, especially on ImageNet (Blalock et al., 2020).
Revised: This overparameterization and history of use makes it popular for pruning studies, especially on ImageNet (Blalock et al., 2020).

Original: There is also a great deal of relevant work from the conditional computation domain, exploring architectures that dynamically find appropriate sub-networks for a given input (Han et al., 2021).
Revised: There is also a great deal of relevant work from the conditional computation domain, exploring architectures that dynamically find appropriate sub-networks for a given input (Han et al., 2021).

Original: In this approach networks are trained to enable selection of sub-networks, rather than simply pruning the network after initial training.
Revised: In this approach networks are trained to enable selection of sub-networks, rather than simply pruning the network after initial training.

Original: Few approaches examine variable feed-forward layer width, for example Bengio et al.
Revised: Few approaches examine variable feed-forward layer width, for example Bengio et al.

Original: (2015) use reinforcement learning to learn which nodes in a fully-connected layer can be masked out on a per-sample basis.
Revised: (2015) use reinforcement learning to learn which nodes in a fully-connected layer can be masked out on a per-sample basis.

Original: Ba & Frey (2013) achieve a similar effect through deep belief networks creating layer masks.
Revised: Ba & Frey (2013) achieve a similar effect through deep belief networks creating layer masks.

Original: Most methods rely on selective routing through entire sub-branches or sub-modules, i.e. Sabour et al.
Revised: Most methods rely on selective routing through entire sub-branches or sub-modules, i.e. Sabour et al.

Original: There is a related area of work in recurrent networks which adapts computation in the time (rollout) dimension during inference, mainly by choosing where and when to update a hidden state (Graves, 2016; Campos et al., 2017; Jernite et al., 2017).
Revised: There is a related area of work in recurrent networks which adapts computation in the time (rollout) dimension during inference, mainly by choosing where and when to update a hidden state (Graves, 2016; Campos et al., 2017; Jernite et al., 2017).

Original: In contrast to these areas, Triangular Dropout specifically seeks dense representations that capture the entirety of the input space, and does so in a non-recurrent manner.
Revised: In contrast to these areas, Triangular Dropout specifically seeks dense representations that capture the entirety of the input space, and does so in a non-recurrent manner.

Original: Network size selection has additional repercussions in the deep reinforcement learning (DRL) domain.
Revised: Network size selection has additional repercussions in the deep reinforcement learning (DRL) domain.

Original: Works like amplified control (Paulhamus et al., 2019) and robust predictable control (Eysenbach et al., 2021) seek compressed policies to perform tasks, using information bottlenecking (Tishby et al., 2001) to enforce this compression.
Revised: Works like amplified control (Paulhamus et al., 2019) and robust predictable control (Eysenbach et al., 2021) seek compressed policies to perform tasks, using information bottlenecking (Tishby et al., 2001) to enforce this compression.

Original: Robust predictable control demonstrates that compression creates policies that are more risk-averse and potentially more transferable.
Revised: Robust predictable control demonstrates that compression creates policies that are more risk-averse and potentially more transferable.

Original: Such compressability is further related to task complexity measurement, an open problem (see the implications of task complexity discussed in Srivastava et al.
Revised: Such compressability is further related to task complexity measurement, an open problem (see the implications of task complexity discussed in Srivastava et al.

Original: We explore the application of Triangular
Revised: We explore the application of Triangular

Original: Dropout to DRL in Section 6.
Revised: Dropout to DRL in Section 6.

Original:  T RIANGULAR D ROPOUT
Revised:  T RIANGULAR D ROPOUT

Original: Triangular Dropout is simple to implement in practice.
Revised: Triangular Dropout is simple to implement in practice.

Original: For a fully-connected layer of width n , processing a minibatch x of B datapoints, the output y has size ( B, n ) .
Revised: For a fully-connected layer of width n , processing a minibatch x of B datapoints, the output y has size ( B, n ) .

Original: During training, standard dropout (with probability p ) multiplies the output element-wise by a random binary mask M of the same size as the output: y =
Revised: During training, standard dropout (with probability p ) multiplies the output element-wise by a random binary mask M of the same size as the output: y =

Original: M ⊙ α ( x T W + b ) where M ∼ Bernoulli ( p ) where W and b are the layer weights and biases, respectively, and α is some activation function.
Revised: M ⊙ α ( x T W + b ) where M ∼ Bernoulli ( p ) where W and b are the layer weights and biases, respectively, and α is some activation function.

Original: Triangular Dropout has the same form, but the mask M is populated as a binary lower triangular matrix instead of a randomly sampled one:
Revised: Triangular Dropout has the same form, but the mask M is populated as a binary lower triangular matrix instead of a randomly sampled one:

Original: [Equation]
Revised: [Equation]

Original: This is visualized in Figure 1.
Revised: This is visualized in Figure 1.

Original: In the case where B > n , the rows of M Tri are simply repeated to reach the desired size.
Revised: In the case where B > n , the rows of M Tri are simply repeated to reach the desired size.

Original: In the case where B < n , a triangular block matrix may be used.
Revised: In the case where B < n , a triangular block matrix may be used.

Original: For any B , an approximate alternative would be to independently sample rows such that a random number of 1’s are followed by 0’s.
Revised: For any B , an approximate alternative would be to independently sample rows such that a random number of 1’s are followed by 0’s.

Original: The implications of a triangular mask can be viewed in different ways.
Revised: The implications of a triangular mask can be viewed in different ways.

Original: Considering the rows of the output y , the mask creates a set of datapoints which have been processed with the full range of layer widths from 1 to n .
Revised: Considering the rows of the output y , the mask creates a set of datapoints which have been processed with the full range of layer widths from 1 to n .

Original: If the batch size is equal to n , the ith row represents a training sample in which the effective layer width is i .
Revised: If the batch size is equal to n , the ith row represents a training sample in which the effective layer width is i .

Original: Training now amounts to something like a multitask learning problem, in which the “tasks” are processing data with varying network widths.
Revised: Training now amounts to something like a multitask learning problem, in which the “tasks” are processing data with varying network widths.

Original: We have an even distribution of data from all these tasks, and the loss function reflects training over this set of architectures (widths):
Revised: We have an even distribution of data from all these tasks, and the loss function reflects training over this set of architectures (widths):

Original: [Equation]
Revised: [Equation]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: → effective
Revised: → effective

Original: [Table]
Revised: [Table]

Original:  
Revised:  

Original: Taking this multitask view, we do not re-scale the layer outputs as in normal dropout; we want the downstream network to be able to handle any of the ”tasks” (any of the potential widths of this layer).
Revised: Taking this multitask view, we do not re-scale the layer outputs as in normal dropout; we want the downstream network to be able to handle any of the ”tasks” (any of the potential widths of this layer).

Original: However, the “tasks” in this view are not independent.
Revised: However, the “tasks” in this view are not independent.

Original: Examining the columns of the output, the jth column represents the output of the jth layer node, masked out for the first j rows.
Revised: Examining the columns of the output, the jth column represents the output of the jth layer node, masked out for the first j rows.

Original: Because the mask is triangular, all of the samples which are non-zero at column j are necessarily also non-zero at columns 1 through j − 1 .
Revised: Because the mask is triangular, all of the samples which are non-zero at column j are necessarily also non-zero at columns 1 through j − 1 .

Original: That is, parameters relevant to the jth node are only updated for samples in which the previous j − 1 nodes (smaller widths) are also updated.
Revised: That is, parameters relevant to the jth node are only updated for samples in which the previous j − 1 nodes (smaller widths) are also updated.

Original: This is not true for subsequent nodes: nodes j + 1 and onwards may or may not be masked when node j is utilized.
Revised: This is not true for subsequent nodes: nodes j + 1 and onwards may or may not be masked when node j is utilized.

Original: This cascading dependence is what masks Triangular Dropout work.
Revised: This cascading dependence is what masks Triangular Dropout work.

Original: The parameters related to node j can be learned with a guarantee that the preceding j − 1 outputs in terms of width are also present.
Revised: The parameters related to node j can be learned with a guarantee that the preceding j − 1 outputs in terms of width are also present.

Original: However, they cannot rely as heavily on outputs j + 1 and beyond, and are essentially restricted from fully co-optimizing with these outputs.
Revised: However, they cannot rely as heavily on outputs j + 1 and beyond, and are essentially restricted from fully co-optimizing with these outputs.

Original: During test time, it is up to the user to decide on the masking strategy.
Revised: During test time, it is up to the user to decide on the masking strategy.

Original: Providing a mask filled with ones makes the layer approximately equivalent to a typical fully-connected layer, or masking out the last N − j nodes causes the layer to behave as though it has width j .
Revised: Providing a mask filled with ones makes the layer approximately equivalent to a typical fully-connected layer, or masking out the last N − j nodes causes the layer to behave as though it has width j .

Original: To permanently ablate the layer down to some desired width, the relevant model parameters (portions of W and b from this layer and the next) simply need to be copied into a smaller model, as shown in Figure 1.
Revised: To permanently ablate the layer down to some desired width, the relevant model parameters (portions of W and b from this layer and the next) simply need to be copied into a smaller model, as shown in Figure 1.

Original:  V ARIABLE C OMPRESSION IN A UTOENCODERS
Revised:  V ARIABLE C OMPRESSION IN A UTOENCODERS

Original: To further detail the properties of Triangular Dropout and compare it to alternative methods, we demonstrate the construction of an autoencoder with a selectable latent size.
Revised: To further detail the properties of Triangular Dropout and compare it to alternative methods, we demonstrate the construction of an autoencoder with a selectable latent size.

Original: We train an autoencoder on MNIST (Deng, 2012) with a latent size n , and use Triangular Dropout on the latent layer.
Revised: We train an autoencoder on MNIST (Deng, 2012) with a latent size n , and use Triangular Dropout on the latent layer.

Original: After training, we can select any level of compression ≤ n and receive both a valid encoding and reconstruction.
Revised: After training, we can select any level of compression ≤ n and receive both a valid encoding and reconstruction.

Revised: We also train autoencoders on the CelebA dataset (Liu et al., 2015) (image size reduced to 64x64) in order to explore Triangular Dropout on a more difficult autoencoding task.

Original: 4. B ASELINES
Revised: 4. B ASELINES

Original: We compare  results quantitatively (reconstruction loss) and qualitatively (visualization of reconstruction) to two baselines: n autoencoders that are trained for specific latent sizes 1 through n , and PCAAE, which trains an autoencoder of size n progressively.
Revised: We compare MNIST results quantitatively (reconstruction loss) and qualitatively (visualization of reconstruction) to two baselines: n autoencoders that are trained for specific latent sizes 1 through n , and PCAAE, which trains an autoencoder of size n progressively.

Original: PCAAE trains one latent variable at a time and includes a correlation minimization loss term.
Revised: PCAAE trains one latent variable at a time and includes a correlation minimization loss term.

Original: PCAAE also trains a separate encoder for each latent variable, and a new decoder for each width.
Revised: PCAAE also trains a separate encoder for each latent variable, and a new decoder for each width.

Original: This means that, like the set of standard autoencoders, PCAAE requires n training runs.
Revised: This means that, like the set of standard autoencoders, PCAAE requires n training runs.

Original: Compared to standard autoencoders, our expectation is that Triangular Dropout must sacrifice some compression in exchange for the introduction of some one-way dependence among features (the first i features are not very dependent on the i + 1 th feature).
Revised: Compared to standard autoencoders, our expectation is that Triangular Dropout must sacrifice some compression in exchange for the introduction of some one-way dependence among features (the first i features are not very dependent on the i + 1 th feature).

Original: Conversely, we would not expect a typical autoencoder to have this property.
Revised: Conversely, we would not expect a typical autoencoder to have this property.

Original: The encoded features from a typical autoencoder, having no encouragement to be independent or disentangled, should only be meaningful when the full feature descriptor is retained.
Revised: The encoded features from a typical autoencoder, having no encouragement to be independent or disentangled, should only be meaningful when the full feature descriptor is retained.

Original: We would expect that PCAAE further sacrifices reconstruction loss for an even more stringent independence among learned features.
Revised: We would expect that PCAAE further sacrifices reconstruction loss for an even more stringent independence among learned features.

Original: However, the features are independent, the reconstructions may still be meaningful when less important features are removed from the final model.
Revised: However, the features are independent, the reconstructions may still be meaningful when less important features are removed from the final model.

Original: 4. E XPERIMENTS AND R ESULTS
Revised: 4. E XPERIMENTS AND R ESULTS

Original: A convolutional autoencoder with latent size 32 was trained with PCAAE and Triangular Dropout to reconstruct MNIST digits.
Revised: A convolutional autoencoder with latent size 32 was trained with PCAAE and Triangular Dropout to reconstruct MNIST digits.

Original: The smaller widths of PCAAE were saved off during training.
Revised: The smaller widths of PCAAE were saved off during training.

Original: Additionally, standard autoencoders were trained for sizes 1 through 32.
Revised: Additionally, standard autoencoders were trained for sizes 1 through 32.

Original: We examine the reconstruction loss of the models as the latent width is increased during training and decreased via ablation of the final 32-width model.
Revised: We examine the reconstruction loss of the models as the latent width is increased during training and decreased via ablation of the final 32-width model.

Original: Triangular Dropout falls into the latter category by design, but has performance similar to the checkpoint models trained at specific sizes.
Revised: Triangular Dropout falls into the latter category by design, but has performance similar to the checkpoint models trained at specific sizes.

Original: These results are shown in Figure 2.
Revised: These results are shown in Figure 2.

Original: Full architecture and training details are available in the Appendix.
Revised: Full architecture and training details are available in the Appendix.

Original: Notably, our hypotheses regarding feature independence vs reconstruction loss hold true: for a given latent size z, the most performant model (without feature ablation) is a standard autoencoder trained at that width, the least performant is PCAAE which introduces a correlation loss, and somewhere inbetween is Triangular Dropout.
Revised: Notably, our hypotheses regarding feature independence vs reconstruction loss hold true: for a given latent size z, the most performant model (without feature ablation) is a standard autoencoder trained at that width, the least performant is PCAAE which introduces a correlation loss, and somewhere inbetween is Triangular Dropout.

Original: The baseline models with ablated encodings do not have comparable reconstruction losses unless the ablation is very minor (most latent features retained).
Revised: The baseline models with ablated encodings do not have comparable reconstruction losses unless the ablation is very minor (most latent features retained).

Original: Triangular
Revised: Triangular

Original: Dropout, trained only at size 32, can be ablated down to very small sizes while only performing slightly worse than individual autoencoders at those sizes.
Revised: Dropout, trained only at size 32, can be ablated down to very small sizes while only performing slightly worse than individual autoencoders at those sizes.

Original: Triangular Dropout roughly encompasses the superset of these autoencoders, while only requiring a single training run.
Revised: Triangular Dropout roughly encompasses the superset of these autoencoders, while only requiring a single training run.

Original: Figure 3 shows the qualitative reconstruction in each of these cases, for selected latent widths.
Revised: Figure 3 shows the qualitative reconstruction in each of these cases, for selected latent widths.

Original: The standard autoencoders trained at increasing sizes (Figure 3, A) show increasing reconstruction fidelity, with recognizable digits appearing around 4-6 latent features.
Revised: The standard autoencoders trained at increasing sizes (Figure 3, A) show increasing reconstruction fidelity, with recognizable digits appearing around 4-6 latent features.

Original: When removing features from the size 32 autoencoder (Figure 3, B), the encoding are clearly corrupted unless most features are present.
Revised: When removing features from the size 32 autoencoder (Figure 3, B), the encoding are clearly corrupted unless most features are present.

Original: PCAAE (Figure 3, C and D) has only similar reconstructions to the standard autoencoders once sufficient features are present, and can also content with significant ablations.
Revised: PCAAE (Figure 3, C and D) has only similar reconstructions to the standard autoencoders once sufficient features are present, and can also content with significant ablations.

Original: Finally, Triangular Dropout (Figure 3, E) has similar reconstructions to the individual autoencoders, even though its latent size is determined by ablation.
Revised: Finally, Triangular Dropout (Figure 3, E) has similar reconstructions to the individual autoencoders, even though its latent size is determined by ablation.

Original: A user of this model could select some latent size approximately between 6 and 32 and expect it to yield recognizable digits.
Revised: A user of this model could select some latent size approximately between 6 and 32 and expect it to yield recognizable digits.

Revised: We also explored the application of Triangular Dropout to the CelebA dataset, and find similar effects.

Revised: This model had a latent size of 128.

Revised: Full architectural and training details are available in the Appendix.

Revised: As seen in Figure 3 F, the most compressed features of the autoencoder also capture the most significant features, with additional detail captured by subsequent outputs.

Original:   P ARAMETER R EDUCTION IN L ARGE N ETWORKS
Revised: We also noted that Triangular Dropout may have unusual behavior with batch normalization (Ioffe & Szegedy, 2015) or variational encodings (Kingma & Welling, 2014), which shift or redefine outputs with a value of zero. These artifacts are primarily seen when width is severely reduced. Reconstructions using these architectures are available in the Appendix.  P ARAMETER R EDUCTION IN L ARGE N ETWORKS

Original: The results of our MNIST autoencoding experiments demonstrate the primary property of a Triangular Dropout layer: it can be reduced in width after training and still provide meaningful output.
Revised: The results of our MNIST autoencoding experiments demonstrate the primary property of a Triangular Dropout layer: it can be reduced in width after training and still provide meaningful output.

Original: We now consider the ramifications of such a mechanism in the supervised learning domain.
Revised: We now consider the ramifications of such a mechanism in the supervised learning domain.

Original: Many powerful models for image feature extraction and natural language are so large that they cannot be easily utilized by the average practitioner at their full sizes.
Revised: Many powerful models for image feature extraction and natural language are so large that they cannot be easily utilized by the average practitioner at their full sizes.

Original: To make them more accessible and study the effects of scale, multiple specific sizes are often trained.
Revised: To make them more accessible and study the effects of scale, multiple specific sizes are often trained.

Original: For example, VGG (Simonyan& Zisserman, 2015) is often used at one of 4 sizes (VGG11, VGG13, VGG16, and VGG19), and so is GPT-

Revised: For example, VGG (Simonyan Zisserman, 2015) is often used at one of 4 sizes (VGG11, VGG13, VGG16, and VGG19), and so is GPT-

Original: (Radford et al., 2019) (sizes small, medium, large, and extra large).
Revised: (Radford et al., 2019) (sizes small, medium, large, and extra large).

Original: We envision Triangular
Revised: We envision Triangular

Original: Dropout as a stepping stone towards single master models that can be reduced in size arbitrarily to fit the needs of any given user.
Revised: Dropout as a stepping stone towards single master models that can be reduced in size arbitrarily to fit the needs of any given user.

Original: In this manner, Triangular Dropout could be considered a pruning technique in which the model is trained in preparation for structured pruning.
Revised: In this manner, Triangular Dropout could be considered a pruning technique in which the model is trained in preparation for structured pruning.

Original: To motivate this view, we retrain the fully connected classification portion of VGG19 on ImageNet using Triangular Dropout for the two hidden layers of size 4096.
Revised: To motivate this view, we retrain the fully connected classification portion of VGG19 on ImageNet using Triangular Dropout for the two hidden layers of size 4096.

Original: We then investigate the fidelity of the model when run at reduced widths.
Revised: We then investigate the fidelity of the model when run at reduced widths.

Original: We concentrate on this portion of the model because it is fully-connected and contains the majority of VGG’s parameters (roughly 123 million out ofmillion in total).
Revised: We concentrate on this portion of the model because it is fully-connected and contains the majority of VGG’s parameters (roughly 123 million out ofmillion in total).

Original: Specific training details are available in the appendix.
Revised: Specific training details are available in the appendix.

Original: 5. M ODEL A CCURACY F INDINGS
Revised: 5. M ODEL A CCURACY F INDINGS

Original: VGG19 retrained with Triangular Dropout does not achieve quite the accuracy of the original
Revised: VGG19 retrained with Triangular Dropout does not achieve quite the accuracy of the original

Original: VGG19, achieving 69.7 percent accuracy when not reduced in width, compared to 72.4 percent accuracy of the original VGG19 model.
Revised: VGG19, achieving 69.7 percent accuracy when not reduced in width, compared to 72.4 percent accuracy of the original VGG19 model.

Original: This may or may not be significant depending on the application; this would be considered a significant difference in an often-used benchmark like ImageNet, but in a unique real world problem may be acceptable in exchange for the properties that Triangular
Revised: This may or may not be significant depending on the application; this would be considered a significant difference in an often-used benchmark like ImageNet, but in a unique real world problem may be acceptable in exchange for the properties that Triangular

Original: Dropout imparts to the network.
Revised: Dropout imparts to the network.

Original: Additionally, the success of the model with two layers of Triangular Dropout indicates that Triangular Dropout layers can potentially be stacked in sequence to create a multi-layer MLP with adjustable width.
Revised: Additionally, the success of the model with two layers of Triangular Dropout indicates that Triangular Dropout layers can potentially be stacked in sequence to create a multi-layer MLP with adjustable width.

Original: In Section 6 we investigate a network with 3 hidden layers, all of which utilize Triangular
Revised: In Section 6 we investigate a network with 3 hidden layers, all of which utilize Triangular

Original: Dropout.
Revised: Dropout.

Original: 5. R EDUCED W IDTH
Revised: 5. R EDUCED W IDTH

Original: Our primary finding is that Triangular Dropout allows the classification portion of VGG19 to be reduced in width after training, resulting in a dense sub-network that trades accuracy for narrowness.
Revised: Our primary finding is that Triangular Dropout allows the classification portion of VGG19 to be reduced in width after training, resulting in a dense sub-network that trades accuracy for narrowness.

Original: This a known tradeoff from prior work in network pruning (see pruning results for VGG16 on
Revised: This a known tradeoff from prior work in network pruning (see pruning results for VGG16 on

Original: ImageNet in Blalock et al.
Revised: ImageNet in Blalock et al.

Original: (2020), figure 3), which Triangular Dropout makes available without introducing sparsity, requiring fine-tuning, or iterative training.
Revised: (2020), figure 3), which Triangular Dropout makes available without introducing sparsity, requiring fine-tuning, or iterative training.

Original: Our results using full-batch Triangular
Revised: Our results using full-batch Triangular

Original: Dropout (batch size 4096) are summarized in Table 1 and graphically in Figure 4 with the curve in bold.
Revised: Dropout (batch size 4096) are summarized in Table 1 and graphically in Figure 4 with the curve in bold.

Original: As seen in this plot, an accuracy of around 50 percent can be achieved with only 1 percent of the original classifier parameters.
Revised: As seen in this plot, an accuracy of around 50 percent can be achieved with only 1 percent of the original classifier parameters.

Original: This corresponds to hidden layers that are roughly 10 percent as wide as in the full model.
Revised: This corresponds to hidden layers that are roughly 10 percent as wide as in the full model.

Original: The hidden layers can be reduced in size by half and only incur a 0.
Revised: The hidden layers can be reduced in size by half and only incur a 0.

Original: percent loss in classification accuracy.
Revised: percent loss in classification accuracy.

Original: This corresponds to ablating more than half of the classifier parameters, and roughly 46 percent of all the parameters in VGG19.
Revised: This corresponds to ablating more than half of the classifier parameters, and roughly 46 percent of all the parameters in VGG19.

Original: 5. E FFECTS OF B ATCH S IZE
Revised: 5. E FFECTS OF B ATCH S IZE

Original: Since training with such a large batch size is not always practical for large image models [Table] we also explore training with batch sizes that are fractions of the full-size batches.
Revised: Since training with such a large batch size is not always practical for large image models, we also explore training with batch sizes that are fractions of the full-size batches.

Original: In this setting [Table] the triangular mask takes the form of a block matrix with blocks of size 1 row by d columns, where d = 4096/
Revised: In this setting, the triangular mask takes the form of a block matrix with blocks of size 1 row by d columns, where d = 4096/

Original: B. The width ablation curves are shown in Figure 4.
Revised: B. The width ablation curves are shown in Figure 4.

Original: The primary result is that the full-size batches appear the most stable and perhaps best-performing, but all batch sizes have similar trends of accuracy vs width.
Revised: The primary result is that the full-size batches appear the most stable and perhaps best-performing, but all batch sizes have similar trends of accuracy vs width.

Original: Notably, this tradeoff does seem to begin breaking down with very small batches, as seen by batch sizes 128 and 256 having irregular curve profiles.
Revised: Notably, this tradeoff does seem to begin breaking down with very small batches, as seen by batch sizes 128 and 256 having irregular curve profiles.

Original: [Table]
Revised: [Table]

Original: is also shown (teal).

Original:  C OMPRESSED RL P OLICIES
Revised:  C OMPRESSED RL P OLICIES

Original: In this section, we explore the application of Triangular Dropout to sequential decision making.
Revised: In this section, we explore the application of Triangular Dropout to sequential decision making.

Original: Specifically, we first train RL policies on four locomotion control tasks from the MuJoCo (Todorov et al., 2012) environment suite within OpenAI Gym (Brockman et al., 2016).
Revised: Specifically, we first train RL policies on four locomotion control tasks from the MuJoCo (Todorov et al., 2012) environment suite within OpenAI Gym (Brockman et al., 2016).

Original: We then then distill the learned policies into secondary networks that use Triangular Dropout, either in a single hidden layer or for all hidden layers.
Revised: We then then distill the learned policies into secondary networks that use Triangular Dropout, either in a single hidden layer or for all hidden layers.

Original: We examine the performance of these distilled policies for different levels of width reduction, providing measures for both the compressibility of these policies and the complexity of the tasks they solve.
Revised: We examine the performance of these distilled policies for different levels of width reduction, providing measures for both the compressibility of these policies and the complexity of the tasks they solve.

Original: Finally, we select a compressed architecture that still performs well for each task and evaluate its performance when retrained from scratch.
Revised: Finally, we select a compressed architecture that still performs well for each task and evaluate its performance when retrained from scratch.

Original: We examine four environments: Hopper-v3, HalfCheetah-v3, Walker2d-v3, and Ant-v3, all of which come from OpenAI Gym’s MuJoCo environment collection.
Revised: We examine four environments: Hopper-v3, HalfCheetah-v3, Walker2d-v3, and Ant-v3, all of which come from OpenAI Gym’s MuJoCo environment collection.

Original: We selected these tasks because they have similar objectives (locomote forward at the highest speed possible), but may differ in complexity.
Revised: We selected these tasks because they have similar objectives (locomote forward at the highest speed possible), but may differ in complexity.

Original: 6. P OLICY D ISTILLATION WITH T RIANGULAR D ROPOUT
Revised: 6. P OLICY D ISTILLATION WITH T RIANGULAR D ROPOUT

Original: We first determine if Triangular Dropout can be used to create variable-width policy networks, and compare the performance of the ablated policies to the original experts.
Revised: We first determine if Triangular Dropout can be used to create variable-width policy networks, and compare the performance of the ablated policies to the original experts.

Original: We also wish to explore if the compressibility of the policy (how much it can be reduced in width without severely affecting performance) is a potential metric of task complexity, an important and open problem in reinforcement learning, multitask learning, and meta-learning.
Revised: We also wish to explore if the compressibility of the policy (how much it can be reduced in width without severely affecting performance) is a potential metric of task complexity, an important and open problem in reinforcement learning, multitask learning, and meta-learning.

Original: We do not use Triangular Dropout in RL directly because the masking changes the policy’s behavior during data collection.
Revised: We do not use Triangular Dropout in RL directly because the masking changes the policy’s behavior during data collection.

Original: For each environment, we first train an expert policy (three fully-connected layers of size 48) using proximal policy optimization (PPO) (Schulman et al., 2017), and then imitate them through supervised learning.
Revised: For each environment, we first train an expert policy (three fully-connected layers of size 48) using proximal policy optimization (PPO) (Schulman et al., 2017), and then imitate them through supervised learning.

Original: To imitate these expert policies, we first gather (state, action) pairs from several thousand evaluation runs with occasional random actions taken to prevent over-sampling of the most likely trajectories.
Revised: To imitate these expert policies, we first gather (state, action) pairs from several thousand evaluation runs with occasional random actions taken to prevent over-sampling of the most likely trajectories.

Original: We randomly down-select to 100,000 data points and train a new model to imitate this policy in a supervised fashion.
Revised: We randomly down-select to 100,000 data points and train a new model to imitate this policy in a supervised fashion.

Original: The new model consists of 3 fully-connected layers of size 48, with either the middle layer using Triangular Dropout, or all layers.
Revised: The new model consists of 3 fully-connected layers of size 48, with either the middle layer using Triangular Dropout, or all layers.

Original: After training, we can reduce the width of the Triangular Dropout layers to enforce a compression in the mapping of states to actions.
Revised: After training, we can reduce the width of the Triangular Dropout layers to enforce a compression in the mapping of states to actions.

Original: Training details of both the experts and imitation networks are available in the Appendix.
Revised: Training details of both the experts and imitation networks are available in the Appendix.

Original: We examine the performance of the ablated policies by finding the smallest layer widths that maintain at least 90% performance (speed) compared to the original experts.
Revised: We examine the performance of the ablated policies by finding the smallest layer widths that maintain at least 90% performance (speed) compared to the original experts.

Original: We measure performance in terms of average top speed over 100 episodes, which is a smoother metric than raw reward (and is the primary goal of the environments).
Revised: We measure performance in terms of average top speed over 100 episodes, which is a smoother metric than raw reward (and is the primary goal of the environments).

Original: We find that in the case with a single middle Triangular Dropout layer, the policies for Hopper, HalfCheetah, Walker2d, and Ant can be compressed to middle bottlenecks of 8, 2, 2, and 7 respectively.
Revised: We find that in the case with a single middle Triangular Dropout layer, the policies for Hopper, HalfCheetah, Walker2d, and Ant can be compressed to middle bottlenecks of 8, 2, 2, and 7 respectively.

Original: For imitation networks consisting only of hidden
Revised: For imitation networks consisting only of hidden

Original: Triangular Dropout layers, we find widths of 8, 6, 6, and 9 recover at least 90% of the expert performance, respectively.
Revised: Triangular Dropout layers, we find widths of 8, 6, 6, and 9 recover at least 90% of the expert performance, respectively.

Original: This implies that Hopper and Ant may be more complex than HalfCheetah and Walker, since they seem to require more parameters in their solutions.
Revised: This implies that Hopper and Ant may be more complex than HalfCheetah and Walker, since they seem to require more parameters in their solutions.

Original: It is also interesting that
Revised: It is also interesting that

Original: HalfCheetah and Walker2d can be performant with a bottleneck of size 2, which is far less than the action space of these environments (size 6).
Revised: HalfCheetah and Walker2d can be performant with a bottleneck of size 2, which is far less than the action space of these environments (size 6).

Original: As described in Paulhamus et al.
Revised: As described in Paulhamus et al.

Original: (2019), this may imply that the solutions reside in a two-degree-of-freedom subspace of possible actions.
Revised: (2019), this may imply that the solutions reside in a two-degree-of-freedom subspace of possible actions.

Original: 6. C OMPARISON TO R ETRAINED A RCHITECTURES
Revised: 6. C OMPARISON TO R ETRAINED A RCHITECTURES

Original: Finally, we take the 90% performance ablated architectures and retrain them from scratch in the same manner as the original experts.
Revised: Finally, we take the 90% performance ablated architectures and retrain them from scratch in the same manner as the original experts.

Original: Here the reduced Triangular Dropout layers have been replaced with fully-connected layers of the reduced size.
Revised: Here the reduced Triangular Dropout layers have been replaced with fully-connected layers of the reduced size.

Original: Comparing the retrained experts with the original expert policies, the retrained experts had comparable (and in some cases better) top speeds to the originals, despite their vastly reduced network size.
Revised: Comparing the retrained experts with the original expert policies, the retrained experts had comparable (and in some cases better) top speeds to the originals, despite their vastly reduced network size.

Original: Compared to the reduced networks with
Revised: Compared to the reduced networks with

Original: Triangular Dropout (before retraining), there are cases in which retraining yields performance gains, but this was not always true.
Revised: Triangular Dropout (before retraining), there are cases in which retraining yields performance gains, but this was not always true.

Original: For some problems simply reducing the policy width after imitating with Triangular Dropout will maintain performance.
Revised: For some problems simply reducing the policy width after imitating with Triangular Dropout will maintain performance.

Original: A detailed summary of the ablation experiments in this section are shown in Figure 5.
Revised: A detailed summary of the ablation experiments in this section are shown in Figure 5.

Original: RL training curves and hyperparameters are available in the appendix, along with numerical values for top speed and average reward of all policies.
Revised: RL training curves and hyperparameters are available in the appendix, along with numerical values for top speed and average reward of all policies.

Original:  D ISCUSSION
Revised:  D ISCUSSION

Original: Throughout our experimentation, we found several characteristics of Triangular Dropout that were surprising.
Revised: Throughout our experimentation, we found several characteristics of Triangular Dropout that were surprising.

Original: Firstly, the results indicate that features learned with Triangular Dropout are indeed oneway dependent.
Revised: Firstly, the results indicate that features learned with Triangular Dropout are indeed oneway dependent.

Original: That is, the n + 1 th feature relies on the nth to be meaningful, but not the other way around.
Revised: That is, the n + 1 th feature relies on the nth to be meaningful, but not the other way around.

Original: It is not immediately clear that this property should form inherently, since for a given training batch there are many examples for which adjacent features are both optimized.
Revised: It is not immediately clear that this property should form inherently, since for a given training batch there are many examples for which adjacent features are both optimized.

Original: Perhaps our most significant finding was that Triangular Dropout seems to encourage compression in learned features, enabling structured pruning after training.
Revised: Perhaps our most significant finding was that Triangular Dropout seems to encourage compression in learned features, enabling structured pruning after training.

Original: In many cases the network used far less of its available width than we had allocated, and to our surprise did not spread representation over all available resources.
Revised: In many cases the network used far less of its available width than we had allocated, and to our surprise did not spread representation over all available resources.

Original: We believe Triangular Dropout to be an interesting mechanism that could lead to networks which are self-organizing and variable in size.
Revised: We believe Triangular Dropout to be an interesting mechanism that could lead to networks which are self-organizing and variable in size.

Original: In addition to the useful properties discussed above, a network that can be pruned to dense sub-networks without retraining has many practical benefits, especially as AI models get larger and mobile and embedded platforms become increasingly widespread.
Revised: In addition to the useful properties discussed above, a network that can be pruned to dense sub-networks without retraining has many practical benefits, especially as AI models get larger and mobile and embedded platforms become increasingly widespread.

Original:  F UTURE D IRECTIONS
Revised:  F UTURE D IRECTIONS

Original: There are several promising directions in which to continue this work.
Revised: There are several promising directions in which to continue this work.

Original: Triangular Dropout has only been designed and tested for fully-connected layers, and immediate future work could examine reforming Triangular Dropout for convolutions or transformers.
Revised: Triangular Dropout has only been designed and tested for fully-connected layers, and immediate future work could examine reforming Triangular Dropout for convolutions or transformers.

Original: Most prior work in structured pruning has focused on convolutions.
Revised: Most prior work in structured pruning has focused on convolutions.

Original: Additionally, Triangular Dropout allows a network layer to be resized in width, but not depth.
Revised: Additionally, Triangular Dropout allows a network layer to be resized in width, but not depth.

Original: A corresponding mechanism which would allow for variable-depth networks post-training would be similarly useful, and very powerful when combined with variable width.
Revised: A corresponding mechanism which would allow for variable-depth networks post-training would be similarly useful, and very powerful when combined with variable width.

Original: Investigating solutions for applying Triangular Dropout directly to on-policy RL would also be worthwhile, as we believe our technique holds promise for developing compressed policies.
Revised: Investigating solutions for applying Triangular Dropout directly to on-policy RL would also be worthwhile, as we believe our technique holds promise for developing compressed policies.

Original: A A PPENDIX A. A UTOENCODER E XPERIMENTS
Revised: A A PPENDIX A. A UTOENCODER E XPERIMENTS

Original: In this section we detail architectures, training routines, and additional experiments concerning Triangular Dropout applied to MNIST autoencoders.
Revised: In this section we detail architectures, training routines, and additional experiments concerning Triangular Dropout applied to MNIST autoencoders.

Original: A.1.  A RCHITECTURE AND T RAINING D ETAILS
Revised: A.1. MNIST A RCHITECTURE AND T RAINING D ETAILS

Original: In all  cases the network architectures were as follows, given some encoder output size z :
Revised: For all MNIST cases the network architectures were as follows, given some encoder output size z :

Original: Model
Revised: Model

Revised: Type

Original: [Table]
Revised: [Table]

Revised: Encoder

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: linear

Revised: [Table]

Original: Decoder
Revised: Decoder

Revised: [Table]

Original: [Table]
Revised: [Table]

Revised: Relu 32 Filters, 3x

Revised: [Table]

Revised: [Table]

Revised: Transp

Revised: [Table]

Revised: [Table]

Revised: , 7x

Revised: [Table]

Revised: [Table]

Revised: Transp

Revised: [Table]

Revised: [Table]

Revised: 32 Filters, 9x

Revised: [Table]

Revised: [Table]

Revised: Convolution sigmoid 1 Filter, 3x

Original:  Models are trained with batch size 1024 using the Adam optimizer, with a learning rate of 0.005 that decreases by a factor of 10 if the average epoch loss does not decrease by at least 2 percent over the last 15 epochs.
Revised: [Table] [Table] , Padding Models are trained with batch size 1024 using the Adam optimizer, with a learning rate of 0.005 that decreases by a factor of 10 if the average epoch loss does not decrease by at least 2 percent over the last 15 epochs.

Original: This is repeated for 5 decreases of the learning rate.
Revised: This is repeated for 5 decreases of the learning rate.

Original: MNIST images were normalized to values between 0 and 1, and binary cross entropy was used as the loss for reconstruction.
Revised: MNIST images were normalized to values between 0 and 1, and binary cross entropy was used as the loss for reconstruction.

Original: A.1. A VAILABLE W IDTH VERSUS C OMPRESSION
Revised: A.1. A VAILABLE W IDTH VERSUS C OMPRESSION

Original: We additionally performed experiments to test if the available encoding size affected the learned representations in Triangular Dropout.
Revised: We additionally performed experiments to test if the available encoding size affected the learned representations in Triangular Dropout.

Original: If extra width is available, does the model utilize that width or is it ignored?
Revised: If extra width is available, does the model utilize that width or is it ignored?

Original: In Figure 6 we show a plot of autoencoders trained using Triangular Dropout with encoding widths up to 32, however they were trained with available widths of 4, 8, 16, 32, 64, 128, 256, and 512.
Revised: In Figure 7 we show a plot of autoencoders trained using Triangular Dropout with encoding widths up to 32, however they were trained with available widths of 4, 8, 16, 32, 64, 128, 256, and 512.

Original:  trained with
Revised: of autoencoders trained with

Original: Triangular Dropout and a variety of available latent widths.
Revised: Triangular Dropout and a variety of available latent widths.

Original: We find that there is a clear pattern of models with more available width being less compressed, but the overall difference in performance diminishes as available width increases.
Revised: We find that there is a clear pattern of models with more available width being less compressed, but the overall difference in performance diminishes as available width increases.

Original: A. VGG19 T RAINING D ETAILS
Revised: A.1. C ELEB A A RCHITECTURE AND T RAINING D ETAILS

Revised: We use the following architecture for our CelebA experiments. The VAE results described below simply use two parallel encoding layers for the mean and variation, respectively. Models were trained for 100 epochs with batch size 128.

Revised: We used the Adam optimizer with learning rate 0.001, decreasing by a factor of 10 every 30 epochs

Revised: [Table]

Revised: C ELEB A R ECONSTRUCTIONS WITH A LTERNATIVE A RCHITECTURES

Revised: Please see reconstruction examples on the following page

Original:  Our version of VGG19 with Triangular Dropout was created by retraining the classification portion of VGG19.
Revised: [Table] A. VGG19 T RAINING D ETAILS Our version of VGG19 with Triangular Dropout was created by retraining the classification portion of VGG19.

Original: The convolutional layers were frozen and not retrained.
Revised: The convolutional layers were frozen and not retrained.

Original: The convolutional layers output 25088 features for a given input, which then pass through two hidden layers of size 4096, and finally output as 1000 features to be interpreted as classification scores for the image.
Revised: The convolutional layers output 25088 features for a given input, which then pass through two hidden layers of size 4096, and finally output as 1000 features to be interpreted as classification scores for the image.

Original: Our model simply replaced the two hidden layers with Triangular Dropout layers of the same size.
Revised: Our model simply replaced the two hidden layers with Triangular Dropout layers of the same size.

Original: We trained our model for 100 epochs using a batch size of 4096 in order to apply a full triangular mask to the batch.
Revised: We trained our model for 100 epochs using a batch size of 4096 in order to apply a full triangular mask to the batch.

Original: We used SGD with an initial learning rate of 0.01, reducing the learning rate by a factor of 10 every 30 epochs.
Revised: We used SGD with an initial learning rate of 0.01, reducing the learning rate by a factor of 10 every 30 epochs.

Original: It is unclear if our model would have behaved differently if the entire architecture were retrained, which may have enabled Triangular Dropout to be applied to the feature descriptors of length 25088.
Revised: It is unclear if our model would have behaved differently if the entire architecture were retrained, which may have enabled Triangular Dropout to be applied to the feature descriptors of length 25088.

Original: This is of interested because VGG19’s 25088 features are often used as a starting point for other image processing tasks.
Revised: This is of interested because VGG19’s 25088 features are often used as a starting point for other image processing tasks.

Original: A. RL T RAINING S CHEMES
Revised: A. RL T RAINING S CHEMES

Original: In this section we detail hyperparameters and training routines needed to reproduce our experiments from Section 6.
Revised: In this section we detail hyperparameters and training routines needed to reproduce our experiments from Section 6.

Original: A.3. PPO H YPERPARAMETERS AND T RAINING D ETAILS
Revised: A.3. PPO H YPERPARAMETERS AND T RAINING D ETAILS

Original: The PPO expert discussed in Section 6 consisted of separate policy and value networks.
Revised: The PPO expert discussed in Section 6 consisted of separate policy and value networks.

Original: The networks were MLPs with three hidden layers of size 48 (or less for the retrained policies), with input and output sizes determined by the environment.
Revised: The networks were MLPs with three hidden layers of size 48 (or less for the retrained policies), with input and output sizes determined by the environment.

Original: Value networks consisted of two layers of width 128.
Revised: Value networks consisted of two layers of width 128.

Original: All layers (besides value output) used hyperbolic tangent as the activation function.
Revised: All layers (besides value output) used hyperbolic tangent as the activation function.

Original: We used the implementation of PPO from Stable Baselines 3 (Raffin et al., 2019).
Revised: We used the implementation of PPO from Stable Baselines 3 (Raffin et al., 2019).

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Hyperparameters A.3. PPO L EARNING C URVES AND S TATISTICS
Revised: Hyperparameters A.3. PPO L EARNING C URVES AND S TATISTICS

Original: For reference, we provide the learning curves and final performance of our PPO agents.
Revised: For reference, we provide the learning curves and final performance of our PPO agents.

Original: Please reference the figures on the subsequent pages.
Revised: Please reference the figures on the subsequent pages.

Original: A.3. I MITATION N ETWORK T RAINING D ETAILS
Revised: A.3. I MITATION N ETWORK T RAINING D ETAILS

Original: The imitation networks consisted of MLPs with 3 hidden layers of width 128, the middle layer using Triangular Dropout.
Revised: The imitation networks consisted of MLPs with 3 hidden layers of width 128, the middle layer using Triangular Dropout.

Original: Internal layers used the Relu activation function while the final layer used hyperbolic tangent.
Revised: Internal layers used the Relu activation function while the final layer used hyperbolic tangent.

Original: The networks were trained in a supervised fashion on 100,000 (state, action) pairs collected from running the PPO-trained policies on their respective environments.
Revised: The networks were trained in a supervised fashion on 100,000 (state, action) pairs collected from running the PPO-trained policies on their respective environments.

Original: The networks were trained for 240 epochs with a batch size of 1024, using the Adam optimizer with a learning rate of 0.01.
Revised: The networks were trained for 240 epochs with a batch size of 1024, using the Adam optimizer with a learning rate of 0.01.

Original: Every 80 epochs, the learning rate was decreased by a factor of 10.
Revised: Every 80 epochs, the learning rate was decreased by a factor of 10.