We present Shapeshifter Networks (SSNs), a flexible neural network framework that improves performance and reduces memory requirements on a diverse set of scenarios over standard neural networks. Our approach is based on the observation that many neural networks are severely overparameterized, resulting in significant waste in computational resources as well as being susceptible to overfitting. SSNs address this by learning where and how to share parameters between layers in a neural network while avoiding degenerate solutions that result in underfitting. Specifically, we automatically construct parameter groups that identify where parameter sharing is most beneficial. Then, we map each group's weights to construct layerswith learned combinations of candidates from a shared parameter pool. SSNs can share parameters across layers even when they have different sizes , perform different operations , and/or operate on features from different modalities . We evaluate our approach on a diverse set of tasks , including image classification, bidirectional image-sentence retrieval, and phrase grounding, creating high performing models even when using as little as 1\% of the parameters . We also apply SSNs to knowledge distillation, where we obtain state-of-the-art results when combined with traditional distillation methods .
Fitting a model into GPU memory during training is an increasing concern as models continue to grow. To address this issue, we present Shapeshifter Networks (SSNs), a flexible neural network framework that decouples layers from model weights, enabling us to implement any neural network with an arbitrary number of parameters. In SSNs each layer obtains weights from a parameter store that decides where and how to allocate parameters to layers. This can result in sharing parameters across layers even when they have different sizes or perform different operations . SSNs do not require any modifications to a model's loss function or architecture, making them easy to use. Our approach can create parameter efficient networks by using a relatively small number of weights, or can improve a model's performance by adding additional model capacity during training without affecting the computational resources required at test time . We evaluate SSNs using seven network architectures across diverse tasks that include image classification, bidirectional image-sentence retrieval, and phrase grounding, creating high performing models even when using as little as 1\% of the parameters .
R
We
Fitting a model into GPU memory during training is an increasing concern as models continue to grow. To address this issue, we
R
improves performance and reduces memory requirements on a diverse set of scenarios over standard neural networks. Our approach is based on the observation that many neural networks are severely overparameterized, resulting in significant waste in computational resources as well as being susceptible to overfitting. SSNs address this by learning
decouples layers from model weights, enabling us to implement any neural network with an arbitrary number of parameters. In SSNs each layer obtains weights from a parameter store that decides
R
share parameters between layers in a neural network while avoiding degenerate solutions that result in underfitting. Specifically, we automatically construct parameter groups that identify where parameter sharing is most beneficial. Then, we map each group's weights to construct layerswith learned combinations of candidates from a shared parameter pool. SSNs can share
allocate parameters to layers. This can result in sharing
R
,
or
R
, and/or operate on features from different modalities
. SSNs do not require any modifications to a model's loss function or architecture, making them easy to use. Our approach can create parameter efficient networks by using a relatively small number of weights, or can improve a model's performance by adding additional model capacity during training without affecting the computational resources required at test time
R
our approach on a diverse set of tasks , including
SSNs using seven network architectures across diverse tasks that include
D
. We also apply SSNs to knowledge distillation, where we obtain state-of-the-art results when combined with traditional distillation methods