PAD -Net: An Efficient Framework for Dynamic Networks (2023)

Abstract: Dynamic networks, e.g., Dynamic Convolution (DY-Conv) and the Mixture of Experts (MoE), have been extensively explored as they can considerably improve the model’s representation power with acceptable computational cost. The common practice in implementing dynamic networks is to convert the given static layers into fully dynamic ones where all parameters are dynamic (at least within a single layer) and vary with the input. However, such a fully dynamic setting may cause redundant parameters and high deployment costs, limiting the applicability of dynamic networks to a broader range of tasks and models. The main contributions of our work are challenging the basic commonsense in dynamic networks and proposing a partially dynamic network, namely PAD-Net, to transform the redundant dynamic parameters into static ones. Also, we further design Iterative Mode Partition to partition dynamic and static parameters efficiently. Our method is comprehensively supported by large-scale experiments with two typical advanced dynamic architectures, i.e., DY-Conv and MoE, on both image classification and GLUE benchmarks. Encouragingly, we surpass the fully dynamic networks by +0.7\% top-1 acc with only 30% dynamic parameters for ResNet-50 and +1.9\% average score in language understanding with only 50% dynamic parameters for BERT. Code will be released at: https://github.com/Shwai-He/PAD-Net .

## introduction deep neural networks have been continuously pushing the state-of-the-art performance in the tasks of computer vision @xcite and natural language processing (dai and @xcite @xcite in past years, at the cost of increasing training cost @xcite . however, most prevalent architectures perform infer- * corresponding author ence statically where both the computational graph and network parameters are fixed once after training, which limits the representation power. dynamic networks @xcite , as opposed to static ones, adapt their parameters or architectures to each specific input, improving the model representation power with acceptable computational cost, e.g., switch transformers @xcite . the common practice of implementing dynamic networks is transforming static networks (or modules) with counterpart dynamic ones. for example, dynamic convolution (dy-conv) @xcite replaces traditional convolution by adopting k adaptive convolutional kernels; mixture of experts (moe) @xcite replaces a single fully connected layer with multiple feed-forward neural networks in a parallel manner. the success of dynamic networks motivates practitioners to design dynamic networks @xcite , which often follow a fully dynamic approach, where all parameters are dynamic (at least within a single layer) and change with the input. previous works @xcite show that dynamic networks often outperform their static counterpart, and using more dynamic layers intriguingly leads to ever-increasing performance. for instance, dynamic convolution promotes the performance on the imagenet when more static convolution layers turn into dynamic ones. however, these prior works do not explain the need for a fully dynamic mechanism, and it remains unclear whether to convert static parameters into dynamic and to what extent if yes. on the other hand, such a fully dynamic manner is resource expensive and may cause redundancy, limiting the applicability of dynamic networks. for instance, the total parameters of bert-base equipped with moe are ~506.3m (with 8 experts) compared to only ~108.9m for vanilla bert-base. in addition, an moe layer also multiplies the computation. it seems more is better when transforming static layers into dynamic ones, but how about the dynamic parameters within a dynamic network: do all of them lead to the promotion? this urges us to reflect whether there exist redundant dynamic parameters, in fully dynamic network layers? based on the above scrutinization, we hypothesize that less is more for dynamic parameters in fully dynamic networks. motivated by this hypothesis, we propose the iterative mode partition (imp) algorithm to progressively convert less important dynamic parameters into static ones for higher efficiency, while maintaining performance at a competitive level. given a fully dynamic network initialized with all parameters in dynamic mode, we attempt to partition a subset of static parameters out from them. specifically, we iteratively transform dynamic parameters based on their impact on loss values. if the transformation of the i-th element of dynamic parameters results in only a minimal loss difference, we safely make it static. given a desired dynamic ratio (the proportion of dynamic parameters), we can balance the trade-off between dynamic and static parameters. since static parameters are less costly to deploy, we prune redundant parameters after mode partition, obtaining a lightweight architecture, namely partially dynamic networks (pad-net), which contains two modes of parameters (dynamic parameters that vary with inputs and static parameters that are fixed during inference). empirically, we extensively validate this hypothesis and our proposed pad-net, including glue benchmark @xcite for moe and visual image classification @xcite for dynamic convolution. experiment results reveal that we successfully converted redundant dynamic parameters into static ones and pad-net achieves the highest performance in all tasks with lightweight architectures. given the superiority of pad-net in both effectiveness and efficiency, we show that less dynamic is more efficient in fully dynamic networks, successfully verifying the above hypothesis. the inspiration of partially dynamic can be extended to other dynamic networks and even inform future efficient architectures designation. in short, our contributions are threefold: ## conclusion and future work in this work, we first reveal parameter redundancy and high deployment costs of fully dynamic networks. to resolve these problems, we proposed the partially dynamic network (pad-net) to advance both performance and efficiency. pad-net demonstrated its superiority on moe and dy-conv frameworks. extensive experiments on both nlp and cv tasks empirically show its effectiveness and efficiency against fully dynamic networks, significantly improving performance with much fewer dynamic parameters and less computation. our proposed method could be extensively integrated with other mainstream architectures and inspire future work in efficient neural network designation and other fields.