Paper: SWAP-NAS: Sample-Wise Activation Patterns for Ultra-fast NAS (ICLR-2024)

Reviewer comment (Theory):
The method assumes ReLU as the activation function, which is a big limitation as more and more transformer based networks use GELU.

Author response:
We totally agree that SWAP should be extended beyond ReLU, e.g. Transformers with GELU activation functions. This forms a key part of our ongoing work. As GELU is not a piecewise non-linear function like ReLU, the primary focus of our new work is on how to binarise the post-activation value from GELU. Unlike adopting the Signum as the indicator function for ReLU networks, we need to design a different suitable indicator function, which can binarise activation values to 0 and 1.   

For this study, the challenge is the absence of dedicated NAS benchmarks for Transformers, akin to NAS-Bench-X01 for ReLU networks. To rigorously verify the applicability and effectiveness of SWAP-Score on GELU activation, as we did with ReLU networks in this work, we will prepare and train a large variety of Transformers to obtain their ground-truth performance. This process is undoubtedly resource intensive. We are actively working in this direction and hope to be able to present it as a complete but separate study real soon.

Reviewer comment (Experiments):
Can this method apply to transformer-based NAS?

Author response:
In a nutshell, it is highly likely. We are working on this. Please see more detailed explanation above.

Reviewer comment (Theory):
Limited Applicability to ReLU-based Networks: One limitation of this approach is its dependency on neural networks using ReLU activations. While ReLU is widely used, it represents a subset of possible activation functions, which narrows the scope of application and might not encompass the full diversity of network architectures.

Author response:
We completely acknowledge this point as extending beyond ReLU has been in our plan for quite a while. This work specifically focuses on ReLU-based networks due to several reasons. Firstly, the proposed SWAP concept is derived from studies on ReLU-based networks, especially in terms of network performance estimation. Secondly, we aim to address the unsolved limitations of these studies, such as the saturation problem (reached the upper bound) and poor correlation. Additionally, as you mentioned, NAS benchmarks and algorithms are predominately ReLU based, making it more feasible for large-scale comparisons, especially compared with other training-free metrics.  

We fully recognise the significance of expanding SWAP, e.g. to Transformers with GELU. This is indeed our ongoing work. Applying SWAP on a certain kind of activation function primarily depends on the indicator function, such as adopting Signum to binarise the activation values from ReLU. The major challenge of this extension is the lack of established NAS benchmarks for non-ReLU networks, and the poor availability of trained networks. Nevertheless, we are committed to exploring the effectiveness and generalization of SWAP across different activation functions and networks, aiming to present it as a complete but separate study.

Reviewer comment (Novelty):
Similarity to NWOT Format: It's worth noting that the format of the proposed zero-cost proxy shares similarities with existing methods like NWOT. While not necessarily a weakness, this overlap should be acknowledged and explored further to understand the distinctions and innovations brought by SWAP-NAS.

Author response:
Thank you for pointing that out. NWOT and SWAP indeed share some similarities, as both methods aim to estimate networks’ performance by examining activation patterns. However, there are fundamental differences in their methodologies. In NWOT, authors first calculate the Hamming distance between pair of binary codes (rows in the standard activation patterns), they called it the activation overlap between data points (as we mentioned in the Related Work section in Page 3). Then, they use the Hamming distances to form a kernel matrix $K_{H}$. Finally, the score $s$ is obtained by computing the logarithm of the determinant of the kernel matrix $K_{H}$.  

In contrast, SWAP-Score is obtained by computing the cardinality of sample-wise activation patterns. This process is less computationally intensive than the NWOT score calculation. In addition, the proposed sample-wise activation patterns dose not only address the limitation of easitimating networks’ performance by standard activation patterns, but also introduces a novel paradigm for subsequent work. Specifically, SWAP is a new form of activation patterns that offer higher capacity for distinct patterns while preserving the information from standard activation patterns. We see it as a distinctive innovation of significant value to the community. 

If the above explanation adds value, we will include a brief version in the revised manuscript.

Reviewer comment (Presentation):
Absence of Correlation Visualization: The paper lacks visualizations that illustrate the correlation between SWAP and performance, which could provide valuable insights into the method's behavior and its implications for optimization. The inclusion of such visual aids would strengthen the paper's argument and help readers grasp the methodology's practical significance.

Author response:
Thank you for the nice suggestion, we have included the correlation visualisations as Fig. 16 & 17 in Appendix E of the revised version.

Reviewer comment (Novelty):
Missing references:  
PINAT: A Permutation INvariance Augmented Transformer for NAS Predictor  AAAI 2023  
TNASP: A Transformer-based NAS Predictor with a Self-evolution Framework  NeurIPS 2021  
Please cite and compare with them in Table 1 and Table 2.

Author response:
Thanks for pointing that out, we have included them in Tables 1 & 2 of the revised version.

Reviewer comment (Experiments):
The method is proposed to be used in the ReLU activation function-based network, which is not been proven to work well on other nonlinear activation function-based networks or not.

Author response:
Thank you for raising this issue. Going beyond ReLU has been in our plan since the early stage of this study and is now working in progress. As discussed in the future work section, verifying the effectiveness of SWAP-Score on other non-linear activation function-based networks is indeed crucial and valuable. We are actively pursuing this direction. The key point of extending SWAP to other kinds of activation function is adopting or developing a suitable indicator function, such as adopting the Signum function as an indicator function for the activation values from the ReLU in this work. The reason that we primarily focus on ReLU networks in this work is, the majority of popular NAS-Benchmarks, including the recently proposed NAS-Bench-Suite-Zero for validating training-free metrics, are predominantly built upon ReLU networks. These benchmarks form the foundation of our current study and provide solid platforms for large-scale validation experiments. 

Expanding our research to include networks with different activation functions, such as Transformers with GELU activations, involves a sizeable amount of work of its own. This entails more than developing a suitable indicator function, but preparation of a diverse range of Transformer architectures, obtaining their ground-truth performance on across a range of tasks. This is our ongoing work, and we would love to present it as a separate study in due time.

Reviewer comment (Experiments):
The paper only give numerical numbers on NAS Search Space, and did not show or analyze any searched neural architectures, to back the claim that Sample-Wise Activation Patterns would measure the network’s expressivity more accurately.

Author response:
Thank you for the excellent suggestion. Accordingly, we have added detailed visualisations of the correlation between networks’ ground-truth performance and SWAP-Scores, for all the combinations of search spaces and tasks (Appendix E of the revised version). Further, we have added visualisations of neural architectures discovered by SWAP-NAS (Appendix F of the revised version). Those new visualisations can demonstrate the advantages of the proposed method more intuitively as per your suggestion.

We would like to stress that the major focus of this work is to propose and validate the effectiveness of the SWAP-Score as a training-free metric. The networks’ expressivity is an abstract concept. Its specific manifestation is the ground-truth performance. As mentioned in the paper, a network with higher expressivity means it is more capable of learning complex functions. Consequently, this kind of network is more likely to perform better. Thus, the high correlation between SWAP-Scores and networks’ ground-truth performance is strong evidence that SWAP-Score can more accurately measure the networks’ expressivity (Fig. 4). Moreover, our experiments of validating the effectiveness of SWAP-Score were conducted through NAS-Bench-Suite-Zero, which is a standardized framework designed to facilitate a fair comparison for different training-free metrics.

Reviewer comment (Experiments):
How would SWAP-NAS integrate with pruning-based search method such as the one used in TE-NAS?

Author response:
The integration of SWAP-Score with pruning and other non-evolutionary search methods is indeed feasible and potentially valuable.  As both SWAP-NAS and TE-NAS adopt DARTS space, there is no difference from the search space perspective, except one is via an evolution-based search algorithm and the other is via a pruning-based. Notably, one of the training-free metrics utilised in TE-NAS, the number of linear regions, serves as a training-free metric via revealing the network’s expressivity as well. Thus, the transition from using the number of linear regions to SWAP-Score in a pruning-based NAS is straightforward. We are grateful for the suggestion and are indeed open to exploring the application of SWAP-Score in a wider range of search algorithms in our future work.

Reviewer comment (Reproducibility):
How to intergate regularised SWAP-NAS with FLOPs or Latency budget constrain?

Author response:
We see no obvious obstacle in the integration of SWAP-Score with FLOPs or latency. Both model parameters and FLOPs can be obtained once the model is constructed. Thus, FLOPs can be incorporated into our regularisation function as the model parameters. Regarding latency, it can be efficiently measured by passing a sample input though the untrained network and measuring the time required for the forward pass. This measurement can be done in parallel with the calculation of SWAP-Score. The result can also be added into the regularization function. An alternative solution is using multi-objective search considering both SWAP-Score and latency at the same time. 

It is important to note that our use of a regularisation function to control model size stems from a widely recognised limitation of training-free metrics, which often favor larger models. This limitation has been underscored in several studies, such as [1, 2].  

Again, thank you for this insightful comment which is very valuable for our next phase of the study. 

&nbsp;
&nbsp;

##### [1] Xuefei Ning, Changcheng Tang, Wenshuo Li, Zixuan Zhou, Shuang Liang, Huazhong Yang, and Yu Wang. Evaluating efficient performance estimators of neural architectures. Advances in Neural Information Processing Systems, 34:12265–12277, 2021. 

##### [2] Arjun Krishnakumar, Colin White, Arber Zela, Renbo Tu, Mahmoud Safari, and Frank Hutter. Nas- bench-suite-zero: Accelerating research on zero cost proxies. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022.