Paper: Resource Efficient Test-Time Training with Slimmable Network (ICLR-2024)

Reviewer comment (Evaluation):
The definition "Efficient" is vague in the title, that the model result of your method in inference is efficient, but not during the training process（it actually costs more time as claimed in your experiment). Does the efficiency come from utilizing the slimmable models and you proposed a TTT method for them?

Author response:
**A1-1:** Good comment. The term "Efficient" in our paper encompasses two key aspects:

Firstly, efficiency is achieved through the ability to seamlessly switch to smaller and more efficient sub-networks. Despite the test-time training process of our slimTTT taking slightly more time than other methods, we can utilize smaller sub-networks as if they were the largest networks (e.g., employing the R-50 (0.75×) sub-network as the largest network). Remarkably, we observe that the results obtained from the R-50 (0.75×) sub-network demonstrate comparable performance to TTAC (R-50) and even surpass TTT++ (R-50), offering a more efficient alternative.

Secondly, efficiency is also evident in the streamlined approach of deploying and training only one model containing multiple sub-networks for adaptive switching. In contrast, employing previous TTT methods on real world scenarios necessitates deploying and training several models independently to accommodate varying resource constraints. The uploading, downloading, and training processes associated with multiple models of other methods contribute significantly to resource consumption, further underscoring the efficiency of our approach.

**A1-2:** We indeed address the resource-efficient test-time training challenges by employing slimmable networks. However, simply incorporating slimmable networks into test-time training poses several challenges: 1. Different widths of networks may interfere with each other, leading to inconsistent update directions, preventing models of various widths from achieving optimal optimization. This issue is more pronounced in test-time training due to the lack of ground truth labels as well as the distribution shift. Therefore, we propose WCL and LCR to solve this challenge, which promotes the consistency between different width of networks in both feature space and prediction space. 2. The GFA is applied to ensure that the test features do not deviate far from the source feature distribution during the whole adaptation process and it can apply a restriction over the multi-view features for each width of sub-networks and ensure the correctness of the multi-view representations.

In the following analytical experiment, we attempt a straightforward application of slimmable networks in test-time training without employing our consistency method (referred to as naive SlimTTT). Unfortunately, this approach yields unsatisfactory results, particularly when applied in an online manner. This is attributed to the fact that our consistency loss functions—WCL, LCR, and GFA—leverage the diverse views captured by sub-networks of varying widths. This facilitates knowledge interaction among slimmable networks, a crucial aspect in the test-time training setting.

| Method | manner | width | Cifar10-C Err. Avg. | manner | width | Cifar10-C Err. Avg. |
| --- | --- | --- | --- | --- | --- | --- |
| naive SlimTTT | offline | 1.0 | 10.80 | online | 1.0 | 14.78 |
|  |  | 0.75 | 11.54 |  | 0.75 | 15.64 |
|  |  | 0.5 | 12.59 |  | 0.5 | 16.72 |
|  |  | 0.25 | 15.99 |  | 0.25 | 18.83 |
| SlimTTT (ours) | offline | 1.0 | **8.33** | online | 1.0 | **10.17** |
|  |  | 0.75 | **8.63** |  | 0.75 | **10.55** |
|  |  | 0.5 | **9.12** |  | 0.5 | **11.41** |
|  |  | 0.25 | **10.68** |  | 0.25 | **14.01** |

Reviewer comment (Theory):
There are some basic assumptions that should be explained.

Author response:
**A2:** Thank you for your suggestion. In the test-time training phase, we follow the approach of TTT++ and TTAC, utilizing an offline training manner. After deploying our slimmable networks to different devices according to their maximum resource constrains, we assume that the test-time training is conducting when the devices have its maximum resources (e.g., the devices are charging). 

However, motivated by Reviewer 1Qhs, now we find that even though the available compute changes frequently during test-time training in an online manner, our SlimTTT exhibits low sensitivity to changes in resources as shown below. We monitored the sensitivity of the performance in such settings on CIFAR10-C (Gaussian Noise), as different model parameters are updated based on the available compute. We implemented 12 time steps with dynamic resources throughout the period. For example, the resources change after time step 2, so we need to update the 0.75 parameters of the model rather than the whole model.

Comparing the performance on dynamically changing resources (the upper half of the table) to that on constant resources (the lower half of the table), the results below reveal that our SlimTTT exhibits low sensitivity to changes in resources. We believe that this dynamic evaluation you mentioned is both important and practical, and we intend to incorporate these findings into our new paper version to encourage future research.

|Dynamic resources|Time-step|0|1|2|3|4|5|6|7|8|9|10|11|12|
|-|-|-|-|-|-|-|-|-|-|-|-|-|-|-
||FLOPs (G)|4.1|4.1|4.1|2.3|2.3|1.1|1.1|0.3|0.3|1.1|1.1|2.3|2.3
||max width|1|1|1|0.75|0.75|0.5|0.5|0.25|0.25|0.5|0.5|0.75|0.75
||Err. (1.0)|19.53|20.12|17.84|16.31|14.92|14.78|14.73|14.36|13.85|13.55|14.06|13.99|13.82
||Err. (0.75)|22.27|21.29|18.75|17.19|16.02|15.76|15.74|15.33|14.84|14.49|14.95|14.81|14.78
||Err. (0.5)|25.00|23.63|20.57|19.14|17.97|17.51|16.96|16.41|15.89|15.35|15.59|15.43|15.38
||Err. (0.25)|28.13|28.52|25.26|23.93|22.27|21.81|21.04|20.41|19.75|19.18|19.32|18.95|18.96
|**Constant resources**|**Time-step**|**0**|**1**|**2**|**3**|**4**|**5**|**6**|**7**|**8**|**9**|**10**|**11**|**12**|
||FLOPs (G)|4.1|4.1|4.1|4.1|4.1|4.1|4.1|4.1|4.1|4.1|4.1|4.1|4.1
||max width |1|1|1|1|1|1|1|1|1|1|1|1|1
||Err. (1.0)|19.53|20.12|17.84|16.41|15.08|14.84|14.73|14.26|13.76|13.36|13.39|13.38|13.37
||Err. (0.75)|22.27|21.29|18.75|17.09|16.02|15.76|15.68|15.14|14.54|14.06|14.24|14.16|14.24
||Err. (0.5)|25.00|23.63|20.57|18.85|17.73|17.51|17.02|16.36|15.76|15.16|15.34|15.10|15.08
||Err. (0.25)|28.13|28.52|25.26|24.22|22.27|21.68|20.87|20.12|19.31|18.79|18.93|18.59|18.51

Reviewer comment (Experiments):
In some works about explainable networks, the grad-cam result shows what the model is focusing on and indicates the semantical feature that is used to make decisions. Does the bigger-width network contain the attention information of smaller networks well in all these backbones? If not, is the bigger network always better?

Author response:
**A4:** Good comment. Bigger-width networks encompass more parameters, potentially containing attention information from smaller networks in certain instances. However, as illustrated in Fig.1 in our introduction, our key observation shows that bigger-width networks can capture entirely distinct perspectives compared to smaller networks. Hence, it is not accurate to claim that larger-width networks always include the attention information of smaller networks.

Given that larger-width networks have more parameters, they typically exhibit better overall performance than the smaller networks. Nevertheless, in scenarios where larger-width networks capture entirely different perspectives compared to smaller networks, the smaller networks can complement the knowledge of larger networks. This mutual supplementation leads to direct knowledge exchange and mutual promotion, elucidating why our method proves effective.

Reviewer comment (Experiments):
In some cases of classification, one class can include several visual patterns (eg. some datasets have coarse classes like “vehicle” including items that do not look exactly the same.) Will these have an impact if you align all networks to the supernet, as it may focus on several instance-level features? I'm also curious if this will have a bigger impact on the contrastive learning that expands the positive data with the different representations from 2 sizes of model.

Author response:
**A5:** Thanks for the insightful comment. In our study, a single image (not images belonging to the same class) undergoes two different data augmentations and is then processed by four networks of varying widths, resulting in eight features. These features collectively form the positive data for that particular image. Consequently, the positive data for an image doesn't only involve the largest model size but also encompasses features from other model sizes.

Furthermore, the multi-views we emphasize in our paper refer to distinct discriminative features that different widths of sub-networks can capture (e.g., the wheel or the car window, both classifiable as a car). Despite datasets having broad classes like "vehicle" that include items with dissimilar appearances, our approach leverages knowledge interaction during test-time training that enables different widths of networks to capture a wide range of views (discriminative features) associated with coarse classes like "vehicle". Therefore, though the fact that the class "vehicle" may encompass items with varying appearances, our networks are capable of capturing the discriminative features specific to each, ultimately leading to enhanced performance compared to individual training.

Additionally, we conducted an experiment on CIFAR10-C where we don’t align smaller networks to the supernet but to the neighbor width network (e.g., 0.75× width align to the 1.0× width, 0.5× width align to the 0.75× width…). The results indicate that aligning smaller networks to different sizes of the model does not significantly impact contrastive learning.

| Method | width | Err. Avg. |
| --- | --- | --- |
| neighbor SlimTTT | 1.0 | 8.55 |
|  | 0.75 | 8.94 |
|  | 0.5 | 9.49 |
|  | 0.25 | 11.04 |
| SlimTTT | 1.0 | **8.33** |
|  | 0.75 | **8.63** |
|  | 0.5 | **9.12** |
|  | 0.25 | **10.68** |

Reviewer comment (Experiments):
One of the primary claims of the paper is the varying inference budget during the inference (also highlighted as Challenge II in Figure 1). Since the paper targets a practical setting of dynamic resource allocation during inference, it is imperative to consider experiments where the available compute changes frequently during inference (test time training). It would be interesting to monitor the sensitivity of the performance in those settings since different parameters of the model will be updated based on the available compute. Low-performance sensitivity on dynamically changing resources will make the method more reliable for practical use cases. Moreover, it’ll promote future research to address the challenges faced in such a setting.

Author response:
**A1:** Good comment. As your suggestion, we conducted experiments on CIFAR10-C (Gaussian_Noise) and CIFAR100-C (Gaussian_Noise) wherein the available compute (e.g., FLOPs) varies frequently during the test-time training phase. In this experiment, we monitored the sensitivity of the performance in such settings, as different model parameters are updated based on the available compute. Specifically, our evaluation spans over 12 time-steps with dynamic resources throughout the period. For example, the resources change after time-step 2 (from 4.1G to 2.3G), so we need to update the 0.75 parameters of the model rather than the whole model.

Comparing the performance on dynamically changing resources (the upper half of the table) to that on constant resources (the lower half of the table), the results below surprisingly reveal that our SlimTTT exhibits low sensitivity to changes in resources (i.e., as of time step 12, the performance of the network trained with dynamic resources is comparable to the performance of the network trained with constant resources), while other TTT methods can not conduct test-time training under such dynamic training resource conditions. We believe that this dynamic evaluation you mentioned is both important and practical, and we intend to incorporate these findings into our new paper version to encourage future research. 

**CIFAR10-C (Gaussian_Noise):**

|Dynamic resources|Time-step|0|1| 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 |
|-|-|-|-|-|-|-|-|-|-|-|-|-|-|-|
||FLOPs | 4.1G | 4.1G | 4.1G | 2.3G | 2.3G | 1.1G | 1.1G | 0.3G | 0.3G | 1.1G | 1.1G | 2.3G | 2.3G |
||max width | 1 | 1 | 1 | 0.75 | 0.75 | 0.5 | 0.5 | 0.25 | 0.25 | 0.5 | 0.5 | 0.75 | 0.75 |
||Err. (1.0) | 19.53 | 20.12 | 17.84 | 16.31 | 14.92 | 14.78 | 14.73 | 14.36 | 13.85 | 13.55 | 14.06 | 13.99 | 13.82 |
||Err. (0.75) | 22.27 | 21.29 | 18.75 | 17.19 | 16.02 | 15.76 | 15.74 | 15.33 | 14.84 | 14.49 | 14.95 | 14.81 | 14.78 |
||Err. (0.5) | 25.00 | 23.63 | 20.57 | 19.14 | 17.97 | 17.51 | 16.96 | 16.41 | 15.89 | 15.35 | 15.59 | 15.43 | 15.38 |
||Err. (0.25) | 28.13 | 28.52 | 25.26 | 23.93 | 22.27 | 21.81 | 21.04 | 20.41 | 19.75 | 19.18 | 19.32 | 18.95 | 18.96 |
|**Constant resources** | **Time-step** | **0** | **1** | **2** | **3** | **4** | **5** | **6** | **7** | **8** | **9** | **10** | **11** | **12** |
||FLOPs | 4.1G | 4.1G | 4.1G | 4.1G | 4.1G | 4.1G | 4.1G | 4.1G | 4.1G | 4.1G | 4.1G | 4.1G | 4.1G |
||max width | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 |
||Err. (1.0) | 19.53 | 20.12 | 17.84 | 16.41 | 15.08 | 14.84 | 14.73 | 14.26 | 13.76 | 13.36 | 13.39 | 13.38 | 13.37 |
||Err. (0.75) | 22.27 | 21.29 | 18.75 | 17.09 | 16.02 | 15.76 | 15.68 | 15.14 | 14.54 | 14.06 | 14.24 | 14.16 | 14.24 |
||Err. (0.5) | 25.00 | 23.63 | 20.57 | 18.85 | 17.73 | 17.51 | 17.02 | 16.36 | 15.76 | 15.16 | 15.34 | 15.1 | 15.08 |
||Err. (0.25) | 28.13 | 28.52 | 25.26 | 24.22 | 22.27 | 21.68 | 20.87 | 20.12 | 19.31 | 18.79 | 18.93 | 18.59 | 18.51 |

**CIFAR100-C (Gaussian_Noise):**

|Dynamic resources|Time-step| 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 |
|-|-|-|-|-|-|-|-|-|-|-|-|-|-|-|
||FLOPs | 2.3G | 2.3G | 2.3G | 4.1G | 4.1G | 0.3G | 0.3G | 1.1G | 1.1G | 2.3G | 2.3G | 1.1G | 1.1G |
||max width | 0.75 | 0.75 | 0.75 | 1 | 1 | 0.25 | 0.25 | 0.5 | 0.5 | 0.75 | 0.75 | 0.5 | 0.5 |
||Err. (1.0) | 64.06 | 60.94 | 59.90 | 57.81 | 55.47 | 53.52 | 51.79 | 51.56 | 50.95 | 49.84 | 49.64 | 48.76 | 48.62 |
||Err. (0.75) | 62.50 | 58.59 | 55.73 | 55.47 | 53.75 | 52.34 | 50.45 | 50.29 | 49.65 | 48.59 | 48.22 | 47.27 | 47.12 |
||Err. (0.5) | 63.28 | 59.77 | 58.33 | 58.40 | 56.56 | 54.82 | 52.34 | 52.25 | 51.22 | 50.39 | 50.50 | 49.54 | 49.70 |
||Err. (0.25) | 75.78 | 73.83 | 72.14 | 69.14 | 66.88 | 65.23 | 62.61 | 61.91 | 60.94 | 59.77 | 59.73 | 58.85 | 59.07 |
|**Constant resources** | **Time-step** | **0** | **1** | **2** | **3** | **4** | **5** | **6** | **7** | **8** | **9** | **10** | **11** | **12** |
||FLOPs | 4.1G | 4.1G | 4.1G | 4.1G | 4.1G | 4.1G | 4.1G | 4.1G | 4.1G | 4.1G | 4.1G | 4.1G | 4.1G |
||max width | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 |
||Err. (1.0) | 62.50 | 58.2 | 56.51 | 54.69 | 52.50 | 50.78 | 48.66 | 48.14 | 47.31 | 46.56 | 46.09 | 45.31 | 45.13 |
||Err. (0.75) | 62.50 | 58.59 | 55.73 | 55.47 | 53.75 | 52.21 | 50.22 | 49.71 | 48.96 | 48.12 | 47.73 | 46.81 | 46.57 |
||Err. (0.5) | 63.28 | 59.77 | 58.33 | 58.40 | 56.56 | 54.82 | 52.23 | 51.76 | 50.52 | 49.45 | 49.43 | 48.50 | 48.62 |
||Err. (0.25) | 75.78 | 73.83 | 72.14 | 69.14 | 66.88 | 65.23 | 62.72 | 62.01 | 61.02 | 59.84 | 59.87 | 59.05 | 59.25 |

Reviewer comment (Experiments):
The proposed framework highly depends on the source domain training however, the training cost and the convergence rate comparison for various architecture is missing from the paper. It would be good to add a comparison of training times/ convergence rates of the proposed architecture to make the approach more transparent for real-world deployment use cases.

Author response:
**A2:** Thank you for your suggestions. We provide the results below to show the training times of the proposed architecture (ResNet50, MobileNetV1, MobileNetV2 and ViT-Tiny) and we provide the convergence rate comparison (2000 epochs) for various architecture (ResNet50, MobileNetV1, MobileNetV2) in our new version of supplementary (highlighted in blue).

Training Times per epoch:

| Backbone | Training Time/epoch |
| --- | --- |
| ResNet50 | 173.48s |
| MobileNetV1 | 28.31s |
| MobileNetV2 | 43.07s |
| ViT-Tiny | 202.7s |

Reviewer comment (Presentation):
It would be better to update the caption of Table 3 to highlight the explanation of numbers presented in red for easier readability.

Author response:
**A1:** Thank you for your suggestion. We have modified the caption of Table 3 to “The average error rate (%) results of SlimTTT with different backbones on CIFAR10-C. The numbers presented in red indicate the improvement of our method compared to TTAC”, and have updated it in our paper (highlighted in blue).

Reviewer comment (Presentation):
While making the comparison with existing methods, it would be good to highlight the dependency on the availability of the source dataset. Adding another column for a clear distinction between TTA and TTT approaches will make the paper more transparent.

Author response:
**A2:** Thank you for your suggestion. We will add another column like we show below to highlight the dependency on the availability of the source dataset in Table1 in the future version. “SSL head” in the table means whether we need to train an auxiliary branch in source training phase, “Availability of the source” means whether we can access the source data during test-time training.

|  | BN (TTA) | TTT-R (TTT) | SHOT (Source-free) | TENT (TTA) | TTT++ (TTT) | TTAC (TTT) | SlimTTT (TTT) |
| --- | --- | --- | --- | --- | --- | --- | --- |
| SSL head | × | √ | × | × | √ | √ | √ |
| Availability of the source data | × | × | × | × | × | × | × |

Reviewer comment (Novelty):
It seems that this work is pretty much slimmable network, but just in the setting of test-time training. I don’t see what are the unique challenges in applying slimmable networks in the test-time training? The test-time training seems to be the same as training-time training, but just without labels. Then the problem seems to be how to apply slimmable networks in the un-supervised setting, which has been studied in previous works [1].

Author response:
**Q1-1:** The test-time training seems to be the same as training-time training, but just without labels.

**A1-1:** Good comment. Test-time training (TTT) improves the model’s generalization on specific test data distribution in a way that is beyond the capabilities of training-time training. Since the distribution of the test data can be diverse and different from training data distribution, it is highly difficult for training-time training alone to train a model that can generalize to all possible test data distribution. TTT, on the other hand, provides a solution to enhance the model’s generalization on specific test data distribution. Particularly, TTT takes a training-time trained model, and then updates the model using unlabeled test data to directly adapt it towards the test distribution. The adapted model is finally used for inference. Therefore, TTT is quite different to training-time training, regardless of whether it has training labels or not.

**Q1-2:** Then the problem seems to be how to apply slimmable networks in the un-supervised setting, which has been studied in previous works [1].

**A1-2:** In spite of both using unlabeled data, TTT and unsupervised learning are quite different. The goal of TTT is to improve the supervised learning model’s generalization performance on test data by directly adapting the model towards them.  By contrast, unsupervised learning aims to learn robust representation that serves as a decent starting point for downstream tasks. Beyond this fundamental discrepancy, SlimTTT differs from [1] in methods. Firstly, there are distinctions between our contrastive learning approach and [1]. SlimTTT aggregates the multi-views captured by networks of varying widths and employs a multi-positive NCE loss, whereas [1] follows the standard SimCLR-MoCo paradigm to train each sub-networks. Our SlimTTT better facilitates knowledge interaction among networks of different widths. Secondly, we employ LCR and GFA to additionally promote the consistency of these multi-views, enhancing the networks' ability to handle distribution shifts.

To further demonstrate the superiority of TTT compared to unsupervised/self-supervised pretraining in the scenario of adapting to specific test distribution, we consider the following alternative involving [1]. We replace our supervised training-time training on CIFAR10 (C10) by unsupervised learning on both C10 and C10-C (Gaussian Noise (G_N)) using [1] and then finetune on C10. We do not conduct test-time training for this alternative and directly test on C10-C (G_N). Nevertheless, the results reveal that the method proposed in [1] struggles to handle distribution shifts in this process, leading to subpar performance on C10-C (G_N). Additionally, it's worth noting that our SlimTTT is adaptable to an online TTT setting, a capability not present in the approach proposed in [1].

|Method|Protocol|Training-time training|Test-time training|Inference|Width|Err. (G_N)|
|-|-|-|-|-|-|-
|[1]|Self-supervised|unsupervised|None|on C10-C|1.0|16.06
|||on C10 + C10-C|||0.75|15.44
|||then supervised|||0.5|15.69
|||on C10|||0.25|16.91
|SlimTTT|Test-time training|supervised|unsupervised|on C10-C|1.0| **10.41**
|||on C10|on C10-C||0.75| **11.04**
||||||0.5| **11.80**
||||||0.25| **13.96**

**Q1-3:** what are the unique challenges in applying slimmable networks in the test-time training?

**A1-3:** Simply incorporating slimmable networks into TTT poses several challenges: 1. Different widths of networks may interfere with each other, leading to inconsistent update directions, preventing models from achieving optimal optimization. This issue is more pronounced in TTT due to the lack of labels as well as the distribution shift. Therefore, we propose WCL and LCR to solve this challenge, which promotes the consistency between different width of networks. 2. The GFA is applied to ensure that the test features do not deviate far from the source feature distribution and it can apply a restriction over the multi-view features for each width of networks.

In the following analytical experiment on C10-C, we attempt a straightforward application of slimmable networks in test-time training without employing our consistency method (referred to as naive SlimTTT). Unfortunately, this approach yields unsatisfactory results, particularly when applied in an online manner. This is attributed to the fact that our consistency loss functions—WCL, LCR, and GFA—leverage the diverse views captured by networks of varying widths. This facilitates knowledge interaction among slimmable networks, a crucial aspect in the test-time training setting.

|Method|width|manner|Err. Avg.|manner|Err. Avg.
|-|-|-|-|-|-
|naive SlimTTT|1.0|offline|10.80|online|14.78
||0.75||11.54||15.64
||0.5||12.59||16.72
||0.25||15.99||18.83
|SlimTTT|1.0|offline|**8.33**|online|**10.17**
||0.75||**8.63**||**10.55**
||0.5||**9.12**||**11.41**
||0.25||**10.68**||**14.01**

Reviewer comment (Novelty):
The motivation that different sub-networks could capture different image features has been studied in [2].

Author response:
**Q2:** The motivation that different sub-networks could capture different image features has been studied in [2].

**A2:** Thank you for bringing this to our attention. MutualNet [2] observes that distinct model-resolution configurations can capture varying image features. However, it's essential to note that the observations and the underlying motivation differ between MutualNet and our approach.

MutualNet claims that due to the shared parameters, the sub-networks can learn multi-scale representations, in terms of pairing images of different resolutions with networks of various widths. They supervise the sub-networks by the supernet prediction with largest image resolution. Thus, the knowledge transfer direction is only from supernet to the sub-networks. Unlike MutualNet, our SlimTTT proposes consistency loss function WCL to explicitly combine supernet and other sub-networks to do contrastive learning in feature space, which enables explicit knowledge transfer for both direction.

Moreover, MutualNet identifies that distinct model-resolution configurations, such as 1.0×-224 and 0.75×-128, can emphasize different image features. Specifically, larger networks with higher resolutions tend to focus more on details, while smaller networks with lower resolutions prioritize global structures. It's noteworthy that the image features captured by various networks are influenced by the resolution of the image itself, while the multi-view in our SlimTTT rely solely on the width of sub-networks. 

Furthermore, the concept of multi-view captured by our SlimTTT differs from the image features identified by MutualNet. Drawing on the principles outlined in [3], we demonstrate that sub-networks of varying widths can capture distinct discriminative multi-views of an object, which is supported by visualization experiments and a theoretical toy example. For instance, certain widths of sub-networks might focus on the head of a dog, while others capture the tail of the same dog. These discriminative features are entirely distinct and can both contribute effectively to the classification of a dog. These multi-views are different from the detailed or global features proposed by MutualNet, which hinge on the resolution of the image.

Finally, we conducted the following experiment to demonstrate that the distinct image features captured by MutualNet using different image resolutions are orthogonal to the multi-views we identified. We treat images of different resolutions as diverse forms of data augmentation. In contrast to MutualNet randomly inputting images of a certain resolution into a network of a particular width, each width of network can observe images of all resolutions once a time in our method. This approach increases the views we claimed in our paper, and through our designed WCL and LCR, we promote consistency between these views. As shown in the table below, we achieve better results on ImageNet-C in online manner after adding images of different resolutions. This indicates that MutualNet and our SlimTTT are orthogonal. It's important to note that these results are preliminary, and we plan to delve deeper into the benefits that images of different resolutions can bring to our method in the future.

| Method | width | Gaussian_Noise | Snow | Brightness |
| --- | --- | --- | --- | --- |
| SlimTTT | 1.0 | 61.37 | 42.05 | 31.44 |
|  | 0.75 | 62.55 | 43.78 | 33.03 |
|  | 0.5 | 64.93 | 47.07 | 36.17 |
|  | 0.25 | 70.95 | 54.89 | 42.89 |
| SlimTTT + resolution | 1.0 | 61.17 | 41.74 | 31.50 |
|  | 0.75 | 63.27 | 43.03 | 32.77 |
|  | 0.5 | 64.52 | 49.03 | 35.83 |
|  | 0.25 | 70.05 | 54.37 | 42.77 |

Therefore, both the concept of multi-views and the rationale for their inclusion in our SlimTTT differ from the approach taken by MutualNet.

[3] Towards Understanding Ensemble, Knowledge Distillation and Self-Distillation in Deep Learning.

Reviewer comment (Experiments):
In Table 1, the other methods should also use ResNet-50 with different widths to have a fair comparison.

Author response:
**Q3:** In Table 1, the other methods should also use ResNet-50 with different widths to have a fair comparison.

**A3:** Thank you for your suggestions. We conducted a comparison of our method with other TTT methods using ResNet-50 with varying widths on Cifar10-C and Cifar100-C, and the results are presented in the table below. The findings demonstrate that our method consistently outperforms other TTT methods, attributed to the unique loss we designed to guarantee the consistency of networks with different widths.

| Method | width | Cifar10-C Err. Avg. | Cifar100-C Err. Avg. |
| --- | --- | --- | --- |
| TTT++ | 1.0 | 9.68 | 32.71 |
| TTAC | 1.0 | 9.24 | 31.60 |
| SlimTTT (ours) | 1.0 | **8.33** | **29.36** |
| TTT++ | 0.75 | 10.62 | 34.72 |
| TTAC | 0.75 | 10.08 | 33.90 |
| SlimTTT (ours) | 0.75 | **8.63** | **30.34** |
| TTT++ | 0.5 | 11.47 | 36.42 |
| TTAC | 0.5 | 11.35 | 35.73 |
| SlimTTT (ours) | 0.5 | **9.12** | **32.07** |
| TTT++ | 0.25 | 12.88 | 40.32 |
| TTAC | 0.25 | 13.46 | 40.06 |
| SlimTTT (ours) | 0.25 | **10.68** | **37.71** |

Reviewer comment (Experiments):
This work finetuned the pre-trained slimmable networks on ImageNet and ImageNet-C, what about other works?

Author response:
**Q4:** This work finetuned the pre-trained slimmable networks on ImageNet and ImageNet-C, what about other works?

**A4:** Thank you for pointing it out. In our study, we initially conduct standard training-time training procedure for slimmable networks on training dataset ImageNet. Subsequently, we update/finetune the pre-trained slimmable network on test dataset ImageNet-C **(without labels)** without accessing ImageNet anymore. The full procedure follows the previous works like TTT++ and TTAC. It's important to note that all the methods compared in our paper follow this standardized process to ensure a fair and equitable comparison. We will rephrase this process in our new version of paper to avoid confusions.

Reviewer comment (Evaluation):
In Table 5, why is the training cost is comparable or even faster than TTT++ and TTAC? The proposed method needs to forward and backward multiple times, I am assuming it should be more training expensive.

Author response:
**Q5:** In Table 5, why is the training cost is comparable or even faster than TTT++ and TTAC? The proposed method needs to forward and backward multiple times, I am assuming it should be more training expensive.

**A5:** Good comment. In TTT++ and TTAC, maintaining a queue is necessary for projection alignment to update the self-supervised head (SSH). Additionally, TTAC involves classwise alignment and filtering. Although our SlimTTT needs to forward and backward multiple times, our method doesn’t need to use class alignment or ssh alignment like TTT++ and TTAC, thus the total training cost of our SlimTTT can be comparable with TTT++ and TTAC.

To fairly compare the training cost of our method with TTT++ and TTAC, we conduct experiment on CIFAR10-C (batchsize=8) in NVIDIA Jetson TX2 to veirfy the training cost of different module. The results are shown in the table below, the total training costs of our SlimTTT are comparable with TTT++ and TTAC, which means our SlimTTT is simple but effective.

|  | ssl_con | pre_reg | class_align | ext_align | ssh_align | optimize | total | Err. Avg. |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| TTT++ (ResNet50) | 0.7906s | 0 | 0 | 0.4964s | 0.7467s | 0.0820s | 2.1157s | 9.80 |
| TTAC (ResNet50) | 0.6773s | 0 | 0.6042s | 0.3993s | 0.6173s | 0.0938s | 2.3919s | 8.52 |
| SlimTTT (ResNet50) | 1.1958s | 0.1461s | 0 | 0.6794s | 0 | 0.1207s | 2.1420s | 8.33 |