Paper: Efficient Heterogeneous Meta-Learning via Channel Shuffling Modulation (ICLR-2024)

Reviewer comment (Experiments):
What is the impact of the lambda in front of the prototype loss on the model's performance?

Author response:
1. We provide below the results of varying the $\lambda$ parameter in the MRN-GB/MRN-GS objective function. We used the 1-shot 5-way Omniglot dataset and 200 training epochs for each experiment (Note that in the full experiment in our manuscript we used 2000 training epochs, hence the accuracy values reported below are somewhat lower).

| $\lambda$         | 0.2   | 0.5   | 1.0       | 2.0       | 5.0   |
|-------------------|-------|-------|-----------|-----------|-------|
| MRN-GB (Omniglot) | 0.827 | 0.829 | **0.888** | 0.861     | 0.840 |
| MRN-GS (Omniglot) | 0.843 | 0.848 | 0.861     | **0.885** | 0.867 |

The above result seems to suggest that the parameter $\lambda$ cannot be too large, causing the loss function to behave similarly to the ProtoNet loss; or too small, causing the task embedding module to capture unmeaningful patterns.

Reviewer comment (Experiments):
Did the authors experiment with adding routers only in shallow or deep layers of the network? I'm curious about the influence of routers on the model across different layers.

Author response:
2.  In all of our experiments, we used a base architecture with 4 convolution blocks. For our methods, we added one routing layer after every block, hence the routers are distributed at various depths of the network. The influence of the routers at each depth is an interesting experiment to conduct. We have provided some preliminary results below (also on the 5-way 1-shot Omniglot dataset), in which we compared the performance of our methods using all 4 routers vs. the performance of using only one router at different depths (e.g., larger depth means being closer to the output layer). 

| Router Depth      | 1     | 2     | 3     | 4     | All   |
|-------------------|-------|-------|-------|-------|-------|
| MRN-GB (Omniglot) | 0.873 | 0.875 | 0.869 | 0.747 | 0.888 |
| MRN-GS (Omniglot) | 0.851 | 0.867 | 0.835 | 0.805 | 0.861 |

Our result suggests that having the routing layer placed earlier on in the architecture is the most effective practice (e.g., putting a routing layer after the second convolution block gives the best performance). However, combining many routing layers might improve performance, such as in the case of MRN-GB. We will investigate this behavior further in future work.

Reviewer comment (Experiments):
Concerning the complexity introduced by the Router to the model, could the authors provide comparisons in terms of parameter count and training efficiency with HSML?

Author response:
3. Parameter count and training efficiency comparison have been provided in Table 3 of our manuscript.

Reviewer comment (Writing):
The paper is not self-contained and requires prior knowledge of prototypical networks and Gumbel-Sinkhorn layers. A reader not familiar with both topics will have to consult the references for basic definitions of key components of the proposed method (namely $L_{proto}$ and $GS(.)$). A minimal overview of these should have been included in the main body of the paper, or at least in the appendix.

Author response:
We will provide an overview of related concepts in the revision of our paper.

Reviewer comment (Theory):
Even though the method reduces the nominal parameter complexity of the required routing network, it might still be over parametrizing the space by a log factor. Specifically, in order to define an ordering of $C$ channels one should require no more than $C$ values (each with at least $\log_2C$ bits).

Author response:
Thank you for explaining. We assume that you might be talking about the reference "Differentiable Sorting Networks for Scalable Sorting and Ranking Supervision" (Petersen et al., 2021). We now understand and agree that, in the simplest form, each sorting layer needs only maintain a C-dimensional vector to encode a shuffling of C channels. However, in practice, we suppose that extra parameterization would be required to meaningfully learn this vector from data. Nonetheless, this is definitely an interesting alternative that we will try to implement and compare against in future revisions.

Reviewer comment (Theory):
The reviewer suspects that one can derive a (continuous) shuffling using embeddings of size $C$ instead of $C \log C$ by using a sorting network (e.g. Bitonic sort) in a similar fashion as the Benes network is used in this work, but with a different switch type: $(U_j, channel_j)$ pairs are sorted by the key values and deltas between key values (passed a sigmoid) can form continuous switches.

Author response:
Thank you for the interesting suggestion regarding the use of sorting networks. It is true that other sorting networks can be used in place of the Benes routing network for our strategy. This approach would be very similar in spirit to our method. Whereas our Gumbel Benes layer is conditioned on the task embedding vector, a sorting layer would theoretically derive contextual information from learning a value function that scores the convolution channels. Furthermore, a bitonic sorting network will also require $C\log^2 C$ comparators, so we would expect this mechanism to have, at best, similar complexity to our method.

Reviewer comment (Evaluation):
The results of all methods, presented in the Table 1 and Table 2, are usually within their standard deviations. Because of that, I will not support the claim that the presented method is significantly better than others.

Author response:
1. Regarding significance, even though the margin of improvement is not as large as you would prefer, we would like to highlight that our contribution also includes a parameter efficient meta learning formulation, as demonstrated in Table 3.

Reviewer comment (Experiments):
The methods that are chosen for the comparison are not a current state-of-the-art methods, having a few years. I strongly encourage authors to include comparison with other methods like [1] or [2].

Author response:
3. We have compared our method to a lot of benchmarks in the manuscript, one of which (i.e. the URL method) was published very recently in 2022. Due to the limited rebuttal time, we may not be able to provide results to the suggested methods. In particular, we couldn’t find an implementation of reference [1]. On the other hand, FEAT (reference [2]) adopts a Transformer architecture that does not immediately fit with our permutation framework, thus it would require significant changes to provide an apples-to-apples comparison. That being said, we will try our best to provide such results in our future revision.

Reviewer comment (Experiments):
The comparison are considered only across a single neural network architecture. I admit that the scenarios of using the Gumbel-Benes routing layers are limited into CNN architectures. However, there is not the reason to skip the comparison on highly popular in Meta-Learning, ResNet architectures. I will suggest for sure comparison on at least ResNet-10 (as in, e.g., [3]).

Author response:
2. Following your suggestion, we have conducted follow up experiments on the more difficult 1-shot, 5-way Mini-ImageNet setting and its Jigsaw counterpart. We adopt similar training settings to the 5-shot, 5-way experiments in the main manuscript. That is, we trained every baseline for 2000 epochs, sampling 32 training tasks of 16 images per epoch (1-shot, 15-queries). For testing, we sampled 5 test tasks with 1 training and 15 test images per task. Due to the limited rebuttal time, we only focus on comparing with other heterogeneous Meta learning baselines (i.e., MMAML, HSMAML, URL). We have also provided the result using ResNet-18 on these scenarios to answer your other question. The results of these experiments are detailed in the Table below:

| Dataset/Method                   | MRN-GB    | MRN-GS | MMAML | HSML      | URL   |
|----------------------------------|-----------|--------|-------|-----------|-------|
| Mini-ImageNet                    | **0.504** | 0.496  | 0.432 | 0.472     | 0.496 |
| Jigsaw Mini-ImageNet             | **0.480** | 0.464  | 0.432 | 0.408     | 0.448 |
| Mini-ImageNet (ResNet-18)        | 0.547     | 0.526  | 0.541 | **0.555** | 0.470 |
| Jigsaw Mini-ImageNet (ResNet-18) | **0.525** | 0.299  | 0.490 | 0.448     | 0.501 |

Reviewer comment (Experiments):
The lack of more challenging 1-shot 5-way classification on Mini-Imagenet dataset.

Author response:
2. Following your suggestion, we have conducted follow up experiments on the more difficult 1-shot, 5-way Mini-ImageNet setting and its Jigsaw counterpart. We adopt similar training settings to the 5-shot, 5-way experiments in the main manuscript. That is, we trained every baseline for 2000 epochs, sampling 32 training tasks of 16 images per epoch (1-shot, 15-queries). For testing, we sampled 5 test tasks with 1 training and 15 test images per task. Due to the limited rebuttal time, we only focus on comparing with other heterogeneous Meta learning baselines (i.e., MMAML, HSMAML, URL). We have also provided the result using ResNet-18 on these scenarios to answer your other question. The results of these experiments are detailed in the Table below:

| Dataset/Method                   | MRN-GB    | MRN-GS | MMAML | HSML      | URL   |
|----------------------------------|-----------|--------|-------|-----------|-------|
| Mini-ImageNet                    | **0.504** | 0.496  | 0.432 | 0.472     | 0.496 |
| Jigsaw Mini-ImageNet             | **0.480** | 0.464  | 0.432 | 0.408     | 0.448 |
| Mini-ImageNet (ResNet-18)        | 0.547     | 0.526  | 0.541 | **0.555** | 0.470 |
| Jigsaw Mini-ImageNet (ResNet-18) | **0.525** | 0.299  | 0.490 | 0.448     | 0.501 |

Reviewer comment (Novelty):
As far as I know, there is the first example of the Benes networks-inspired Meta-Learning approach. However, there were others works based on the Benes routing, e.g, [4]. I would like to see the Related Work section given credits to that works and comparing the proposed method with them.

Author response:
4. Thank you for your suggestion. We will provide a discussion of other DL works that use the Benes Network in our revision. We however hope you agree that our work is the first that uses this concept in a meta-learning context.

Reviewer comment (Experiments):
Provide results for the 1-shot 5-way classification setting on Mini-Imagenet.

Author response:
2. Following your suggestion, we have conducted follow up experiments on the more difficult 1-shot, 5-way Mini-ImageNet setting and its Jigsaw counterpart. We adopt similar training settings to the 5-shot, 5-way experiments in the main manuscript. That is, we trained every baseline for 2000 epochs, sampling 32 training tasks of 16 images per epoch (1-shot, 15-queries). For testing, we sampled 5 test tasks with 1 training and 15 test images per task. Due to the limited rebuttal time, we only focus on comparing with other heterogeneous Meta learning baselines (i.e., MMAML, HSMAML, URL). We have also provided the result using ResNet-18 on these scenarios to answer your other question. The results of these experiments are detailed in the Table below:

| Dataset/Method                   | MRN-GB    | MRN-GS | MMAML | HSML      | URL   |
|----------------------------------|-----------|--------|-------|-----------|-------|
| Mini-ImageNet                    | **0.504** | 0.496  | 0.432 | 0.472     | 0.496 |
| Jigsaw Mini-ImageNet             | **0.480** | 0.464  | 0.432 | 0.408     | 0.448 |
| Mini-ImageNet (ResNet-18)        | 0.547     | 0.526  | 0.541 | **0.555** | 0.470 |
| Jigsaw Mini-ImageNet (ResNet-18) | **0.525** | 0.299  | 0.490 | 0.448     | 0.501 |

Reviewer comment (Experiments):
Please, compare with state-of-the-art methods as suggested in Weaknesses section.

Author response:
3. We have compared our method to a lot of benchmarks in the manuscript, one of which (i.e. the URL method) was published very recently in 2022. Due to the limited rebuttal time, we may not be able to provide results to the suggested methods. In particular, we couldn’t find an implementation of reference [1]. On the other hand, FEAT (reference [2]) adopts a Transformer architecture that does not immediately fit with our permutation framework, thus it would require significant changes to provide an apples-to-apples comparison. That being said, we will try our best to provide such results in our future revision.

Reviewer comment (Experiments):
The work will be better if you could test the Gumbel-Benes layers in ResNets also.

Author response:
2. Following your suggestion, we have conducted follow up experiments on the more difficult 1-shot, 5-way Mini-ImageNet setting and its Jigsaw counterpart. We adopt similar training settings to the 5-shot, 5-way experiments in the main manuscript. That is, we trained every baseline for 2000 epochs, sampling 32 training tasks of 16 images per epoch (1-shot, 15-queries). For testing, we sampled 5 test tasks with 1 training and 15 test images per task. Due to the limited rebuttal time, we only focus on comparing with other heterogeneous Meta learning baselines (i.e., MMAML, HSMAML, URL). We have also provided the result using ResNet-18 on these scenarios to answer your other question. The results of these experiments are detailed in the Table below:

| Dataset/Method                   | MRN-GB    | MRN-GS | MMAML | HSML      | URL   |
|----------------------------------|-----------|--------|-------|-----------|-------|
| Mini-ImageNet                    | **0.504** | 0.496  | 0.432 | 0.472     | 0.496 |
| Jigsaw Mini-ImageNet             | **0.480** | 0.464  | 0.432 | 0.408     | 0.448 |
| Mini-ImageNet (ResNet-18)        | 0.547     | 0.526  | 0.541 | **0.555** | 0.470 |
| Jigsaw Mini-ImageNet (ResNet-18) | **0.525** | 0.299  | 0.490 | 0.448     | 0.501 |

Reviewer comment (Novelty):
Add the Related Work section regarding other deep learning works utilizing the Benes networks/routing algorithm.

Author response:
4. Thank you for your suggestion. We will provide a discussion of other DL works that use the Benes Network in our revision. We however hope you agree that our work is the first that uses this concept in a meta-learning context.

Reviewer comment (Novelty):
The authors describe the heterogeneous meta-learning problem from the MAML perspective. However, similar to heterogeneous meta-learning, researchers also used the terminology multi-domain/cross-domain to describe the diverse task distribution when dealing with few-shot meta-learning problems, e.g., [R1]. Besides Li et al. (2022), other related works applying feature modulation can also be discussed, such as [R2, R3].

Author response:
Thank you for your suggestion; we will provide a discussion of the suggested related works in our revision.

Reviewer comment (Evaluation):
I am wondering why permutation and channel shuffling work compared with modulation. The intuition of modulation assumes a strong shared backbone, and each dataset/distribution performs task-specific adjustments for its data. For the permutation case, it is better to visualize what permutation can be learned for different distributions.

Author response:
We strongly agree that the intuition of modulation assumes a strong shared backbone, and each dataset/distribution performs task-specific adjustments for its data. However, the degree of adjustments will depend on the heterogeneity of the task distribution. Additive modulations might suffice for task distributions consisting of closely related tasks. On the other hand, more challenging distribution of tasks tend to require more significant modifications, as explored in previous studies of heterogeneous meta learning via modulation. Our hypothesis is that structural modulations, such as a shuffling of the embedded information channels, would be more effective at modulating between tasks of different modalities. In fact, this is not an arbitrary design choice as it is also supported by previous empirical evidence. For example, random shuffling has been used in the context of compact neural architecture due to its efficiency in encoding information.