Paper: A Hierarchical Bayesian Model for Few-Shot Meta Learning (ICLR-2024)

Reviewer comment (Novelty):
The authors claim in the abstract to "propose a novel hierarchical Bayesian model for the few-shot meta learning problem". This is not accurate as the model and variants thereof have been studied extensively in the literature [1-10].

Author response:
**1. The claim of *”A novel hierarchical Bayesian model for FSL”* in the abstract.** 

Thanks for the feedback. We agree that several other papers took hierarchical Bayesian perspectives on FSL. Our contribution lies in a specific parametrization and inference algorithm that allows fast and effective learning of such HBMs in large neural networks. We have adjusted our wording accordingly in the revised paper.

Reviewer comment (Experiments):
The authors claim that a Bayesian treatment of "only a small fraction" of the model's weights "considerably limits the benefits from uncertainty modeling". The authors cite and compare to the conditional neural process [11], which is a conditional model and thus not a Bayesian treatment of the few-shot learning problem, as well as to the attentive neural process [12], which has been shown in [6] to revert to a mainly deterministic approach. Therefore, these methods are not considered suitable baselines to support the authors' claim of improved uncertainty modeling.

Author response:
**3. The claim of "*Bayesian treatment of only a small fraction of the network limits the benefits from uncertainty modeling*". About CNP/ANP baselines to support claims on improved uncertainty modeling.**

With regard to improved uncertainty modeling, we mention the CNP/ANP for their non-Bayesian modeling. In our evaluation on uncertainty calibration (Table 5), we did not compare ours to CNP/ANP but mainly to MetaQDA, a Bayesian model that is state of the art among those that only provide a Bayesian treatment of the output layer (VERSA, Gordon ICLR’19 and DKT Patacchiola’20, etc). The comparison here supports the claim that our Bayesian treatment of the entire network is beneficial.

Reviewer comment (Evaluation):
Most of the experiments are classification tasks with reported accuracy. The reviewer asks the authors to elaborate in which sense these experiments and metrics measure uncertainty quantification.

Author response:
**5. Most of the experiments are classification tasks. Why ECE metric?**

Our main target problem is few-shot meta learning (FSL). And those classification experiments (miniImageNet, tieredImageNet) are recognised as the mainstream benchmarks for FSL. We follow prior Bayesian FSL methods (eg: MetaQDA - Zhang ICCV’21; DKT - Patacchiola NeurIPS’20; Amortised Meta - Ravi ICLR’19) to assess uncertainty quantification via ECE and R-ECE on these standard benchmarks.

Reviewer comment (Evaluation):
The reviewer acknowledges that calibration is measured in Table 6 but maintains that most of the presented experimental evidence is unsuitable to support the central claims.

Author response:
**6. Calibration measures in Table 6.**

Please refer to (Tran et al. 2020) and (Cui et al. 2020) . Together with the test likelihood score, we believe that the ECE score is the mainstream measure for uncertainty calibration (Hospedales et al., 2022; Wang et al., 2020a). We go beyond the evaluations of existing learners that mainly focused on uncertainty quantification in classification (ECE, Tab 5), and also report uncertainty quantification in regression (Tab 6).

Reviewer comment (Evaluation):
The reviewer encourages the authors to provide more experimental evidence for their claims, in ways suitable for measuring predictive uncertainty quality, for example:  
- Results in terms of predictive log marginal likelihood [2,4-6,10]  
- Calibration measurements for further experiments, possibly including reliability diagrams [8]  
- Additional experiments measuring uncertainty quantification, such as more probabilistic regression experiments (like the presented line-sine) or tasks requiring well-calibrated uncertainties for exploration-exploitation trade-offs (e.g., Bayesian optimization or bandits as in [3,6,8]).

Author response:
**7. Provide results in terms of predictive log marginal likelihood.**

As requested by the reviewer, we have done experiments with the SineLine dataset to report the predictive log-marginal-likelihood scores (LMLHD). We try to match the experimental settings from [6] so that the results are comparable. More specifically, we test on 256 tasks, each of which consists of 64 samples with a varying number of context samples (1,2,4,6,8,12, and 16). The number of posterior samples used to compute/approximate the predictive marginal likelihood is 1024. The results are as follows (the higher the better uncertainty quantification and generalisation). As shown, our NIW-Meta has higher scores, especially for larger context size, implying that capturing uncertainty in full model parameters is important for generalisation capability.

| Context size | 1 | 2 | 4 | 6 | 8 | 12 | 16 |
| ------------- |-----|-----|-----|-----|-----|-----|-----|
| (Volpp et al., ICLR’23) | -18.38 | -15.98 | -13.69 | -12.74 | -11.75 | -11.14 | -10.23 |
| NIW-Meta (Ours) | -17.02 | -14.74 | -12.60 | -8.14 | -3.51 | -1.55 | -1.33 |

Reviewer comment (Theory):
The reviewer asks for more details on whether the two-stage optimization approach — (i) reducing the optimization over $(L_{1:N}, L_0)$ to one over $L_0$ by solving for $L_i^*$ with fixed $L_0$, and (ii) solving for $L_0$ — is guaranteed to find the same solution as the original joint optimization.

Author response:
**11. Is the two-stage optimization approach (described in the paragraph before Eq.(7)) guaranteed to find the same solution as the original joint optimization problem?** 

Yes, the two-stage approach guarantees to find the same solution as the original joint problem.  As we mentioned in the paper, this is essentially the same as $\min_{x,y} f(x,y) = \min_x f(x,y^*(x))$ where $y^*(x) = \arg\min_y f(x,y)$ for each given x – by the very definition of the joint minimisation problem.

Reviewer comment (Reproducibility):
The reviewer is interested in more architectural details on how the proposed approach combines with set-based backbones like CNP, as mentioned at the end of Section 3.

Author response:
**12. Some more architectural details about how the combination of the authors proposed approach with set-based backbones like CNP works, cf. end of Sec. 3.**

We basically follow the architectures from (Gao et al. 2022) https://arxiv.org/pdf/2203.04905.pdf
For instance, for the ShapeNet1D case, the networks to which our NIW-Meta was applied are:

* CNP: 
    * Rsupp = enc2( enc1(Xsupp), Ysupp )    where enc1 = 4-layer conv net, enc2 = 2-layer MLP
    * Zsupp = max-pool(Rsupp)
    * Yqry_prediction = dec( enc1(Xqry), Zsupp )    where dec = 3-layer MLP

* ANP: 
    * We replace “max-pool” in CNP by “multi-head-attn( enc1(Xsupp), Rsupp, enc1(Xqry) )”

* C+R (Conv-Net + Ridge regression head):
    * Yqry_prediction = W* @ \phi(Xqry) where \phi() is 4-layer conv + 2-layer MLP, and 
    * W* = argmin_W || W @ \phi(Xsupp) - Ysupp ||^2 + \lambda*||W||^2 (a  closed-form solution)

For further details and other datasets, please refer to (Gao et al. 2022).

Reviewer comment (Evaluation):
Not all results come with error bounds. The reviewer asks how the error bounds of the presented results were computed.

Author response:
**13. Not all results come with error bounds. The boldfaced results. Table 3/4 captions do not say that classification accuracy is reported.**

* Error bars:  Now we have added the SEMs for the results in Table 3 to show the statistical significance. Please see the revised paper for the full results and also the table below for only DINO/s CIFAR. Even though for some tasks there are some overlaps of the error intervals between our proposed approach and the competing methods, overall our approach is superior to the existing methods with statistical significance to some extent.

| CIFAR, DINO/s | 1-shot | 5-shot |
| ------------- |:-------------:| -----:|
| ProtoNet      | $81.1 \pm 0.29$ | $92.5 \pm 0.13$ |
| MetaOpt      | $70.2 \pm 0.22$ | $84.1 \pm 0.27$ |
| MetaQDA    | $77.2 \pm 0.34$ | $90.1 \pm 0.18$ |
| **NIW-Meta (Ours)** |  $\bf{82.8 \pm 0.26}$ | $\bf{92.9 \pm 0.11}$ |

Reviewer comment (Presentation):
The boldfaced results do not always seem to be better within error bounds. The reviewer asks what the authors' "protocol" is for boldfacing results.

Author response:
**13. Not all results come with error bounds. The boldfaced results. Table 3/4 captions do not say that classification accuracy is reported.**

* Error bars:  Now we have added the SEMs for the results in Table 3 to show the statistical significance. Please see the revised paper for the full results and also the table below for only DINO/s CIFAR. Even though for some tasks there are some overlaps of the error intervals between our proposed approach and the competing methods, overall our approach is superior to the existing methods with statistical significance to some extent.

| CIFAR, DINO/s | 1-shot | 5-shot |
| ------------- |:-------------:| -----:|
| ProtoNet      | $81.1 \pm 0.29$ | $92.5 \pm 0.13$ |
| MetaOpt      | $70.2 \pm 0.22$ | $84.1 \pm 0.27$ |
| MetaQDA    | $77.2 \pm 0.34$ | $90.1 \pm 0.18$ |
| **NIW-Meta (Ours)** |  $\bf{82.8 \pm 0.26}$ | $\bf{92.9 \pm 0.11}$ |

 * Boldfaced results: We marked the ones with the largest average values. 

* For Table 3/4: In our revised paper, we have mentioned that they are classification accuracy. Thank you for pointing this out.

Reviewer comment (Presentation):
The captions of Tables 3 and 4 do not state that classification accuracy is reported.

Author response:
**13. Not all results come with error bounds. The boldfaced results. Table 3/4 captions do not say that classification accuracy is reported.**

* Error bars:  Now we have added the SEMs for the results in Table 3 to show the statistical significance. Please see the revised paper for the full results and also the table below for only DINO/s CIFAR. Even though for some tasks there are some overlaps of the error intervals between our proposed approach and the competing methods, overall our approach is superior to the existing methods with statistical significance to some extent.

| CIFAR, DINO/s | 1-shot | 5-shot |
| ------------- |:-------------:| -----:|
| ProtoNet      | $81.1 \pm 0.29$ | $92.5 \pm 0.13$ |
| MetaOpt      | $70.2 \pm 0.22$ | $84.1 \pm 0.27$ |
| MetaQDA    | $77.2 \pm 0.34$ | $90.1 \pm 0.18$ |
| **NIW-Meta (Ours)** |  $\bf{82.8 \pm 0.26}$ | $\bf{92.9 \pm 0.11}$ |

 * Boldfaced results: We marked the ones with the largest average values. 

* For Table 3/4: In our revised paper, we have mentioned that they are classification accuracy. Thank you for pointing this out.

Reviewer comment (Theory):
Please provide more background knowledge for the Normal-Inverse-Wishart model, and why this is justified in an application for this model.

Author response:
The main reason why NIW model is adopted is that it allows a closed-form solution to the approximate episodic optimisation problem due to conjugacy with Gaussian. This is what enables us to achieve fast task-wise adaptation in (episodic) few-shot meta learning. This improved efficiency is part of what allows us to fully apply Bayesian learning to state of the art feature extractors to achieve excellent results, where many existing Bayesian attempts are restricted to relatively toy problems (PMAML, etc), or only a one Bayesian layer within a NNet (VERSA, MetaQDA, etc). 

Regarding the background knowledge, we recommend and refer to K. Murphy's MLPP book (https://probml.github.io/pml-book/book1.html), especially the probability and Bayesian statistics sections (eg, Ch.2, Ch.4.6, and Ch.A.8.3). It is overwhelming for us to make the paper fully self-contained by including all required background materials in it. Instead, we have cited these background materials in the revised paper.

Reviewer comment (Novelty):
This model is similar to hierarchical recurrent VAE, if so, what are the advantages of this model?

Author response:
At first glance models following the Hierarchical Bayesian structures may look similar to each other. But the main difference is specific modeling and approximate inference strategies employed, in particular tailored for the specific FSL problem in this paper. We need fast task-wise adaptation, ie, fast solution for the inner variable variational optimisation, and to this end we came up with novel quadratic likelihood approximation under the NIW conjugate prior-posterior modeling. 

Another novelty is the theoretical analysis, which to the best of our knowledge has not been done for other HBMs.

Reviewer comment (Experiments):
If possible, not only few-shot learning but other related tasks could be tested to validate the generalisation ability of the model.

Author response:
Our approach can, in principle, be applicable to any other problems that involve multiple different data distributions (e.g., client/local data distributions in Federated or distributed learning). But we focus in this paper on the FSL problem mainly, leaving the application to other problems as future work.

Reviewer comment (Experiments):
There are multiple approximations, most importantly the ELBO and the quadratic approximation, in the proposed framework. Though well-motivated, the lack of empirical evolution of the effectiveness of the approximations other than the downstream performance undermines the significance of the work. Especially given that the theoretical analysis in section 5 also relies on the quality of the approximation.

Author response:
> **1. Effectiveness of the multiple approximations, the ELBO and the quadratic approximation, in the proposed framework.**

Both ELBO and the quadratic approximation are essential parts of our approach. ELBO is one of the few approximate methods to make Bayesian inference feasible, and is also used by other Bayesian methods such as VERSA (Gordon ICLR’19). And our quadratic approximation is what distinguishes it from other (hierarchical) Bayesian methods such as Probabilistic MAML (PMAML).

Reviewer comment (Evaluation):
In section 7, the empirical results for some tasks only show average values rather than showing confidence intervals. And some superior performances lack statistical significance.

Author response:
> **2. Empirical results for some tasks only show average values without confidence intervals.**

We now have added the SEMs for the results in Table 3 to show the statistical significance. Please see the revised paper for the full results and also the table in our official comments for all reviewers for only DINO/s CIFAR. Even though for some tasks there are some overlaps of the error intervals between our proposed approach and the competing methods, overall our approach is superior to the existing methods with statistical significance to some extent.

Reviewer comment (Experiments):
Could the author comment on the choice of the parameters introduced by the proposed method, including the number of SGLD iterations, test-time variational inference steps for (13) or (63,64), and the number of test-time model samples?

Author response:
> **3. Choice of the hyperparameters introduced by the proposed method.**

Most hyperparameters are discussed and reported in Sec. D (Impl. details and experimental settings) of Appendix. For the test-time number of VI steps, we use 3 (mini/tiered-ImageNet), 2 (SineLine), and 20 (ShapeNets) iterations. The test-time number of posterior model samples is: 5 (mini/tiered-ImageNet and SineLine) and 2 (ShapeNets) samples.

Reviewer comment (Experiments):
The calibration performance is validated using only a single data set.

Author response:
> **3. The calibration performance is validated using only a single data set.**

Actually, we evaluated calibration on two datasets/tasks: Both miniImageNet (Tab 5) and Sine-Line (Tab 6). We have now also done new experiments to evaluate calibration in terms of the predictive marginal likelihood at test time. Please see our responses to **EN3o**’s Q.3(c).

Reviewer comment (Theory):
Regarding Section 5, what kind of information about the proposed method can we get from the final forms of the theorems, eq. (14) and eq. (15)? For example, can we say anything about the choice of the hyperparameter \Lambda or about the relationship between the quality of the approximation of the posterior distributions and the generalization error bound?

Author response:
> **4. What kind of information about the proposed method can we get from the final forms of the theorems, Eq.(14) and Eq.(15)?**

The theorems say that our hierarchical Bayesian modeling and our approximate posterior family choice (ie, NIW), are sufficiently good enough for provable test performance. That is, our ultimate optimisation objective Eq.(6), once optimised properly, can lead to a model that generalises well to unseen test episodes with the provided certificates.

Reviewer comment (Evaluation):
Regarding Figure 5, first the figures can be misleading because the computational complexities of your method look constant with the common parameter with MAML (which is not, the parameters for only MAML) in the current presentation. NIW-Meta, more precisely SGLD, requires multiple iterations as it skips the first burn-in iterations. How about comparing the actual training times of both methods when they are optimized to show the similar performance?

Author response:
> **5. Comparing the actual training times in Fig. 5.**

Figure 5 is actually already a fair comparison. Figure 5(b) already shows the full actual training times for competing methods. (Due to the far inferior performance of MAML, we are not able to compare the methods at similar performance.) However, we realise that there are some unclear points which may cause misunderstanding. For instance, in [Fig.5 (b) Left], we have two NIW-Meta models with different numbers of SGLD iterations (#SGLD=2 and #SGLD=5). The plots *already include* the time for both the number of burn-in steps, which is 2 burn-in steps in this case, and SGLD iterations (so, more precisely, they run 2+2 and 5+2 SGLD iterations, respectively, and compared to MAML with 1~5 inner iterations). It is our mistake not to mention this clearly enough, and we have clarified this in our revised paper.

Reviewer comment (Presentation):
Though this is potentially a matter of personal preference, the flow of the paper could be improved by putting the toy experiment either before the derivations or after the theoretical analysis. Otherwise, it seems a bit out of place.

Author response:
> **2. Layout of the toy experiment section. Adding a proof sketch for Theorem.**

* **Toy experiment section**: We can move the section after the theoretical analysis section. We will think about which way is the best for presentation.

Reviewer comment (Presentation):
Adding a proof sketch for Theorems 5.1 and 5.2 in the main paper would be helpful even though the full proofs are provided in the appendix.

Author response:
> **2. Layout of the toy experiment section. Adding a proof sketch for Theorem.**

* **Proof sketch for theorems**: We only put the final theoretical results in the main paper due to the lack of space. But we agree with the reviewer that it would be helpful to provide some overview or sketch of the proofs. We will figure out how we can include them concisely should the paper be accepted.

Reviewer comment (Theory):
Could distributions other than NIW be considered for the prior and variational posterior? What are the limitations of using NIW? In other words, what limitations does the choice of the prior impose on the effectiveness of the approach and how does it compare to existing few-shot learning frameworks?

Author response:
> **3. Choice of the NIW distribution. Limitations of using NIW.**

The reason for employing NIW is mainly because it enables closed-form solutions to local episodic optimisation, which is essential for fast task adaptation in episodic FSL.

A Potential limitation of NIW might be that NIW is a single-modal distribution, thus of limited flexibility compared to the unrestricted, say nonparametric distribution families. For example, if mixing classification (object recognition) and regression (pose estimation) episodes which might prefer position invariant and position equivariant feature extraction respectively, then a uni-modal prior might be overly restrictive and multi-modal prior might be preferred. Still, it’s unclear how to perform *efficient* task adaptation with a multi-modal prior. 

**Impact of the NIW assumption compared to other FSL methods**: As we mentioned in Sec.3.1 (Interpretation), those MAML, ProtoNet and Reptile are all shown to be special cases of our framework, even when we followed our NIW prior-posterior family. So there is no substantive limitation compared to existing FSL methods because of our NIW family choice.