Paper: Distribution Calibration For Few-Shot Learning by Bayesian Relation Inference (ICLR-2024)

Reviewer comment (Writing):
What is the definition of the relation graph? What are the nodes and edges?

Author response:
***Q1:*** *What is the definition of the relation graph? What are the nodes and edges?*

  ***A1:*** Thank you for raising this issue. In each relation graph in this paper( Summary graph $\boldsymbol{M}$ & Gaussian relation graph $\boldsymbol{A}$ et.al.), **nodes are either base or target class embeddings, and the edges represent the relation intensity between target class and each base class.** Specifically, target class node embedding $N\_{t} \\in \mathbb{R}^{\rm{N}\_{f}}$ is the feature generated by the backbone network( In our paper, we introduce pre-trained Resnet 18 as the backbone network) from a single target image. For the $i$-th base class node embedding $N_{b_{i}} \in \mathbb{R}^{\rm{N}_{f}}$, we first generate the features of all the images in the $i$-th base class through the backbone network, after which we take the average of the features as the base class node embedding.

Reviewer comment (Writing):
What is the range of the mean in Eq. (1)?

Author response:
***Q2:*** *What is the range of the mean in Eq. (1)?*

  ***A2:*** Thank you for pointing out this problem which we overlooked. **We rewrite Eq. (1) as follow to make the expression clearer**:

  > ​ $$\begin{equation}\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\boldsymbol{N\_{b\_{i}}} = mean(\\sum\_{X\_{Y=y\_{i}}}\boldsymbol{f}\_{\operatorname{Res18}} (X\_{Y=y\_{i}}))                             \tag{1}\end{equation}$$

  Since the output of $\boldsymbol{f}_{\operatorname{Res18}} (\cdot)$ is bounded to(0, 1), **the range of the mean in Eq. (1) is bounded to(0, 1).**

Reviewer comment (Writing):
What is a "splicing" of two node embeddings?

Author response:
***Q3:*** *What is a "splicing" of two node embeddings?*

  ***A3:*** Thanks for your question. "Splicing" in our paper represents a simple splice of two embeddings. For example, $\boldsymbol{N_{t}} \in \mathbb{R}^{\rm{N_{f}}}$ and $\boldsymbol{N_{b_{i}}} \in \mathbb{R}^{\rm{N_{f}}}$ are two tensors of the same dimension and $\left[\boldsymbol{N_{t}}, \boldsymbol{N_{b_{i}}}\right] \in \mathbb{R}^{2\rm{N_{f}}}$ represents the splicing of them. We have the first $\rm{N_{f}}$ features in $\left[\boldsymbol{N_{t}}, \boldsymbol{N_{b_{i}}}\right]$ are equal to $\boldsymbol{N_{t}}$ and the last $\rm{N_{f}}$ features in $\left[\boldsymbol{N_{t}}, \boldsymbol{N_{b_{i}}}\right]$ are equal to $\boldsymbol{N_{b_{i}}}$.

Reviewer comment (Writing):
The domains and codomains of \( L \) were not clearly stated.

Author response:
***Q5:*** *The domains and codomains of $L$ were not clearly stated.*

  ***A5:*** Thank you for raising this question. $\mathrm{L}\_{\mathrm{mean}}(\cdot)$ and $\mathrm{L}\_{\mathrm{std}}(\cdot)$ in Section 3.1, page 4, are implemented by **neural networks**. The **input** of each $L$ is $\boldsymbol{E}\_{\mathrm{e_i}} \in \mathbb{R}^{\rm{2N_{f}}}$ which is the edge embedding generated in Eq.(2) in Section 3.1, page 3 and the **output** of each $L$ is a single number. **$\mathrm{L}\_{\mathrm{mean}}(\cdot)$ is a simple linear layer and $\mathrm{L}\_{\mathrm{std}}(\cdot)$ is consist of a linear layer and a Softplus layer.**  Hence the domains of each $L$ is bounded to $(0, 1)$, the codomain of $\mathrm{L}\_{\mathrm{mean}}(\cdot)$ is bounded to ($-\infty, +\infty$) and  the codomain of  $\mathrm{L}\_{\mathrm{std}}(\cdot)$ is bounded to ($0, +\infty$).

Reviewer comment (Experiments):
The main weakness is limited experiments. The task of skin-cancer classification is famous and important. However, there are other datasets concerning ImageNet, Food-101 and so on. If required, we can generate the imbalanced datasets from the original dataset.

Author response:
***I1:*** Thank you for your constructive comment. The proposed Bayesian distribution calibration model is a generic model which can achieve satisfying performance on various datasets. We chose the Dermnet dataset for our experiments due to the fact that the model has a certain degree of interpretability, which is more important for medical scenarios. To further validate the performance of the model, we will experiment the proposed model on other datasets. Due to time as well as arithmetic issues, **we choose the MiniImagenet dataset for the additional experiments**, and the experiment result will be published in the supplementary material, thank you!

Reviewer comment (Experiments):
More generalization and advantages are widely explored in the various fields, and if possible, the conventional algorithms for few-shot learning can be added to baseline algorithms (cannot be fair, but it can provide some implications).

Author response:
***I2:*** Thank you for your suggestions for experiment. Adding the conventional algorithms for few-show learning to baseline algorithms is a fascinating experiment that can provide some implications. **We will compare the performance of the proposed Bayesian distributional calibration method with the conventional Resnet18 model on Dermnet dataset,** and the experiment result will be published in the supplementary material soon, thank you!

Reviewer comment (Writing):
The presentation is not kind for the reader, especially not familiar with few-shot learning and mathematical formulation of variation inference (maybe there are many omissions in the math expression).

Author response:
**I3:** Thank you for pointing this out, which is an oversight in our writing. We made extensive revisions to the Method section (Section 3 *Bayesian Relational Inference* on page 3~7) of the manuscript to allow the reader to understand the model details more intuitively. Specifically, we made the following revisions to the Method section:

  - For some variables, **we indicate their dimension**. The description of the variable is modified to be similar to the following form:

    > $\\boldsymbol{N_{t}} \\in \\mathbb{R}^{\\rm{N_{f}}}$ represents the target node embedding, $\\boldsymbol{N_{b}}\\in \\mathbb{R}^{\\rm{N_{b}} \\times \\rm{N_{f}}}$ represents the base node embeddings

  - **We rewrite the assumptions related to the Bernoulli distribution as well as the computational part of the paper**, and the physical meanings represented by variables are more precise. The revised presentation is similar to the follow form:

    > $\\mathrm{L}\_{\\mathrm{mean}}(\\cdot)$ and $\mathrm{L}\_{\mathrm{std}}(\cdot)$ are implemented by neural networks for estimating the mean and standard deviation of the Gaussian distribution $\mathcal{N}\left(\mu_{i},\ \sigma_{\mathrm{i}}^{2}\right)$ which can calculate the approximation of the Bernoulli distribution $\mathcal{B}(n, \lambda_{\mathrm{i}})$ with $n\rightarrow \infty$ and $\lambda_{\mathrm{i}} \rightarrow 0$

  - **We rewrite the ill-expressed formulas.** For example:

    > ​   $$\\begin {equation} \quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\\widetilde{\\alpha}\_{\mathrm{i}}= \sqrt{(m\_{\mathrm{i}} \times (1.0 - m\_{\mathrm{i}}))} \times \varepsilon\_{\mathrm{i}}+m\_{\mathrm{i}}      \tag{6} \end{equation}$$

  - For the theorems used in the paper we proof them in the **supplementary material A.3.**

Reviewer comment (Writing):
Please clarify the notation $ \mathcal{B}(n, \tilde{\lambda}_i)$ in the equation (16) since there is no definition.

Author response:
**A1:** Thanks for pointing out this issue. We did not clarify this  Bernoulli distribution clearly in the paper previously. This is an assumption for summary graph mentioned in Section 3.1, where we assume that **each edge of the summary graph is a sampling of a Bernoulli distribution $\mathcal{B}(n, \lambda)$ with $n\rightarrow \infty$ and $\lambda \rightarrow 0$.** We **modify the expression in Section 3.1** to explicitly state the assumptions associated with that Bernoulli distribution. Besides, **we explicitly state the basis for conversion to these two KL terms above equation (16) in Section 3.3.** The modifications to Section 3.1 and above for equation (16) in Section 3.3 are as follows, respectively:

  > **In section 3.1:**
  >
  > Hence we assume that the summary graph of the coupling is a sampling of a Bernoulli distribution $\mathcal{B}(n, \lambda)$ with $n\rightarrow \infty$ and $\lambda \rightarrow 0$.
  > 
  > 

  >
  > **In Section 3.3:** 
  > Since each variable in $\mathbf{S}$ is affected by $\tilde{\mathbf{A}}$ in Eq.(6), we have $s\_{i} \mid \tilde{\alpha}\_{i} \sim \mathcal{N}\left(\tilde{\alpha}\_{i} * \mu\_{i}, \tilde{\alpha}\_{i} * \sigma\_{i}^{2}\right)$. Besides, each element in Gaussian graph is conditioned on the Binomial variable for the same edge of the summary graph. Hence The $\mathrm{KL}$ term can be further written as:
  >
  > ​$$\begin{aligned}  &\sum_{\mathrm{i}}^{\rm{N_{b}}}\Bigg \\{ \\mathrm{KL}\bigg(\mathcal{B}(n, \lambda_{\mathrm{i}}) \| \mathcal{B}\left(n, \lambda_{\mathrm{i}}^{(0)}\right)\bigg)    \\\\ \quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad +\mathbb{E}\_{\mathrm{\tilde{\alpha}}\_{\mathrm{i}}}\bigg[\mathrm{KL}\bigg( \mathcal{N}&\left(\tilde{\alpha}\_{\mathrm{i}} * \mu\_{i},\  \tilde{\alpha}\_{\mathrm{i}} * \sigma\_{\mathrm{i}}^{2}\right)) \| \mathcal{N}\Big(\tilde{\alpha}\_{\mathrm{i}} * \mu\_{i}^{(0)}, \tilde{\alpha}\_{\mathrm{i}} * \sigma\_{\mathrm{i}}^{(0)^{2}}\Big)\bigg)\bigg]\Bigg\\}
   \end{aligned}$$

Reviewer comment (Theory):
In equation (15), I cannot understand the $\\sum_{i=1}^M$ since there is not explicit $i$ in arguments of $p$ and $q$. Please clarify this issue.

Author response:
**A2:** Thank you for raising this issue. $M$ is the number of Gaussian relation graphs generated in one batch. We add to our paper the note on the meaning of $M$. Equation (15) is now as follow:

  > ​$$\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad \sum\_{i=1}^{M}\bigg(\mathrm{KL}\left(q\left(\tilde{\mathbf{A}}\_{i}, \mathbf{S}\_{i} \mid \mathbf{N}\_{\mathrm{t}}, \mathbf{N}\_{\mathrm{1}: \mathrm{n\_{b}}}\right) \|  p\left(\tilde{\mathbf{A}}\_{i}, \mathbf{S}\_{i} \mid \mathbf{N}\_{\mathrm{t}}, \mathbf{N}\_{\mathrm{1}: \mathrm{n\_{b}}}\right)\right)\bigg)$$

Reviewer comment (Reproducibility):
Can you clarify the combination strategy of multi-view graphs? Are the strategies different between training and test steps? Can you have other strategies for the combination of multi-view graphs?

Author response:
**A3:**  Thanks for raising these insight questions. Multi-view Gaussian Graph is one of the key component of the proposed model which is used to generate a large number of fusion features for few-shot classification tasks. Our answers to each of the sub-questions are as follows:

  ***(1)*** Inspired by human unconscious relation thinking, each edge in the summary graph is regarded as sampling from a Bernoulli distribution $\mathcal{B}(\,n, \lambda\,)$ with $n\rightarrow \infty$ and $\lambda \rightarrow 0$. However, **the summary graph suffers from the problem of numerical denseness and contains more redundant information.** Existing work points out that the graph generated by human unconscious relation thinking should be sparse (Huang et al. (2020)). Inspired by previous work, our proposed multi-view **Gaussian graph generation strategy generates sparse Gaussian graph** by generating Gaussian variables $s$ associated with the edges of the summary graph and performing a series of Gaussian transformations on it. Since the Gaussian variable contains the **standard Gaussian variable $\epsilon^{'}$**, it is possible to sample $\epsilon^{'}$ multiple times, **generate a large number of Gaussian graphs from multiple perspectives**, and use these Gaussian graphs to generate a sufficient number of fusion features for few-shot classification tasks. **The fusion features of different views generated by multi-view Gaussian graphs can be calibrated to the distribution of the target classes.**

  ***(2)*** The **training process** uses a conventional model, the purpose of which is to **optimize the Bayesian relation inference component** to **ensure that the generated summary graphs correctly represent the relationship between the target class and the base classes**, and therefore **only a single Gaussian graph** and its corresponding fusion feature need to be generated for the conventional classification task. The **validation and testing phase** generates **a large number of Gaussian graphs** as well as fusion features with the aim of training simple logistic regression classifiers for use in few-shot classification tasks. 

  ***(3)*** The multi-view Gaussian graph generation strategy in the proposed model is rather naive, and other ways of Gaussian graph generation can be used, which **may achieve better results**. For example, **a multi-view generation strategy with mask is used so that each Gaussian graph randomly focuses on the relationship between some of the base classes and target class** to generate a more representative Gaussian graph.

Reviewer comment (Experiments):
Considered graphs are limited since there is no connections between baseline classes. What’s your arguments about this problem.

Author response:
**A4:** Thank you for raising this insight question. **We considered whether the relations between baseline classed should be generated or not and ultimately decided not to.** The core idea of the Bayesian relational inference component is to calibrate the distribution of target class based on its input features. **Intuitively, it is only necessary to find the relation between each base classes and the target class to generate fusion features.** Moreover, the **base class features are unchanged** for any input target class. Thus inferring relations between base classes **can be computationally redundant**, and we eliminate the inference of relationships between base classes to **obtain a more lightweight model.**

Reviewer comment (Theory):
The term "Bernoulli distribution" used in the paper is actually the binomial distribution.

Author response:
***I2:*** Thank you for pointing out our clerical error, we have corrected the part of the manuscript that was incorrectly written as "Bernoulli distribution" to "binomial distribution".

Reviewer comment (Writing):
It needs to take some effort to understand the logic in some paragraphs.

Author response:
***I3:*** Thank you for your suggestion for our paper. In previous revisions, we have endeavored to make the formulas in the manuscript less difficult to understand, but some of the mathematical transformations used (e.g., the transform from binomial distribution to Gaussian distribution) could not be recounted in detail due to space limitations in the article. We show these theorems in the supplementary material and marked in the **manuscript** (e.g., *Please refer to the supplementary material A.3.2 for the calculation of this value* in Section 3.3, page 6). **In addition we have made some wording changes to improve the readability of the article.**

Reviewer comment (Presentation):
A previous paper on distribution calibration [Shuo Yang, Lu Liu, and Min Xu. Free lunch for few-shot learning: Distribution calibration. arXiv preprint arXiv:2101.06395, 2021] provides some visualizations of the distribution that can facilitate understanding (Figure 1, Figure 2, and Figure 6). It would be helpful to provide some visualizations.

Author response:
***I4:*** Thank you very much for your suggestion, the article you mentioned is an inspiring paper and we will refer to it to enrich the content of our manuscript and make it more understandable. Due to time constraints, **these changes will be finalized after the rebuttal and uploaded to the anonymous GitHub**.

Reviewer comment (Experiments):
Why does the training phase use a linear classifier, but the inference phase uses a logistic regression classifier?

Author response:
**A1:** Thank you for your constructive question. In the paper, the training process is modeled after conventional deep learning with the aim of optimizing the Bayesian relation inference component. Due to the **high number of classes** in training data (more than 200 classes), the **logistic regression classifier is not as effective as the linear classifier** for this type of problem. Whereas only five-class tasks are required during validation and testing, logistic regression classifier is faster to train. Moreover, existing work (Yang et.al. 2021) indicates that the logistic regression classifier outperforms many other classifiers in distributional calibration tasks.

Reviewer comment (Writing):
What is the prior graph and posterior graph in Equation 15?

Author response:
**A2:** The edges of the post graph are $m_{\mathrm{i}}$, and the prior graph is obtained by fitting the mean and variance of the Gaussian distribution of its edges and sampling them using a linear neural network.

Reviewer comment (Theory):
How is $m_i$ derived in Equation 5?

Author response:
**A3:** We assume that the summary graph of the coupling is a sampling of a binomial distribution $\mathcal{B}(\,n, \lambda\,)$ with $n\rightarrow \infty$ and $\lambda \rightarrow 0$.} However, due to the $n\rightarrow \infty$ of this binomial distribution, we **cannot directly solve for the specific values of the edges** of the summary graph. By **theorem 1 shown in supplementary material A.3.1**, we can use the method shown in the manuscript to derive $m_{\mathrm{i}}$. Specifically, $m_{\mathrm{i}}$ in the summary graph is **sampling of Gaussian proxies for the hypothetical binomial distribution** being used to **compute the KL term** and, by Theorem 1, we are assured that the **error of the computation using this $m_{\mathrm{i}}$ is acceptable.** The following is the definition for $m$ in Theorem 1:

  > according to exist works, we have that: $f_{1}(x)$ attains its minimum on the interval $(0,1)$ and $f_{2}(x)-f_{2}^{*}$ is bounded on the interval $(0, \sqrt{2} / 2-   1 / 2)$, with:
  >
  > $x=m=\frac{1+l-\sqrt{1+l^{2}}}{2}, \text { where } l=\frac{2 \sigma^{2}}{1-2 \mu}$