Paper: Continuous Indeterminate Probability Neural Network (ICLR-2024)

Reviewer comment (Writing):
In the abstract, the phrase "pushed this classification capability to infinity" is unclear.

Author response:
It means that the model is very flexible to classification tasks, we can use the same neural network without changing the number of output nodes to classify arbitrary categories. For example, in Fig. 9, we used two random variables (4 nodes in the last layer) to classify into 2 or 10 categories. And we can also use same model to classify a very large number of categories without limitation.

Reviewer comment (Writing):
The introduction's motivation is based on the IPNN, which is a model the reader is not familiar with at that point in the paper.

Author response:
We apologize for the additional reference to our other paper, which may have disrupted your reading.

Reviewer comment (Writing):
In Section 2.1, the statement "VAE uses neural network as the approximate solution of decoder" is unclear, as in a VAE the decoder is defined as a modeling choice and the encoder approximates the posterior probability.

Author response:
It means that the decoder of VAE is a model, but the decoder of our CIPAE is a probability equation, that means we do not need to design the decoder.

Reviewer comment (Writing):
In Section 2.2, the overall explanation is hard to follow and would benefit from a better example or toy problem (for instance, the classification task from Figure 1).

Author response:
Actually, the toy example in Section 3 is a very good toy example to CIPNN.

Reviewer comment (Writing):
The meaning of "Observers" when "introducing Observers and treating the outcome of each random experiment as indeterminate probability distribution" is unclear, and Observers are not mentioned later in the section.

Author response:
In CIPNN, the label can be understood as Observer 1, the model output can be understood as Observer 3.  
(Observer 2 is the case for IPNN.)  

In addition, we'd like to answer this question from a high-level.  

The world is understood through observers. In the case of a coin toss, the outcome cannot be known unless observed. That is, the ground truth is not able to be known, we only know the observations.  

Given that the use of observers is unavoidable, and the imperfect observations maybe more general in the real world? (Perfect observations are the special case.)  

Additionally, we do not need to restrict the observers, as they may have different ways of understanding the world. For example, in our coin toss example in Section 3, Observer$_3$ understands the outcome as a Gaussian distribution.  

The questions now is: what truly matters?  

The Change! While different observers may have different understandings of the world, the Change still follows the Ground Truth (albeit with some errors). Our proposed equation is designed to evaluate this Change.

Reviewer comment (Writing):
In Section 3, the need for both Observer 2 and Observer 3 is unclear, and the difference between them is not explained. Observer 2 appears irrelevant for the subsequent discussion.

Author response:
In CIPNN, the label can be understood as Observer 1, the model output can be understood as Observer 3.  
(Observer 2 is the case for IPNN.)  

In addition, we'd like to answer this question from a high-level.  

The world is understood through observers. In the case of a coin toss, the outcome cannot be known unless observed. That is, the ground truth is not able to be known, we only know the observations.  

Given that the use of observers is unavoidable, and the imperfect observations maybe more general in the real world? (Perfect observations are the special case.)  

Additionally, we do not need to restrict the observers, as they may have different ways of understanding the world. For example, in our coin toss example in Section 3, Observer$_3$ understands the outcome as a Gaussian distribution.  

The questions now is: what truly matters?  

The Change! While different observers may have different understandings of the world, the Change still follows the Ground Truth (albeit with some errors). Our proposed equation is designed to evaluate this Change.

Reviewer comment (Writing):
In Section 6, the phrase "training strategy of Gaussian distribution" is unclear and needs clarification.

Author response:
It means that using a distribution other than Gaussian, such as a uniform distribution or others, is also acceptable.

Reviewer comment (Experiments):
In Section 7.1, "Simple-Softmax" is compared in Table 3, but it is not defined, and it performs significantly better than the proposed model.

Author response:
Our proposed model is not for a performance improving, as you can see its performance is even poor than the most simple neural networks, so it doesn't need to compare it with other better models.

Reviewer comment (Evaluation):
In Section 7.2, the statement that similar auto-encoder task results "further verify that CIPAE is the analytical solution" is unclear and not justified from simply looking at a figure.

Author response:
CIPAE is the analytical solution is a theoretical analysis results, and the results in Fig. 5 is an evidence.

Reviewer comment (Novelty):
I guess the whole paper should start with a deeper explanation of differences between the proposed approach and a VAE model.

Author response:
**Q1**: Add some introduction to differences between the proposed model and VAE-like approaches

A1: OK. The framework of CIPAE and VAE are very similar, but the two models are derived from different theories.
1. The KL regularization term is an irremovable part in ELBO. Our KL regularization term is additional for over-fitting problem, and it is allowed to use other regularization method for CIPAE.
2. If we use a model as the decoder for CIPAE (not Eq. (1) in the paper), here is the detailed difference:

**VAE**:

$$ ELBO =-D_{KL}(q_{\phi}(\mathbf{z}\mid x^{(i)}) \parallel p_{\theta}(\mathbf{z})) + E_{q_{\phi}(\mathbf{z}\mid x^{(i)})} \left[\log p_{\theta}(x^{(i)}\mid \mathbf{z}) \right] \text{   (a)}$$


**Approximate CIPAE** (Not the original CIPAE):
$$
\underset{\text{(b}}{\underbrace{\log P^{\mathbf{z}}(y \mid x_{t})}}
=\underset{\text{(c)}}{\underbrace{\log \int\limits_{\mathbf{z}}P(y,\mathbf{z}\mid x_{t})}}
=\underset{\text{(d)}}{\underbrace{\log \int\limits_{\mathbf{z}}\left ( P_{\theta}(y\mid\mathbf{z}) 
\cdot P_{\phi}(\mathbf{z}\mid x_{t}) \right )}}
= \underset{\text{(e)}}{\underbrace{\log E_{P_{\phi}(\mathbf{z}\mid x_{t})}
\left[P_{\theta}\left(y\mid \mathbf{z}\right)\right] }}
$$
From (b) to (c): marginalization.  
From (c) to (d): according to axiom 3 that given $\mathbf{z}$, $x$ and $y$ (image pixels) is mutual independent.   
From (d) to (e): the decoder is approximated with a model $\theta$.  

If we set the regularization term of CIPAE to be the same as that of VAE, our loss will be almost identical to that of VAE. The only difference is that the $\log$ term is outside of the expectation term $\mathbb{E}$. For the case where the Monte Carlo number C = 1, the losses of VAE and approximate CIPAE will be exactly the same.

This analysis results are quite interesting, that VAE and approximate CIPAE are derived from very different theory, however their final code implementations can be same in a special case (C=1).

Reviewer comment (Experiments):
An ablation study concerning the complexity is missing. The authors say that the C hyperparameter can be set to 1 “as long as the batch size is high enough…” (page 7, bottom), but still they use C=2 in the experiments.

Author response:
**Q5**: What is the impact of C value and the batch size on quality, trading and inference time, etc.? Could you provide some ablation study?

A5: Thank you for you suggestions, here are our ablation study results on MNIST with 100-D latent space. 

|Monte Carlo\Batch size|1|8|32|64|
|-|-|-|-|-|
|1|85.6|90.1|91.9|92.9|
|2|90.3|94.5|95.5|95.7|
|8|94.9|96.6|97.4|97.5|
|16|95.8|97.2|97.5|97.5|

where regularization factor $\gamma = 0.7$ for the batch size = 1, $\gamma =0.8$ for other tests, forget number $T=500$. And mean value of 3 times tests is reported (Fewer points test only done 1 times due to too slowly.).

As we can see from the above table, we can set a small Monte Carlo number for big batch size.

Reviewer comment (Novelty):
Add some introduction to differences between the proposed model and VAE-like approaches, or perhaps the whole sequence should be accompanied by comparisons to corresponding steps in a VAE-type model.

Author response:
**Q1**: Add some introduction to differences between the proposed model and VAE-like approaches

A1: OK. The framework of CIPAE and VAE are very similar, but the two models are derived from different theories.
1. The KL regularization term is an irremovable part in ELBO. Our KL regularization term is additional for over-fitting problem, and it is allowed to use other regularization method for CIPAE.
2. If we use a model as the decoder for CIPAE (not Eq. (1) in the paper), here is the detailed difference:

**VAE**:

$$ ELBO =-D_{KL}(q_{\phi}(\mathbf{z}\mid x^{(i)}) \parallel p_{\theta}(\mathbf{z})) + E_{q_{\phi}(\mathbf{z}\mid x^{(i)})} \left[\log p_{\theta}(x^{(i)}\mid \mathbf{z}) \right] \text{   (a)}$$


**Approximate CIPAE** (Not the original CIPAE):
$$
\underset{\text{(b}}{\underbrace{\log P^{\mathbf{z}}(y \mid x_{t})}}
=\underset{\text{(c)}}{\underbrace{\log \int\limits_{\mathbf{z}}P(y,\mathbf{z}\mid x_{t})}}
=\underset{\text{(d)}}{\underbrace{\log \int\limits_{\mathbf{z}}\left ( P_{\theta}(y\mid\mathbf{z}) 
\cdot P_{\phi}(\mathbf{z}\mid x_{t}) \right )}}
= \underset{\text{(e)}}{\underbrace{\log E_{P_{\phi}(\mathbf{z}\mid x_{t})}
\left[P_{\theta}\left(y\mid \mathbf{z}\right)\right] }}
$$
From (b) to (c): marginalization.  
From (c) to (d): according to axiom 3 that given $\mathbf{z}$, $x$ and $y$ (image pixels) is mutual independent.   
From (d) to (e): the decoder is approximated with a model $\theta$.  

If we set the regularization term of CIPAE to be the same as that of VAE, our loss will be almost identical to that of VAE. The only difference is that the $\log$ term is outside of the expectation term $\mathbb{E}$. For the case where the Monte Carlo number C = 1, the losses of VAE and approximate CIPAE will be exactly the same.

This analysis results are quite interesting, that VAE and approximate CIPAE are derived from very different theory, however their final code implementations can be same in a special case (C=1).

Reviewer comment (Theory):
Equation (21), being the basis for training, needs a deeper explanation. Why use the max functions both in numerator and denominator?

Author response:
**Q2**: Equation (21), being the basis for training, needs a deeper explanation. Why use the max functions both in numerator and denominator?

A2: Actually, the max functions are not important, because we usually set $\epsilon=10^{-20}$. Therefore, the max functions are not activated for the importantsample space.

Reviewer comment (Evaluation):
When comparing the proposed CIPAE with VAE, section 7.2, the visualizations of the latent space become somehow different, with parts of the R² latent for VAE empty. Does it come from different latent definition in both cases? Or is it just a result of showing only the [-20, 20]×[-20, 20] square? This might not seem to be a fair comparison unless explained.

Author response:
**Q3**: ...the visualizations of the latent space become somehow different, with parts of the R^2 latent for VAE empty.

A3: You can get a sense of the difference by comparing Fig. 4a and Fig. 4b. In Fig. 4a, the effective points are confined to a small space, whereas the valid space of CIPNN is larger, as shown in Fig. 4b.
Similarly, for CIPAE, although it is valid in a larger space, the values of Eq. (22) and Eq. (23) are very small, and the test or train points will not fall within that region.

Reviewer comment (Experiments):
From a practical point of view: what is the training and inference comparison between VAE (as well as other WAE, etc.) approach and the proposed ones?

Author response:
**Q4**: From a practical point of view: what is the training and inference comparison between VAE and CIPAE?

A4: We are afraid that we cannot finished it during rebuttal. Maybe after rebuttal?

Reviewer comment (Experiments):
What is the impact of C value and the batch size on quality, training and inference time, etc.? Could you provide some ablation study?

Author response:
**Q5**: What is the impact of C value and the batch size on quality, trading and inference time, etc.? Could you provide some ablation study?

A5: Thank you for you suggestions, here are our ablation study results on MNIST with 100-D latent space. 

|Monte Carlo\Batch size|1|8|32|64|
|-|-|-|-|-|
|1|85.6|90.1|91.9|92.9|
|2|90.3|94.5|95.5|95.7|
|8|94.9|96.6|97.4|97.5|
|16|95.8|97.2|97.5|97.5|

where regularization factor $\gamma = 0.7$ for the batch size = 1, $\gamma =0.8$ for other tests, forget number $T=500$. And mean value of 3 times tests is reported (Fewer points test only done 1 times due to too slowly.).

As we can see from the above table, we can set a small Monte Carlo number for big batch size.

Reviewer comment (Writing):
In conclusion the authors state that the proposed model is actually composed of two parts: first detects attributes, and the second (i.e., classification?) is a probabilistic model which may be used for reasoning. A 1-D example shown in figure 3, that the authors refer to, shows that the first part performs a kind of clustering. Is that so? Please elaborate on that, since it would greatly enlarge the readability of the paper.

Author response:
**Q6**: In conclusion ...first part is for detects attributes... A 1-D example shown in figure 3, shows that the first part performs a kind of clustering, is that so? 

A6: Suppose we need to classify a large number of images across an extremely large number of categories, and the latent space is multi-dimensional. It becomes important if one random variable can summarize the attributes of the image, for example, the model learned that $z^{1}$ is for color, $z^{2}$ is for shape, and so on. In this way, the large joint space will be more efficient, because each $z$ is for special information. And in our opinion, clustering during classification is kind of summarizing the image attributes.

Reviewer comment (Presentation):
Please, if possible, make the figures a bit larger, just to make them somehow more readable. This refers particularly to figures 1, 3, and perhaps 2 and 4 too.

Author response:
**Q8**: Please, if possible, make the figures a bit larger...

A8: We also want larger figures, but it will make our contents more than 9 pages, this is a regrettable compromise.

Reviewer comment (Experiments):
In section 7.1 you claim that the CIPNN tends to put 1, 4, 7, 9 MNIST numbers in one cluster — this is hardly visible in the figures. How is that model used for classification? Which inputs were used in each round? Could you elaborate on that a bit?

Author response:
**Q9**: In section 7.1 you claim that the CIPNN tends to put 1, 4, 7, 9 MNIST numbers in one cluster — this is hardly visible in the figures. How is that model used for classification?

A9: The model and settings (with different random seed) are the same for each round of classification. 

In addition, we reorganized the results of Fig. 3 with the following table by ignoring the value direction of $z^{1}$.

|Round| Cluster1|Cluster2|
|-|-|-|
|1|1,7,9,4|8,5,6,3,2,0|
|2|7,9,4,1|8,3,5,3,6,0|
|3|7,9,4,1|6,8,2,5,3,0|
|4|7,9,4,1|8,5,6,3,2,0|
|5|7,9,4,1|8,3,5,2,6,0|
|6|1,7,9,4|6,2,8,5,3,0|
|7|1,7,9,4|8,5,6,3,2,0|
|8|7,3,1,9,4|8,5,2,6,0|
|9|7,1,9,4|8,3,5,2,6,0|
|10|1,7,9,4|3,8,5,2,6,0|

As you can see, '1,7,9,4' is clustered $9/10$ times into one cluster.

Reviewer comment (Writing):
Equations are sometimes complex (lots of variables and indices), e.g., sequence from (9) to (14). Could you please make them easier to follow?

Author response:
**Q10**: Equations are sometimes complex (lots of variables and indices), e.g. Sequence from (9) to (14). Could you, please, make them easier to follow?

A10: Thank you for your reminding. However, as you can see, our proposed indeterminate equation in IPNN is also complex to read.

Reviewer comment (Presentation):
I find the motivation of this paper unclear. I had difficulty following the progression from Section 2.2 to Section 3 and then to the CIPNN model.

Author response:
We apologize for the additional reference to our other paper, which may have disrupted your reading.

Reviewer comment (Theory):
The indeterminate probability theory is not surprising to me, and I believe it can be easily derived via the definition of conditional probability. Equations (1) and (2) hold for any $z$, but it's not clear to me which specific type of $z$ we are expecting in the learning process.

Author response:
At the very least, CIPAE has a very special property:: The famous VAE model requires a decoder model, whereas the decoder of CIPAE is a probability equation, and therefore does not require a model. Have you ever seen an image auto-encoder whose decoder does not require a model?

Reviewer comment (Theory):
I find Proposition 1 to be weak from my perspective. Specifically, Proposition 1 states, ''If $P(y_l | z^1, ... , z^N) \to \infty$, CIPNN converges to the global minimum.'' This is equivalent to saying that successful classification depends on our ability to learn a set of favorable variables, namely $z^1, ... , z^N$. However, the main challenge lies in determining the existence of these 'good' variables and how we can identify and obtain such $z^1, ... , z^N$ with theoretical guarantees.

Author response:
Your understanding is correct, the classification depends on our ability to learn a set of favorable variables, namely $z^1, ... , z^N$. Firstly, through experimentation, we can see that our models are able to learn these 'good' variables. Additionally, the main loss in Eq. (15) is an application of maximum likelihood which serves as a theoretical guarantee.

Reviewer comment (Experiments):
I did not find the comparison with existing approaches. Also, the numerical results did not indicates the improved performance is indeed from the introduction of $z^1, ... , z^N$.

Author response:
Our proposed model is not for a performance improving, as you can see its performance is even poor than the most simple neural networks, so it doesn't need to compare it with other better models.