Paper: Indeterminate Probability Theory (ICLR-2024)

Reviewer comment (Theory):
What do you want to say in Section 2? It seems that the example does not go beyond classical probability theory, i.e., all definitions, quantities and calculations are consistent with definitions and axioms in classical probability theory.

Author response:
**Q1**: What do you want to say in Section 2? It seems that the example does not go beyond classical probability theory, i.e., all definitions, quantities and calculations are consistent with definitions and axioms in classical probability theory.

A1: The example in Section 2 employs our proposed candidate axiom 3, which serves as a helpful introduction to our theory. Our intention is to provide readers with a seamless transition into our theory by using this example.

Reviewer comment (Theory):
What is new in your indeterminate probability theory? Specifically, it is unclear why eq. (4) must be 0 or 1 in classical probability theory. Could the authors give some references? The authors should give references, clear derivations or rigorous counter-examples when refuting something in classical probability theory, as it is based on rigorous mathematics.

Author response:
**Q2**: What is new in your indeterminate probability theory?

A2: We suggest that you take a look at the simple coin toss example in Appendix B. If you can solve it using other probability theories, then our theory would not offer anything new.

**Q3**: Specifically, I am confused why eq. (4) must be 0 or 1 in classical probability theory. Could the authors give some references? 

A3: The point here is that Eq. (1) in the paper is only applicable when Eq. (4) is 0 or 1.

**Q4**: The authors should give references, clear derivations or rigorous counter-examples when refuting something in classical probability theory, as it is based on rigorous mathematics.

A4: Our proposed equation does not refute anything in classical probability theory. The classical probability Eq. (1) is a special case to our equation.

Reviewer comment (Theory):
How general is your proposed theory? For example, does your theory enable A and Y to be any kind of random variables, and can the two-phase protocol in your theory model any data-generation process? If not, then the generality of your theory should be discussed.

Author response:
**Q5**: How general your proposed theory is? For example, does your theory enable A and Y to be any kind of random variables, and can the two-phase protocol in your theory model any data-generation process?

A5: The generalization of our theory is a conclusion from our proposed new candidate axioms. Therefore, if even a single exception is found, it would not be appropriate to assert that the theory is general.
Moreover, we have applied our theory to thousand dimensional continuous latent spaces without any special requirements for the variables or datasets. That is why we say that the proposed equation is general.

Reviewer comment (Theory):
In page 5 the authors say "...Otherwise, Candidate Axiom 2 and Candidate Axiom 3 cannot both be true". It is strange to discuss the soundness of an axiom once it is proposed.

Author response:
**Q6**: In page 5 the authors say "...Otherwise, Candidate Axiom 2 and Candidate Axiom 3 cannot both be true". In my opinion, it is strange to discuss the soundness of an axiom once it is proposed.

A6: We have a differing opinion on this point. Since the axioms are newly proposed, readers should initially approach them with skepticism. The soundness of new axioms should not only be discussed, but also given the highest priority for verification, as they serve as the foundation for a new theory. (For example, it is wrong to claim naïve Bayes independent assumption as an axiom.)

Reviewer comment (Theory):
The paper is very quick on some technical details while being very verbose on rather basic things such as classical applications of probabilistic conditioning. It is also somewhat cryptic in terms of language and a bit naive about some aspects. For instance, at the top of page 2, it is stated that one cannot apply Bayes’ rule in a continuous setting, which is incorrect as it has been done numerous times. The meaning of “indeterminate” at this point is also obscure. Similarly, it is not clear for the naive Bayes case what exactly is meant by \(P(A^j=a^j_{i_j}|Y=y_l)\) being not solvable. This quantity can certainly be estimated from data, even with noisy observations or untrue assumptions (potentially leading to biased estimates, but still estimable).

Author response:
**Q1**: For instance, P2 top, it is not true that one cannot apply Bayes rule in continuous setting, and it has been numerous, numerous times.

A1: We respectfully wish to remind you that you have misunderstood our point in P2 top. We merely stated that continuous variable is not applicable to Eq. (1), not Bayes rule.

**Q2**: it is not clear for the naive Bayes what exactly means $P(A^j=a^j_{i_j}|Y=y_l)$ being not always solvable? 

A2: The coin toss example in Appendix B is an evidence to this point.

Reviewer comment (Novelty):
It is unclear what the authors really mean by “indeterminate”: does it mean that the observational process is noisy, or that the obtained probabilities are ill-known, requiring one to consider sets of possible probabilities? The paper suggests the first case, yet in such a situation it is not apparent how the proposed approach differs from learning from noisy data where the noise process is known or can be estimated. There is already a huge literature on learning from noisy and/or imprecise data, so the paper should be positioned with respect to that work. If the main idea is to model \(P(y_{obs}=y|y_{true}=y) < 1\) (whether \(y\) is the output value or a feature value) and proceed from there, this is not new. If “indeterminate” means ill-defined, then there is an entire literature on that as well (e.g., following Peter Walley’s work on imprecise probabilities). Claiming to build a new theory of probability should be supported by precise explanations of why previous theories do not address the problem.

Author response:
**Q3**: is it that the observational process is noisy, or that the obtained probabilities are ill-known and hence that one should consider sets of possible probabilities? 
...
if the main idea of the paper is to have $P(y_{obs}=y|y_{true}=y)<1$ ($y$ here can be either the output value or a feature value)?

A3: Your formulation does not align with our theory. Hence, we will adopt your idea to frame our case as follows: $P(y_{obs}=y|x_{k})<1$ and $P(y_{true}=y|x_{k})=1$. Our focus is solely on the indeterminate observation for the $k^{th}$ random experiment. And our objective is to utilize $A_{obs}$ ($P(A_{obs}=a|x_{k})<1$) to infer $y_{obs}$. More details you can find in Q5 of Reviewer MWjN.


**Q4**: Similarly, if indeterminate means ill-defined, then there is a whole literature about that (see, e.g., work following the book on Peter Walley on imprecise probabilities and similar).

A4: Thank you for sharing Peter Walley's imprecise probabilities theory. From our limited understanding, this theory differs from ours in that it represents probabilities using sets or intervals of values, rather than a single point. 

**2nd_Q2**: As said in the review, the word "indeterminate" usually refers to coarse/missing/imprecise aspects, something that is not really discussed here. 

2nd_A2: I did not respond to your general questions in our initial response, because I don't know how to offer a rebuttal to this point. (As you have written 'some for which the authors can offer a rebuttal, mostly when it concerns the content and *not the form of the paper*')

**2nd_Q3**: Third positioning itself from other works that are very close to the presented idea, such as noisy observations (note that noise can be defined instance-wise as well).

2nd_A3: To my understanding, the 'indeterminate' is only a similar word to 'noisy', it is also difficult for me to connect this paper to it. For example, the outcome of a Coin toss example in this paper is interpreted as a Gaussian distribution, this is hard to make a connection to noisy observations.

Reviewer comment (Evaluation):
It is unclear why the current experiments, which merely show accuracy results for standard problems, demonstrate that the theory is “valid.” One would equally question a statistical learning theory or uncertainty theory whose axioms cannot be tested and falsified. All serious theories of uncertainty, especially probabilistic ones, have been subject to falsifiability (the Ellsberg paradox is a good example). Additionally, since Softmax does not have particularly strong theoretical properties, it should not be considered a strong baseline for testing the axioms of a theory.

Author response:
**Q5**: I would equally question a statistical learning theory or more generally an uncertainty theory whose axioms cannot be the subject of tests and falsification?

A5: To deny a new axiom, all you need is to find a single exception, even a simple toy example would suffice.

**Q6**: All theories of uncertainty I know of that are a bit serious in terms of operationally are subject to falsifiability, and this especially true for probabilistic theories (see the Ellsberg paradox for a good example of attempted falsification).

A6: Thank you for sharing the Ellsberg paradox with us. However, this example differs from our case. The reason why Axioms 1, 2, and 3 cannot be falsified is that they have no exceptions.

**Q7**: since Softmax does not enjoy peculiarly good properties from a theoretical perspective, 

A7: We agree that softmax properties are not good enough. In our applications, softmax is only used for IPNN, and we have encountered local minimum problems with this function, as outlined in Appendix E.7. However, our CIPNN and MTS forecasting method rely on Gaussian distribution, not softmax function.

**2nd_Q5**: "Axioms 1, 2, and 3 cannot be falsified is that they have no exception"...so as to know its limits and be open to criticism.

2nd_A5: In my opinion, this is the most important contributions of our paper. If the proposed candidate axioms have exception and can be falsified, it would be not correct to refer to them as axioms. 
Besides, I am open to criticism, I write these independent assumptions as **candidate** axioms, because I think they are very important. And these candidate axioms will remain valid until someone finds an exception.

Reviewer comment (Theory):
My question can be summarized very simply as to whether or not indeterminate probability theory can be expressed in terms of a definition of probability space using abstract probability space.

Author response:
**Q1**: ... if it can be expressed as abstract probability space.

A1: Yes, we will provide a detailed response starting from Q3.

Reviewer comment (Theory):
Since this auxiliary random variable is not needed to describe the system, we will try to eliminate it in some way. In a conventional standard Bayesian analysis, this can be done by eliminating the auxiliary random variable by marginalization. However, a problem arises if the auxiliary random variable is shared by all observers — the system loses observer independence (Axiom 2 of the proposed probability theory) when it is eliminated.

Author response:
As you have mentioned, if the auxiliary (or latent) random variables $\mathbb{A}$ is eliminated through marginalization, the system loses observer independence. On the other hand, if we do not eliminate it, the current probability theory stops at step (b). (Of course, approximation methods or analytical solutions for special datasets can handle it further.)

Reviewer comment (Theory):
The proposed probability theory simply ignores the auxiliary random variable while simultaneously assuming Axiom 2.

Author response:
**Q2**: your own understanding ... the proposed probability theory simply ignores the auxiliary random variable while simultaneously assuming Axiom 2.

A2: We are surprised by your new understanding of this theory, and we agree with you.
We will provide further details to support this understanding.

$$
\underset{\text{(a)}}{\underbrace{{\color{Red}P^{\mathbb{A}}\left ( y_{l}\mid x_{n+1}   \right ) }}}
= \underset{\text{(b)}}{\underbrace{\sum_{\mathbb{A}}P\left ( y_{l},\mathbb{A}\mid x_{n+1}   \right )}}
=\underset{\text{(c)}}{\underbrace{
    \sum_{\mathbb{A}}\left (P\left ( y_{l}\mid \mathbb{A} \right )
\cdot P(\mathbb{A}\mid x_{n+1}) \right )}}
=\underset{\text{(d)}}{\underbrace{
    \sum_{\mathbb{A}}\left (\frac{ {\textstyle \sum_{k=1}^{n}\left ( 
    {\color{Red} P(y_{l}\mid x_{k})}\cdot  P(\mathbb{A}\mid x_{k}) \right ) } } {{\textstyle \sum_{k=1}^{n}P(\mathbb{A}\mid x_{k})  } }
\cdot P(\mathbb{A}\mid x_{n+1}) \right )}}
= \underset{\text{(e)}}{\underbrace{\text{Eq. (15)}}}
$$

From (a) to (b): marginalization.  
From (b) to (c): candidate axiom 3.  
From (c) to (d): candidate axiom 2.  
From (d) to (e): candidate axiom 1.
 
Where superscript $\mathbb{A}$ serves only as an indicator to distinguish the two red terms. For continuous random variables, we change $\sum_{\mathbb{A}}$ to $\int_{\mathbb{z}}$.

As you have mentioned, if the auxiliary (or latent) random variables $\mathbb{A}$ is eliminated through marginalization, the system loses observer independence. On the other hand, if we do not eliminate it, the current probability theory stops at step (b). (Of course, approximation methods or analytical solutions for special datasets can handle it further.)

Reviewer comment (Theory):
Following this intuition, my interest is in what the authors' system would look like if it were represented in an abstract probability space — a situation where all randomness in the world is governed by an abstract space $\Theta$, where all randomness is lost if the abstract space is determined at a point $\theta\in\Theta$, and where all variables can be described deterministically. In this space, random variables are represented as a projection of the world as a map to an object, e.g., $Y(\theta), X(\theta), A(\theta)$ can be uniquely determined for a given source $\theta$. The question is: can the authors' system be represented using such a conventional abstract probability space, or is it a deviation from that rule?

Author response:
**Q3**: a situation where all randomness in the world is governed by an abstract space $\Theta$, where all randomness is lost if the abstract space is determined at a point $\theta \in \Theta$ , and where all variables can be described deterministically.

A3: We like your point and also such kind of high-level discussion. However, we are concerned that the point may have problem at the microscopic level according to 'quantum uncertainty'.  

Our further opinion is that the world can be very good predicted if we have sufficient multivariate latent variables $\mathbb{A}$ (or $\mathbb{z}$).
This can be expressed as:
$$P^{\mathbb{A}}\left ( Y\mid x_{n+1}   \right ) $$

Where $\mathbb{A}$ is latent variables and $Y$ is our interested variables. By utilizing a sufficient number of historical observations of $\mathbb{A}$ adn $Y$, we can use this equation to make a prediction. Although this calculation is very complex, it is of polynomial complexity rather than exponential. Additionally,  this theory can be applied to time series forecasting.