Paper: Learning Conditional Invariances through Non-Commutativity (ICLR-2024)

Reviewer comment (Writing):
The theory part of this work is very difficult to understand.

Author response:
We thank the reviewer for precisely pointing out the theoretical aspects of our work that require further clarification. To this end, we have introduced a number of subsections, primarily in the Appendix, that attempt to address such lack of clarity. Below, we discuss our changes in further detail.

0. **General theoretical clarity:** Through our update, we make the notions of optimality (**Appendix A.3.4, Definitions 3 and 4**) and risk (**Appendix A.3.3**) more explicit, elaborate upon the necessity of introducing the construct of operators and their connection with encoders (**Appendix A.3.2**), as well as formalize the notion of separation on the space of domains (**Appendix A.3.1**). Following the suggestions of Reviewer tufY, we also expand upon the proof of Theorem 2, disambiguate the usage of $\delta$ by introducing $\Delta$ as a separate symbol for inter-domain distance, as well as state the assumption of asymmetry within a separate mathematical environment.

Reviewer comment (Writing):
There is a lack of necessary explanations of notations and definitions. See the questions for details.

Author response:
We thank the reviewer for precisely pointing out the theoretical aspects of our work that require further clarification. To this end, we have introduced a number of subsections, primarily in the Appendix, that attempt to address such lack of clarity. Below, we discuss our changes in further detail.

0. **General theoretical clarity:** Through our update, we make the notions of optimality (**Appendix A.3.4, Definitions 3 and 4**) and risk (**Appendix A.3.3**) more explicit, elaborate upon the necessity of introducing the construct of operators and their connection with encoders (**Appendix A.3.2**), as well as formalize the notion of separation on the space of domains (**Appendix A.3.1**). Following the suggestions of Reviewer tufY, we also expand upon the proof of Theorem 2, disambiguate the usage of $\delta$ by introducing $\Delta$ as a separate symbol for inter-domain distance, as well as state the assumption of asymmetry within a separate mathematical environment.

Reviewer comment (Writing):
In definition 1, what does an operator mean in the context of invariant learning (or domain adaptation)? Can you give an example of an operator in an invariant learning algorithm (e.g., DANN)?

Author response:
2. and 3. **Meaning and example of operators, and their connection with encoders:** We have now added a dedicated subsection, **Appendix A.3.2 (Operator-Encoder Duality)**, that discusses all of these. It first establishes the motivation for introducing the notion of operators into our theoretical framework, and then discusses the connection between the abstract operators and more concrete / practical, real-world encoders (which are essentially neural networks in our case). It accompanies the discussion with examples, both from the paradigm of NCI (non-commutatively invariant operator) and DANN (commutatively invariant operator) to make the ideas more relatable to practice.

Reviewer comment (Writing):
You only give the definition of NCI for operators (in definition 1). What is a NCI encoder $\phi^{\star}$? Again, what does optimal mean in Result 1?

Author response:
1. **Optimality of** $\Phi^*$: We have now concretized all the notions of optimality, i.e., domain-specific ($\Phi^*$) and domain-agnostic ($\varphi^*$) in **Appendix A.3.4, Definitions 3 (Domain-Specific Optimality) and 4 (Optimal-on-Average)**, which details the exact conditions under which an encoder can be deemed as being optimal. From these definitions, it can be seen that the objective function that is minimized is the empirical risk, denoted by $R(\cdot)$, further details about which are provided in Section 3.4 (Training with NCI) of the main paper and Appendix A.3.3 (Risks).

2. and 3. **Meaning and example of operators, and their connection with encoders:** We have now added a dedicated subsection, **Appendix A.3.2 (Operator-Encoder Duality)**, that discusses all of these. It first establishes the motivation for introducing the notion of operators into our theoretical framework, and then discusses the connection between the abstract operators and more concrete / practical, real-world encoders (which are essentially neural networks in our case). It accompanies the discussion with examples, both from the paradigm of NCI (non-commutatively invariant operator) and DANN (commutatively invariant operator) to make the ideas more relatable to practice.

Reviewer comment (Theory):
In Theorem 1, you write $s=\tau + \delta$. But how to do calculations between domain $s$ and $\tau$? You do not give any definition.

Author response:
4. **Calculations in the space of domains:** We have now described in **Appendix A.3.1 (Metric Space of Domains)**, how the discrepancy between two domains can be calculated. In summary, the idea of $H_\eta$-divergence (Definition 5) allows us to quantify the separation between $s$ and $\tau$ as $s = \tau + \Delta$. In practice, this is calculated by the discriminator network $\eta$ that measures the separability of domains in the representation space of the encoder $\varphi$ through optimizing $\mathcal{L}_\eta$ (Equation 2).

Reviewer comment (Writing):
All the risks are not defined.

Author response:
5. **Risks:** We have now introduced **Appendix A.3.3 (Risks)** that explicitly defines the general form of the risks under consideration. In general, we minimize the empirical risk in all cases. Their specific forms depend on the kind of function that is being learned, as stated in Section 3.4 (Training with NCI) of the main paper.