Paper: Enhancing Neural Subset Selection: Integrating Background Information into Set Representations (ICLR-2024)

Reviewer comment (Reproducibility):
Equation 4 describes the neural network construction. However, I am unclear about the objective function to optimize the neural network. Also, after optimization, how do you use this neural network to select a subset?

Author response:
We greatly appreciate the time and effort you have invested. In response to your concerns, we have provided detailed clarifications.
### ***Comments 1: The optimization objective and how to select an optimal subset during inference?***
***ANSWER:*** Thank you for your insightful question, as the construction of an optimization objective and inference process are indeed key aspects of neural subset selection [1, 2, 3]. We aimed to balance detailed explanation with readability for a broad audience, which is why we initially provided a high-level overview in the **Introduction Section**. To specifically address your concerns, we are now including more detailed descriptions of the optimization and inference processes.

Our formulation of the optimization objective is based on the framework established in [1]. Specifically, the optimization objective is to address Equation 1 in our paper by adopting an implicit learning strategy grounded in probabilistic reasoning. This approach can be succinctly formulated as follows:
$$ argmax_\theta\ \mathbb{E}_{\mathbb{P}(V, S)} [\log p _\theta (S^{*}| V)] $$
$$s.t.  p _\theta (S | V) \propto  F _\theta (S ; V), \forall  S \in 2^V, $$

The important step in addressing this problem involves constructing an appropriate set mass function $p_\theta (S|V)$ that is monotonically increasing in relation to the utility function $F_\theta (S;V)$. To achieve this, we can employ the Energy-Based Model (EBM):
$$
p_\theta (S|V) = \frac{\mathrm{exp}( F_\theta (S; V))}{Z}, \; Z := \sum\nolimits_{S'\subseteq V}  \mathrm{exp}( F_\theta (S'; V)),
$$
In practice, we approximate the EBM by solving a variational approximation
$$
    \phi^* =  argmin_{\phi} D(q_\phi(Y|S,V)) || p_\theta (S|V)),
$$
During the training phase, we need an EquiNet, denoted as $Y = \operatorname{EquiNet}(V;\phi): 2^V \rightarrow [0,1]^{|V|}.$ This network takes the ground set $V$ as input and outputs probabilities indicating the likelihood of each element $x \in V$ being part of the optimal subset $S^*$. In the inference stage, EquiNet is employed to predict the optimal subset for a given ground set $V$, using a TopN rounding approach. For detailed information on the implementation and derivation of the aforementioned objective, please refer to [1].

[1] Ou Z, Xu T, Su Q, et al. "Learning Set Functions Under the Optimal Subset Oracle via Equivariant Variational Inference."NeurIPS, 2022.

[2] Tschiatschek S, Sahin A, Krause A. "Differentiable submodular maximization." IJCAI, 2018.

[3] Zhang D W, Burghouts G J, Snoek C G M. "Set prediction without imposing structure as conditional density estimation." ICLR, 2021.

Reviewer comment (Theory):
In equation 4, how do you divide a superset into several subsets? There are an exponential number of combinations.

Author response:
### ***Comments 2: In equation 4, how do you divide a superset into several subsets?***
***ANSWER:*** Theorem 3.5 and Eq.4 are general frameworks to establish the relationship between $Y$ and $(S,V)$. This approach **does not necessitate** dividing a superset into multiple subsets; instead, it requires processing only once for a specific pair of $(S,V).$ In the context of neural subset selection with the optimal supervision (OS) oracle, addressing the variational approximation employs Monte-Carlo (MC) sampling. For a given $V_i,$ we only generate $m$ subsets during training, consistently setting this number to $5$ across various tasks. Consequently, this **eliminates the need for an exponential number of combinations**. It is important to highlight that one of the foremost advantages of Neural Subset Selection in the context of OS Oracle is its ability to **significantly reduce the computational burden** associated with processing an exponential number of $(S,V).$

Reviewer comment (Reproducibility):
What is the number of learnable parameters for each baseline method and the proposed method?

Author response:
### ***Comments 3: What is the number of learnable parameters for baselines and the proposed method?***
***ANSWER:*** In regard to the parameters, we have already done ablation studies in **Table 3** and provided a discussion in **Section 4.4** of our paper. To further demonstrate that the improvements achieved by our method are **not merely due to additional parameters**, we present an additional table  here using the CeleA dataset. This table compares EquiVSet (v1) and EquiVSet (v2) — variants of EquiVSet where we have incorporated a Conv(32, 3, 2) layer and a Conv(64, 4, 2) layer into the EquiVSet backbone, respectively. Detailed descriptions of these backbones are available in Appendix E.2. Notably, despite having the largest number of parameters, EquiVSet (v2) is outperformed by INSET, indicating that **our method's efficacy is not solely parameter-dependent**.

|  | DeepSet | Set-Transformer | EquiVSet | EquiVSet-v1  | EquiVSet-v2 | INSET
|--|--|--|--|--|--|--|
| Parameter |  651181 | 1288686 | 1782680 | 2045080 | **3421592** | 2162181
|MJC| 0.440$\pm$0.006| 0.527$\pm$0.008 | 0.549$\pm$0.005 | 0.554$\pm$0.007 | 0.560$\pm$0.005 | **0.580$\pm$0.012**

Thank you in advance for dedicating your time and attention to our response. We are confident that the clarifications and additional information provided here comprehensively address your concerns. With this in mind, we respectfully and earnestly request that you re-evaluate our work, considering the explanations we have offered.

Reviewer comment (Experiments):
Lack of ablation studies.

Author response:
Thank you for highlighting the absence of ablation studies in our work. Indeed, our method, INSET, **does not introduce any new hyperparameters** to the EquiVSet [1] framework. We use the **exact same** hyperparameters as EquiVSet in all of our experiments, ensuring a **fair comparison**. Meanwhile, INSET can significantly outperform baseline models across various datasets and tasks, demonstrating its **substantial efficacy**.

To further verify the robustness of INSET, we have now conducted ablation studies focusing on the Monte-Carlo (MC) sample numbers for each input pair $\{(V_i, S_i^*)\}$. In the context of neural subset selection tasks, our primary aim is to train the model  $\theta$ to predict the optimal subset $S^*$  from a given ground set $V$.  During training, we sample m subsets from $V$ to optimize our model parameters  $θ$, thereby maximizing the conditional probability distribution $p_\theta (S^* | V)$ among of all pairs of $(S,V)$ for for a given V. In our main experiments, we adhere to EquiVSet's protocol by setting the sample number $m$ to 5 across all the tasks. Even with varying the value of $m$, the results **consistently** demonstrate that INSET **significantly outperforms** EquiVSet. Please note that the performance of EquiVSet is reported by **selecting the best results** achieved after tuning the value of $m$.

| | EquiVSet | m=1 | m=2 |m=5 | m=7 |m=8 | m=10 
|--|--|--|--|--|--|--|--|
| Toys |  0.704$\pm$0.004 | 0.752$\pm$0.006 | 0.753$\pm$0.005 | 0.769$\pm$0.005 | 0.768$\pm$0.003 | 0.767$\pm$0.003| **0.771$\pm$0.004**
| Gear| 0.745$\pm$0.013 | 0.788$\pm$0.015|  0.775$\pm$0.020| 0.808$\pm$0.012 | 0.813$\pm$0.010 | 0.821$\pm$0.015 | **0.846$\pm$0.011**
| Bath| 0.820$\pm$0.005 | 0.821$\pm$0.010|  0.851$\pm$0.008| 0.862$\pm$0.005| 0.874$\pm$0.006 | 0.861$\pm$0.005 | **0.874$\pm$0.003**
| Health| 72.0$\pm$0.010 | 0.749$\pm$0.015|  0.763$\pm$0.012| 0.812$\pm$0.005 | **0.824$\pm$0.008** | 0.808$\pm$0.005 | 0.811$\pm$0.005

Reviewer comment (Experiments):
The proposed method is not evaluated on a wide distribution of datasets.

Author response:
It is important to clarify that our experiments encompass three tasks: product recommendation, set anomaly detection, and compound selection, which involve the processing of tabular data, images, and 3D Cartesian coordinates. Specifically, we conduct extensive experiments on **these tasks using six datasets**. Notably, for the product recommendation task, the datasets consist of 12 categories, effectively representing **12 sub-datasets**.

Moreover, we have also conducted synthetic experiments in Appendix F.1 to assess INSET's capability to learn complex set functions.  Additionally, to provide further evidence of INSET's effectiveness, we have performed set anomaly detection tasks using the CIFAR-10 dataset. We are also incorporating additional filters for compound selection tasks for a wider distribution of datasets. For more information, please refer to Appendix F.3 and F.5 in the revised submission."

|  | Random | PGM | DeepSet | Set-Transformer | EquiVSet | INSET
| :-: | :-: | :-: | :-: | :-: | :-: | :-: |
| CIFAR-10 | 0.193 |  0.450$\pm$0.020 | 0.316$\pm$0.008 | 0.654$\pm$0.023 | 0.603$\pm$0.012 | **0.742$\pm$0.020**
| PDBBind | 0.073 | 0.350$\pm$0.009 | 0.323$\pm$ 0.004 | 0.355$\pm$0.010 | 0.357$\pm$0.005 | **0.371$\pm$0.010**
| BindingDB | 0.027 | 0.176$\pm$0.006 | 0.165$\pm$0.005 | 0.183$\pm$0.004 | 0.188$\pm$0.006 | **0.198$\pm$0.005**

The latest results provide further evidence of INSET's superior performance compared to the baselines. Furthermore, it is worth mentioning that our experimental setup includes a **significantly larger number** of experiments compared to DeepSet (Sec. 4.3) [2] and PGM (Sec. 5.3) [3].

[1] Ou Z, Xu T, Su Q, et al. "Learning Set Functions Under the Optimal Subset Oracle via Equivariant Variational Inference."NeurIPS, 2022.

[2] Zaheer M, Kottur S, Ravanbakhsh S, et al. "Deep Sets." NeurIPS, 2017.

[3] Tschiatschek S, Sahin A, Krause A. "Differentiable submodular maximization." IJCAI, 2018.

Reviewer comment (Theory):
Will similar findings hold if the dataset contains imbalance? If so, what degree of imbalance do the guarantees still hold?

Author response:
INSET is designed to significantly enhance the models' capacity to effectively learn $P(Y|S,V)$ or $F(S,V)$. According to Theorem 3.5, this enhancement **holds true consistently** when the tasks involve modeling the relationship between (S,V) and $Y$. To provide empirical evidence, we conduct additional experiments that demonstrate INSET's consistent superiority over the baselines, even in scenarios with imbalanced ground set sizes. Specifically, we train the model on the two-moons datasets (for detailed information, please refer to Appendix F.1) using fixed ground set sizes of 100, and evaluate its performance on various data sizes ranging from 200 to 1000.

|  | 200  | 400 | 600 | 800| 1000 |
|--|--|--|--|--|--|
|EquiVSet | 0.538 $\pm$ 0.002  | 0.513 $\pm$ 0.003 | 0.482 $\pm$ 0.002 | 0.473  $\pm$ 0.005 | 0.471 $\pm$ 0.003
|INSET |  0.547 $\pm$ 0.002 | 0.518 $\pm$ 0.005 | 0.502 $\pm$ 0.003 | 0.486 $\pm$ 0.002 | 0.485 $\pm$ 0.002

The results clearly show that INSET **consistently enhances** the performance of EquiVSet, regardless of any imbalances.

Reviewer comment (Experiments):
Baselines do not consider the information from superset, but these baselines be improved by adding the invariant sufficient statistic of the superset?

Author response:
Certainly, Theorem 3.5 offers a comprehensive framework for modeling the relationship between $Y$ and $(S,V)$, which is also **applicable to the baselines**. However, integrating this invariant sufficient statistic directly into DeepSet and Set-Transformer presents challenges, as they do not explicitly learn a neural subset function $F(S,V)$. Our method, INSET, has employed DeepSet as its backbone. To demonstrate that Set-Transformer can also derive benefits from INSET, we utilize Set-Transformer as our backbone to showcase this.

|  | Random  |  Set-Transformer | Set-Transformer + INSET
|--|--|--|--|
| Toys |0.083 | 0.625 $\pm$ 0.020 | **0.769 $\pm$ 0.010**|
| Gear | 0.077 | 0.647 $\pm$ 0.006| **0.825 $\pm$ 0.021** |
| Carseats | 0.066 | 0.220 $\pm$ 0.010| **0.230 $\pm$ 0.031**|
| Bath | 0.076 | 0.716 $\pm$ 0.005| **0.862 $\pm$ 0.005**|
| Health |0.076  |0.690 $\pm$ 0.010 | **0.852 $\pm$ 0.009** |
| Diaper | 0.084 |0.789 $\pm$ 0.005 | **0.896 $\pm$ 0.005** |
| Bedding | 0.079 | 0.760 $\pm$ 0.020| **0.885 $\pm$ 0.013** |
| Feeding | 0.093 | 0.753 $\pm$ 0.006| **0.902 $\pm$ 0.004** |

By employing Set-Transformer as our backbone, we enable it to explicitly learn the relationship between $Y$ and $(S,V)$. The empirical results clearly demonstrate a **significant improvement** in performance as a result.

Reviewer comment (Writing):
The presentation of some of the mathematical details needs improvement. In particular, it seems that some of the notations are overloaded (i.e., the same notation is used with different interpretations) or not clearly defined. For example, the notation $S$ appears as a *subset* of the ground set $V$ in the Introduction, but in Section 3.1 (Background), the notation $S$ appears as an *element* of $V$ that takes a matrix form. The relationship between elements $x_i \in \mathcal{X}$ and $S_i$ is not clearly defined either.

Author response:
### ***Comments 1: Relationship Between $x_i, S_i,$ and $V.$***
***ANSWER:*** We regard $V$ as a set composed of $n$ elements, denoted as $x_i$, i.e., $V=\\\{x_1, x_2,..., x_n\\\}$. In order to facilitate the proposition of Property 1, we describe $V$ as a collection of several disjoint subsets, specifically $V = \\\{S_1, \dots, S_m\\\}$, where $S_i \in \mathbb{R}^{n_i \times d}$. Here, $n_i$ represents the size of subset $S_i \subset V$, that is, $S_i = \\\{x_{1_i}, x_{2_i},..., x_{n_i}\\\}.$

Reviewer comment (Writing):
It is not entirely clear what the function value $Y \in \mathcal{Y}$ is really referring to, which also appears without an explicit discussion of its meaning in the Introduction as part of the variational distribution $q(Y|S,V)$. Is $Y \in \mathcal{Y}$ supposed to be the utility function value (which was also introduced with the notation $U = F_{\theta}(S,V)$ in the Introduction)? The confusion arising from notational ambiguity makes the paper less readable.

Author response:
### ***Comments 2: The definition of $Y.$***
***ANSWER:*** The generality of our Theorem 3.1 allows it to be applied to both U=F(Y|S,V) and P(Y|S,V) for different tasks. Specifically, when considering the task of Neural Subset Selection in Optimal Subset (OS) oracles, which involves learning P(Y|S,V), we define Y as a $|V|$ independent Bernoulli distribution, which is parameterized by $Y \in [0,1]^{|V|}$, representing the odds or probabilities of selecting element $x_i \in V$ in a output subset $S$.

Reviewer comment (Writing):
Can the authors clearly define what $Y$ is? The footnote mentions that $Y_i$ is the "probability of element $i$ being selected", but this description is ambiguous.

Author response:
Thank you for your valuable suggestions! We are now revising our manuscript based on your suggestions. To address your questions, we would like to provide the following clarifications:

Firstly, we want to clarify that $Y \in [0, 1]^{|V|}$ represents a set of $|V|$ **independent Bernoulli distributions** rather than a categorical distribution with $V$ classes. Hence, it is not required for the elements of $Y$ to satisfy the constraint $\sum_i Y_i = 1$. Additionally, we define the expression of $q(Y|S,V)$ as follows:
$$
q(Y|S,V) = \prod_{i \in S}Y_i \prod_{i \not\in S}(1-Y_i), Y\in[0,1]^{|V|}.
$$
Next, we would like to explain why $q(Y|S,V)$ can approximate $P(S|V)$. Consider $Y \in [0, 1]^{|V|}$ and $S = \\\{0, 1\\\}^{|V|}$. In this case, $Y$ can be viewed as a stochastic version of $S$ since $Y$ can also be generated as $Y = \\\{0, 1\\\}^{|V|}$ while still satisfying the constraint $Y \in [0, 1]^{|V|}$.

To facilitate comprehension, let us consider an illustrative scenario. Suppose we have a ground set $V = \\\{x_1, x_2, x_3\\\}$, and the optimal subset $S^*$ is $\\\{x_1, x_2\\\}$, which can be represented as $[1, 1, 0]$. Specifically, we define $P(S^*|V) = 1$, indicating that $S^*$ is the correct subset, while for any $S \neq S^*$, we have $P(S|V) = 0$.

Now, let's examine the case when $Y = [1, 1, 0].$ In this situation, we can calculate that $q(Y|S^,V) = 1.$ This implies that $q(Y|S^,V)$ accurately represents the probability of observing $S^*$ given $V$, and it correctly assigns a high probability to the optimal subset.

We hope these clarifications help provide a better understanding of our framework. Once again, we appreciate your constructive suggestions and look forward to further discussion.

Reviewer comment (Theory):
How is the neural network construction in Eq. (4) explicitly related to $p_{\theta}(S,V)$ (or $F_{\theta}(S,V)$)?

Author response:
### ***Comments 4: How is the neural network construction in Eq. (4) explicitly related to $p_\theta(S,V)$ or $F_\theta(S,V).$***
***ANSWER:*** Once neural networks are trained, their outputs become fixed for a given input, such as (S,V). Thus, Eq. (4) represents the explicit structure used to construct models for learning the deterministic function $\theta(S,V)$ (to differentiate it from the utility function U=F(S,V)). Using this function, we can construct the conditional distribution $q(Y|S,V)$ according to Theorem 3.5. Specifically, we employ the Mean-Field Variational Inference (MFVI) method introduced by [1] (Section 3.2) to approximate the distribution $q(Y|S,V)$, referred to as $\psi$ in [1]. 

To prevent overwhelming readers with an abundance of notations and equations, we have deliberately omitted the detailed construction of q(Y|S,V) and the derivation of variational approximation in our paper. This decision was motivated by two factors. Firstly, our theorem and Eq. 4 offer a general framework for modeling the relationship between $Y$ and $(S,V)$, instead of focusing on the neural subset selection tasks. Secondly, in order to ensure clarity of our motivation, we have provided a high-level description of these concepts in the Introduction section. For readers interested in the details of these concepts, we strongly recommend referring to [1] (Section 3) for a more comprehensive understanding. For the implementation details of q(Y|S,V) and $\theta(S,V)$, we suggest consulting our accompanying code located at (./model/modules.py).

Thanks for your time and suggestions again. We would appreciate knowing if you have any additional feedback or suggestions.

[1] Ou Z, Xu T, Su Q, et al., "Learning Set Functions Under the Optimal Subset Oracle via Equivariant Variational Inference."NeurIPS, 2022.