Paper: Learning from Label Proportions: Bootstrapping Supervised Learners via Belief Propagation (ICLR-2024)

Reviewer comment (Novelty):
The proposed approach is complex; a simple framework is desirable.

Author response:
**Ans:** We think our algorithm is a simple 2-Step Iterative algorithm with the Belief Propagation Step and Embedding Refinement Step that are easy to implement in code. In order to be comprehensive, we detail the Belief Propagation step. Given a Ising model, there are several standard packages to perform Belief Propagation like pgMax that we use.

Reviewer comment (Theory):
The proposed approach assumes the case of disjoint bags; it cannot handle non-disjoint bags.

Author response:
**Ans:** Our method is general enough for us to accommodate overlapping bags, the only change would be in the potential terms in the Gibbs Distribution formed, and thus the factor graph would differ from the disjoint case. This does not affect the algorithm procedure but only the outcome as expected, in fact learning non-disjoint bags should be an easier problem intuitively as more information in terms of bag constraints is available to the learner. We only discuss the disjoint bag case in the paper as it is of more practical relevance and is the one previously studied in popular literature as highlighted in Section 2. We reiterate that to run our algorithm on non-disjoint bags requires no code change and it will run out of the box when provided with non-disjoint bags as input.

Reviewer comment (Theory):
The paper should discuss theoretical aspects of the proposed approach. For example, I am interested in the time and space complexity of the proposed approach since the proposed approach has rather complex framework. What are the computational and memory costs of the proposed approach?

Author response:
Ans. We refer the reviewer to Section 6.1 in the main paper and section A.2.1 in the supplement for detailed discussion in the time complexity. With respect to the space complexity, we would like to mention that the only overhead for our approach is for a potential matrix for the factor graph to store the node and pairwise potentials. This only amounts to O(V+E) space complexity as we would have V unary and E pairwise terms. Note that V = data_size, and E = O((num_neighbours + bag_size) * data_size). This is because if two points are not in the K-NN neighborhood of each other and if they don’t belong together in any bag, then there would be no pairwise term corresponding to it as discussed in section 4.1. Hence the space complexity is merely O((num_neighbours + bag_size) * data_size), thus linear.

Reviewer comment (Theory):
The graph structure has a significant impact on the proposed approach. How do you determine the number of nearest neighbors? Is there any theoretical background to determine the graph structure?

Author response:
**Ans:** The graph is an outcome of the bag constraints and the covariate pairwise constraints in the Gibbs Distribution, both of which are given to the learner. The choice of neighbors is a hyper-parameter and we have shown that even using a single Nearest Neighbor suffices in A.1.1.

Reviewer comment (Theory):
As shown in Algorithm 1, the proposed approach uses the iterative computations. Does the proposed approach have a theoretical property to converge? How do you determine the number of iterations, R?

Author response:
**Ans:** We refer the reviewer to Section 4.3 where we define our iterative procedure and choices explicitly. To reiterate, “In principle, we could iterate it several times to refine embeddings progressively but we stop when the new iteration does not clearly improve performance on validation set“. We further refer the reviewer to section A.2.2 in the appendix where we note the performance of the algorithm on different iterations and highlight why we empirically choose iteration 2 as the stopping iteration.

Reviewer comment (Experiments):
Since the proposed approach uses a k-NN graph, it needs a high computational time to construct the graph. In Section 6.1, the paper shows the processing time of only the proposed approach. Is the proposed approach more efficient than the previous approaches?

Author response:
**Ans:** Compared to data setup time which is common to DLLP and our method the overall algorithm time is not that significant. The training, data setup times are identical, the only overhead is the BP time. Moreover, the time taken for the kNN graph construction and the required pre-processing is merely **80.49s** (+- 6.07s) for the Adult Dataset (barely a minute for 1 nearest neighbor, for a ~50k sized dataset), which is almost 10 times smaller than the common data setup time and comparable to the training time for the MLP (**85s**). It is also in the same range as the BP time for 128 sized bags, and much faster than BP for larger bags. Thus the graph construction for kNN does not need high computational time and is not a bottleneck. As we have highlighted in section A.1.1, 1NN suffices to retrieve the majority of the performance Thus we are able to achieve a superior performance over the baselines with only additional 80s required for obtaining the neighbor graph to form our covariate factors before Belief Propagation.

Reviewer comment (Evaluation):
Lack of explanation/intuition about results. Are there any hypotheses as to why the results of your method are worse in cases with small bag sizes but better in cases with large bag sizes?

Author response:
**Ans.** The performance of DLLP on small bags is the result of relatively higher supervision available as compared to larger bags where the number of bags can be as few as 20! (so just 20 label counts in all for the dataset) In presence of the high supervision provided by small bags, even other algorithms are able to perform comparable to our method. In case of small bag sizes our performance is not necessarily worse but in the same ballpark, within the confidence intervals, and is really close to the instance wise performance. So there is much less margin for improvement in the case of small bags as compared to large bags (*~1% for bags of size 8 vs ~10% for bags of size 2048* for Adult dataset). We would like to reiterate that even on small bags sizes we still are competitive with, if not better than, all baselines across all datasets.

On large bags where the other methods collapse due to extremely weak supervision, our novel algorithm still holds up.

We further provide an approximate Linearized BP analysis that shows that for large bags regime for Adult, inverse temperatures chosen actually yield convergence for the Linearized BP, in a **new Section C** in the updated manuscript. We feel this analysis of the theoretical guarantee of the algorithm’s performance helps provide the requested theoretical backing for our strong empirical results, especially on Larger Bags.

Reviewer comment (Novelty):
Lack of discussion about work from the field of weak supervision, where there have been similar problems studied in the context of combining weak supervision (labels similar to aggregate labels over bags) and covariate information via clustering [1] and via label propagation [2]. In both cases, a model is trained after pseudolabels are generated (although no iterative refinement is done as these methods start from pretrained representations and supervised models are directly fit on the pseudolabels).

Author response:
**Ans.** We have comprehensively cited all prior works in the LLP domain and compared against several strong baselines. We’ll be happy to cite and discuss other weak supervision works, but they don’t necessarily directly apply to our setup as we have a specific form of weak supervision in the form of bag level labels. Heeding the reviewer's suggestion, we have updated our manuscript’s section B in the supplement to include additional discussion on Weak Supervision and would request the reviewer to take a look at the same.

Reviewer comment (Writing):
A few typos that I noted (that don’t overall affect my score):  
* Last line of page 7: “the iteration seem to help improve performance”, should be “iteration seems to help improve performance”  
* “Bag constraints” in section 6.1 shouldn’t be capitalized

Author response:
**Ans.** We thank the reviewer for pointing out these typos and have corrected them in the revised version of the manuscript. All the changes are highlighted in blue text.

Reviewer comment (Writing):
The writing of this paper needs great improvement, e.g., the connection between the LLP problem and the parity checks should be clearly elaborated. The intuition behind this should be clarified.

Author response:
**Ans.** We would be happy to elaborate on the connection between the LLP problem and the parity check paradigm. In error correction, parity check reinforces even parity in a ‘bag’. In our case, we want to make sure it equals the bag level aggregate labels. Further, we also have pairwise constraints (parity of size 2) with nearest neighbors that impose similarity. In coding theory constraints are enforced in a hard manner. However, using the Gibbs measure (an energy function), we impose a soft version of these constraints.

Reviewer comment (Novelty):
The bag-level and instance-level constraints are commonplace, making the novelty quite limited.

Author response:
**Ans.** As far as we are aware, no one has attempted to utilize the constraints in such a way (formulating a Gibbs distribution for an Ising Model, performing BP) for the LLP problem, achieving a practical and scalable solution, thus we feel this is quite a novel approach to incorporate those constraints.

Reviewer comment (Experiments):
The experiments were only conducted on Binary Classification problems, while the multi-class classification is a more general case.

Author response:
**Ans.** We would like to emphasize that the binary classification LLP problem is of importance and such datasets and applications can be found in healthcare or advertising (conversion modeling). Please additionally refer to [1] below for a detailed discussion on the significance of the problem and the rising interest in the domain, especially due to privacy constraints. In fact we have demonstrated our techniques on Criteo which is a large and standard dataset in the advertising domain. Prior works like references *(Ardehaly & Culotta, 2017; Saket et al., 2022; Scott & Zhang, 2020; Busa-Fekete et al., 2023; Patrini et al., 2014)* in the manuscript do deal with only the binary LLP problem. 

[1] O'Brien, Conor, et al. "Challenges and approaches to privacy preserving post-click conversion prediction." arXiv preprint arXiv:2201.12666 (2022).

Reviewer comment (Theory):
The formulation of the equations should be carefully checked, e.g., in Eq.(1), according to the definition of $y(S_i)$ (the third row in Section 3), the first term equals 0. Besides, the derivation from Eq.(1) to Eq.(2) should be provided for clear understanding.

Author response:
**Ans.** We would like to emphasize that instance level labels are not available. Only bag level aggregates are available. Thus the usage of $y_j$ in the section 3 line 4 corresponds to true instance labels, which are not available to the learner. On the other hand, the second usage of $y_j$ in the Gibbs Distribution is for the predicted labels which are the ones on which the constraints are imposed softly via the least squared loss. We have updated the manuscript to clarify this and replaced the first usage of $y_j$ by $y^{true}_j$ in section 3, line 4.

We would like to emphasize that Gibbs distributions impose constraints softly on instance wise predictions with a least squares loss between the sum of predicted labels and the true bag level sum (first term reviewer is referring to). Configurations that will have 0 loss will have low energy due to that term. Note that any configuration that matches the bag level sum will have zero loss; not necessarily the true one. Also, there could be configurations that may have non zero loss for that term but are preferred due to optimizing other constraints by our algorithm.

Reviewer comment (Evaluation):
The proposed method performs well on large bag sizes, but the authors do not give explanations why this is the case.

Author response:
**Ans.** The performance of DLLP on small bags is the result of relatively higher supervision available as compared to larger bags where the number of bags can be as few as 20! (so just 20 label counts in all for the dataset) In presence of the high supervision provided by small bags, even other algorithms are able to perform comparable to our method. In case of small bag sizes our performance is not necessarily worse but in the same ballpark, within the confidence intervals, and is really close to the instance wise performance. So there is much less margin for improvement in the case of small bags as compared to large bags (**~1% for bags of size 8 vs ~10% for bags of size 2048** for Adult dataset).

On large bags where the other methods collapse due to extremely weak supervision, our novel algorithm still holds up.

We further provide an approximate Linearized BP analysis that shows that for large bags regime for Adult, inverse temperatures chosen actually yield convergence for the Linearized BP, in a **new Section C** in the updated manuscript. We feel this analysis of the theoretical guarantee of the algorithm’s performance helps provide the requested theoretical backing for our strong empirical results, especially on Larger Bags.

Reviewer comment (Evaluation):
According to the results of Tables 4 and 5, the running time of the proposed method is far more than that of DLLP, besides, the running time of other methods is not reported.

Author response:
**Ans.** We observed the time performance of other baselines, but only provided DLLP as an indicator. We have further provided the numbers for EasyLLP and GenBags on Adult and Criteo Datasets, for the training time for both algorithms, thus appending 2 columns to Table 5 Section A.2.1 and Table 4 in Section 6.1, and reinforcing the marginal overhead our method incurs. We reiterate that the Data Setup Time (Column 4) is common to all baselines and methods as well as it comprises the data loading and preprocessing times, which is identical for all methods including ours (which we have made more clear in the updated version of the manuscript). As one can see, GenBags takes more time to train the same MLP as our method (up to 5 times more on Adult and 3 times more on Criteo), and with much higher std. dev. And thus GenBags are much more time consuming, with similar or worse performance. Also, on Criteo, MLP training time is more for EasyLLP than for our method. We have updated Table 5 in the manuscript to include the 2 additional baseline’s training time as well.

We reiterate that the only additional overhead as compared to the baselines is the BP step which for large bags gives enormous performance gains. We have also updated the manuscript to make this clear.

Reviewer comment (Theory):
Why do you introduce the Gibbs distribution in the modeling? The reasons should be detailedly clarified.

Author response:
**Ans.** Gibbs distribution is a joint distribution over the unknown labels. By adding terms that enforce bag constraints softly and nearness constraints softly, we enforce the constraints on the label distribution. Then, we marginalize this joint distribution using BP to obtain pseudo labels which are used for the supervised learning step.

Reviewer comment (Experiments):
Are the instances in each bag all labeled? If yes, then the bag level counts will equal the size of the bag.

Author response:
**Ans.** No. The main setup is that we only have bag level labels, but have no access to any instance label in any bag. Hence, each bag has only one label which is equal to the mean of all instance-level labels in the bag. But we would like to train a classifier to predict instance labels on the test, thus we formulate a Gibbs Distribution and account for the uncertainty via enforcing the constraints softly. Not even one instance of the bag has labels available to the learner during training, so it’s not clear what the reviewer’s doubt is.

Reviewer comment (Experiments):
Is the size of each bag in this paper equal? What if the sizes of each bag are not equal?

Author response:
**Ans.** Yes, while we deal with constant sized bags, the algorithm is directly applicable to the case of variable bag sizes, as only the potential terms would change in the Gibbs Distribution, and thus a different factor graph, leading to a different solution as expected. We need no algorithm change to incorporate datasets with variable bag sizes, and follow the fixed bag size setup following popular literature discussed in Section 2. We reiterate that to run our algorithm on variable sized bags requires no code change and it will run out of the box when provided with variable sized bags as input.

Reviewer comment (Evaluation):
According to the results of Tables 1-3, it is weird that the performance of DLLP (published in 2017) is better than that of GenBags (published in 2022) and EasyLLP (published in 2023). Please explain these.

Author response:
**Ans.** It is true that in many cases, DLLP outperforms many results published later empirically. However, we would like to point out that GenBags, whose novelty as per the authors is “provable error bounds while being bag distribution agnostic and model agnostic” has non trivial performance in larger bag size regimes, as seen in Table 3 for CIFAR. EasyLLP is as the author's quote “a flexible and simple-to-implement debiasing approach based on aggregate labels, which operates on arbitrary loss functions” is highly competitive on smaller bag sizes on UCI Datasets, as seen in Table 1. EasyLLP and GenBags in themselves are interesting approaches to tackle this problem and have been published at prestigious conferences (NeurIPS and AISTATS respectively) and thus have been considered relevant and important by the community.  

We hope this clears all the doubts the reviewer had. We are happy to discuss or clarify further questions as well.

Reviewer comment (Evaluation):
The proposed algorithm is slower than other algorithms it is compared to.

Author response:
**Ans:** We would like to emphasize that the running time of Belief Propagation is only linear in (bag_size + num_neighbours), and even for a dataset with 1 million samples, total run time for 128 sized bags is merely 106 minutes, where the MLP time is akin to the baselines with 10 minutes. On bag size 2048, for an adult dataset with 50k instances, it only takes 17 mins, which is negligible for the performance improvement obtained.

Also, we would like to emphasize that the Data Setup time (Col 4) reported in Tables 4 & 5 includes time taken for data loading, model initialization etc. which would also contribute to DLLP total time. So DLLP total time is actually DLLP training time (Col 1) + Data Setup time for DLLP. As can be seen in the tables, DLLP training time (Col 1) is comparable to Ours - MLP training time (Col 4) as expected. So the only overhead of our method over DLLP is the BP time (Col 3) which, as clarified above, is linear in (bag_size + num_neighbours). We further refer the reviewer to Section 6.1 regarding time complexity of BP step.

Reviewer comment (Writing):
Can you provide some intuition into the architecture of the network g_L? For example what is the function of the average pooling and how it is applied.

Author response:
**Ans:** The intuition is to obtain a single representation for the “bag”, the aggregate embedding through average polling of the instances in the bag is used to train on the bag label as in a usual supervised learning setup. As we highlight in Section 6.3, the aggregate embedding helps reinforce the bag constraint and amongst various kinds of pooling that can be done, mean pooling works the best consistently. The intuition is to simply obtain an “average” combined embedding representing the entire bag, and to obtain this one must utilize ways to combine individual embeddings of the instances comprising the bag into one single embedding. Usage of attention is one technique as we try by doing PMA as described in Section A.5. The average pooling function that our ablations from Section 6.3 point to being the best is mean pooling that is obtained by simply taking a mean of the all the instances of the bag to output one aggregate embedding for the bag via the following tensorflow code:

` pooled_embed = tf.reduce_mean(instance_embed, axis=1) `

Reviewer comment (Theory):
This is more of a comment: BP is supposed to be more precise on sparse factor graphs. Your factor graph is not sparse due to the term with $\lambda_b$ coupling all the variables within one bag. Do you think there is a way to modify the Gibbs measure to keep the desired properties but having a sparse factor graph?

Author response:
**Ans:** It is also precise on forests, we also prove that the 1NN section of the factor graph is tree like in sec A.3 but the bags do introduce cycles. We can’t prove convergence to true marginals due to the presence of cycles in general. However, the approximation via Loopy BP is good enough as we demonstrate empirically.

We would like to refer the reviewer to Section A.3 in the appendix where we provide some intuition as to why our algorithm works via showing that the nearness constraints through 1-NN produces a cycle free factor graph. However, when bags are randomly formed, invariably cycles are formed. Starting from the classical paper (Frey et al., 1997) to the very recent works on message passing (Newman, 2023), loopy belief propagation does not have convergence guarantees in general. In fact, the former points out the empirical success in decoding error correcting codes even on loopy graphs which precisely inspired us.

We further provide an approximate Linearized BP analysis that shows that for large bags regime for Adult, inverse temperatures chosen actually yield convergence for the Linearized BP, in a **new Section C** in the updated manuscript.

We cannot easily see a way to create a cycle free graph with bag constraints.

Reviewer comment (Reproducibility):
Can you comment on the convergence of the BP iterations? Did BP converge? did the convergence time change with the size of the training set? Did you use some trick to make it converge?

Author response:
**Ans:** BP usually converges in 200 iterations. We did not explicitly use any trick to converge the BP. For larger sizes of the training set, it does take more iterations to converge than for a smaller training set but we discover that 200 iterations suffice across all datasets.

We hope this clears all the doubts the reviewer had. We are happy to discuss or clarify further questions as well.

Reviewer comment (Theory):
In the setup section (section 3, p3), it lacks the assumptions and descriptions on the data distribution (x,y), and bag distribution.

Author response:
**Ans.** We would like to reiterate that we do not work under any specific assumptions for the data. We only assume that the instances in each bag are sampled randomly without replacement. As mentioned in section 5, we follow the standard practice in popular LLP literature *(Chen et al, 2023; Ardehaly & Culotta, 2017; Patrini et al, 2014; Busa-Fekete et al., 2023)* of creating disjoint random bags where we sample instances without replacement from the training set, and keep repeating this for each bag, bag-size: k number of times. We are solving for the most general case of bag construction. Thus, there is no distributional assumption on the dataset or bagging.

Reviewer comment (Theory):
For example, for distributions, the proposed algorithm may not work and/or could not converge.

Author response:
**Ans.** We would like to reiterate that we do not work under any specific assumptions for the data. We only assume that the instances in each bag are sampled randomly without replacement. As mentioned in section 5, we follow the standard practice in popular LLP literature *(Chen et al, 2023; Ardehaly & Culotta, 2017; Patrini et al, 2014; Busa-Fekete et al., 2023)* of creating disjoint random bags where we sample instances without replacement from the training set, and keep repeating this for each bag, bag-size: k number of times. We are solving for the most general case of bag construction. Thus, there is no distributional assumption on the dataset or bagging.

Reviewer comment (Theory):
Without data distribution assumptions, it will limit the guidance for practitioners.

Author response:
**Ans.** We would like to reiterate that we do not work under any specific assumptions for the data. We only assume that the instances in each bag are sampled randomly without replacement. As mentioned in section 5, we follow the standard practice in popular LLP literature *(Chen et al, 2023; Ardehaly & Culotta, 2017; Patrini et al, 2014; Busa-Fekete et al., 2023)* of creating disjoint random bags where we sample instances without replacement from the training set, and keep repeating this for each bag, bag-size: k number of times. We are solving for the most general case of bag construction. Thus, there is no distributional assumption on the dataset or bagging.

Reviewer comment (Theory):
There is no analysis of the theoretical guarantee of the algorithm's performance.

Author response:
**Ans** We would like to refer the reviewer to Section A.3 in the appendix where we provide some intuition as to why our algorithm works via showing that the nearness constraints through 1-NN produces a cycle free factor graph. However, when bags are randomly formed, invariably cycles are formed. Starting from the classical paper (Frey et al., 1997) to the very recent works on message passing (Newman, 2023), loopy belief propagation does not have convergence guarantees in general. In fact, the former points out the empirical success in decoding error correcting codes even on loopy graphs which precisely inspired us.

We further provide an approximate Linearized BP analysis that shows that for large bags regime for Adult, inverse temperatures chosen actually yield convergence for the Linearized BP, in a **new Section C** in the updated manuscript. We feel this analysis of the theoretical guarantee of the algorithm’s performance helps provide the requested theoretical backing for our strong empirical results.

Reviewer comment (Evaluation):
The proposed algorithm is much slower than the baseline algorithm, and the running time is about one order slower (Table 4). However, the performance of the proposed algorithm in Tables 2 and 3 is not significantly better in many setups.

Author response:
**Ans.** We would like to emphasize that the running time of Belief Propagation is only linear in (bag_size + num_neighbours), and even for a dataset with 1 million samples, total run time for 128 sized bags is merely 106 minutes, where the MLP time is akin to the baselines with 10 minutes. On bag size 2048, for an adult dataset with 50k instances, it only takes 17 mins, which is negligible for the performance improvement obtained.

Also, we would like to emphasize that the Data Setup time (Col 4) reported in Tables 4 & 5 includes time taken for data loading, model initialization etc. which would also contribute to DLLP total time. So DLLP total time is actually DLLP training time (Col 1) + Data Setup time for DLLP. As can be seen in the tables, DLLP training time (Col 1) is comparable to Ours - MLP training time (Col 4) as expected. So the only overhead of our method over DLLP is the BP time (Col 3) which, as clarified above, is linear in (bag_size + num_neighbours). We further refer the reviewer to Section 6.1 regarding time complexity of BP step.
Table 2 contains results on Criteo, which is a very hard, tabular dataset. Even small amounts of improvement on this data is non-trivial and significant. Over the last 7 years, the dataset has seen a 2% improvement in AUC while our method is able to produce up to **0.8%** improvement over DLLP *(Ardehaly & Culotta, 2017)*. In Table 3, the performance improvement in CIFAR-S is up to **7.4%**. CIFAR-B (balanced dataset) is a much easier dataset to learn from, as there are a lot of instances with label 1. Our method performs comparable to the baselines on this. The gap to the instance-wise performance is also minimal on CIFAR-B.

Reviewer comment (Experiments):
The 4 datasets in the experimental analysis are not real data on the bag-level. The bags are manually created.

Author response:
**Ans.** We would like to refer the reviewer to several papers from the LLP literature *(Chen et al, 2023; Ardehaly & Culotta, 2017; Patrini et al, 2014; Busa-Fekete et al., 2023)* who follow the same procedure to create random bags on usual datasets. The use of Adult, Marketing, Criteo, CIFAR is well-established in the same prior literature. We request the reviewer to please take a look at Section 2 of the paper where related work from the LLP perspective is discussed. All of these prior work deal with creation of “manual” bags following some bag creation techniques.

In industrial applications as well, stricter tracking regulations especially with respect to clicks as in PCM *(https://webkit.org/blog/11529/introducing-private-click-measurement-pcm/, https://developer.chrome.com/docs/privacy-sandbox/)*, have led to development of aggregation techniques with aggregation into random bags of uniform size as described by us is one of the most popular.

So our setting does reflect a real-world setup and is important to study not only from a research perspective but also from an implementation perspective for future regulation ready systems.

To reiterate, there is no “real bag data” that we are aware of that has been benchmarked on publicly. We would be happy to append experiments on datasets that the reviewer suggests in particular. As mentioned in section 5, we follow the standard procedure of creating disjoint random bags where we sample instances without replacement from the training set, and keep repeating this for each bag, bag-size: k number of times.

Reviewer comment (Writing):
Some notations are not reader-friendly.

Author response:
**Ans.** We appreciate the reviewer pointing out these issues and would be happy to clarify that [m] denotes the indices {1,2,3 … m}, from which the indices corresponding to the instances in that bag S_i are obtained. | x | denotes an indicator variable which is 1 if and only if both i and j belong to the same bag. We have clarified these notations in the updated manuscript.

Reviewer comment (Writing):
In the 3rd line in section 3 of p3, [m] is not defined.

Author response:
**Ans.** We appreciate the reviewer pointing out these issues and would be happy to clarify that [m] denotes the indices {1,2,3 … m}, from which the indices corresponding to the instances in that bag S_i are obtained. | x | denotes an indicator variable which is 1 if and only if both i and j belong to the same bag. We have clarified these notations in the updated manuscript.

Reviewer comment (Reproducibility):
Nearly all parameters are tuned. There is no guidance on how to choose them in practice for quick application. For example, there is no guideline for the stopping rule of the algorithm to ensure convergence.

Author response:
**Ans.** We would like to refer the reviewer to section A.5 in the appendix where we explicitly state the default hyper-parameters and the stopping rule of the algorithm for convergence. For reproducibility purposes we also provide exact hyper-parameter values per dataset in section A.6. We would also release the code publicly with the camera ready version of the paper, further enhancing reproducibility. The choice of hyper-parameters varies from dataset to dataset, and while we cannot comment on the optimal choice of parameters for a new dataset a user might experiment on, we provide the entire implementation and experimentation detail and in near future shall provide code that we believe that should help the user extend our algorithm with ease.

Reviewer comment (Theory):
In the proposed algorithm, the pair-wise calculation could lead to a high order time complexity. For example, capturing k-nearest neighbors for every point x_i is a very slow process when the data size is large.

Author response:
**Ans.** We refer the reviewer to Section 6.1, where we clearly state that the time complexity of Belief Propagation is linear in the number of neighbors per point. We further establish the effectiveness of 1NN in capturing the majority of the gains and thus even a small number of neighbors per point is sufficient, even when the data size is large.

Further to provide empirical evidence, compared to data setup time which is common to DLLP and our method the overall algorithm time is not that significant. Moreover, the time taken for the kNN graph construction and the required pre-processing is merely 80.49s (+- 6.07s) for the Adult Dataset (barely a minute for 1 nearest neighbor, for a ~50k sized dataset), which is almost 10 times smaller than the common data setup time and comparable to the training time for the MLP (85s). It is also in the same range as the BP time for 128 sized bags, and much faster than BP for larger bags. Thus the graph construction for kNN does not need high computational time and is not a bottleneck. As we have highlighted in section A.1.1, 1NN suffices to retrieve the majority of the performance Thus we are able to achieve a superior performance over the baselines with only additional 80s required for obtaining the neighbor graph to form our covariate factors before Belief Propagation.

Reviewer comment (Evaluation):
What's the time performance of other baseline algorithms?

Author response:
**Ans.** We observed the time performance of other baselines, but only provided DLLP as an indicator. We have further provided the numbers for EasyLLP and GenBags on Adult and Criteo Datasets, for the training time for both algorithms, thus appending 2 columns to Table 5 Section A.2.1 and Table 4 in Section 6.1, and reinforcing the marginal overhead our method incurs. We reiterate that the Data Setup Time (Column 4) is common to all baselines and methods as well as it comprises the data loading and preprocessing times, which is identical for all methods including ours (which we have made more clear in the updated version of the manuscript). As one can see, GenBags takes more time to train the same MLP as our method (up to 5 times more on Adult and 3 times more on Criteo), and with much higher std. dev. And thus GenBags are much more time consuming, with similar or worse performance. Also, on Criteo, MLP training time is more for EasyLLP than for our method. We have updated Table 5 in the manuscript to include the 2 additional baseline’s training time as well.

Reviewer comment (Theory):
What's the distribution and bagging assumption required for the proposed algorithm?

Author response:
**Ans.** We would like to reiterate that we do not work under any specific assumptions for the data. We only assume that the instances in each bag are sampled randomly without replacement. As mentioned in section 5, we follow the standard practice in popular LLP literature *(Chen et al, 2023; Ardehaly & Culotta, 2017; Patrini et al, 2014; Busa-Fekete et al., 2023)* of creating disjoint random bags where we sample instances without replacement from the training set, and keep repeating this for each bag, bag-size: k number of times. We are solving for the most general case of bag construction. Thus, there is no distributional assumption on the dataset or bagging.