From: ICLR.cc/2023/Conference/Program_Chairs
title: Paper Decision
decision: Reject
metareview:_summary,_strengths_and_weaknesses: The authors conduct a large-scale empirical study of active learning methods on tabular data using neural networks, showing that 'simple' margin-based methods outperforms (or is at least non-inferior) to more recent methods in most experimental settings. Thus, at least for practical settings, in the absence of additional problem-specific information, margin-based AL seems easy and performant. More specifically, experiments are conducted on the OpenML-CC18 benchmark (69 datasets) to compare 15+ AL querying strategies under three different {initial data, batch size} settings with and without pre-training -- showing 'simple' multi-class margin to generally be the best AL querying strategy.

The consensus strengths of this work include:
- This is ostensibly the largest empirical study of active learning querying strategies, resulting in a finding that many practitioners have claimed/suspected, but haven't had strong evidence to support.
- For tabular data, this work provides an empirical framework that can be used to evaluate future claimed advanced convincingly for practical settings.
- The core conclusion is convincing; 'simple' margin-based active learning works best in practice (possibly modulo incorporating dataset-specific domain knowledge, etc.)

The weaknesses discussed in the reviews (and my own reading) of this work include:
- While the quantity of empirical results are significant and the claimed findings are convincing, there was limited discussion regarding how/why particular methods were chosen (even if the list is relatively exhaustive). It would be a more rigorous result if the authors followed the format of a systematic review in terms of specifying inclusion criteria, etc.
- From a novelty perspective, there (obviously) aren't any methodological contributions (as this isn't the point of the paper). However, also from an empirical methods perspective, this is using an available benchmark and evaluation methods. Thus, while a potentially interesting finding and good benchmark for future work, there also aren't any methods proposed for finding out when some querying strategies are particularly strong (e.g., dimensionality of the data). 
- In the same vein as the previous item, there is limited interpretation of the results beyond the primary conclusion. There must have been additional observations that point the way to future work.
- While neural networks are increasingly used in tabular data settings, tree-based ensembles remain widely used, especially in the 'no additional information specific to the application" setting that this paper emphasizes. Thus, while not strictly necessary, adding these results would strengthen the impact amongst practitioners (who are a key consumer for this work).
- It isn't clear if the authors are going to release the code, etc. to make this a benchmark for future work on active learning. It would be helpful to minimize friction for testing future proposed methods with a simple way to plug the method in and get the same tables/figures out.

Overall, the paper is well-written, easy to follow, and convincingly supports an interesting result. Additionally, the paper was improved during the rebuttal period. After discussion, the consensus was that the findings in this paper would be of interest to the community, but additional interpretation of the results are needed (i.e., secondary findings that point to future AL work; experimental methods that facilitate deep analysis of contrasts in methods) for this to be adopted as the standard benchmarking evaluation for AL research. Additionally, while secondary, all of the reviewers would like to see tree-based ensemble results (possibly more than the pre-training results) for the tabular setting.

justification_for_why_not_higher_score: To have a higher score, we would like to see:
- A more rigorous statement regarding the inclusion criteria for different methods (and associated rationale)
- Increased interpretation of the results, including a possible deep dive into secondary experiments.
- Results using tree-based ensembles.
justification_for_why_not_lower_score: N/A

From: ICLR.cc/2023/Conference/Paper5484/Authors
title: RE: Response
comment: Thanks for the reply.

The paper you reference proposes using an ensemble of deep networks for active learning and compares it to MC Dropout (Bayesian ensembling) and a density-centric baseline. In section 4.3, they compare ensembles to MC Dropout, ablating factors like the number of ensemble members / dropout passes, the random initialization of the ensemble members, and random mini-batching.

If Margin sampling had factors that would be interesting to ablate like this, we would have done it already. As to why method A outperforms method B, as you state, the reasons can sometimes be complex and up to interpretation, really. **If you have any specific concrete analysis in mind that is missing, we'll be happy to add it.**

In our previous note, we mentioned that we ran experiments where we looked at the 2x2 matrix of (is the base model pre-trained) x (is the acquisition model pre-trained). We found that it was important for the base model and acquisition model to be consistent in their pre-training -- either pre-train both or pre-train neither. We mentioned that the base and acquisition models should be the same, which made sense, and we felt that including this in the paper was not too valuable and could be distracting.

Our evaluation setting is standard and follows the references you provided. We run a lot of baselines on a large number of datasets and do each with many trials, so the findings we share are rigorous and convincing.

We'd like to note that many baselines are variants of Margin (e.g. Margin + random sampling, Margin + clustering), so comparing Margin against these variants serves as evidence that the modification made was ineffective. For example, incorporating clustering with Margin did not really boost performance in the settings we considered here.

Our analysis is based on valid statistical tools that are tailored for stating which method is better. During the rebuttal, we also further extended this analysis to include "Probability of Improvement" plots (does one method outperform another when the dataset is unknown). 

As we state in the title and abstract, the goal of this study is to provide strong statistically backed empirical analysis for comparing margin to other AL methods. We feel that the evidence and explanations we provide in the paper are very convincing and insightful for practitioners and researchers alike.

From: ICLR.cc/2023/Conference/Paper5484/Reviewer_TZgZ
title: response.
comment: I'm glad to see author did enhance the information output of the experimental results figures, however, the experimental analysis problem as I mentioned in question 1 still made me unsatisfied. What I mean is "A outperform than B is because xxx", not just "according to Figure xx, A is outperform than B". A good example of how to conduct experimental analysis can see https://openaccess.thecvf.com/content_cvpr_2018/papers/Beluch_The_Power_of_CVPR_2018_paper.pdf .

From: ICLR.cc/2023/Conference/Paper5484/Authors
title: Further Questions?
comment: Thank you again for the review. We believe we addressed your concerns, but please let us know soon if there is anything else we can do or if you have any further questions, as the end of the discussion window is fast approaching!

From: ICLR.cc/2023/Conference/Paper5484/Authors
title: Further questions?
comment: Thank you again for the review. We believe we addressed your concerns, but please let us know soon if there is anything else we can do or if you have any further questions, as the end of the discussion window is fast approaching!

From: ICLR.cc/2023/Conference/Paper5484/Authors
title: Response
comment: To address your statistical testing concern, we looked at two options:

(1) Compute win ratio plots, where all datasets are treated as a single huge aggregate dataset. In this case we do 16 x 16 tests. In each test (for methods m1 and m2) the null is "does the mean for m1 equal the mean for m2", where the underlying dataset is unknown (uniform)).

Win ratios here are either 0/0, 0/1, 1/0. Using p-value 0.01, we found most entries to be 0/0 (no statistical difference) with Margin beating (1/0-ing) some of the worse baselines.


(2) Probability of Improvement. This method is suggested in Deep Reinforcement Learning at the Edge of the Statistical Precipice (https://arxiv.org/abs/2108.13264 ; outstanding paper at Neurips 2021).

Let $X$ and $Y$ be the scalar performance metric (higher is better) of algorithms $X$ and $Y$, and $X_m$ ($Y_m$) be the performance of $X$ ($Y$) on dataset $m$. Suppose we observe $N$ samples of $X_m$ and $K$ samples of $Y_m$. We use the Mann-Whitney U-statistic:

$ P(X_m > Y_m) = \frac{1}{NK} \sum_{i=1}^{N}\sum_{j=1}^{K}S(x_{m,i}, y_{m, j})$


$P(X > Y) = \frac{1}{M} \sum_{i=1}^{M} P(X_m > Y_m)$,


$S(x,y)$ is 1 if $y < x$, 1/2 if $y=x$ and 0 if $y >x$.

$x_{m,i}$ represents the performance of $X$ on trial $i$ on dataset $m$.
We perform stratified bootstrap sampling (re-sampling 200 times from $X_{m,1:N}$ and $Y_{m,1:K}$ for each dataset $m$) and then show violin plots of the bootstrap sampling distribution of the U-statistic (i.e. probability of improvement). If the upper CI is higher than a threshold of 0.75, then the results are said to be statistically meaningful as per the Neyman-Pearson statistical testing criterion.

Here, we found that the probability is less than around 0.50 and the quantiles are well below 0.75 so we can say that when the dataset / task is unknown, no method outperforms Margin in any reasonably statistically meaningful way.

We have updated the paper to include probability of improvement (PI) plots in all settings. They are currently in the Appendix, but if reviewers prefer these plots over the win ratio ones, we can move the PI plots to the main text.

From: ICLR.cc/2023/Conference/Paper5484/Authors
title: Response
comment: Thank you for the review.

**RE: Model choice**

Indeed, the paper studies deep neural networks and decision trees are commonly used for tabular datasets in practice. We'd like to note that the SCARF paper shows that with pre-training (which we, crucially, employ in this work), deep networks of the size we consider in this work often outperform XGBoost on exactly the same datasets we use here (pages 21-24 of https://arxiv.org/abs/2106.15147 show test accuracy of XGBoost vs. pre-trained neural networks for every dataset). Practitioners are increasingly adopting neural networks, and this work is timely as we are providing concrete guidance around active learning for these practitioners. We have updated the paper to include discussion on this point.


**RE: How we selected AL baselines**
We surveyed the literature and identified methods, new and old, that gave strong results. We tried to sample from a spectrum (e.g. uncertainty-based, clustering-based, Bayesian, etc).
Following your suggestion, we have added BADGE as a baseline method!


**RE: Evaluation criterion**
Another reviewer brought up this point as well. We've replaced the "accuracy at round k" metric with "area under the budget curve" (AUBC), which captures the methods' performance across all acquisition rounds.

**RE: Why the specific settings for small, medium, and large**
We looked at the histogram of dataset sizes as well as the test accuracy on the initial / seed set for various seed set sizes, with the goal of identifying interesting data regimes. In particular, we wanted to study the settings where AL *does* help (i.e. at least one method does better than random). Going bigger than our "large" setting, we noticed that the results were not insightful, since there was enough initial data for random to attain high accuracy and actively selecting points provided diminishing returns.

From: ICLR.cc/2023/Conference/Paper5484/Authors
title: Response
comment: Thank you for the review.

RE: (1) Pre-training with SCARF is performed using the same exact training data as that used for supervised learning, so there is no mismatch between source and target domains, as you mention. We can think of AL as having two models -- a "base" model, whose test accuracy we report, and the "acquisition" model, whose uncertainties or embeddings we use for AL. The acquisition model chooses the points to acquire and the base model is trained on them. Earlier, we did run experiments where we looked at the 2x2 matrix of (is the base model pre-trained) x (is the acquisition model pre-trained). We found that it was important for the base model and acquisition model to be consistent in their pre-training -- either pre-train both or pre-train neither. It makes sense that for effective AL, the base and acquisition models should be the same, so these experiments proved to be more sanity checks, and adding them to the paper provided little value, so we decided to keep it simple and show two cases -- when both models are either pre-trained or not pre-trained.

RE: (2). Great suggestion! We have updated all our plots to represent *Area Under the Budget Curve* rather than test accuracy at a particular round. RE: zooming in on the line plots -- this is a good suggestion but the legend currently takes too much space so fitting the zoom box is challenging. If the paper is accepted, we can figure out the optimal presentation strategy for the camera ready.

RE: (3) Following your suggestion, we have removed the claim that PowerBALD's possible performance boost may be due to random sampling effects, since this claim is unsubstantiated and is outside the scope of our work.

From: ICLR.cc/2023/Conference/Paper5484/Authors
title: Response to all reviewers
comment: Thank you all for your reviews.

Noteworthy changes made to the paper:
* Area under the Budget Curve (AUBC) is the primary metric now, instead of "test accuracy at round k", as it measures the performance of AL methods across all acquisition rounds.
* Unfiltered box plots are shown in the main text, and the little points on them have been removed as they were distracting.
* Batch Active learning by Diverse Gradient Embeddings (BADGE) is added as a method.
* Probability of Improvement (PI), which basically measures P(method M outperforms Margin when the dataset is unknown) is introduced as an alternative to the win ratio plots, since one reviewer noted that the win plots conduct too many statistical tests. Our response to said reviewer has more details about this. PI plots are shown in the Appendix but they can be moved to the main text for the camera ready.

Multiple reviewers pointed out that in practice decision trees, not neural networks, are often used on tabular datasets, so our work which focuses on deep nets may be limited. We'd like to note that the SCARF paper shows that with pre-training (which we, crucially, employ in this work), deep networks of the size we consider in this work often outperform XGBoost on exactly the same datasets we use here (pages 21-24 of https://arxiv.org/abs/2106.15147 show test accuracy of XGBoost vs. pre-trained neural networks for every dataset). Practitioners are increasingly adopting neural networks, and this work is timely as we are providing concrete guidance around active learning for these practitioners. We have updated the paper to include discussion on this point.

From: ICLR.cc/2023/Conference/Paper5484/Reviewer_58Ep
title: Thank you for clarifying your testing procedure
comment: Apologies that I did not correctly grasp your setup in the initial review. Nonetheless, my concern is that you are performing 16 x 16 x 69 = 17,664 statistical tests. At this scale of multiple hypothesis testing, there is a high risk of false positive errors when rejecting null hypotheses. This lends doubt to broader statements about whether a method is in fact better than others.

To me, there are two possible mitigations:
- Reduce the number of statistical tests: In my initial review, I suggested one possible experiment question, where we ask “not knowing anything about the data set being used, is method A actually different from method B in general”? An ideal setup should have at best 16 x 16 = 256 tests, which is far less than 17k.  This setup also makes a bit more sense from a practitioner perspective because the practitioner doesn’t know the actual characteristics of the data sets, and just wants to know on an aggregate level, is method A actually better.   Note the current setup asks, “for a given data set, is method A actually different from method B”; but as a practitioner, I don’t care if A is better than B for one of the data sets that you tested, I just want to know whether it is better than B in general. 
- Incorporate some sort of statistical correction factor to address the risk of false positives. There are many correction methods available, but it will be important to consider one that can handle your scale of experiments.

Nonetheless, I would like to commend the authors for taking the time to perform statistical analyses. I feel that designing the experiment well becomes crucial, especially when considering so many tests.

Also I would appreciate if details on the experiment setup are provided (e.g., in Appendix) so that we don’t have to refer to the SCARF paper.

From: ICLR.cc/2023/Conference/Paper5484/Authors
title: Requesting clarification about your review
comment: Thank you for your review. We are in the process of incorporating reviewers' feedback into a revision of the paper.

**Statistical Testing**

We'd like to clarify our approach to statistical testing and to better understand what you are proposing instead.

We currently do a hypothesis test for every (method1, method2, dataset) triple. Each triple has 20 observations (trials) and we compute the mean across trials. The null hypothesis we wish to reject is "method1 and method2 have equal means on the dataset". We use the Welch's t-test since the variances between the two methods may be unequal.

We believe you are suggesting that we instead compare method1 and method2 across 69 datasets * 20 trials = 1380 samples?
The samples might then represent sampling hierarchically, first the dataset, uniformly over all datasets, then from the distribution of accuracy conditioned on dataset, and we would be testing whether the average accuracy of method1 and method2 differ, not knowing the latent dataset.

We believe that doing the test for each dataset (i.e. the triple) makes the most sense. The goal of the statistical test really is to address variation in accuracy due to stochastic factors like random initialization of deep learning models, random mini-batching, etc, as opposed to randomness in the dataset selection -- why treat the dataset as random if you don't have to?

Formulating a null for each dataset to the tune of "method1 (say Margin)'s mean is not greater than or equal to all other method's means on the dataset"  is possible, but we believe a pairwise comparison for each dataset provides more insight for the reviewer.

Note: as mentioned in the text, our evaluation setup exactly matches that of Bahri et. al (SCARF: Self-Supervised Contrastive Learning using Random Feature Corruption), which was an ICLR 2022 spotlight paper.
 
**Decision Trees**

You point out that the paper's focus is too narrow as most practitioners use decision trees, random forests, etc on tabular data.
We'd like to note that the SCARF paper shows that with pre-training (which we, crucially, employ in this work), deep networks of the size we consider in this work often outperform XGBoost on exactly the same datasets we use here (pages 21-24 of https://arxiv.org/abs/2106.15147 show test accuracy of XGBoost vs. pre-trained neural networks for every dataset). Practitioners are increasingly adopting neural networks, and this work is timely as we are providing concrete guidance around active learning for these practitioners. We are rewording the paper so it's clear that it's about specifically about active learning using deep neural networks.

**Increasing p-values of Box Plots**

The dilemma inherent with the box plots is: we'd like to show the relative improvements of different methods across the spectrum of datasets. We'd also, however, like to show gains that are statistically meaningful. Picking a small p-value means that some methods' box plots may have little or no points, which makes the box plot presentation less useful. As mentioned in the text itself, we increased the p-value to provide a good tradeoff between filtering and having enough points to make the box plots useful. We have unfiltered plots in the appendix. If you think it's better to move the unfiltered plots to the main text and put the filtered plots in the appendix, we are happy to.


**If possible, please let us know your thinking soon so that we can revise the paper by the revision deadline.**

From: ICLR.cc/2023/Conference/Paper5484/Reviewer_TZgZ
confidence: 5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.
summary_of_the_paper: This paper is an empirical study of deep active learning (especially for margin based approaches). This paper consider different tasks (openml) and how model pre-training affect the performance.
strength_and_weaknesses: Strength: this empirical study tests many DAL approaches (mostly are margin-based approaches) on openml datasets. A meaningful finding is that margin-based approachs perform consistently better than random sampling across multiple various tasks.

Weakness: some directions need to dig deeper.

1. For w/ and w/o pre-training techniques, this paper only provide an overall performance, however, how w/ and w/o pre-training techniques affect the AL performance is complex, it is related to the similarity btw target and source domain and perform different on different tasks like [r1] (section 3.4, considering dataset with distribution shift). The author should dig more informations either from existing experimental results or conduct more experiments.

2. For evaluation methods, both win matrix and box plots only present the performance on one round. As for Figure 4, there are too many curves with mixing colors. It's really hard to distinguish which is better across varying batches. Can use Area Under the Budget Curve (AUBC) [r1] or AUCLOG, which is the AUC of the learning curve when the X-axis is in log-scale [r2] or the average performance across multiple rounds (not recommended) to present an overall performance. Aslo, Figure 4 (and the similar figures) need to be re-designed by selecting more distinguishable color schemes or zoom in on some areas like [r3].

3. A deeper and more confidential analysis of the experimental results is needed. For instance, at the end of Page 7, "Methods that tied
or slightly underperformed margin on the win plots have a comparable relative gain over random, whereas the gains for others are near zero or negative (in the case of BALD). We observe that Power-BALD outperforms BALD – this is, most likely, not because BALD is ill-suited for the batch AL setting but because PowerBALD’s additive noising process mimics (biased) random sampling." Firstly, "most likely" seems not confidential. Comprehensive study needs reasonable and confident analysis. Then, Why BALD is ill-suited for the batch AL setting? Does the author try different batch settings or there are some publications support the opinion? 

[r1] Zhan X, Wang Q, Huang K, et al. A comparative survey of deep active learning[J]. arXiv preprint arXiv:2203.13450, 2022.

[r2] Mozafari B, Sarkar P, Franklin M, et al. Scaling up crowd-sourcing to very large datasets: a case for active learning[J]. Proceedings of the VLDB Endowment, 2014, 8(2): 125-136.

[r3] Kothawade S, Beck N, Killamsetty K, et al. Similar: Submodular information measures based active learning in realistic scenarios[J]. Advances in Neural Information Processing Systems, 2021, 34: 18685-18697.
clarity,_quality,_novelty_and_reproducibility: This paper is clealy written and easy to follow, it tests many margin-based DAL approaches across 69 datasets. From the aspect of the empirical study, the quality is high, the novelty is not appliable for empirical study. The author does not provide the re-implementations of their experiments thus it is not reproducible. 
summary_of_the_review: This is a well-written emprical study with sufficient comparative experiments about margin-based DAL approaches, however, the experimental analysis is somewhat weak and need to be improved.
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
recommendation: 5: marginally below the acceptance threshold

From: ICLR.cc/2023/Conference/Paper5484/Reviewer_h2sD
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: This paper compares a variety of active learning algorithms on 69 tabular datasets using a multilayer perceptron model (with and without unlabeled pretraining). The results show that margin-based (i.e. best versus second best) uncertainty sampling performs the best.
strength_and_weaknesses: Strengths:
 - This paper empirically analyzes active learning algorithms (which is much needed) rather than defining yet another algorithm.
 - This paper studies tabular datasets which are under-explored in active learning experiments. 
 - This paper compares a large variety of active learning algorithms and large number of datasets.


Weaknesses:
 - This paper only evaluates with one model, a multilayer perceptron, which may be suboptimal for tabular datasets compared to tree-based methods.
 - With the very large number of active learning algorithms, it is admittedly hard to pick a list. However, I was surprised to see BADGE and BAIT were not on the list. While BADGE is computationally intensive, for image classification tasks, it works very well.
 - The evaluation criteria (performance after k rounds in 3 different batch settings) seems a bit arbitrary without much discussion.

clarity,_quality,_novelty_and_reproducibility: The paper is clearly written and is original in its choice of models/datasets for an active learning empirical survey. 

In terms of writing, I think more discussion regarding choices in the evaluation would be very helpful. For example:
 - Why the specific settings for small, medium, large?
 - Why performance after k iterations rather than area under the active learning curve, or some other metric?
 - Why the particular MLP model rather than tree-based methods?
 - What was the process for choosing the active learning methods?
summary_of_the_review: This type of active learning evaluation paper (especially for tabular datasets) is a very useful addition to the active learning literature. However, the quality of the evaluation could be improved in some ways though.
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
recommendation: 6: marginally above the acceptance threshold

From: ICLR.cc/2023/Conference/Paper5484/Reviewer_58Ep
confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: This paper performs a comparison of different active learning strategies on tabular data sets when trained with deep learning models that are pre-trained with self-supervised learning. The key result is that margin sampling, which is also one of the easiest active learning strategies, consistently outperforms other strategies.

strength_and_weaknesses: Strengths
- The experiments cover a large number of data sets and seeds
- The paper concludes with clear guidelines for practitioners

Weaknesses
- The focus of the study seems narrow. The paper focuses on (1) tabular data with (2) deep learning models. This is limited as the vast majority of tabular ML in practice is done not with DL but with GBT, RF, etc.
- The statistical testing seems inappropriate in that the number of tests that are being made is massive. My understanding may be incorrect so please correct me if I am wrong, but there are 49 x 20 x 3 tests performed and then the results are collated. At the very least it should be possible to reduce the number of tests to 16 x 16 by considering multiple measurements (i.e., seeds) and just doing a single test per method x method pair. An even simpler analysis would be for each data set, to perform a single ANOVA (if assumptions are met) with margin and other methods to demonstrate that margin actually outperforms all methods on each data set. This reduces the number of tests down to 49. There are statistical setups that could also be considered.
- Why do you use a larger p-value in one setting and not in the other? The p-value is supposed to be a priori fixed and not modified to increase the number of wins. If the result is not significant for the pre-determined p-value then that in itself is a result

clarity,_quality,_novelty_and_reproducibility: The paper is mostly clear, but does require prior knowledge of the experimental setup in a previous paper.
summary_of_the_review: Overall, the paper seems limited in scope and performs statistical analysis that is not clear. It could be my understanding of the stat analysis is wrong and if I am incorrect, I am happy to increase my score, but I am not yet sure about how to address the scope limitations.


==============
AFTER REBUTTAL

I appreciate the authors for carefully considering my comments and exploring a more robust statistical setup to compare their methods. I have raised my score accordingly.
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 6: marginally above the acceptance threshold