From: ICLR.cc/2023/Conference/Paper3182/Authors
title: Submission Withdrawn by the Authors

From: ICLR.cc/2023/Conference/Paper3182/Reviewer_wW3E
title: Acknowledgement of Response to Reviewer wW3E
comment: Thank you for your response. I have raised my score to 5.

From: ICLR.cc/2023/Conference/Paper3182/Reviewer_bFoz
title: Response from the reviewer bFoz
comment: Thank you for the clarification. Many of my concerns have been addressed by the authors and I will raise my initial score from 1 to 5.

From: ICLR.cc/2023/Conference/Paper3182/Authors
title: Response to Reviewer tZq9
comment: We thank the reviewer for their feedback, and are glad to see they are impressed by the empirical results and simplicity of the technique.

From: ICLR.cc/2023/Conference/Paper3182/Authors
title: Response to Reviewer VAE1
comment: We thank the reviewer for their insightful comments and recommendations.

**Theoretical Guarantee:** In section 4, we show theoretically that the confidence-based version of StackEm is guaranteed to have less churn than $f_{\mathrm{new}}$ while maintaining the same level of accuracy.

**Gradient Compatibility and Stacking:** We rewrite some of section 3 to make it more explicit why gradient incompatibility motivates the need for a stacking-based methodology. Namely, the forgetting of samples throughout the course of training a single model suggests that churn is inevitable to some extent. Training a new model $f_{\mathrm{new}}$ from a different random initialization on a larger dataset introduces additional causes for churn beyond the forgetting observed when training a single model. Even if these additional causes are dealt with, forgetting still happens when training $f_{\mathrm{new}}$. Rather than trying to regularize or modify the training procedure for $f_{\mathrm{new}}$, it is simpler to use $f_{\mathrm{new}}$ on samples where it is more likely to be accurate, and revert to $f_{\mathrm{base}}$ otherwise. Since methods like distillation have a hyperparameter which balances between matching predictions of $f_{\mathrm{base}}$ and enabling $f_{\mathrm{new}}$ to achieve maximal accuracy, they inevitably sacrifice performance on one of the competing goals, whereas a stacking-based methodology does not since $f_{\mathrm{new}}$ is trained independently of $f_{\mathrm{base}}$.

**More Baselines:** We add two more baselines to our results: warm start where $f_{\mathrm{new}}$ is initialized with the parameters of $f_{\mathrm{base}}$, and anchor loss which is a variant of distillation meant to increase the plasticity of $f_{\mathrm{new}}$ by applying distillation only to samples correctly predicted by $f_{\mathrm{base}}$. These extra results can be found in the appendix as they do not fit in the main results table. We note that both of these new baselines underperform distillation and StackEm, so they do not change any of our conclusions.

**Detailed Structure of Learned Model:** We now clarify in the text that a learned meta-model $h^{*}:\mathbb{R}^{d} \times \mathbb{R}^{d} \rightarrow \mathbb{R}^{k}$ 

takes as input the outputs of $f_{\mathrm{base}}$ and $f_{\mathrm{new}}$. $h^{*}$ can be any model which assigns a per-class score to a given sample. In our case, we consider logistic regression, random forests, and gradient boosting models.

**Reorganizing Related Works:** We rewrite this section to make it have the following clearer structure. First, we discuss the types of churn which have been mentioned in the literature: variance, model adaptation, and model upgrade. We briefly mention the settings where each type of churn is relevant, and then emphasize that we focus on the model adaptation type. Then we follow the structure mentioned in the introduction where we group existing churn reduction techniques into the two categories of distillation-based and ensemble-based methods. Lastly, we note that of the churn reduction methods discussed, some are designed for specific tasks (e.g. image retrieval), and therefore cannot directly be used for other tasks (e.g. classification), yet at a high level some of the ideas behind those methods  can be adapted to other tasks. We conclude that there is no method for classification tasks that has the effectiveness of ensembles, yet is much more efficient, hence the need for StackEm.

**Explanation for Margin:** The margin as defined in sec. 3 was introduced by Pleiss et al. and is an adapted version of the linear approximation of the true distance to decision boundary by Elsayed et al. 

Pleiss et al. demonstrate that the margin at a given time step T will be higher for more difficult samples as well as noisy label samples. For a given sample and time step T, the margin is affected by that sample's loss gradient, as well as the loss gradients of all other samples in the batch. We expect and show that there are incompatible samples test samples throughout the course of training which have small margin and large gradient norm and are therefore responsible for churn.

**Churn Results With Value > 1:** The values reported in the table are percentages, hence why they are greater than 1. We now clarify this in the text.

Pleiss, G., Zhang, T., Elenberg, E. R., & Weinberger, K. Q. (2020). Identifying Mislabeled Data using the Area Under the Margin Ranking. arXiv. http://arxiv.org/abs/2001.10528

Elsayed, G. F., Krishnan, D., Mobahi, H., Regan, K., Bengio, S., & Research, G. (n.d.). Large Margin Deep Networks for Classification. https://arxiv.org/pdf/1803.05598v1.pdf

From: ICLR.cc/2023/Conference/Paper3182/Authors
title: Response to Reviewer bFoz
comment: We thank the reviewer for their comments regarding notation. We now explicitly define all notation on page 3.

**Page 6 Random Quantity in $\mathcal{F}_{\mathrm{base}} (x)$:** We are stating an assumption used by Zhao et al. that the output logits of a model are normally distributed, thus the output logits of a ensemble of such models are also normally distributed. The quantity is random in the input x which we now capitalize to make it clear that it is a random variable.

**Page 6 Randomness in $P(\hat{y}_{\mathrm{base}} =y \mid x)$**: We have entirely change the notation in this section to make the sources of randomness clearer.

**Page 6 Randomness in $f^{(i)}(x) \sim \mathcal{N}(\mu, \Sigma)$:** We change the notation to $f^{(i)}(X) \sim \mathcal{N}(\mu, \Sigma)$ to make it clear that the input to the model is a random point in the training data, and that the output logits follow a normal distribution.

**Page 7 $H_{\phi}$ and $\mathcal{H}$ not Defined:** We apologize for the typo as $H_{\phi}$ should be $G_{\phi}$. We now define $\mathcal{H}$ as the hypothesis space of functions for the Learned Combination model.

**Page 7 What "search over" Means:** By "search over" we mean that we try all 3 ways of combining models (Confidence, KL-Divergence, and Learned Model), and report the best one.

**Page 6 Calibration and Notation:** We now make explicit what the expectations in this section are over, and explicitly define the notation to avoid any confusion. We note that it is assumed that both the base and new model are perfectly calibrated. This means that the base model can only predict class 1 with probability 1.0 for every sample if it is always correct in doing so, i.e. the data only has a single class. Otherwise, this would violate our calibration assumption

From: ICLR.cc/2023/Conference/Paper3182/Authors
title: Response to Reviewer wW3E
comment: We thank the reviewer for their feedback and address their concerns below.

**Multiple Updates:** We recongize the limitation of our experiments only considering a single update, but note that this is the standard in the churn reduction literature.

**Cross Validation and Stacking:** We acknowledge that only the method from section 4.2 satisfies the traditional definition of stacking, and clarify that by stacking we mean any method the takes a combination of model outputs to generate a final output.

We add text to section 4.2 to make it clear that we perform 5-fold cross validation  to choose the model with best cross validation error rather than training error. We note that this does not change the conclusions about "Confidence" being superior to "Learned Combination" for churn reduction.

**Performance Improvement Due to Increased Model Size:**  We thank the reviewer for raising this observation about increaded model size. We note that $f_{\mathrm{new}}$ is already trained from scratch in our setup. While combining StackEm with distillation might further reduce churn, one of the main advantages of StackEm is not having to change the training process of the underlying models, so this would defeat the purpose. It is also not obvious what a single neural network equivalent in architecture to the two member ensemble would look like. 

**Why $f_{\mathrm{new}}$ Is Trained from Scratch:**Training $f_{\mathrm{new}}$ from scratch is the standard approach in the literature. Initializing $f_{\mathrm{new}}$ from $f_{\mathrm{base}}$ has been shown to be a very weak baseline for churn reduction by Jiang et al. which in some cases causes $f_{\mathrm{new}}$ to have worse performance than if trained from scratch. This generalization gap when using warm starts has also been observed by Ash and Adams, and requires tricks to overcome which violates one of the main advantages of StackEm: not requiring any modifications to existing model training procedures.

**Cite Wolpert's Original Paper:** We now add this citation to our paper.

**Rephrasing "Finally, for StackEm we search over...":** We rephrase this sentence to the following:
"Finally, for StackEm only the Learned Combination requires hyperparameter tuning, namely a search over the space of models used to combine the outputs of $f_{\mathrm{base}}$ and $f_{\mathrm{new}}$. We consider logistic regression, random forests, and gradient boosting models, with the hyperparameters used for each listed in the appendix.

**Table 1 Clarification:** The "No Regularization" column represents training  $f_{\mathrm{new}}$ with no attempt to reduce churn w.r.t.  $f_{\mathrm{old}}$, and is thus the worst case scenario in terms of churn.

**Significance Test Used:** We now mention in the text that a one-sided t-test with unequal variances was used to determine statistical significance with a significance level of 0.05.


Ash, J., & Adams, R. P. (2020). On warm-starting neural network training. Advances in Neural Information Processing Systems, 33, 3884–3894.

From: ICLR.cc/2023/Conference/Paper3182/Authors
title: Response
comment: We thank the reviewer for the useful and constructive feedback.

**Motivation for Memory Mechanism:** The need for a memory mechanism comes from the observation that even during the course of training $f_{\mathrm{base}}$ we end up with unstable predictions between epochs as the average batch gradient is not a descent direction for all samples. We argue that if churn cannot be avoided from one optimization iteration to another during training, then the same holds true at test time where we cannot leverage the data or use it for regularization purposes. We clarify in the text that by memory mechansim, we mean any method which remembers the exact predictions of $f_{\mathrm{base}}$, which inlcudes keeping a copy of $f_{\mathrm{base}}$ across updates.

**Repetitions on Different Data Splits:** We clarify that our experiments are on different splits of the original training data like in Jiang et al., 2021 since $f_{\mathrm{base}}$ is trained on a random subset of 60% of samples from D_train which varies between random seeds. 

**Model Selection Criteria:** We clarify in the text that model selection in practice is done based on the Churn at Cold Accuracy metric defined by Jiang et al. 2021 where the method with lowest churn that at least matches the accuracy of the baseline model would be selected. I.e. we are not willing to sacrifice performance relative to baseline in order to reduce churn. Since "Confidence" achieves the lowest churn while doing at least as well at the baseline, it is preferred.

**Additional Calibration Figures:** We add similar figures for all other datasets to the appendix.

**Reorganizing Related Works:** We rewrite this section to make it have the following clearer structure. First, we discuss the types of churn which have been mentioned in the literature: variance, model adaptation, and model upgrade. We briefly mention the settings where each type of churn is relevant, and then emphasize that we focus on the model adaptation type. Then we follow the structure mentioned in the introduction where we group existing churn reduction techniques in the the categories of distillation-based and ensemble-based methods. Lastly, we note that of the churn reduction methods discussed, some are designed for specific tasks (e.g. image retrieval), and therefore cannot directly be used for other tasks (e.g. classification), yet at a high level some of the ideas behind those methods  can be adapted to other tasks. We conclude that there is no method for classification tasks that has the effectiveness of ensembles, yet is much more efficient, hence the need for StackEm.

**Notation Use for Margin Definition:** We apologize for lack of clarity in the notation, and change the definition of margin to be consistent with the remainder of the text. 

**Churn Definition Hard Constraint:** We thank the reviewer for pointing out the difference between the definition we use and that of Jiang et al. We add text following the definition stating that we use a hard notion instead of a soft one like Jiang et al.

From: ICLR.cc/2023/Conference/Paper3182/Reviewer_VAE1
confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: This paper proposes a predictive churn reduction method that combines base and new model outputs to generate stable predictions with fewer negative flips. The authors first analyze the existence of negative flips through the perspective of inter-sample gradient compatibility, implying that the proposed method should remember the knowledge of the base model without constraining the learning of the new model. Then they design a stacking-based predictive churn reduction method that can generate a new prediction capable of correcting the mistakes of both new and old models.
The effectiveness of the proposed StackEm has been evaluated on several computer vision benchmark datasets by showing superior performance against a distillation-based method and comparable performance to an ensemble-based method.

strength_and_weaknesses: Strengths:
1. This paper tackles an important but easily neglected research problem, which has large practical value.
2. The proposed method is simple and intuitive but effective. And its empirical performance looks good.


Weaknesses:
1. It seems that the proposed method lack of theoretical guarantee, which can be improved by providing some rigorous analysis. Also, the presented analysis of inter-sample gradient compatibility in Sec. 3 cannot motivate the stacking-based methodology design.
2. More baselines discussed in the paper should be compared. Simply using the empirical investigation results of (Jiang et al., 2021) is not enough.
3. Many details require further clarification.
   - In sec. 3, the margin $M^t(x, y)$ is introduced without explanation.
   - The evaluation metric $C_\text{rel}$ (eq1) should be less than 1. Thus the results in the tables seem strange.
   - What is the detailed structure (input, model parameters, etc.) of Learned Model meta-models?
4. The writing is poor and requires large efforts to polish. Please see my further comment in the next part.

clarity,_quality,_novelty_and_reproducibility: There exists a large room for improving clarity, including
- The “Negative Flip Reduction” part in Sec. 2 (Related Works) is cumbersome and requires reorganization. First, the authors highlight the two categories of ensemble-based methods and distillation-based methods in the introduction, while here, they do not follow this categorization. Instead, two different categorizations are further introduced and mixed together. One is the context-based one (model adaptation, variance, model upgrade), and the other is the task-based one (classification, image retrieval, structured prediction). Second, there is no conclusion to this wordy discussion.
- Many sentences are hard to understand. For example, 
 “they can lead to errors on samples the previous model correctly predicted causing per sample regression in performance known as predictive churn.” and
“We propose a missing middle-ground solution called StackEm”
- Many small typos. As on page 3, “Bhojanapalli et al. (2021) investigated many of the factors responsible for for the variance notion of churn”.


Although the proposed predictive churn reduction method is simple and effective, it is based on stacking, and its technical novelty is limited. 

As for reproducibility, the authors claim to provide the source code after acceptance.

summary_of_the_review: Given the listed weaknesses and other comments on clarity and novelty, I stand for the rejection of this paper.
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 3: reject, not good enough

From: ICLR.cc/2023/Conference/Paper3182/Reviewer_mXtq
confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: The paper tackles the issue of relevant churn introduced by model adaptation, i.e., the fraction of predictions that flip from correct to incorrect when a base model is trained on new incoming data. The proposed method relies on a meta-model that combines the base and new model predictions. Results show superior performance to previous distillation methods (in terms of both accuracy and relevant churn), and comparable performance to ensemble methods (in terms of relevant churn).
strength_and_weaknesses: ### Strengths
- Research on churn has been driven by concerns on real-world deep learning systems. I appreciate that the paper motivates several of its design decisions with similar real-world concerns: targeting relevant churn because it is most disruptive to users, looking for training time efficiency over ensembles, and simplifying model training and deployment for already established pipelines.
- The apparent simplicity of the proposed method is a strength of the paper (for the reason presented above).

### Weaknesses
- My main confusion with the paper is the motivation for the memory mechanism via stacking outlined in Section 3. I think I understand each point individually, but I’m not sure I follow how the existence of incompatible examples during training suggests the need for a memory mechanism. The presented argument doesn’t provide insights into test time performance, which is what we care about. 
- In Table 1, I appreciate that the paper presents multiple repetitions for each experiment. However, in this specific setup, I believe it’d be more convincing to have repetitions based on different splits of the data (similar to Jiang et al., 2021).
- I’m a bit confused on how StackEm is evaluated. Is the idea to perform model selection between the ”Confidence”, ”KL-Divergence”, and “Learned” variants? Is model selection performed based on relevant churn only? 
- The paper states that it investigates the role of calibration in reducing churn, and enumerates it as one of its contributions. However, only one dataset is explored here (CIFAR-100). Results on the other datasets used in the paper would strengthen the conclusions in this contribution. 

clarity,_quality,_novelty_and_reproducibility: ### Clarity
- I got lost while reading the related work on negative flip reduction. I suggest breaking it up into paragraphs.
- Please specify what notation is being used when defining the margin in Section 3. 
- I believe [Jiang et al., 2021] define churn in terms of a soft constraint, not a hard one, as presented in Section 3. I’d specify this or change the citation. 

### Reproducibility
The appendix states that the code will be open-sourced.

### Quality and Novelty
The paper has some minor clarity issues. My main gripes are with two of its stated contributions: the connection to incompatible examples and the investigation of calibration. The clear contributions of the paper are thus limited. 

summary_of_the_review: The paper presents a simple method with encouraging results. However, of the four stated contributions (connection to incompatible examples, introduction of the method, results, investigation of calibration), I was only convinced of two of them (method, results of the method). The contributions are thus limited. At the moment, I vote to reject. 
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: Not applicable
flag_for_ethics_review: NO.
recommendation: 5: marginally below the acceptance threshold

From: ICLR.cc/2023/Conference/Paper3182/Reviewer_wW3E
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: The submission presents algorithms designed for scenarios where a classification model has been built, new data becomes available, another model is built from the extended dataset, and the aim is to minimize the number of new instances where the old model produces the correct classification while the new model produces an incorrect classification (denoted "relevant churn"), while also exploiting the additional information in the new data to improve classification accuracy overall. In the experiments presented in the submission, the best method for reducing relevant churn is to use both models for prediction, picking the predicted classification of the model that is the more confident one. A "stacking" method based on a learned meta-classifier and one based on KL-divergence are also evaluated. Interestingly, calibration using temperature scaling does not reduce relevant churn.
strength_and_weaknesses: - In real-world applications, models are rarely re-built just once
- Stacking is applied without cross-validation
- Comparison is performed only w.r.t. large ensemble

+ The proposed method is simple and works well
+ There is an interesting empirical analysis of "relevant churn"
+ The paper is well written and easy to follow
clarity,_quality,_novelty_and_reproducibility: The submission is very clear, the proposed method appears original, and the quality of what is presented seems quite high.
summary_of_the_review: The scenario considered in the submission does not seem very relevant to real-world applications. In practical applications, new data will not just become available once, and the model does not need to be re-built just once.

Really, only the method in Section 4.2., where a combination is learned, is what most people would consider a from of stacking. Also, typically, cross-validation or use of a hold-out set to generate the meta-level data is considered a crucial component of stacking to prevent overfitting, but this does not seem to be done in the "stacking" method in Section 4.2.

The proposed approach outperforms distillation, but this may be due primarily to the increased size of the model, which consists of two networks. One baseline that should also be compared to is a single neural network model that is equivalent in architecture to the two-member ensemble model with the "maximum probability" combination rule but trained from scratch on the extended dataset, both with and without the distillation loss.

Other comments and questions:

Cite Wolpert's original paper on stacked generalization as well.

Why is f_new trained from scratch? What happens if the proposed method is applied when f_new is initialized from f_base?

"Finally, for StackEm we search over the Confidence, KLDivergence, and Learned Model meta-models." -- rephrase

What is shown in the column labeled "No Regularization" in Table 1?

What significance test is used for Table 1 and what is the significance level?

"logisic regression"
correctness: 4: All of the claims and statements are well-supported and correct.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
details_of_ethics_concerns: N/A
recommendation: 5: marginally below the acceptance threshold

From: ICLR.cc/2023/Conference/Paper3182/Reviewer_tZq9
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: This paper introduces a novel and relatively simple technique for reducing the churn between an old, production model and a new, experimental model. Churn is defined as the number of correct predictions made by the old model that are flipped to incorrect in the new model. The setting in which this is preferable over (or in addition to) raw accuracy is when churn causes confusion, frustration, or lack of trust by the end consumers of a model's predictions. The technique involves first training a new model without consideration to the old model, and then constructing a meta-model which outputs either the old prediction or the new prediction, depending on whichever one is more confident, using the maximum logit value as a proxy for model confidence.
strength_and_weaknesses: Strengths:
* Very simple technique
* Efficient to train when retraining is infrequent
* As discussed by the author(s), this provides a somewhat middle ground between the common techniques of using ensembles or using variations of distillation to reduce churn
* Appears to perform very well on empirical studies

Weaknesses:
* If retraining is frequent, the size of the launched model will grow with each retraining, and the technique will become unsustainable
* Depends on the logits of each model to be well-calibrated, so that they can be meaningfully compared against each other
clarity,_quality,_novelty_and_reproducibility: Very easy to understand. Clearly written. Adequate amount of empirical results.
summary_of_the_review: I like this paper because it's a clever idea that's super easy to implement. It gives practitioners a new tool to try out for churn reduction which is meaningfully different from prior work and seems to be very effective.
correctness: 4: All of the claims and statements are well-supported and correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
recommendation: 8: accept, good paper

From: ICLR.cc/2023/Conference/Paper3182/Reviewer_bFoz
confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: The paper proposes a new method called StackEm to reduce negative flips using the idea of stacking. The authors claimed to have superior performance against previous methods (distillation and ensembles) in various real datasets. 
strength_and_weaknesses: **Strength**
 - The research problem of reducing negative flips is interesting. 

**Weaknesses**
 - Please see the Clarity, Quality, Novelty And Reproducibility section below.
clarity,_quality,_novelty_and_reproducibility: 1. Many mathematical notations used in the paper are not defined, making the paper very unclear. 
 - Page 3: How are $(x,y) \in \mathcal{D}$ defined?
 - Page 6: What is the random quantity in $\mathcal{F}_{\mathrm{base}} (x)$? 
 - Page 6: Is $ \hat{y}\_{base} $  in  $ P(\hat{y}_{\mathrm{base}} =y \mid x) $ random? It is not clear what randomness is considered in this equation. Same issues apply to $f^{(i)}(x) \sim \mathcal{N}(\mu, \Sigma)$ and equations derived from it.
 - Page 6: $\hat{\sigma}$ is not defined. 
 - Page 7: $H_{\phi}$ and $\mathcal{H}$ are not defined. 

2. On Page 7, "Finally, for StackEm we search over the Confidence, KL-Divergence, and Learned Model meta-models" is not clear. What exactly does "search over" mean?

3. On Page 6, I am less sure if the equation $\mathbb{E} [ P (\hat{\sigma} = y \mid x) ] =  \mathbb{E} \left[ \max_{k} \left( \mathrm{argmax} G_{\phi} (x)  \right) \right] $ is true: the left-hand side depends on the label $y$, but the right-hand side does not. Maybe this confusion is due to the lack of clear notation. Also, what if the baseline model always predicts the class 1 with 1.0 probability (the highest confidence) in binary classification settings $f_{\mathrm{base}} (x) = (0,1)$? Then, for any sample, $\sigma(x) = 1.0$ and becomes a trivial classifier. 


summary_of_the_review: I believe the submitted paper is not ready for publication: the clarity should be improved and the empirical performance shown in the Experiments section is not surprising. 

correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 1: The contributions are neither significant nor novel.
flag_for_ethics_review: NO.
details_of_ethics_concerns: N/A
recommendation: 5: marginally below the acceptance threshold