Paper: ARIES: A Corpus of Scientific Paper Edits Made in Response to Peer Reviews (ICLR-2024)

Reviewer comment (Experiments):
The core manual data is very tiny (196 alignments, 42 reviews total) and so some of the value of the dataset really rests on the quality of the synthetic alignment work detailed in A.5.

Author response:
It is true that the number of comments and reviews is relatively small, although we would note that we actually have over 10k comment-edit candidate pairs in our alignment experiments, and for edit generation our analysis produced statistically significant conclusions despite the small number of annotations.

Reviewer comment (Evaluation):
It's hard to have a good intuition about the quality of their IAA with Jaccard overlap of 65% in the reviews, but it does call into question whether the segmentation/identification of relevant review comments is clear. Insofar as there is the whole field of peer review segmentation and typing (e.g. Xua et al. 2021, Cheng et al. 2021, Kennard et al. 2021, Dycke et al. 2023), it might be worth checking if any of that is relevant for use (some of those may also be relevant to the analysis types).

Author response:
Of note, we did find that 88% of spans overlap between annotators; the reason for token-level jaccard being lower is that different amounts of context were selected (but qualitatively the overall meanings were similar).

Many of those prior works don't seem to report agreement for segmentation, but for those that do, they also find a fair amount of noise in the annotation.  While not perfectly apples-to-apples with our metric, Kennard et al (2022, DISAPERE) reports Cohen's kappa in the range of 0.5-0.6 for action/aspect annotation (their Table 5), while Hua et al (2019, AMPERE) reports Krippendorf's alpha of 0.61 for type annotation and Cohen's kappa of 0.93 for proposition segmentation (their page 2).  Our comment extraction consists of identifying "actionable" comments, so it can be thought of as a combination of segmentation and typing in that regard.

Reviewer comment (Theory):
If this is envisioned as part of a workflow, does it really make sense for the model to be freely determining in effect whether to comply, disagree, make promises, etc.? Wouldn't those components ideally make sense as a starting assumption for an edit?

Author response:
This depends on the result to some extent.  Given our result that the model does not match humans very well when writing edits, it would seem that in practice a user should provide some guidance to start the edit.  On the other hand, if models improve or if we can reliably determine which comments a model can address well, then it might make sense to let the model do some work without user input.

Another factor to consider is that it takes some work for an author to provide guidance on each edit, so it may be easier to receive a candidate edit, decide whether to accept/reject, and potentially revise the edit as needed, even if this results in more generated edits being rejected by the user.  The accept/reject approach has been used in some past work, but more user studies will be needed to compare the different ways of integrating these assistants into practical workflows.

Reviewer comment (Novelty):
The motivation and the usefulness of the proposed tasks are not clear. For the first task, comment-edit alignment, there are a number of papers that require the authors' answer letter to the editors that describe how the authors have responded to each of the comments raised by the reviewers. For such journal papers, there is no need to find correspondence between the edited parts of the revision. The second task is more puzzling. As the authors claim, GPT-4 often addresses responses on a superficial level without including technical details. It is totally unclear why such superficial or pretending responses are necessary.

Author response:
> For the first task, comment-edit alignment, there are a number of papers that require the authors' answer letter to the editors that describe how the authors have responded to each of the comments raised by the reviewers. For such journal papers, there is no need to find correspondence between the edited parts of the revision.

The purpose of the comment-edit alignment task is threefold: (1) it provides a challenging reasoning task to test the capabilities of LLMs, (2) if solved, it would provide a way to automatically obtain large-scale aligned data, which could be used to study peer reviewing and to train edit- and review-generation models, and (3) because many venues do not require authors to describe the edits or do not enforce those rules, automatic alignment could still be useful in tools that assist with tracking changes during peer review discussions.


> The second task is more puzzling. As the authors claim, GPT-4 often addresses responses on a superficial level without including technical details. It is totally unclear why such superficial or pretending responses are necessary.

Similar to comment-edit alignment, the edit generation task is motivated as both a challenging reasoning task and as a potentially useful task in writing-assistance applications.  Of course, we would not expect GPT to generate perfect edits due to both reasoning limitations and data limitations, but it is not clear in advance how well it could do.

One might imagine that GPT could produce edits that are very realistic, perhaps with some placeholders for missing information, and that an author could simply make some minor adjustments to update the paper.  However, what we find in this work is that the kinds of edits GPT generates differ from the kinds of edits that humans generate in several significant ways, such as having a high tendency to paraphrase the comment being addressed (which human authors rarely do).

We hope that our findings and dataset can motivate and assist future work on improved edit generation techniques.

Reviewer comment (Evaluation):
Even though the comment-edit alignment task is a binary classification task, the overall results are very low, much lower than 50%, which may imply that this data or this task is an ill-formed one.

Author response:
> Even though the comment-edit alignment task is a binary classification task, the overall results are very low, much lower than 50%, which may imply that this data or this task is an ill-formed one.

> Why are the results of micro scores for the comment-edit alignment so low, even though it is a binary classification task? What is the proportion between the positive and negative pairs, and what scores will be obtained under random guess?

The scores are below 50% because it is not a balanced binary classification task; the number of positives is quite small (1.7%) because most comments do not correspond to most edits (and vice versa); in general we would expect that with N comments and M edits per paper, the fraction of positive-labeled pairs would be proportional to 1/(N*M).

We have computed the scores of a random-guess classifier (using the test proportion of positive examples as the positive-guess probability) and added the results to Table 1.  All methods outperform random on almost all metrics, with the exception of deberta-biencoder and linkbert-crossencoder on micro-ao-f1; these have a fairly tight distribution of probability outputs and the threshold chosen on the dev set was a poor match for the test set.

Reviewer comment (Evaluation):
Why are the results of micro scores for the comment-edit alignment so low, even though it is a binary classification task? What is the proportion between the positive and negative pairs, and what scores will be obtained under random guess?

Author response:
> Even though the comment-edit alignment task is a binary classification task, the overall results are very low, much lower than 50%, which may imply that this data or this task is an ill-formed one.

> Why are the results of micro scores for the comment-edit alignment so low, even though it is a binary classification task? What is the proportion between the positive and negative pairs, and what scores will be obtained under random guess?

The scores are below 50% because it is not a balanced binary classification task; the number of positives is quite small (1.7%) because most comments do not correspond to most edits (and vice versa); in general we would expect that with N comments and M edits per paper, the fraction of positive-labeled pairs would be proportional to 1/(N*M).

We have computed the scores of a random-guess classifier (using the test proportion of positive examples as the positive-guess probability) and added the results to Table 1.  All methods outperform random on almost all metrics, with the exception of deberta-biencoder and linkbert-crossencoder on micro-ao-f1; these have a fairly tight distribution of probability outputs and the threshold chosen on the dev set was a poor match for the test set.

Reviewer comment (Evaluation):
In the macro evaluation, what is the reason that F1 scores are lower than both precision and recall? How are those F1 scores calculated?

Author response:
> In the macro evaluation, what is the reason that F1 scores are lower than both precision and recall? How are those F1 scores calculated?

The F1 is lower than P/R because the harmonic mean is lower than the arithmetic mean, and we are doing an arithmetic-mean macro average.  That is, we are individually computing the P/R/F1 for each group of comment-edit pairs (grouped by comment originally, and now by paper; see below for details) and taking the arithmetic mean of those to get the macro-F1.  We could imagine an extreme case where half of those groups have P=10, R=100, F1=18 and the other half have P=100, R=10, F1=18 and the macro-averages are ultimately P=55, R=55, F1=18.

Reviewer comment (Evaluation):
According to the precision and recall scores for the macro evaluation, the performance of BM25 looks better than or at least competitive with those of GPT-4 multi-edit. It would be better to discuss the differences between the results of those models.

Author response:
> According to the precision and recall scores for the macro evaluation, the performance of BM25 looks better than or at least competitive with those of GPT-4 multi-edit. It would be better to discuss the differences between the results of those models.

The high performance of BM25 is largely due to a bias in macro-f1, which favors methods that are conservative in their predictions (i.e., tending to output fewer positive predictions).  We agree that this makes it somewhat confusing to interpret, so we have changed the macro-average to group by paper instead of by comment, which reduces the effect of the bias.  For completeness, we explain the bias below:

The macro-f1 bias in question stems from the fact that some comments do not have any corresponding edits (i.e., when authors did not address a comment), and if a model correctly predicts zero alignments in these cases, we consider the F1 as 100% (the F1 is technically undefined in this case).  Getting a few perfect 100 scores can skew the results, and thus a model that is biased to output fewer positives has an advantage.  This is exacerbated by the fact that some comments correspond to multiple edits; models biased to output more positive predictions are more likely to do well when there are several positive labels, and yet these cases are effectively down-weighted by macro-averaging.  Thus, while GPT is much better at identifying any given comment-edit pair (as indicated by the micro-average), the macro average has the effect of up-weighting its false positives and down-weighting its true positives.

We have added some discussion of this bias in Appendix F of the paper.

Reviewer comment (Experiments):
This may be a common criticism for any paper showing that LLMs do not perform well on a certain task: It is possible that the poor performance of GPT-4 is due to inappropriate instructions or prompts. More analyses are needed on the effect of instructions. For example, if chain-of-thoughts prompting is used, would GPT-4 generate "deeper" edits?

Author response:
A proper chain-of-thought analysis may be difficult to do in the given time, but we did spend ~20 manual iterations to find a good prompt (using train examples to test).  Since the design space is quite large for this task, we limited scope to single-pass prompts, but in future work we would like to explore more options to improve GPT's ability to work with technical text, such as fine-tuning, multi-persona self-discussion, chain-of-thought, and perhaps user intervention.

Reviewer comment (Experiments):
Could you try chain-of-thoughts prompting or other more advanced techniques to see whether the performance of GPT-4 can be improved?

Author response:
A proper chain-of-thought analysis may be difficult to do in the given time, but we did spend ~20 manual iterations to find a good prompt (using train examples to test).  Since the design space is quite large for this task, we limited scope to single-pass prompts, but in future work we would like to explore more options to improve GPT's ability to work with technical text, such as fine-tuning, multi-persona self-discussion, chain-of-thought, and perhaps user intervention.

Reviewer comment (Reproducibility):
I would suggest directly writing "SPECTER2" rather than "SPECTER" in Table 1. Also, could you specify which adapter is used for SPECTER2?

Author response:
Thank you for pointing this out, we have updated the paper to specify the model.  In preliminary experiments we tried the PRX adapter and the base model (with no adapter) and found that the base model was better on the dev set, so that is the one we report.