DISAPERE: A Dataset for Discourse Structure in Peer Review Discussions
Abstract
At the foundation of scientific evaluation is the labor-intensive process of peer review.
This critical task requires participants to consume vast amounts of highly technical text.
Prior work has annotated different aspects of review argumentation, but discourse relations between reviews and rebuttals have yet to be examined.
We present DISAPERE, a labeled dataset of 20k sentences contained in 506 review-rebuttal pairs in English, annotated by experts.
DISAPERE synthesizes label sets from prior work and extends them to include fine-grained annotation of the rebuttal sentences, characterizing their context in the review and the authors’ stance towards review arguments.
Further, we annotate every review and rebuttal sentence.
We show that discourse cues from rebuttals can shed light on the quality and interpretation of reviews.
Further, an understanding of the argumentative strategies employed by the reviewers and authors provides useful signal for area chairs and other decision makers.
Introduction
Peer review performs the essential role of quality control in the dissemination of scientific knowledge.
The recent rapid increase in academic output places an immense burden on decision makers such as area chairs and editors, as their decisions must take into account not only extensive manuscripts, but enormous additional amounts of technical text including reviews, rebuttals, and other discussions.
One long term goal of research in peer review is to support decision makers in managing their workload by providing tools to help them efficiently absorb the discussions they must read.
While machine learning should not be used to produce condensed accounts of the peer review text due to the risk of amplifying biases (Zhao et al., 2017), ML tools could nevertheless help manage information overload by identifying patterns in the data, such as argumentative strategies, goals, and intentions.
Any such research requires an extensive labeled dataset.
While the OpenReview platform (Soergel et al., 2013) has made it easy to obtain unlabeled public peer review text, labeling this data for supervised NLP requires highly qualified annotators.
Correct interpretation of the discourse structure of the text requires an understanding of the technical content, precluding the use of standard crowdsourcing techniques.
Prior work on discourse in peer review has focused this qualified labor force on labeling arguments extracted from the text, which enables the complete annotation of more examples, at the expense of research on non-argumentative behaviors in peer review.
While there has been extensive research and deep analysis of different aspects of peer review, the taxonomies used to describe review argumentation are disparate and not directly compatible.
Finally, there has been limited research into understanding the discourse relations between rebuttals and reviews (Cheng et al., 2020;
Bao et al., 2021), and none so far into the discourse structure of rebuttals.
This paper presents DISAPERE ( DI scourse
S tructure in A cademic PE er RE view), a dataset focusing on the interaction between reviewer and author 1 .
We give reviews and rebuttals equal importance, and emphasize the relations between them.
To enable the study of behaviors beyond the core arguments, we also annotate every sentence of both the review and rebuttal, and provide fine-grained labels for non-argumentative types.
We annotate at the sentence level not only for completeness but also to avoid the propagation of errors from argument detection.
We annotate four properties ( REVIEW - ACTION , FINE - REVIEWACTION , ASPECT , POLARITY ) of each review sentence, where the set of properties and their values were developed by synthesizing taxonomies from prior work.
We also annotate each sentence of a rebuttal with a fine-grained label indicating the author’s intentions and commitment, and a link to the set of review sentences that form its context.
[-It would be helpful to note in the description of Table 
-]
Figure 1 shows the DISAPERE annotation scheme on a minimal, fictional example review-rebuttal pair.
DISAPERE is intended as a comprehensive and high-quality test collection, along with training data to fine-tune models.
Our annotations are carried out by graduate students in computer science who have undergone training and calibration, amounting to over 850 person-hours of annotation work.
Much of the test data is double-annotated, and we report inter-annotator agreement on all aspects of the annotation.
We describe the performance of state of the art models on the tasks of predicting labels and contexts, showing that interesting ambiguities in the data provide the NLP community with research challenges.
We also show an example that demonstrates how decision makers could use models like these to understand trends and inform policies for future conferences (§ 5).
The contributions of this paper are as follows:
(1) a new labeled training dataset of 506 reviewrebuttal pairs (over 20k sentences) of peer review discussion text in English, where review sentences are annotated
with four properties, and rebuttal sentences are annotated with context and labels from a novel scheme to describe discourse structure; (2) a taxonomy of discourse labels synthesizing prior work on discourse in peer review and extending it to add useful subcategories; (3) a summary of the performance of baseline models on the dataset (§ 6); (4) examples of analyses on the dataset that could benefit peer review decision makers (§§and 5), and (5) extensive annotation guidelines and software to support future labeling efforts.
Related work
The design of this dataset draws upon extensive, but disparate prior work on this topic.
Many works, some addressed below, have taken advantage of the availability of review text hosted on OpenReview.
Argument-level review labeling
Prior work has developed label sets that address different phenomena.
Hua et al.
(2019) introduced the study of discourse structure in peer review by annotating argumentative propositions in the AMPERE dataset with a set of labels tailored to the peer review domain ( EVALUATION , REQUEST , FACT ,
REFERENCE , and QUOTE ).
Similarly, Fromm et al.
(2020)’s
AMSR dataset frames the problem as an argumentation process, in which the stance of each argument towards the paper’s acceptance or rejection is of paramount importance.
Both view peer review as argumentation, using argument mining techniques to highlight spans of interest.
While its goal is not to examine discourse structure per se, Yuan et al.
(2021) uses polarity labels to indicate each argument’s support or attack of the authors’ bid for acceptance.
Besides polarity, these examples follow Chakraborty et al.
by annotating each argument with the aspect of the paper it comments on.
In contrast to Yuan et al. (2021), we do not attempt or recommend generat- ing peer review text, instead focusing on analyzing human-generated text in peer review.
Review-rebuttal interactions
We also expand on work by Cheng et al.
(2020), who first annotated discourse relations between sentences in reviews and rebuttals.
While Cheng et al.
present new deep learning architectures, in this paper we focus on the creation and comprehensive annotation of a new dataset, illustrated with results from some less specialized baseline models.
Other research into rebuttals includes Gao et al.
Besides their main finding that reviewers rarely change their rating in response to rebuttals, they find that more specific, convincing and explicit responses are more likely to elicit a score change.
Observations from this paper are formalized into rebuttal action labels in DISAPERE.
Comparison of datasets
In DISAPERE we attempted to unify these schemas to form a single hierarchical schema for review discourse structure.
We then expanded this hierarchical schema to introduce fine-grained classes for implicit and explicit requests made by the reviewers.
The details of the correspondence between DISAPERE labels and those from prior work are summarized in Appendix A.
In contrast to prior work, DISAPERE labels discourse phenomena at the sentence level rather than the argument level.
This enables more thorough coverage of the text while avoiding the propagation of errors from machine learning models earlier in the annotation pipeline.
While using manually defined discourse units (above or below the sentence level) may more precisely capture some discourse information, a separate pass of discourse segmentation can hinder the use of discourse datasets, as achieving consistent and replicable annotation of argument units is known to be highly challenging (Trautmann et al., 2020), and also because few works actually tackle unit segmentation (Ajjour et al., 2017).
Dataset
Each example in DISAPERE consists of a pair of texts: a review and a rebuttal.
Labels for reviews and rebuttal sentences are described below.
Review sentence labels are summarized in Table 2, and rebuttal sentence labels in Table
[Table]
A M
P E R
E
A M
S R
A S
A PR e v i e w
A P E
D I S
A P
E R
E
[Table]
3. Review sentence labels 3.1. Review actions
REVIEW - ACTION annotations characterize a sentence’s intended function in the review.
Annotators label each sentence with one of six coarsegrained sentence types including evaluative and fact sentences, request sentences (including questions, which are requests for information), as well as non-argument types: social , and structuring for organization of the text.
3.1. Fine-grained review actions
We also extend two of these review actions with subtypes: structuring sentences include headers, quotations, or summarization sentences, and request sentences are subdivided by the nature of the request, distinguishing between clarification of factual information, requests for new experiments,
[Table]
I
[Table]
WA C T
I
ON
[Table]
F I N E - R E
V I E
W - A
C T
I
ON
S t r u c t
[Table]
e
[Table]
e s t
[Table]
Labels from all categories besides REVIEW - ACTION are optional, and thus may not add up to 100%.
requests for an explanation (e.g. of motivations or claims), requests for edits, and identification of minor typos.
3.1. Aspect and polarity
ASPECT annotations follow the ACL review form (Chakraborty et al., 2020; Yuan et al., 2021).
These distinguish clarity , originality , soundness/correctness , replicability , substance , impact/motivation , and meaningful comparison .
Following Yuan et al. (2021), arguments with an ASPECT are also annotated for POLARITY .
We label positive and negative polarities.
ASPECT and POLARITY are applied to sentences whose REVIEW - ACTION value is evaluative or request .
3. Rebuttal sentence labels
We annotate two properties of each rebuttal sentence: a REBUTTAL - ACTION label characterizing its intent, and its CONTEXT in the review in the form of a subset of review sentences.
3.2. Rebuttal actions
The 14 rebuttal actions (Table 3) are divided into three REBUTTAL - STANCE categories ( concur , dispute , non-arg ) based on the author’s stance towards the reviewer’s comments.
(1) concur : The author concurs with the premise of the context.
This includes answering a question or discussing a requested change that has been made to the manuscript, conceding a criticism in an evaluative sentence.
(2) dispute : The author disputes the premise of the context.
The rebuttal sentence may reject a criticism or request, disagree with an underlying fact or assertion, or mitigate criticism (accepting a criticism while, e.g., arguing it to be offset by other properties).
(3) non-arg : Encompasses rebuttal actions including social actions (such as thanking reviewers), and structuring labels, for sentences that organize the review.
Responses to request s are further annotated: if the author concur s, we record whether the task has been completed by the time of the rebuttal, or promised by the camera ready deadline; if the author dispute s, we record whether the task was deemed to be out of scope for the manuscript.
3.2. Rebuttal context
We refer to the set of sentences which a rebuttal sentence is responding to as the context of that
[Table]
sentence, with special labels for when referring to the entire review ( global context ) or the empty set ( no context ).
By not mandating a fixed discourse chunking, these annotations may handle situations when some rebuttal sentences respond to large sections of text, and other rebuttal sentences respond to specific sentences within those sections.
3. Data Source and Annotation
DISAPERE uses English text from scientific discussions on OpenReview (Soergel et al., 2013), which makes peer review reports available for research purposes.
We draw review-rebuttal pairs from the International Conference on Learning
Representations (ICLR) in 2019 and 2020, resulting in text within the domain of machine learning research.
Review-rebuttal pairs are split into train, development and test sets in a 3:1:2 ratio such that all texts associated with any manuscript occur in the same subset.
Overall statistics for the dataset are summarized in Table 4.
Authors are able to respond to each ICLR review by adding a comment.
Although rebuttals are not formally named, we consider direct replies by the author to the initial review comment to constitute a rebuttal.
While multi-turn interactions are possible, we focus on reviews and initial responses, and leave study of extended discussion for future work.
The text is separated into sentences using the spaCy (Honnibal and Montani, 2017) sentence separator.
Annotation was accomplished with a custom
[Table]
annotation tool designed for this task, which is available as part of the code release accompanying DISAPERE.
The tool is described in detail in
Appendix B. Annotators annotate each sentence of a review, then examine the rebuttal sentences in order, selecting sets of review sentences to form their context.
While this linking between sentences does not explicitly align multi-sentence chunks as in pipelined approaches to discourse alignment (Cheng et al., 2020), we note that since multiple sentences may be aligned to the same set of sen- tences in the review, some discourse structure is nevertheless latently implied.
3. Agreement
We report Cohen’s κ (Cohen, 1960) on the IAA of labeling both review and rebuttals, treating each sentence as a labeling unit (Table 5).
The annotators for each example are selected randomly from the pool of 10 annotators.
Cohen’s κ is calculated for sentences annotated at least twice.
Where more than two annotations were produced, we calculate κ between all pairs and normalize by the number of possible pairs.
The results show between moderate and substantial chance-corrected agreement between annotators, for both REBUTTAL - ACTION and REBUTTAL - STANCE labels (Appendix D provides details about agreement on context sentences).
While these IAA scores do illustrate the noise of the task, note that this is not highly unusual for discourse labeling tasks – e.g. Habernal and Gurevych (2017) and Miller et al.
both report α u between 0.4 and 0.5.
[Table]
Analysis 4. Context types
We separate the different types of rebuttal contexts in terms of the number and relative position of selected review sentences in Table 6, along with the four cases in which the context cannot be described as a subset of review sentences.
Notably, 84.81% of sentences are linked to some review context.
A small number of sentences refer to other sentences within the rebuttal, rather than any review context, posing a challenge for future work.
[Table] n t s . s e l ec t e d Multiple contiguous 42.
[Table]
Single sentence
[Table]
Mult.
non-contiguous
[Table]
N o s e n t s .
s e l ec t e d
Global context
[Table]
Context in rebuttal
[Table]
No context
[Table]
Context error
[Table]
Cannot be determined [Table] 4. Alignment One might reasonably hypothesize that the task of alignment between rebuttal and review sentences would be trivial, since authors are likely to respond to each point in the review in order.
We can show that this is not the case.
In Figure 2, we calculate Spearman’s ρ between rebuttal sentence indices and their aligned review sentence indices.
Rebuttals responding to each point in order would achieve ρ = 1 . 0 ; this case is rare.
Many examples with positive ρ < 1 . 0 indicate that authors do respond to points approximately in order, but a simple mapping based on order alone would not capture the correct alignment.
Thus, while linear inductive bias may be beneficial to alignment models, the task of determining rebuttal sentences’ contexts is not trivial.
[Figure]
R e q u e s t
E v a l u a t
i v e F a c t S t r u c t u r i n g S
o c i a l
O t h e r
R e q u e s t
E v a l u a t
i v e F a c t S t r u c t u r i n g S o c i a l O t h e r R e q u e s t
E v a l u a t
i v e F a c t S t r u c t u r i n g S
o c i a l
O t h e r
[Figure]
u m b e r o f r e b u tt a l s e n t e n c e s
4. Author interpretations of criticism
In our taxonomy, each argumentative REBUTTALACTION corresponds to a particular REVIEWACTION , which we refer to as its canonical REVIEW - ACTION (listed in ‘Reply to’ column of
Table 3). For example, answers are generally responses to requests , while conceding criticism is usually a response to an evaluative statement. Annotations revealed that authors often interpreted review sentences as if they embodied REVIEWACTION s besides the canonical one, in a way that furthered the author’s argumentative goal. For example, authors often responded to evaluative statements as if they were requests , perhaps in order to appease a reviewer, although no action was explicitly requested.
Figure 3 shows the distribution of contexts for three different REBUTTAL - ACTION s.
4. Relating discourse features to rating
Figure 4 shows one possible analysis taking into account the rating of the review.
We show the distribution of FINE - REVIEW - ACTION labels of requests with review ratings.
It appears that high-scoring manuscripts are rarely asked to add experiments, and are polished enough to not elicit requests to fix typos.
Interestingly, low-scoring manuscripts have the second-lowest occurrence of typo requests, which could be due to the preponderance of other requests, but this bears further examination.
Application: Agreeability
Gao et al.
(2019) showed that reviewers do not appear to act upon the rebuttals responding their reviews.
It is possible that this is due to paucity of time on the reviewers’ part.
It is also common practice for area chairs to use review variance across a manuscript’s reviews as a practical heuristic to decide which manuscripts need their attention.
We propose that discourse information such as that described by DISAPERE can be used to provide heuristics that are data-driven, yet interpretable, and leverage information from the content of reviews rather than just numerical scores, resulting in better decision making.
One such measure is agreeability , which we define as the ratio of CONCUR sentences to argumentative sentences in a rebuttal, i.e.: agreeability = n concur n concur + n dispute .
We argue that low agreeability can indicate problematic reviews even in cases where the variance in scores does not reveal an issue, as illustrated in Figure 5.
Agreeability is only weakly correlated with rating, with Pearson’s r = 0 .
In Figure 5, 18% (28/159) of manuscripts would not meet the bar for high variance scores (top quartile), although their low agreeability (bottom quartile) indicates that they may merit closer attention from area chairs 3 .
and
Mean agreeability score
V a r . i n r a t i n g a c r o ss r e v i e w e r s
Authors agree with reviewers more
R e v i e w e r s a g r ee w i
t
h e a c h o t
h e r m o r e
Avg. rating 1.  Baselines
Two types of machine learning tasks can be defined in DISAPERE. First, a sentence-level classification task for each of the four review labels and the two levels of rebuttal labels. Second, an alignment task in which, given a rebuttal sentence, the set of review sentences that form its context are to be predicted. The models described below are not intended to introduce innovations in discourse modeling, rather, we intend to show the off-the-shelf performance of state-of-the-art models, and indicate through error analysis the phenomena that are yet to be captured.
6. Sentence classification
For the six classification tasks, we use bert-base (Devlin et al., 2019) to produce sentence embeddings for each sentence, then classify the representation of the [CLS] token using a feedforward network.
We report macro-averaged F1 scores, shown in
Table 7.
In general, F1 is lower for tasks with larger label spaces.
While the performance is reasonable in most cases, there is still room for improvement.
While ASPECT achieves a particularly low F1 score, its κ is within the bounds of moderate agreement; thus, this must be accounted for by the inherent difficulty of the task rather than a deficit in data quality.
[Table]
As one might expect, errors in the classification results largely mirror disagreements in the annotations, which in turn reflect particularly ambiguous utterances.
One example is the occurrence of rhetorical questions, such as (1) in Table 8, incorrectly labeled as request instead of evaluative .
In fact, for sentences such as (1), additional context would disambiguate its type: the reviewer answers the question in the next sentence, and hence both sentences were labeled evaluative .
Similarly, (2) was labeled fact , but since it is an integral part of a reviewer’s argument against the soundness of the paper, should have been labeled evaluative .
Certain reviewers also use conventions that do not fit the general schema we observed when developing DISAPERE.
For example, (3), an opinionated heading, could be considered both structuring and evaluative .
Finally, certain lexical cues a model may pick up on can be quite subtle.
For example, though they share a prefix, sentences (4) and (5) are clearly evaluative and request respectively.
6. Rebuttal context alignment
We model rebuttal context alignment as a ranking task.
Ideally, a model should rank all relevant review sentences higher than non-relevant review sentences.
As a baseline, we use an information retrieval (IR) model based on BM25 that, given a rebuttal sentence ranks all the corresponding review sentences.
We also report results from a neural sentence alignment model based on a twotower Siamese-BERT (S-BERT) model (Reimers and Gurevych, 2019).
We add a NO_MATCH sentence to the review, to which rebuttal sentences without context sets in the review are aligned.
Then, each review and rebuttal sentence is encoded independently using a S-BERT encoder and the similarity between two sentences is computed using cosine
[Table] [Table]
wonder if the authors ever looked at how much [...] determines the performance of the system?
[Table]
various datasets evaluated using mean reciprocal rank (MRR) Mean Average Precision (MAP)
[Table] Surprisingly, the BM25 model outperforms a neural model (Thakur et al., 2021). While this shows that lexical information is a useful signal, both models have significant scope for improvement and lexical overlap is clearly not sufficient for this task.
Importantly, neither of these models account for the context of the rebuttal sentence, and predict each sentence’s context independently. Incorporating this information is likely to lead to performance gains; however, we leave this investigation to future work.
Conclusion
As the burden of academic peer reviewing grows, it is important for program chairs and editors to act upon data-driven insights rather than heuristics, to make the best possible use of participants’ scarce time.
Models trained on data like DISAPERE will allow decision makers to glean deep insights on the interactions occurring during peer review.
Almost all publicly available peer review data is from the domain of artificial intelligence, limiting the scope of DISAPERE and any similar project.
While this means that models trained on
DISAPERE won’t necessarily generalize to all new domains, we hope that with the detailed annotation guidelines and seamless data collection using the software provided with this paper support, users can build on our work, and ensure that their insights are robust to differences over time and across fields.
Ethics
The outcomes of peer review can have outsize effects on the careers of participating scholars. As machine learning models are known to amplify biases, we strongly recommend against using the outputs of any machine learning system to make decisions about individual cases. A dataset like
DISAPERE is best used to survey participants’ behavior. Any interventions based on this information should be subjected to studies in order to ensure that they do not introduce or exacerbate bias. Acknowledgments This material is based upon work supported in part by the National Science Foundation under
Grant Numbers IIS-1763618, IIS-1922090, and IIS-1955567, in part by the Defense Advanced
Research Projects Agency (DARPA) via Contract No.
FA8750-17-C-0106 under Subaward No. 89341790 from the University of Southern California, in part by the Office of Naval Research (ONR) via Contract No. N660011924032 under Subaward No. 123875727 from the University of
Southern California, in part by IBM Research AI through the AI Horizons Network, in part by the Chan Zuckerberg Initiative under the project Scientific Knowledge Base Construction, and in part by the Center for Intelligent Information Retrieval. Any opinions, findings and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect those of the sponsors.
A Rationale for taxonomy construction
Our label sets leverage ideas from and commonalities between existing work in this domain, including AMPERE (Hua et al., 2019), AMSR (Fromm et al., 2020) ASAP-Review (Yuan et al., 2021), and
Gao et al.
• ASAP-Review’s polarity labels approximately correspond to arg-pos and arg-neg labels in AMSR • AMSR and AMPERE each label nonargumentative sentences in a similar manner • aspect labels from ASAP-Review apply only to certain types of sentences; namely request and evaluative sentences from AMPERE’s taxonomy.
• summary is an exception among ASAPReview’s aspect s, behaving similarly to AMPERE’s quote .
We thus include both of these under a structuring category.
• Further, in order to gauge the extent to which authors acquiesced to reviewers’ requests, we introduce a fine-grained categorization of the types of requests.
• Gao et al.
(2019) enumerates some features of rebuttals, including expressing gratitude, promising revisions, and disagreeing with criticisms.
We formalize these observations into our rebuttal label taxonomy.
B Annotation tool
Two modes of annotation are possible.
First, annotators can apply labels on a sentence-by-sentence basis.
Multiple labeling schemas can be annotated simulatenously, with the option of adding constraints so that certain values govern possible values for other properties.
This annotation mode is shown in Figure 6.
The second annotation mode can build on the output of the first annotation mode.
Here, sentences of a focus text (the rebuttal) are presented in sequence, and annotators are permitted to select one or more of the sentences in the reference text (the review) which form the context of the sentence of the focus text.
Further, a label can be applied to the alignment.
This annotation mode is shown in
Figure 7 and Figure 8.
C Annotated review-rebuttal pair
Figure 9 shows a truncated version of a reviewrebuttal pair from the train set of DISAPERE.
D Context overlap analysis
As a proxy for agreement of rebuttal spans we show the types of overlap between spans on rebuttal sentences from 81 examples annotated by two annotators in Table 10.
[Table]
[Table]
[Table]
E Additional Agreement Analysis
While some of the IAA scores on annotation are low, we note that the labels used in this task attempt to characterize relatively complex relationships in text.
To give more insight into such disagreements,
Figure 10 provides a confusion matrix regarding the REBUTTAL - ACTION labels.
Recognizing that there are often situations in which users of a dataset will hope to reduce a label set, we provide some guidance as to which such merges may be acceptable and which are not.
Many disagreements come from three labels which might be said to exist upon a continuum –
ANSWER , MITIGATE CRITICISM and REJECT CRITICISM .
We suggest that in the situation of needing to minimize IAA disagreement, one might consider first merging mitigate criticism into reject criticism .
The kind of disagreements seen between the two are understandable but nuanced: the difference between saying that the reviewer has a point (but that they disagree on the relevance of that point) and disagreeing with the point itself.
Out-of-context rebuttal sentences illustrating this are provided below as examples of this kind of ambiguous situation:
• We note that such rules are indeed limited to some extent, but they still capture a rather expressive fragment of answer set programs with restricted forms of external computations.
• The use of C valp for hyperparameter tuning was incidental and not a central point of our paper.
• We agree that the measure theoretic approach is not always necessary (indeed for angular actions, it is not needed), but it is necessary for a very common scenario – clipped actions.
Furthermore, we note that (as illustrated in the confusion matrix) a wide range of disagreements are hard to distinguish from “answer” labels, as authors often attempt to frame disagreements as simple answers to questions.
[Table]
This paper is an empirical contribution regarding SGD arguing that it presents two different behaviors which the authors name a noise dominated regimen
[Table]
and a curvature dominated
[Table]
I find the observations interesting [Table] but the contribution is empirical and not entirely new.
It would be nice if there were some theoretical results to back up the observations
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
: "Although our primary contributions are empirical, we also provided a detailed theoretical discussion in section 2, where we give a clear and simple account of why the two regimes arise.", " [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
-criticism ", "
[Table] [Table] A cc ep t f o r f u t u r e w o r k A cc ep t p r a i s e
A n s w e r
C on c ede c
r
i t i c
i s m
C on t r ad i c t a ss e r t i on
F o ll o w up que s t i on
M i t i ga t e c
r
i t i c
i s m
N one
O t he r
R e f u t e que s t i on
R e j e c t c
r
i t i c
i s m
R e j e c t r eque s t
S
o c i a l
S t r u c t
u
r
i ng
S u m m a r y
T a sk ha s been done
T a sk w ill be done
Label from Annotator
Accept for future work
Accept praise Answer
Concede criticism
Contradict assertion
Followup question
Mitigate criticism None
Other
Refute question
Reject criticism
Reject request Social
Structuring Summary
Task has been done
Task will be done
Labe l f r o m A nno t a t o r
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]