Exploring Jiu-Jitsu Argumentation for Writing Peer Review Rebuttals (2023)

Abstract: In many domains of argumentation, people’s arguments are driven by so-called attitude roots, i.e., underlying beliefs and world views, and their corresponding attitude themes. Given the strength of these latent drivers of arguments, recent work in psychology suggests that instead of directly countering surface-level reasoning (e.g., falsifying the premises), one should follow an argumentation style inspired by the Jiu-Jitsu “soft” combat system: first, identify an arguer’s attitude roots and themes, and then choose a prototypical rebuttal that is aligned with those drivers instead of trying to invalidate those. In this work, we are the first to explore Jiu-Jitsu argumentation for peer reviews by proposing the novel task of attitude and theme-guided rebuttal generation. To this end, we enrich an existing dataset for discourse structure in peer reviews with attitude roots, attitude themes, and canonical rebuttals. To facilitate this process, we recast established annotation concepts from the domain of peer reviews (e.g., aspects a review sentence is relating to) and train domain-specific models. We then propose strong rebuttal generation strategies, which we benchmark on our novel dataset for the task of end-to-end attitude and theme-guided rebuttal generation and two subtasks.

## introduction peer review, one of the most challenging arenas of argumentation @xcite , is a crucial element for ensuring high quality in science: authors present their findings in the form of a publication, and their peers argue why it should or should not be added to the accepted knowledge in a field. often, the reviews are also followed by an additional rebuttal phase. here, the authors have a chance to convince the reviewers to raise their assessment scores with carefully designed counterarguments. 1 code at https://github.com/ukplab/emnlp2023_ jiu_jitsu_argumentation_for_rebuttals review i think this is a very interesting direction, but the present paper is somewhat unclear. we invite the referee to be speci c about the sections of the original manuscript that need more clari cation, allowing us to revise these sections. canonical rebuttal recently, the analysis of review-rebuttal dynamics has received more and more attention in nlp research @xcite @xcite . for instance, previous efforts focused on understanding the links between reviews and rebuttals @xcite and on categorizing different rebuttal actions (e.g., thanking the reviewers; @xcite . @xcite find that well-structured rebuttals can indeed induce a positive score change. however, writing good rebuttals is challenging, especially for younger researchers and non-native speakers. for supporting them in writing effective rebuttals, argumentation technology can be leveraged. in computational argumentation, existing efforts towards generating counterarguments are based on surface-level reasoning @xcite @xcite . that is, existing research focuses on directly rebutting the opponents' claims based on what is evident from the argument's surface form. in contrast, researchers also proposed theories that explain how one's arguments are driven by underlying beliefs @xcite . one such concept, rooted in psychology, relates to 'attitude roots' and corresponding finer-grained 'attitude themes'. these attitude roots are the underlying beliefs that help sustain surface opinions @xcite . based on this idea, the authors further proposed 'jiu-jitsu argumentation', a rebuttal strategy that relies on understanding and using one's attitude roots to identify generic but customizable counterarguments, termed canonical rebuttals. as those are aligned with the opponent's underlying beliefs, jiu-jitsu argumentation promises to change opinions more effectively than simple surface-level counterargumentation. in this work, we acknowledge the potential of leveraging the latent drivers of arguments and are the first to explore jiu-jitsu argumentation for peer-review rebuttals. concretely, we propose the task of attitude root and theme-guided rebuttal generation. in this context, we explore reusing established concepts from peer review analysis for cheaply obtaining reviewer attitude roots and themes @xcite : reviewing aspects, and reviewing targets (paper sections). we show an example in figure 1 : the example review sentence criticizes the clarity of the paper without being specific about its target. the argument is thus driven by the attitude root clarity and the attitude theme overall. the combination of these two drivers can be mapped to an abstract and generic description of the reviewer's beliefs. next, given a specific rebuttal action, here reject criticism, a canonical rebuttal sentence can be retrieved or generated, serving as a template for further rebuttal refinement. contributions. our contributions are three-fold: ## jiu-jitsu argumentation for peer review rebuttals we provide an introduction to jiu-jitsu argumentation and the task we propose. ## jitsupeer dataset we describe jitsupeer, which allows for attitude and theme-guided rebuttal generation by linking attitude roots and themes in peer reviews to canonical rebuttals based on particular rebuttal actions. to facilitate the building of jitsupeer, we draw on existing corpora, and on established concepts, which we recast to attitude roots and attitude themes. we describe our selection of datasets (cf. §3.1) and then detail how we employ those for building jit-supeer (cf. §3.2). we discuss licensing information in the appendix (cf. ## baseline experiments along with end-to-end canonical rebuttal generation, we propose three novel tasks on our dataset. ## conclusion in this work, we explored jiu-jitsu argumentation for peer reviews, based on the idea that reviewers' comments are driven by their underlying attitudes. for enabling research in this area, we created jitsupeer, a novel data set consisting of review sentences linked to canonical rebuttals, which can serve as templates for writing effective peer review rebuttals. we proposed different nlp tasks on this dataset and benchmarked multiple baseline strategies. we make the annotations for jitsupeer publicly available. we believe that this dataset will serve as a valuable resource to foster further research in computational argumentation for writing effective peer review rebuttals.