MR e D : A Meta-Review Dataset for Structure-Controllable Text Generation (2022)

Abstract: When directly using existing text generation datasets for controllable generation, we are facing the problem of not having the domain knowledge and thus the aspects that could be controlled are limited. A typical example is when using CNN/Daily Mail dataset for controllable text summarization, there is no guided information on the emphasis of summary sentences. A more useful text generator should leverage both the input text and the control signal to guide the generation, which can only be built with deep understanding of the domain knowledge. Motivated by this vision, our paper introduces a new text generation dataset, named MReD. Our new dataset consists of 7,089 meta-reviews and all its 45k meta-review sentences are manually annotated with one of the 9 carefully defined categories, including abstract, strength, decision, etc. We present experimental results on start-of-the-art summarization models, and propose methods for structure-controlled generation with both extractive and abstractive models using our annotated data. By exploring various settings and analyzing the model behavior with respect to the control signal, we demonstrate the challenges of our proposed task and the values of our dataset MReD. Meanwhile, MReD also allows us to have a better understanding of the meta-review domain.

## introduction text generation entered a new era because of the development of neural network based generation techniques. along the dimension of the mapping relation between the input information and the output text, we can roughly group the recent tasks meta-review: [this paper studies n-step returns in off-policy rl and introduces a novel algorithm which adapts the return's horizon n in function of a notion of policy's age.]←abstract @xcite all, the reviewers found that the paper presents interesting observations and promising experimental results.]←strength [however, they also raised concerns in their initial reviews, regarding the clarity of the paper, its theoretical foundations and its positioning (notably regarding the bias/variance tradeoff of uncorrected n-step returns) and parts of the experimental results. ]←weakness [in the absence of rebuttal or revised manuscript from the authors, not much discussion was triggered.]←rebuttal process [based on the initial reviews, the ac cannot recommend accepting this paper, but the authors are encouraged to pursue this interesting research direction.]←decision to some extent, the existing task settings are not so adequate because they do not have a deep understanding of the domains they are working on, i.e., domain knowledge. taking text summarization as an example, the most well-experimented dataset cnn/daily mail @xcite is composed of the training pairs of news content and human-written summary bullets. however, it does not tell why a particular piece of news content should have that corresponding summary, for example for the same earnings report, why one media emphasizes its new business success in the summary, but another emphasizes its net income. obviously, there is not a standard answer regarding right or wrong. for such cases, if we can specify a control signal, e.g., "emphasizing new business", the generated text would make more sense to users using the text generator. to allow controlling not only the intent of a single generated sentence but also the whole structure of a generated passage, we prepare a new dataset mred (short for meta-review dataset) with in-depth understanding of the structure of meta-reviews in a peer-reviewing system, namely the open review system of iclr. mred for the first time allows a generator to be trained by simultaneously taking the text (i.e. reviews) and the structure control signal as input to generate a meta-review which is not only derivable from the reviews but also complies with the control intent. thus from the same input text, the trained generator can generate varied outputs according to the given control signals. for example, if the area chair is inclined to accept a borderline paper, he or she may invoke our generator with a structure of "abstract | strength | decision" to generate a meta-review, or may use a structure of "abstract | weakness | suggestion" otherwise. note that for ease of preparation and explanation, we ground our dataset in the peer review domain. however, the data preparation methodology and proposed models are transferable to other domains, which is indeed what we hope to motivate with this effort. specifically, we collect 7,089 meta-reviews of iclr in recent years @xcite and fully annotate the dataset. each sentence in a meta-review is classified into one of the 9 pre-defined intent categories: abstract, strength, weakness, rating summary, area chair (ac) disagreement, rebuttal process, suggestion, decision, and miscellaneous (misc). to summarize, our contributions are as follows. (1) we introduce a fully-annotated meta-review dataset to make better use of the domain knowledge for text generation. with thorough data analysis, we derive useful insights into the domain characteristics. (2) we propose a new task of controllable generation focusing on controlling the passage macro structures. it offers stronger generation flexibility and applicability for practical use cases. ## mred: meta-review dataset in this paper, we explore a new task, named the structure-controllable text generation, in a new domain, namely the meta-reviews in the peerreviewing system. unlike the previous datasets that mainly focus on domains like news, the domain for meta-reviews is worth-studying because it contains essential and high-density opinions. specifically, during the peer review process of scientific papers, a senior reviewer or area chair will recommend a decision and manually write a meta-review to summarize the opinions from different reviews written by the reviewers. we first introduce the data collection process and then describe the annotation details, followed by dataset analysis. ## related work to facilitate the study of text summarization, earlier datasets are mostly in the news domain with relatively short input passages, such as nyt @xcite , gigaword @xcite , cnn/daily mail @xcite , newsroom @xcite and xsum @xcite . datasets for long documents include @xcite , @xcite . in this paper, we explore text summarization in a new domain (i.e., the peer review domain) and provide a new dataset, i.e., mred. moreover, mred's reference summaries (i.e., meta-reviews) are fully annotated and thus allow us to propose a new task, namely, structurecontrollable text generation. researchers recently explore the peer review domain data for a few tasks, such as peerread @xcite for paper decision predictions, am-pere @xcite for proposition classification in reviews, and rr @xcite for paired-argument extraction from review-rebuttal pairs. additionally, a meta-review dataset is introduced by @xcite without any annotation. our work is the first fully-annotated dataset in this domain for the structure-controllable generation task. there are also some datasets and annotation schemes on research articles @xcite @xcite , which differ in nature from the peer review domain and cannot be easily transferred to our task. a wide range of control perspectives has been explored in controllable generation, including style control (e.g., sentiments @xcite , politeness @xcite , formality @xcite , domains @xcite and persona @xcite ) and content control (e.g., length @xcite , entities @xcite , and keywords @xcite ). our structure-controlled generation differs from these works as we control the high-level output structure, rather than the specific styles or the surface details of which keywords to include in the generated output. our task also differs from content planning @xcite @xcite , which involves explicitly selecting and arranging the input content. instead, we provide the model with the high-level control labels, and let the model decide on its own the relevant styles and contents. ## conclusions this paper introduces a fully-annotated text generation dataset mred in a new domain, i.e., the meta-reviews in the peer review system, and provides thorough data analysis to better understand the data characteristics. with such rich annotations, we propose simple yet effective methods for structure-controllable text generation. extensive experimental results are presented as baselines for future study and thorough result analysis is conducted to shed light on the control mechanisms.