A Dataset of Argumentative Dialogues on Scientific Papers (2023)

Abstract: With recent advances in question-answering models, various datasets have been collected to improve and study the effectiveness of these models on scientific texts. Questions and answers in these datasets explore a scientific paper by seeking factual information from the paper’s content. However, these datasets do not tackle the argumentative content of scientific papers, which is of huge importance in persuasiveness of a scientific discussion. We introduce ArgSciChat, a dataset of 41 argumentative dialogues between scientists on 20 NLP papers. The unique property of our dataset is that it includes both exploratory and argumentative questions and answers in a dialogue discourse on a scientific paper. Moreover, the size of ArgSciChat demonstrates the difficulties in collecting dialogues for specialized domains. Thus, our dataset is a challenging resource to evaluate dialogue agents in low-resource domains, in which collecting training data is costly. We annotate all sentences of dialogues in ArgSciChat and analyze them extensively. The results confirm that dialogues in ArgSciChat include exploratory and argumentative interactions. Furthermore, we use our dataset to fine-tune and evaluate a pre-trained document-grounded dialogue agent. The agent achieves a low performance on our dataset, motivating a need for dialogue agents with a capability to reason and argue about their answers. We publicly release ArgSciChat.

## introduction the enormous and ever-growing number of scientific papers @xcite make scientific text processing imperative. a considerable body of research in nlp is dedicated to developing methods to provide insights from such texts @xcite @xcite @xcite . recent advances in question-answering (qa) models made them a great asset in accessing the content of scientific papers @xcite . to develop and evaluate such qa models, various datasets (section 2) have been collected @xcite @xcite @xcite . for each scientific paper in these datasets, a set of questions with the exploratory intention is defined. these questions explore the content of the paper by seeking factual information. thus, answers in these datasets are mainly limited to yes/no, named entities, or extracted from the text of the paper. scientific answers should be explained with argumentation to be persuasive @xcite . if an answer is not persuasive, it might need an argumentation to be well understood. to build models with the ability to argue about a scientific fact motivates a vital need for a dataset of scientific argumentative dialogues. a scientific argumentative dialogue includes interactions with both exploratory, exp, and argumentative, arg, intents. exp intents address factual information and arg intents aim at the argumentative content of a paper. collecting such a dataset is challenging for two reasons. first, while there are linguistic theories about argumentative dialogues @xcite ; there is no intent set for collecting argumentative questions and answers in the scientific domain. second, discussions in a scientist domain (and an expert domain in a broader perspective) need specialized domain expertise. so, the process of incentivizing scientists and aligning them to chat about a paper should be optimized for scientists. thus, off-the-shelf crowdsourcing platforms, which are applied for data collection in generic domains using crowd subjects, hardly are applicable for our goals in this paper. in this work, we introduce argscichat, the first dataset of text-based argumentative dialogues on nlp papers (section 3). figure 1 shows an example dialogue from our dataset. dialogues are in english and conducted synchronously by subjects who are nlp scientists. each dialogue is about a single paper. each subject plays a different role in the dialogue. e is the expert on the research presented in the paper, and p (for proponent) is the subject aiming to learn about the scientific paper by posing questions to the expert. importantly, e and p argue about the factual content of the paper and each other's opinion based on rationale, r, selected from the paper. to collect argscichat, we define a set of intent classes for interactions (i.e., questions and answers). these intents address exploratory and argumentative interactions. we also develop a new dialogue collection tool in which we account for scientist' preferences during the dialogue collection process by allowing them to choose a paper to chat about. to put our dataset in context, we compare (section 4) it with three recent document-grounded qa datasets, i.e., coqa @xcite , quac @xcite and doc2dial @xcite . we then evaluate the diversity of dialogues in our dataset. finally, we use our dataset to finetune and evaluate led @xcite , a pre-trained dialogue agent for answering questions about a long text in generic domains. the goal of this experiment is to study the difficulty level of our dataset for current dialogue systems. our results (section 5) confirm the difficulty of our dataset. our main contributions are (1) a challenging scientific argumentative dialogue dataset on nlp papers, (2) a new set of intent classes for interactions in argumentative dialogues, (3) a new tool for collecting dialogues between scientists. we hope that our contributions pave the way for building human-like dialogue agents in scientific domains. these agents should foster our access to the factual content of scientific papers by answering our questions and also scientifically arguing about the paper's content. ## related work while the problem of distinguishing between opinionated and factual content is not new @xcite @xcite , there is no dataset consisting of dialogues with both exploratory (exp) and argumentative (arg) interactions in a scientific domain. the most similar datasets to ours deal with qa tasks on scientific papers @xcite @xcite @xcite @xcite . ## methodology our goal is to collect synchronous argumentative dialogues about the content of scientific papers. these dialogues should include exp and arg interactions (i.e., questions and answers) between scientists. thus, we begin with defining a set of intents for interactions in scientific argumentative dialogues. we then introduce our dialogue collection methodology and a tool to collect such dialogues. ## experiments we collected a set of argumentative dialogues on nlp papers. we design a set of experiments to assess if the collected dialogues reflect the properties of scientific argumentative dialogues, and then evaluate the difficulty of our dataset for an advanced dialogue agent. ## conclusions we propose a scientific argumentative dialogue dataset in the nlp scientific domain. in our dataset, dialogue sentences are in free-form english texts, written by nlp experts, and annotated with intent labels encoding the exploratory and argumentative intents of scientific argumentation. to collect this dataset, we define a new methodology, letting scientists introduce their scientific papers, choose the topic and the time of dialogues based on their preferences. in addition to in-depth analysis of our datasets, we report it as challenge for a strong dialogue agent. as shown in our experiments, even a small-sized but high-quality dataset contributes to developing dialogue agents in specialized domains.