ORCHID : A C hinese Debate Corpus for Target-Independent Stance Detection and Argumentative Dialogue Summarization (2023)

Abstract: Dialogue agents have been receiving increasing attention for years, and this trend has been further boosted by the recent progress of large language models (LLMs). Stance detection and dialogue summarization are two core tasks of dialogue agents in application scenarios that involve argumentative dialogues. However, research on these tasks is limited by the insufficiency of public datasets, especially for non-English languages. To address this language resource gap in Chinese, we present ORCHID (Oral Chinese Debate), the first Chinese dataset for benchmarking target-independent stance detection and debate summarization. Our dataset consists of 1,218 real-world debates that were conducted in Chinese on 476 unique topics, containing 2,436 stance-specific summaries and 14,133 fully annotated utterances. Besides providing a versatile testbed for future research, we also conduct an empirical study on the dataset and propose an integrated task. The results show the challenging nature of the dataset and suggest a potential of incorporating stance detection in summarization for argumentative dialogue.

## introduction recent development of large language models (llms) have pushed the general interest on dialogue agents to a new level, and increasingly powerful llms such as gpt series demonstrated promising capabilities across multiple application scenarios. among various tasks that have been assigned to dialogue agents, engaging argumentative dialogues @xcite has long been a challenging one. regardless of specific aims, whether winning a debate @xcite , convincing people @xcite , or † equal contribution. * corresponding author. 1 orchid is publicly available at https://github.com/ xiutian/orchid debate whether technology is ethically neutral? pro con ...tech is not conscious, so it cannot take a neutral position... it is the user's good and evil that gives tech ethical values, not tech itself... it depends on whether tech has biases that lead to differential outcomes... a gun is ethically different from a wheel, for the former tech tends to destroy... pro sum. con sum. tech has no position... ...yet no position is not neutral position... ...a few rounds later opening up minds @xcite , dialogue agents rely on two of foundation abilities: stance detection and summarization. stance detection aims to reveal attitudes of arguments, and the goal of summarization is to collect and condense information in order to build arguments. they collaboratively support comprehending and developing arguments, consequently both abilities are crucial for engaging argumentative dialogues @xcite @xcite . in general natural language processing (nlp), stance detection is to classify the stance ('favor', 'against' and 'none') of a piece of comment with respect to a target entity or claim @xcite @xcite . the other task, text summarization aims to compress information from a large piece of text and produce a concise and comprehensible digest @xcite . dialogue summarization, fittingly, takes dialogues as source text. however, some unique features of argumentative dialogues propose atypical challenges for the two tasks. regarding summarization tasks, argumentative dialogues, such as debates, meetings and online forum discussions, often contain contradictory utterances with conflicting stances @xcite , making them more convoluted to summarize. also, in comparison with written text, spoken dialogues naturally carry more noises such as mispronounces, rephrasing and repeated words that obstruct summarizaiton. meanwhile, unlike typical target-specific stance detection whose targets are explicit entities (e.g., 'metaverse'), stance detection on argumentative dialogues is targetindependent, meaning that the targets of those studies are claims in the form of complete sentences (e.g., 'the commercial value of metaverse is overestimated'). despite the progress made on the tasks, research community's effort has been decelerated by an insufficiency of proper language resources. existing dialogue summarization datasets are dominantly in english @xcite @xcite @xcite . regarding non-english summarization datasets, prior resources in chinese are either focus on one specific domain @xcite @xcite , or without stance-specific summaries @xcite . there still is a lack of multi-domain and annotated chinese dialogue summarization datasets. moreover, most existing stance detection datasets are target-specific regardless of language @xcite . overall, in terms of chinese language resources, the amount of datasets suitable for argumentative dialogue summarization is highly limited, and there currently is no benchmark for target-independent stance detection. to remedy this shortage and facilitate related research, we present orchid (oral chinese debate), to the best of our knowledge, the first chinese debate summarization dataset annotated with multi-granularity and stance-specific summaries, also the first chinese benchmark for targetindependent stance detection. our dataset consists of 14,133 fully annotated utterances and 2,436 stance-specific summaries from 1,218 real-world debates that were conducted in mandarin. we employed an automatic speech recognition (asr) to transcribe raw data, followed by manual postcorrection and annotation. we provide debate summaries on two-levels of granularity, short concise statements and long comprehensive summaries of both stances. stances and debaters are labelled at utterance level. furthermore, we conduct a preliminary empirical study on the constructed data. sharing this novel dataset, we hope to resupply the research community tackling tasks including dialogue summarization, stance detection and other argument mining tasks. in summary, our contributions are three-fold: (1) we introduce orchid, the first chinese dataset for debate summarization and target-independent stance detection; (2) we propose a new integrated task, stance-specific summarization, which is suggested by the experiment results to improve summarization on argumentative dialogues; and (3) we conduct preliminary experiments, benchmarking classical and newly-suggested tasks against our dataset, reporting corresponding results, and setting initial baselines for future work. ## related work: existing datasets we reviewed existing dialogue summarization and stance detection datasets as presented in ## creating orchid having reviewed existing stance detection and dialogue summarization corpora, we determine to construct a new versatile dataset that adapts to both tasks. to this end, we introduce orchid (oral chinese debate), as the name implies, a corpus that features oral debates in chinese. we select oral debates in competition scenario as our source for following reasons: (1) debates are highly argumentative and thematic, and stances of both sides are clearly stated and not subject to change; (2) debate utterances are high-quality in terms of logic and rhetoric @xcite , and such utterances are much less colloquial than daily conversations while retaining partial oral features; (3) since existing stance detection corpora are predominantly written texts, utterances of spoken styles and expressions, which debates offer, could be a valuable addition. the construction of our dataset consists four major stages: data collection, asr-aided transcription, manual annotation, and quality control. we first collect videos from public sources. next, we employ an asr system to obtain raw transcripts, followed by automatic labelling stances and manual correction. finally, we extract and construct stance-specific position statements and conclusive summaries. we also apply several quality control measures during the process. ## conclusions we have presented orchid, a novel dataset for target-independent stance detection and argumentative dialogue summarization in chinese. our dataset features in rhetorical real-world debates, multi-domain topics, stance-annotated utterances and stance-specific summaries, which invite future utilization to progress various nlp tasks. we benchmark several baseline stance detection and summarization methods on our dataset. furthermore, we propose a new integrated task that shows potential in improving summarization on argumentative dialogues. future work can include, for example, devising new models to obtain higher stance-detection accuracy, designing better metrics to better evaluate quality of summarization, and perhaps developing methods to scale up the size and further augment this dataset.