Orca: A Few-shot Benchmark for C hinese Conversational Machine Reading Comprehension (2023)

Abstract: The conversational machine reading comprehension (CMRC) task aims to answer questions in conversations, which has been a hot research topic in recent years because of its wide applications. However, existing CMRC benchmarks in which each conversation is assigned a static passage are inconsistent with real scenarios. Thus, model’s comprehension ability towards real scenarios are hard to evaluate reasonably. To this end, we propose the first Chinese CMRC benchmark Orca and further provide zero-shot/few-shot settings to evaluate model’s generalization ability towards diverse domains. We collect 831 hot-topic driven conversations with 4,742 turns in total. Each turn of a conversation is assigned with a response-related passage, aiming to evaluate model’s comprehension ability more reasonably. The topics of conversations are collected from social media platform and cover 33 domains, trying to be consistent with real scenarios. Importantly, answers in Orca are all well-annotated natural responses rather than the specific spans or short phrase in previous datasets. Besides, we implement three strong baselines to tackle the challenge in Orca. The results indicate the great challenge of our CMRC benchmark.

## introduction conversational machine reading comprehension (cmrc) has aroused an increasing research interest in recent years. accordingly, lots of large-scale cmrc benchmarks @xcite @xcite have been proposed, such as coqa @xcite , quac @xcite . technically, cmrc aims to endow a machine with the ability to understand the given text passage/paragraph and respond appropriately to a set of questions in a conversation. however, despite notable successes in a variety of types and on a sizable scale, there are still a number of important challenges that have rarely been mentioned in prior cmrc efforts: to this end, we collect orca, the first few-shot chinese cmrc benchmark. concretely, we hire professional annotators to annotate the following p of 𝒕𝒖𝒓𝒏𝟒 : 空间站分两种类型：第一种就是单一式的，而第二种是组合式的。而科学家搭建空间站的原因是方便 人类长期在太空中工作，不需要短时间返回地球进行资源补充。there are two types of space stations: single and combined. the reason why scientists build the space station is to facilitate human beings to work in space for a long time, and there is no need to return to the earth for resource replenishment in a short time. components in orca: 1) topic, which consists of several sentences to drive the whole conversation; 2) domain, only one or two words indicate the specific field to which the content of the conversation belongs; 3) conversations, where each turn is assigned a golden knowledgeable passage. importantly, the question and response in each turn are human-crafted, leading to more coherent conversations. and response-related passages are also manually selected from the search engine. figure 1 shows an abridged conversation in orca. in total, we annotate 831 conversations and 4742 turns in orca. we deliberately limit the size of our dataset to build strong cmrc models under few-shot/zeroshot settings, and without the need to collect data for each target domain. our dataset has the following salient features: 1) each turn is assigned with a passage as the knowledge source, i.e., the given passages are dynamically changed rather than only one topic-central passage. orca gives more in-depth background knowledge about the specific topic in this manner, making the following-up dialogue more natural and diversified. meanwhile, orca poses a greater challenge to existing methods since it necessitates modeling both dialogue history and dynamic evidence passages. 2) answers at each turn are natural and informative responses from human annotation rather than certain spans in the provided passage. with this, we can both evaluate models' comprehension ability and generation ability. 3) dialogue topics in orca are collected from november 2021 to november 2022 on weibo 1 , one of the most popular social media platforms in china. this means that the collected data reflects real human inter-1 https://m.weibo.cn/ ests, and has never been included in earlier benchmarks. hence orca can serve as a comprehensive tool for assessing the adaptability of current llms to novel knowledge and emerging questions. moreover, good results on orca are of practical interest. 4) we carefully annotate conversations across 33 domains, such as society, people, celebrity, book, finance. in contrast, as the most commonly-used datasets, coqa only has 7 domains and doqa contains 3 domains. the variety of data domains makes orca a better testbed to evaluate the generalization of cmrc models. in this paper, we implement two strong frameworks to tackle the challenge in orca: 1) in-context learning with llms like @xcite and chatgpt; 2) fine-tuning with language models that are less than 1 billion. in our experiments, the latter includes both end-to-end training and three-stage training. we conduct our experiments under both zero-shot and few-shot settings. the results indicate that orca provides a challenging evaluation testbed for the cmrc systems. ## dataset analysis as a result, orca contains 831 conversations and 4,742 turns. for each conversation, there are 5.71 turns on average, and each turn is assigned with a knowledgeable passage. we randomly split the collected data into support (train) and test set. concretely, the support set and test set contain 200 and 631 conversations, separately. each turn of a conversation needs to be evaluated in experiments. in short, we aim to provide a challenging evaluation testbed that can evaluate models' ability towards real scenarios in a reasonable way. ## models in this section, we introduce three strong baselines to address the challenge in orca: 1) incontext learning with large language models; 2) fine-tuning including end-to-end training and threestage training which consists of three modules: query rewriter, passage reader, and response rewriter (short for qpr). ## experiments in this section, we first introduce experimental settings. then automatic and human evaluation metrics are introduced to evaluate model performances. at last, we present extensive experimental results. ## conclusion we propose the first few-shot benchmark orca for chinese cmrc. unlike existing cmrc datasets that each dialog contains a static passage, we assign each turn a golden response-related passage in orca, which is more consistent with real scenarios. all the conversations in orca are hot-topic driven and collected from various domains, responses are natural and informative via human annotation. in this way, model's comprehension ability and generation ability towards real scenarios can be evaluated reasonably. moreover, zero-shot and few-shot settings are provided to evaluate model's learning ability towards unseen domains with few samples. experimental results of our proposed three baselines indicate that orca provides a challenging testbed for cmrc research.