TED - CDB : A Large-Scale C hinese Discourse Relation Dataset on TED Talks (2020)

Abstract: As different genres are known to differ in their communicative properties and as previously, for Chinese, discourse relations have only been annotated over news text, we have created the TED-CDB dataset. TED-CDB comprises a large set of TED talks in Chinese that have been manually annotated according to the goals and principles of Penn Discourse Treebank, but adapted to features that are not present in English. It serves as a unique Chinese corpus of spoken discourse. Benchmark experiments show that TED-CDB poses a challenge for state-of-the-art discourse relation classifiers, whose F1 performance on 4-way classification is 60%. This is a dramatic drop of 35% from performance on the news text in the Chinese Discourse Treebank. Transfer learning experiments have been carried out with the TED-CDB for both same-language cross-domain transfer and same-domain cross-language transfer. Both demonstrate that the TED-CDB can improve the performance of systems being developed for languages other than Chinese and would be helpful for insufficient or unbalanced data in other corpora. The dataset and our Chinese annotation guidelines will be made freely available.

## introduction recent years have witnessed increasing attention to the properties of discourse for a wide variety of natural language processing (nlp) tasks, e.g., machine translation @xcite , summarization @xcite , machine reading comprehension @xcite . one of those interesting properties is the coherence between clauses and sentences arising from shallow discourse relations. as empirical approaches for modeling discourse relations usually require corpora annotated with 1 https://github.com/wanqiulong0923/ted-cdb such relations, penn discourse treebank (pdtb) @xcite , based on the idea that the discourse relations are grounded in an identifiable set of discourse connectives or altlex expressions, has been widely applied in the field of natural language processing. largely because pdtb is effective to extract discourse semantic features, it serves as a useful substrate for the development and evaluation of neural models in many downstream nlp applications @xcite @xcite . because for chinese, discourse relations have only been annotated over news text and few of the resulting corpora are freely available, we have created the ted-cdb dataset. ted-cdb currently comprises 72 ted talks in chinese (∼268.1k words), annotated with 15,540 discourse relations -almost 3 times as many as the cdtb @xcite . because @xcite have shown that discourse relations in spoken discourse are expressed differently than in written text, for scenarios involving chinese spoken discourse (e.g., dialogue, spoken language translation), ted-cdb boasts unprecedented potential for exploitation and application. our contributions comprise: ## related work most annotations in pdtb style are conducted on written texts originating from news reports. before 2015, there has been just one corpus for spoken discourse @xcite , where pdtb annotations are constructed on italian dialogues. recently, researchers have realized that the pdtb annotation guidelines should be used more widely instead of just being confined to construct corpora of written texts. zeyrek et al. (2018) annotate 6 ted talks for 7 languages. scheffler et al. (2019) build a discourse corpus on twitter conversations. regarding chinese discourse corpora for discourse relations, as illustrated in therefore, the scarcity of chinese datasets, especially the lack of corpora for spoken monologues have significantly inspired to build ted-cdb. ## the ted-cdb corpus this section describes the annotation procedure for ted-cdb, including details on the data, annotation scheme, annotation process and agreement study among the annotators. ## statistics on ted-cdb the number of the intra-sentential relations and inter-sentential relations in pdtb-3 are almost the same, but clearly, we can see that the discourse relations in our corpus are more commonly annotated within the sentence, consisting of 9,847 intra-sentential relations and 5,693 inter-sentential relations. the reason perhaps lies in the use of punctuation, which is quite different in chinese than in english. for example, a comma sometimes serves the same function as a full stop in english @xcite . therefore, a long chinese sentence may require the use of multiple english sentences to express the same content and preserve grammatically @xcite . this may be why there are more intra-sentential relations in chinese than in english. we also compared the cdtb and our ted-cdb with respect to the sense distribution. this is displayed in figure to explore the discourse differences between implicit and the explicit relations, we compare the distribution of top-level senses between the two corpora. figure 2(a) and 2(b) show that there are more contingency relations among the explicits, whereas there are more expansion relations among the implicits. the statistics also tell us that in explicit relations, "因为" (because) and "所以" (so) are the top two most frequent connectives. ## transfer learning via ted-cdb we also conducted transfer learning experiments across discourse corpora in different domains and languages. in particular, we considered the following two tasks for transfer learning: (1) training on ted-cdb snf testing on cdtb and (2) training on ted-cdb and testing on ted-mdb. the former is for transfer learning across domains of the same language, while the latter for transfer learning across seven languages within the same domain. the goal of these transfer learning experiments is to investigate if ted-cdb would be helpful for improving the performance of systems being developed for other languages and for insufficient or unbalanced data in other corpora. ## conclusion we have presented ted-cdb, a large-scale dataset for discourse relations on spoken monologues in chinese. it is equipped with high-quality annotations and linguistic elements tailored for both chinese and the genre of spoken monologue. the benchmark results of pretrained language models suggest that ted-cdb is a challenging dataset, which can be used to promote further development on discourse relation recognition and discourselevel nlp tasks. moreover, we display the ability of ted-cdb to help address the issue of insufficient or unbalanced data on other corpora and improve the performance of models for other languages.