M u CPAD : A Multi-Domain C hinese Predicate-Argument Dataset (2022)

Abstract: During the past decade, neural network models have made tremendous progress on in-domain semantic role labeling (SRL). However, performance drops dramatically under the out-of-domain setting. In order to facilitate research on cross-domain SRL, this paper presents MuCPAD, a multi-domain Chinese predicate-argument dataset, which consists of 30,897 sentences and 92,051 predicates from six different domains. MuCPAD exhibits three important features. 1) Based on a frame-free annotation methodology, we avoid writing complex frames for new predicates. 2) We explicitly annotate omitted core arguments to recover more complete semantic structure, considering that omission of content words is ubiquitous in multi-domain Chinese texts. 3) We compile 53 pages of annotation guidelines and adopt strict double annotation for improving data quality. This paper describes in detail the annotation methodology and annotation process of MuCPAD, and presents in-depth data analysis. We also give benchmark results on cross-domain SRL based on MuCPAD.

## introduction as a fundamental nlp task, semantic role labeling (srl), also known as shallow semantic parsing, aims to capture the major semantic information of a sentence based on predicate-argument structure. basically, srl tries to answer "who did what to whom where and when" @xcite . previous works have shown that srl can help various downstream tasks, including information extraction @xcite , plagiarism detection @xcite , machine translation @xcite , reading comprehension @xcite , etc. figure 1 gives two examples of srl structure. according to the definition of semantic roles, there exist two typical representation forms, i.e., the word-based and the span-based. this work adopts the word-based form, in which an argument corresponds to a single word. in contrast, span-based srl, adopted by most previous datasets, takes a word span as an argument. the direction of arcs is from predicates to arguments, and the labels indicate the types of semantic roles. for example, the arc from "买(bought)" to "裙子(dress)" with a label "patient" means that the semantic role between the predicate "买(bought)" and the argument "裙 子(dress)" is "patient". recently, chinese srl research has achieved tremendous progress, thanks to the rise of deep learning methods @xcite @xcite , especially of powerful pre-trained language models (plms) @xcite @xcite . however, existing studies on chinese srl mainly focus on the in-domain setting, where training and test data are from the same domain @xcite @xcite . srl performance drops dramatically when the domain of test data is different from that of the training data, known as the domain adaptation problem. meanwhile, with the rapid growth of usergenerated web data, cross-domain srl has become an important and challenging task in realistic nlp systems @xcite . however, due to the scarcity of multi-domain labeled data, recent research on srl makes very limited progress in the domain adaptation scenario. as far as we know, there are three publicly available chinese srl datasets, i.e., chinese proposition bank (cpb) @xcite , chinese nombank (cnb) @xcite , and chinese sem-bank (csb) @xcite . all these datasets mainly contain canonical texts from newspapers or magazine/textbook articles. in order to facilitate research on cross-domain srl, this paper presents mucpad, a multi-domain chinese predicate-argument dataset, consisting of 30,897 sentences and 92,051 predicates, from 6 different domains. overall, mucpad has the following important features. we release mucpad along with our annotation guidelines for research usage at https://github.com/suda-la/mucpad . propbank and nombank are built by adding predicate-argument structures to the constituents of syntactic parser trees in penn treebank @xcite . their semantic roles are naturally span-based, instead of word-based. propbank considers verbal predicates, while nombank supplements nominal predicates. following framenet, propbank and nombank use frames to represent semantic meanings of predicates. however, the development of frames is both time-consuming and labor-intensive, and requires annotators to be equipped with strong linguistic background. the texts of propbank and nombank are mainly from the news domain, i.e., wall street journal, except 426 sentences from the brown corpus, which is usually used as an out-of-domain section of prop-bank. it is also noteworthy that there are propbankstyle srl data for other languages, such as portuguese @xcite , arabic @xcite , finnish @xcite , and turkish (sahin and adali, 2018). chinese srl data. cpb @xcite , cnb @xcite , and csb @xcite bank @xcite . the semantic roles are based on pre-defined frames as well. moreover, sentences in cpb and cnb mainly come from canonical texts, such xinhua newswire, hong kong news, and sinorama magazine @xcite . in contrast, csb uses general-purpose role labels, such as "agent" and "patient", and the sentences are mainly from canonical texts such as online articles and news as well. domain adaptation. domain adaptation has been an important and challenging research topic in nlp @xcite @xcite @xcite . @xcite proposed a neural sharedprivate model for the cross-domain slot sequence tagging task, which utilizes separate bilstm encoders to obtain domain-invariant and domainspecific representations, achieving significant improvements on all domains. @xcite proposed parameter generation networks for crossdomain ner. they idea is dynamically generate parameters of network modules (such as bilstms) according to predicted domain distribution. to facilitate cross-domain chinese dependency parsing research, @xcite proposed a large-scale multi-domain dataset for chinese dependency parsing. they organized the nlpcc-2019 shared task on cross-domain dependency parsing @xcite . @xcite rank the first place in the shared task, based on a tri-training approach. however, possibly due to the lack of multidomain data, research on cross-domain srl is scarce so far. we hope our newly annotated muc-pad can promote future research in this direction. ## data annotation this section describe the annotation methodology and annotation process of mucpad in detail. annotation guidelines. after an extensive survey of previous works on srl data annotation, we compile 53 pages of annotation guidelines. we adopt 24 fine-grained general-purpose role labels to capture the semantic relationships between predicates and arguments, as shown in ## analysis on mucpad in this section, we analyze the mucpad dataset from different perspectives to gain more insights. annotation consistency. as aforementioned, each task is assigned to two annotators. if the two submissions are inconsistent, a third expert annotator is asked to handle the inconsistency and decide the final results. the first major row in the predicate-wise consistency ratio is defined as #predannoa∩annob #predannoa∪annob , where the denominator is the total number of predicates submitted by all annotators, and the numerator is the number of predicates with consistent arguments from all annotator pairs. we can see that the predicate-wise annotation consistency ratios in most domains are lower than 60%. even the highest predicate-wise consistency ratio, which is achieved in pc domain, is only 71.23%. for the first major row, "avg" represents the average predicate/argument-wise consistency ratios in six domains. for the second major row, "avg" represents the average accuracy of overall/each label in six domains. boldface indicates the maximum value of each row, underline represents the minimum value of each row. this means that more than a quarter of the annotation tasks need to be further checked by a third expert annotator, demonstrating the difficulty of the srl annotation task and the importance of performing strict double annotation to guarantee data quality. in addition, it is worth noting that the predicatewise consistency ratio in pc domain is much higher than that in the other five domains. we believe this is related to the average number of arguments per predicate. for further investigation, we calculate the average number of arguments and find the number of average arguments per predicate is the lowest in pc domain. therefore, it is relatively easier for the annotators to recognize the arguments in pc domain. ## approach based on our newly annotated multi-domain chinese srl data, we conduct preliminary experiments, aiming to provide benchmark results. specifically, we present a simple basic srl model and enhance the model with the contextualized word representations from bert for further improvements. besides, we also present a mtl framework to improve the srl performance by learning from multiple heterogeneous datasets simultaneously @xcite . in this work, we focus on the predicate-given setting, which means we do argument identification and classification according to the given predicates in one sentence. following previous works @xcite , we treat the predicate-given srl task as a word pair classification problem and try to find the predicate-argument structure ŷ with the highest score: ## conclusions this paper presents a multi-domain chinese predicate-argument dataset, named mucpad, which consists of 30,897 sentences with 92,051 predicates and covers 6 different domains. in particular, we adopt a frame-free annotation methodology, which does not require high-level linguistic background for defining frames for large amounts of new predicates or new senses in multi-domain data. besides, considering that omission of content words is ubiquitous in chinese, we explicitly annotate omitted core arguments with two special designed labels "hidden-subject" and "hidden-object" for better semantic understanding. to ensure annotation quality, we adopt strict double annotation and ask a third expert to handle annotation inconsistency. we also perform analysis on mucpad from different perspectives. finally, we conduct preliminary cross-domain experiments and analysis on mucpad.