C hinese Opinion Role Labeling with Corpus Translation: A Pivot Study (2021)

Abstract: Opinion Role Labeling (ORL), aiming to identify the key roles of opinion, has received increasing interest. Unlike most of the previous works focusing on the English language, in this paper, we present the first work of Chinese ORL. We construct a Chinese dataset by manually translating and projecting annotations from a standard English MPQA dataset. Then, we investigate the effectiveness of cross-lingual transfer methods, including model transfer and corpus translation. We exploit multilingual BERT with Contextual Parameter Generator and Adapter methods to examine the potentials of unsupervised cross-lingual learning and our experiments and analyses for both bilingual and multilingual transfers establish a foundation for the future research of this task.

## introduction fine-grained opinion mining has been a crucial task in natural language processing (nlp) for a long time @xcite @xcite @xcite @xcite and it aims to discover useful structural information of user opinions from unstructured text, which is the relation between expression and entities such as who expressed what kind of sentiment towards what?. the expression conveys attitudes including sentiments, agreements, beliefs, or intentions (e.g., voiced his condolences in figure 1 ); the entities consist of the holder who expresses the opinion (e.g., chen.) and the target which the opinion is expressed to (e.g., the families) @xcite @xcite . here we focus on the opinion role labeling (orl) task which is to identify opinion holders and @xcite holder @xcite expression to [the families] target . targets @xcite when the expressions are given. most of the previous researches focus on the english orl, benefiting from the benchmark mpqa dataset @xcite which includes spanbased annotations of opinion expressions, holders and targets. the task is commonly solved by sequence labeling models with the bio conversion scheme @xcite @xcite . recently, neural bilstm-crf models have achieved state-of-the-art performance on this task @xcite @xcite . however, the studies on other languages are relatively rare due to the scarcity of annotated datasets. to our best knowledge, there is only one exception by @xcite , which has annotated a small-scale dataset for the portuguese language. unsupervised cross-lingual transfer @xcite is one promising way to address the low resource problem for orl. under the neural setting, there are two representative categories of methods: model transfer @xcite @xcite and corpus translation @xcite . the model transfer trains a model on a resource-rich language by using only language-independent features such as multilingual bert @xcite and then apply it to the target language. the corpus translation approach firstly obtains parallel corpora through either human or machine translation and then projects the annotations from the source language to the target side. in this work, we present the first study of the chinese orl. first, we construct a benchmark corpus by manually translating the english mpqa corpus, which involves auto-translation (i.e., automatic sentence translation and opinion aligning) and human refinement. second, we investigate the performance of unsupervised cross-lingual transfer for chinese orl based on the annotated corpus. we investigate the contextual parameter generator networks (pgn) in multilingual bert with adapter (known as parameter efficient in learning) method @xcite (we call it pgn-adapter) and discover the complementarity of the model transfer and corpus translation methods. we conduct experiments on the newly constructed chinese dataset to evaluate our methods, together with the english mpqa corpus @xcite and the portuguese dataset @xcite for cross-lingual transfer. we observe that for the unsupervised cross-lingual transfer from the english corpus, the translation-based method is better than the model transfer, and their combination leads to further improvements. although the scale of the portuguese corpus is much smaller, adding it into the multilingual transfer still outperforms the bilingual counterpart. to summarize, in this paper, we have the following contributions: fine-grained opinion mining there have been a number of studies in fine-grained opinion mining @xcite @xcite . @xcite exploit a semantic role labeller to extract opinion holders and topics. choi et al. (2005) and breck et al. (2007) model the task by sequence labeling with crf to discover opinion holders and recognize opinion expressions, respectively. yang and cardie (2012)'s semi-markov crf model outperforms the standard crf, and irsoy and cardie (2014) and liu et al. (2015) use recurrent neural network for opinion mining. johansson and moschitti (2013) and katiyar and cardie (2016a) propose joint models for opinion expressions, holders and targets. opinion role labeling as for orl, marasović and frank (2018) exploit multi-task learning about how to use srl information to improve orl scores. zhang et al. (2019a) utilize semantic role labeling to enhance orl, where three different integrating approaches are compared. bo et al. (2020) propose a dependency-based graph convolutional networks to enhance orl with syntax information. all these studies focus on the english orl by using supervised models, assuming that a training corpus is already available. in this work, we investigate chinese orl, building a benchmark dataset for chinese manually and then studying unsupervised cross-lingual transferring for the task. cross-lingual transfer learning crosslingual transfer learning has been extensively applied in nlp, including sentiment classification @xcite , pos tagging @xcite @xcite , named entity recognition (zirikly and hagiwara, 2015), semantic role labeling @xcite , and dependency parsing @xcite @xcite . unsupervised cross-lingual transferring has received great interest @xcite , which is our major focus. the work of @xcite is mostly related to our study, which applies model transferring and corpus translation to dependency parsing. our work focuses on orl, applying the two approaches for the chinese language. pgn-adapter pgn-adapter is used for merging different languages to same space by adapter and pgn methods with bert. about the adapter method which a pre-trained network added between the transformer encoder layer, there are many studies on using adapter modules @xcite @xcite . pgn is first proposed by platanios et al. (2018) for universal neural machine translation task. and üstün et al. (2020) integrated that two methods above in dependency parsing which inspired us to merge this idea into our orl task. the transfer method doesn't work in eger et al. raw the president had sidelined masire after accusing him . translated 这位总统在指控马西尔之后退居二线。 revision 在指控马西尔之后，总统把他排挤到了一边。 raw russian guards seize 87 kg of heroin on tajik-afghan border . translated 俄罗斯警卫在塔吉克-阿富汗边境抓获87公斤海洛因。 revision 俄罗斯士兵在塔吉克-阿富汗边境缴获87公斤海洛因。 ## the construction of chinese dataset we manually construct a chinese orl dataset to facilitate our research. in order to reduce the overall cost, we exploit corpus translation to assist the construction process, converting the english mpqa corpus @xcite into chinese. the conversion contains the following four steps by order: (1) sentence translation, (2) manual revision, (3) opinion projection, and (4) manual correction. the first and third steps formalize into automatic corpus translation, which has been used as one approach for unsupervised cross-lingual transfer, and the second and fourth steps are used to ensure the final quality. the whole construction is conducted at the sentence-level. sentence translation neural machine translation (nmt) has achieved state-of-the-art performances for a range of language pairs @xcite . in particular, the state-of-the-art nmt can reach a bleu score over 45 @xcite . thus it is applicable to use nmt for automatic sentence translation. here we first translate all the english sentences of the mpqa dataset into chinese by using the google translator foot_0 automatically. manual revision next, we let several native speakers check the translation quality, and make revisions to the imperfect translations. there can be two types of revisions. on the one hand, the translated sentences may have errors, and human opinion projection third, we project all opinions (expressions, holders and targets) from the english sentence into its chinese translation. before the projection, we use the stanford segmentor tool for word segmentation foot_1 . the overall projection is supported by automatic word alignments, which can be produced by using a word-alignment tool. here we exploit the fast-align tool foot_2 @xcite to calculate the alignment probabilities. figure 2 shows an example to illustrate the projection process. concretely, given an english-chinese sentence pair (e 1 • • • e n , c 1 • • • c m ) and its english-to-chinese alignment probabilities a(c j |e i ), the projection is performed as follows: ## model opinion role labeling aims to discover the opinion arguments given opinion expressions. the task can be modeled as a sequence labeling problem @xcite . we adopt the bmeso scheme to convert spans of opinion arguments into a sequence of word-level boundary tags, where b, m and e denote the beginning, middle and ending words of an argument, respectively, s denotes the word itself is an argument, and o denotes the rest of the words. formally, assuming that the input sentence is @xmath0, ..., w n , and a given opinion expression is @xmath1, ..., w e (1≤ @xmath2, our task is to assign a sequence of boundary tags t 1 , ..., t n . we exploit a bilstm-crf framework based on pgn-adapter to implement our model. figure 3 shows an overview of our model. pgn-adapter this model @xcite is based on the traditional bert architecture @xcite the final encoder representation e i : the parameters in the bert model are frozen except the adapter part, thus our model can be much more parameter-efficient than fine-tuning, and meanwhile, our preliminary results show that the method does not hurt the performance. i e is a language embedding by a multi-layer perceptron mlp lang : further, word-level representations x 1 • • • x n can be derived by averaged pooling over the covered word pieces of each word. bilstm-crf following the front outputs, we apply bi-directional lstms (bilstm) and conditional random fields (crf) @xcite to get high-level features and compute the probability of each candidate output tag sequence @xmath3, • • • , y n . the concrete calculation method is defined as follows: training objective we exploit sentence-level cross-entropy loss for model training, which can be described as: ## experiment results and analysis this section provides an overview of the english-chinese transfer and multilingual transfer experiments on the basis of adding portuguese to the chinese target language. further, we analyze the results in detail. in order to understand the performance of different roles (i.e., the holder and target), we measure the performance variance along with the span length of the arguments. as for the cross-lingual analysis, apart from the mt-based setting, we also add one more semi-automatic setting, i.e., manual translation with automatic alignment. just to be clear in advance, the following method corpustrans is using the fixed multilingual bert embeddings only, no adapter or pgn-adapter method, as the train, dev and test datasets are all in chinese. ## conclusions we presented the first work of chinese orl. first, we manually constructed a chinese dataset with the help of corpus translation. then, we investigated unsupervised cross-lingual transfer for chinese orl. we studied two different approaches, model transfer and corpus translation, respectively. experiments and analyses were performed based on the annotated dataset. results showed that unsupervised cross-lingual transfer is an effective method for chinese orl, and in addition, multisource transfer further improves the results which are promising for future exploration of such crosslingual transfer to other low-resource languages.