BERT -based Cohesion Analysis of J apanese Texts (2020)

Abstract: The meaning of natural language text is supported by cohesion among various kinds of entities, including coreference relations, predicate-argument structures, and bridging anaphora relations. However, predicate-argument structures for nominal predicates and bridging anaphora relations have not been studied well, and their analyses have been still very difficult. Recent advances in neural networks, in particular, self training-based language models including BERT (Devlin et al., 2019), have significantly improved many natural language processing tasks, making it possible to dive into the study on analysis of cohesion in the whole text. In this study, we tackle an integrated analysis of cohesion in Japanese texts. Our results significantly outperformed existing studies in each task, especially about 10 to 20 point improvement both for zero anaphora and coreference resolution. Furthermore, we also showed that coreference resolution is different in nature from the other tasks and should be treated specially.

## introduction the meaning of natural language text is supported by cohesion among various kinds of entities. such cohesion includes coreference relations, predicate-argument structures, and bridging anaphora relations. for example, there are various relations between entities in the japanese text (two sentences) of figure 1 . clarifying these relations is indispensable for computers to understand natural language texts. among these relations, coreference relations and predicate-argument structures for verbs have been actively studied. however, predicate-argument structures for nominal predicates and bridging anaphora relations have not been studied well, and their analyses have been still very difficult. recent advances in neural networks, in particular self training-based language models including bert @xcite , have significantly improved many natural language processing (nlp) tasks. by using these techniques, it is now possible to dive into the study on analysis of cohesion in the whole text, including nominal predicate-argument structures and bridging anaphora relations. in this study, we tackle an integrated analysis of cohesion in japanese texts. to the best of our knowledge, no study has focused on such an integrated analysis of text cohesion. first, we explain the characteristics of each relation in japanese texts. a coreference relation is a relation between nouns that refer to the same real-world entity, and the task of revealing this relation is called coreference resolution (cr). in figure 1 , "author" and "he" refer to the same entity. there is no japanese specific phenomenon in cr. a predicate-argument structure consists of a predicate and its arguments that fill each case of the predicate, such as who does/did what to whom. the task of clarifying these relations is called predicateargument structure analysis (pas analysis). in japanese, an argument is usually labeled by a case maker such as "が" (ga) "を" (wo), and "に" (ni), which roughly correspond to nominative (nom), accusative (acc), and dative (dat), respectively. when an argument is labeled by a case marker and has a dependency relation with a predicate, like the situation of "author-acc" and "know" in figure 1 , their relation is overt. on the other hand, even if an argument and a predicate has a dependency relation, the case marker is sometimes hidden when, for example, a topic maker is attached to the argument. in that situation, we need to clarify their relation: nom, acc, or dat, like the nom relation between "i-top" and "know" in figure 1 . we call it case analysis. japanese nlp has been long suffering from zero anaphora resolution because arguments are very often omitted in japanese texts. in figure 1 , for example, the dat argument of "show" is omitted in the second sentence, and "i" in the first sentence should be referred to. this study does not handle overt situations, since it can be analyzed with 95% or higher accuracy based on the recent accurate parsing technology. when we compare our method with related work, we discuss case analysis and zero anaphora resolution separately. when we consider an integrated cohesion model, we show their combined accuracy. pas is usually considered for verbal predicates (pas analysis for verbal predicates is called vpa, hereafter). an eventive noun is syntactically noun, but it takes arguments and they usually appear in the context. such eventive nouns are called nominal predicates. this study also considers pas analysis for nominal predicates, and it is called nominal predicate-argument structure analysis (npa). there is no overt situation for npa. even if an argument and a nominal predicate has a dependency relation like "visit of me", we have to do case analysis to detect nom. in most cases, like "visit" in figure 1 , we need to do zero anaphora resolution between entities with no dependency. a bridging anaphora relation is an anaphoric relation between an anaphor and its antecedent, where the antecedent makes up a kind of semantic insufficiency of the anaphor. for example, "roof" is the roof of a building, "price" is the price of a product, and "author" is the author of a work. in figure 1 , the antecedent of "author" is "novel", and that of "manuscript" is again "novel". bridging anaphora resolution (bar) does not classify the relation between an anaphor and its antecedent, but just judge if they have a relation or not. because of that, if they have a dependency like "novel-of author" in figure 1 , they are considered as overt and are not analyzed or evaluated by this study, like an overt situation of vpa. on the other hand, the relation between "manuscript" and "novel" are analyzed like zero anaphora resolution. considering relations among these tasks, vpa and npa are very similar, and bar is also similar to them since all three tasks consider important relations between different entities. on the other hand, coreference is a relation between identical entities and essentially different from vpa/npa/bar. taking the difference into account, this paper proposes integrated cohesion analysis models and discusses their performance experimentally. the contribution of this study is three-fold: ## related work although many studies have focused on japanese vpa @xcite @xcite @xcite , few studies worked on solving other related tasks simultaneously. shibata and kurohashi (2018) introduced a mechanism called an entity buffer to capture the saliency of entities in a document, and performed multi-task learning of vpa and cr. they showed that although cr and vpa are heterogeneous tasks, cr is improved by performing vpa. this is because vpa captures the saliency of entities and improves the representation of the entity buffer since entities referenced more by vpa are considered to be more salient. @xcite treated not only vpa but also npa simultaneously and showed that there was a gain on both sides. however, none of the studies performed both cr and npa. in this study, we perform vpa, npa, bar, and cr simultaneously. in english, semantic role labeling (srl) is a task similar to japanese vpa. li et al. ( 2020 ) achieved f1 score of 88.03 in srl using roberta @xcite . unlike in english, argument omission frequently occurs in japanese, and thus japanese vpa is more difficult. in @xcite , the performance of zero anaphora resolution is 58.1% (f1 score), while case analysis, a task of finding arguments that have a relation of case, can be solved with f1 score of 89.5%. there are not many studies on bridging anaphora resolution and coreference resolution in japanese. for bar, @xcite constructed a dictionary of expressions like "a の b" ("b of a" or "a's b") from a large raw corpus, and used this dictionary for the analysis. since bar is a very difficult task, their bar performance is an f1 score of 42.7. in english, bar and cr are performed in a span-based method @xcite @xcite , while in japanese, they are performed in a dependency-based method. although it cannot be directly compared with our study for this reason, hou (2020) and @xcite have improved the accuracy of bar and cr by employing a qa framework. @xcite showed the effectiveness of multi-task learning of cr and bar. although both are heterogeneous tasks each other, both have one thing in common in english: both of them need to extract the spans corresponding to mentions first. in contrast to cr, the corpus size of the english bar is very small. therefore, nn was not able to perform span extraction accurately. the improvement in the accuracy of bar with multi-task learning of cr is partly due to the fact that the cr provides a good span representation for bar. ## proposed method in this study, we perform vpa, npa, bar, and cr all together using bert. this model is called cohesion analysis model (camodel). in this section, we first describe our base model for performing only vpa, and then describe the camodel for multi-task learning of all four tasks. finally, we describe the coreference-aware cohesion analysis model (corefcamodel), which deals with cr specially. ## conclusion we proposed multi-task learning of cohesion analysis including verbal predicate-argument structure analysis, nominal predicate-argument structure analysis, coreference resolution, and bridging anaphora resolution using bert, and investigated the effect of each task on multi-task learning. our model significantly outperformed existing studies in each task, especially about 10 to 20 point improvement both for zero anaphora resolution and coreference resolution. furthermore, we also showed that coreference resolution is different in nature from the other tasks and should be treated specially. in the future, based on this fact, we would like to create a model that can better utilize coreference information for other tasks.