C h EMU -Ref: A Corpus for Modeling Anaphora Resolution in the Chemical Domain (2021)

Abstract: Chemical patents contain rich coreference and bridging links, which are the target of this research. Specially, we introduce a novel annotation scheme, based on which we create the ChEMU-Ref dataset from reaction description snippets in English-language chemical patents. We propose a neural approach to anaphora resolution, which we show to achieve strong results, especially when jointly trained over coreference and bridging links.

## introduction chemical research has contributed greatly to human society and wellbeing, including new medicines and vaccines @xcite . research is heavily reliant on knowledge of existing chemical processes and methods of chemical synthesis, which are documented in chemical research literature and chemical patents. given the rapid growth of both publications and patents in chemistry, the need for automatic methods to extract semi-structured knowledge from chemical texts is becoming increasingly critical @xcite . anaphora resolution is a key component of comprehensive information extraction @xcite . in chemistry, different chemical compounds are mixed and reacted together in different ways to generate novel compounds, and to understand the precise chemical process often involves both resolving anaphoric references and understanding chemical changes/interactions a given entity is involved in. for example, as seen in figure 1 , while the final mention of mixture on line 3 and that on line 4 are both coreferent and chemically identical, in the case of mixture on line 2 and the first mention of mixture on line 3, the chemical composition is the same but a transformation has taken place via the stir and cool actions. our aim in this paper is to both identify anaphoric references in chemical patents, and determine the chemical relation between each linked pair of entities. we propose a domain-specific annotation framework based on five types of anaphora relations combining coreference and bridging. we then construct a dataset following this framework, annotated by chemical experts who achieve high inter-annotator agreement. we additionally extend existing anaphora resolution methods to model anaphora in chemical text, and compare both component-wise and joint models for anaphora resolution. this dataset will be released as part of the upcoming chemu 2021 shared task foot_0 @xcite . our contributions in this paper are as follows: (1) we propose a novel annotation scheme for anaphora resolution in chemical patents; (2) we develop a novel anaphora-resolution dataset based on chemical patents; and (3) we extend a general-purpose coreference resolution method, and achieve strong results via joint training over coreference and bridging with domain-specific fine-tuning. ## related work anaphora occurs in two basic forms: coreference and bridging. coreference occurs when different expressions in a text refer to the same entity in the real world @xcite , while bridging occurs between discrete entities that are linked via lexical semantic, frame-based, or encyclopedic relations @xcite . most existing anaphora datasets focus only on coreference, predominantly using generic relations @xcite , but also using domain-specific relations for knowledge-rich corpora such as biomedical literature @xcite . the conll-2012 dataset (pradhan et al., 2012) is a general corpus consisting of texts from three languages (english, chinese, and arabic). it is annotated based on ontonotes v5.0 @xcite and includes two types of coreference relations: identity, a symmetrical and transitive relation; and appositive, two noun phrases that are adjacent and not linked by a copula. coreference resolution is modelled as a clustering task. the wikicoref corpus was constructed with the same relations, over wikipedia documents @xcite . bionlp-st 2011 @xcite is a domain-specific coreference corpus over abstracts from biomedical literature, focusing mainly on gene-protein coreference, considering four relations: relat (relative pronouns or adjectives, e.g. which), pron (pronouns, e.g. they), dnp (definite or demonstrative noun phrases marked with the, this, etc.), and appos (apposition). instead of modelling coreference resolution as a clustering task, here the direction of coreference links is preserved. as this corpus focuses on gene-protein coreference, the range of coreference phenomena is limited. the craft-cr corpus @xcite adds coreference relations to the colorado richly annotated full text (craft) corpus @xcite , following ontonotes v5.0 with minor adaptions, and including discontinuous expressions, domain-specific proper nouns, and a broad range of mention types. the definition of bridging is somewhat imprecise @xcite , and different corpora have adopted different definitions. based on @xcite , there are two types of bridging: referential bridging, which can be treated as a context-based relation; and lexical bridging, which describes lexical-semantic relations such as holonymy and meronymy. @xcite introduced the arrau corpus of general language texts for bridging, which consists of news, dialogue, and narrative text. in the corpus, entities are limited to noun phrases, and most bridging pairs are lexical relations, with only a small number of instances of referential bridging. isnotes @xcite includes 50 wall street journal (wsj) articles from the ontonotes corpus, and has both coreference and bridging annotations, with most of the bridging pairs being referential. bashi @xcite has both coreference and bridging annotations over 50 wsj articles based on the ontonotes v5.0 guidelines, with most bridging links once again being referential. @xcite developed a corpus called scicorp based on english scientific papers, following the same annotation scheme as bashi. due to limited dataset availability, most research has modelled coreference resolution and bridging separately. there are two basic approaches to coreference resolution. first is mention ranking methods, which aim to score the coreferent probability of mention pairs @xcite @xcite , and make the assumption that mentions have been preidentified, meaning they are heavily reliant on upstream mention detection methods. second is span ranking methods, which combine mention detection with coreference prediction @xcite @xcite @xcite , and tend to perform better. bridging methods can be grouped into: (1) rule-based methods @xcite @xcite ; and (2) machine learning methods @xcite @xcite . rule-based methods have been shown to achieve competitive results on domain-specific corpora, but equally to be domain brittle. @xcite jointly trained a model for coreference resolution and bridging by adapting a span ranking method for coreference @xcite , achieving good performance over various bridging corpora. however, they evaluated their model only on bridging. ## annotation scheme in this section, we introduce our annotation guidelines for anaphora resolution in chemical patents. the complete annotation guidelines are made available at @xcite . ## task definition anaphora resolution can be decomposed into a twostep task: (1) mention detection; and (2) anaphora relation detection. for the evaluation of mention and relation detection, we use precision, recall and f1. one issue here is that, for coreference resolution, anaphors can link to multiple antecedents. many coreference evaluation metrics @xcite @xcite cannot deal with this since they model coreference resolution as a clustering task, where all related antecedents and anaphors occur in one cluster, and assume a given mention occurs in a unique cluster. hence we adopt the approach to evaluation of @xcite , scoring coreference from two perspectives: (1) surface coreference; and (2) atom coreference. surface coreference considers whether the anaphor refers to the closest previous antecedent(s). atom coreference considers whether the anaphor refers to the correct antecedent(s). atom coreference links take the coreferent transitivity into consideration and can be generated from surface coreference links, which we use by default. for the corpus annotation, we use the brat text annotation tool. 2 to date, 220 snippets have been annotated by two chemical experts, a phd candidate and a final year bachelor student in chemistry. four rounds of annotation training were completed prior to beginning official annotation. in each round, the two annotators individually annotated the same 10 snippets (different across each round of annotation), and compared their annotations; annotation guidelines were then refined based on discussion. after several rounds of training, we achieved a high inner-annotator agreement of krippendorff's α = 0.92 @xcite at the mention level, foot_2 and α = 0.84 for relations. in total, 1,500 snippets will be annotated in the final dataset that will be used in the chemu 2021 shared task. the statistics of the current corpus, and train/dev/test set splits that form the basis of our experiments in this paper, are shown in ## methodology we propose a joint neural model for anaphora resolution. 4 similar to @xcite , our model adopts an end-to-end neural conference resolution @xcite , as outlined in figure 2 . assume the snippet has t tokens represented as vector @xmath0, ..., x t }, consisting of fixed pretrained word and character embeddings learned from a convolution neural network (cnn). for mention candidate detection, we follow the assumption of @xcite , considering continuous tokens as a potential span and computing the span score (s m ) for each possible span. specifically, span representation s i is obtained by the concatenation of output token representations (x * i ) from a bidirectional lstm (bilstm) @xcite , the syntactic head representation (h i ) obtained from an attention mechanism @xcite , and a feature vector of the mention (φ(i)): to reduce the number of spans considered, we use a beam of λt candidate mention spans. inspired by @xcite , the mention loss is defined as: gold m is the set of gold mentions that are involved in anaphora relations. for anaphoric relation detection, a span pair embedding is obtained by the concatenation of each span embedding (s m (i), s m (j)) and the element-wise multiplication of the span embeddings (s m (i) • s m (j)) and a feature vector (φ(i, j)) for span pair i and j: as coreference and bridging are different, we consider them separately. for coreference resolution, we follow @xcite in optimizing the marginal log-likelihood of all correct antecedents for a given anaphor: for bridging resolution, as we have four relations, we model it as a multiclass classification task for each span pair. we represent the bridging relation as a one-hot representation and introduce a new relation type no-relation for span pairs that do not have a bridging relation. the loss for bridging is: the total loss is @xmath1, where: in this section, we detail our experiments. we use similar hyperparameters to @xcite . specifically, we use glove embeddings @xcite with window @xmath2. for bilstm, glove embeddings with window @xmath3. character embeddings are learned from a charactor cnn with windows of 3, 4, and 5 characters, each with 50 filters. for bridging prediction, the feed-forward neural networks are composed of two hidden layers with 150 dimensions and rectified linear units @xcite . we separate the gold mentions into those for coreference and bridging. for joint training, the gold mentions are combined. overall, the joint training configuration achieves 54.2% f 1 score for coreference resolution and 71.1% f 1 score for bridging, representing +0.8% and +1.2% f 1 score absolute improvement over the component-wise models. this indicates that joint training improves the performance of both tasks. compared to bridging, the performance of anaphor detection in coreference resolution is lower, particularly in terms of recall, possibly because the data is sparser. to investigate the contribution of each step (mention detection vs. relation detection), we experiment with providing oracle mentions during the training process. we also plot model performance with increasing amounts of training data in figure 3 . while the model performance is starting to plateau, potential gains could be attained with more annotated data. the strong correlation between anaphor detection and relation detection is also self-evident in the graph. to perform error analysis, we analysed the model errors on the dev dataset. as detailed in for coreference resolution, errors can be attributed to three primary phenomena: for bridging, as shown in we further investigated the over-prediction problem in bridging. as shown in the reaction mixture was then heated at 50 o c. for 2.5 h. an additional portion of sodium borohydride (10 mg, 0.27 mmol) was added, and the reaction mixture was heated at 50 o c. for an additional 2 h... ## conclusion we propose a novel annotation scheme for anaphora resolution in chemical patents. for our annotation, we incorporate generic and domainspecific knowledge to define coreference and bridging specific to the chemical domain, based on which we created the novel chemu-ref dataset. our corpus analysis and inner-annotator agreement show the complexity of the task, as well as the high quality of annotation. we model anaphora resolution as two sub-tasks, mention detection and anaphora relation detection, and also propose a joint training model, which outperforms the separately-trained models. by incorporating embeddings pretrained on the chemical domain, we found that domain knowledge boosts performance. with detailed error analysis, we also identified directions to further enhance performance.