Entity Enhancement for Implicit Discourse Relation Classification in the Biomedical Domain (2021)

Abstract: Implicit discourse relation classification is a challenging task, in particular when the text domain is different from the standard Penn Discourse Treebank (PDTB; Prasad et al., 2008) training corpus domain (Wall Street Journal in 1990s). We here tackle the task of implicit discourse relation classification on the biomedical domain, for which the Biomedical Discourse Relation Bank (BioDRB; Prasad et al., 2011) is available. We show that entity information can be used to improve discourse relational argument representation. In a first step, we show that explicitly marked instances that are content-wise similar to the target relations can be used to achieve good performance in the cross-domain setting using a simple unsupervised voting pipeline. As a further step, we show that with the linked entity information from the first step, a transformer which is augmented with entity-related information (KBERT; Liu et al., 2020) sets the new state of the art performance on the dataset, outperforming the large pre-trained BioBERT (Lee et al., 2020) model by 2% points.

## introduction discourse relation classification (drc) involves automatically inferring the logical link between different text segments (such as causal, contrastive, temporal etc.). it has been shown to be a valuable preprocessing step to many downstream natural language processing tasks such as machine translation @xcite , text summarization @xcite and questionanswering @xcite . a main obstacle to a wider usage of automatic dr classifiers however lies in getting the classifiers to work reliably on domains other than the wsj, that discourse relation parsers are usually trained on pdtb @xcite and rst @xcite . moving to a different domain is particularly challenging in drc because the overall distribution of relations typically differs between domains, and because many of the content words that classifiers may rely on are very different between domains. we here focus on the most challenging subtask of implicit discourse relation classification, which involves classifying those relations that are not linked by any explicit connectives like "because" or "but". in order to correctly recognize implicit relations, the classifier needs to recognize subtle surface cues (which may differ between domains) and learn about typical content-related relations. for instance, from the example "it's hot outside, therefore i'd like to eat an icecream", the words "hot outside" and "icecream" are relevant cues for the relation. an overview of typical cues for determining a coherence relation is provided in @xcite . the key to improving automatic drc on a new domain hence consists of better encoding of the discourse relational arguments. as we will show below (in line with earlier findings by @xcite , it makes a big difference to have at least a small amount of in-domain discourse annotated data. we here explore drc on the biomedical domain, which seems particularly suitable because a discourse-annotated corpus is available (biodrb; @xcite , which we can use for evaluation, as well as a setting with a small amount of indomain training data. furthermore, the biomedical domain does have large raw text corpora available. an example instance from biodrb @xcite is shown below: in this paper, we first propose an unsupervised method using information retrieval and knowledge graph techniques for identifying text passages that are similar content-wise to the coherence relation we want to label. the underlying assumption here is that if two instances share the same entities in both the relational arguments, it is possible that they have the same or a similar discourse relation. this part of the method is applicable to any domain for which large amounts of in-domain text are available, but no in-domain discourse relation annotations. we find that this method helps to improve results substantially compared to a bi-lstm baseline model, but doesn't reach state of the art performance (which is set by transformer models). we therefore proceed to enrich a transformer model with the knowledge extracted from the unlabelled texts, using the k-bert model @xcite . the model is fine-tuned on the discourseannotated in-domain biodrb data. we show that this setting sets the new state of the art on discourse relation classification on the biomedical domain, achieving an accuracy of 69.57%. ## related work early approaches on biodrb use probabilistic classifiers such as naïve bayes, maximum entropy, etc. to predict the relation @xcite . @xcite combine representations from different types of embeddings including contextualized word vectors from elmo @xcite and achieve 55.9% accuracy on biodrb for in-domain training, and 29.52% in the cross-domain setting (reported in @xcite ). @xcite also explore the performance of bert @xcite models on the drc task on biodrb using cross-domain (fine-tuning on pdtb, testing on biodrb) as well as in-domain (fine-tuning on biodrb and testing on biodrb) settings. they find a very good performance of the bert model, which they attribute to its "next sentence prediction" task in pre-training. comparing the original bert model to biobert @xcite , which was trained on biomedical text, they however find that biobert has only a limited ability for learning domain specific representations: cross-domain performance is no better than for the bert model, and in-domain performance improvements are moderate at only 1.5% points. given that the entities play an important role in inferring implicit discourse relation in scientific texts, putting an emphasis on entities seems vital for achieving further improvements. in contrast with previous studies that (largely unsuccessfully) attempted to train on explicit discourse relations for learning to classify implicit classifiers in supervised ways, such as @xcite ; @xcite ; @xcite ; @xcite ; @xcite etc., we here propose an unsupervised voting pipeline and achieve good performance even comparing with supervised models like bert and biobert. we believe that the key difference lies in the fact that previous methods tried to learn surface cues from explicit relations and tried to use them for implicits (which does not work, because these features differ between explicits and implicits, see e.g., @xcite ; asr and demberg (2012)), while our method focuses on the content of the discourse relational arguments. ## unsupervised method with information retrieval system the successful usage of a memory network in @xcite showed that instances that share the same relation have close representations. we believe that for sparse data like biodrb, which has only around 2,000 labeled implicit instances in total, it is essential to use similar explicit instances to help find the latent patterns they share. in this section, we introduce an unsupervised method for implicit drc, which is inspired by a recent information retrieval method. the core idea is as follows: we use information retrieval methods to identify explicitly marked coherence relations from the corpus which are content-wise similar to the relation we want to la-bel. we then automatically label these explicitly marked instances (relying on the high drc accuracy of ca. 96% for explicit relations) and assign the majority label from the explicit instances to the implicit instance from our test set. ## drc with an entity-augmented transformer integrating external domain-specific knowledge into the model is beneficial for this task has been found by @xcite , who integrated the conceptnet relations as additional knowledge into the lstm network and achieved better performance on the pdtb. we here aim to explore whether model performance can be further improved by exploiting richer entity representations in specialized texts like the biomedical domain. the pipeline with softmatching proposed in the above section provides us with spo triples from related documents for each implicit relation instance in the test set. we here employ the recently proposed knowledgeenabled language representation model @xcite to integrate the external entity knowledge into the pre-trained language model for better argument representations. ## conclusion in this paper, we address the task of implicit discourse relation classification on biodrb in the biomedical domain. due to the importance of entities in scientific text, we decided to address this problem by identifying explicitly marked relations containing the same instances, and using a simple majority voting system. while this setting showed good performance in the unsupervised setting, much better results are achieved when at least a small amount of labelled data is available. we show that when a transformer model is augmented with entity information from the domain, the previous state of the art on the task is exceeded by 2% points.