Relating Relations: Meta-Relation Extraction from Online Health Forum Posts (2021)

Abstract: Relation extraction is a key task in knowledge extraction, and is commonly defined as the task of identifying relations that hold between entities in text. This thesis proposal addresses the specific task of identifying meta-relations, a higher order family of relations naturally construed as holding between other relations which includes temporal, comparative, and causal relations. More specifically, we aim to develop theoretical underpinnings and practical solutions for the challenges of (1) incorporating meta-relations into conceptualisations and annotation schemes for (lower-order) relations and named entities, (2) obtaining annotations for them with tolerable cognitive load on annotators, (3) creating models capable of reliably extracting meta-relations, and related to that (4) addressing the limited-data problem exacerbated by the introduction of meta-relations into the learning task. We explore recent works in relation extraction and discuss our plans to formally conceptualise meta-relations for the domain of user-generated health texts, and create a new dataset, annotation scheme and models for meta-relation extraction.

## introduction the vast amounts of information now stored digitally in various forms e.g. social media and online forum posts, electronic health records, academic papers, etc., represent large amounts of unstructured data that contain useful information for a variety of purposes. due to the scale of this data, automatic knowledge extraction (ke) holds the promise of allowing the automatic extraction of machine interpretable knowledge from these unstructured sources. relation extraction (re) identifies relations that hold (typically) between entities in text; it is often an important and challenging subtask in ke -and is the primary focus of our thesis. although there has been much work on re and ke @xcite @xcite there still remain considerable problems, including the high cost of creating high-quality annotated data, adequate conceptualisation of higher-order relationships that link events and/or cross sentence boundaries, and extraction from messy user-generated text. in this research proposal we explore recent work in the field of re, and identify some of the limitations that current research still faces. next, we outline the specific problems that our research will address, before presenting a first outline of our approach based on meta-relations. we demonstrate the role of meta-relations with a worked example and outline how they will allow us to extract more richly structured data that more adequately captures the relationships between different types of information, as well as lowering the cognitive load on annotators. we then go on to discuss the data and methods we will use in our research, how these will progress our research, address our objectives, and conclude with a summary of the expected contributions of our thesis. ## research context this thesis will focus on relation extraction from natural language -with a particular focus on user generated text such as in online forum and social media posts. the task of relation extraction is typically approached by identifying named entities in the text, then identifying relations that hold between these entities. re is often a very important component of ke systems, where these entities and relations are then used to populate a knowledge graph. this section will discuss recent re approaches in order to establish the context our research will belong to. relation extraction is a task that has seen much attention over the years, as @xcite shows. previous work has used statistical approaches, feature based methods, and graphbased approaches, but more recently neural machine learning models that use word embeddings as inputs have become the norm. looking at shared tasks in re, we can see particular language models and neural architectures that have been widely adopted and adapted for re, such as the bert language model @xcite . a widely used architecture for re is the recurrent neural network (rnn), such as the bi-lstm @xcite -a very popular model that has seen wide adoption and further developments such as using attention to enhance performance @xcite . bert and bi-lstm based methods are the top performing models in various shared tasks, such as rurebus @xcite , ccks @xcite , fewrel2.0 @xcite , and bionlp 2019's agac and bb tasks @xcite . @xcite also identify various challenges still faced by re systems: current re models often work in a simplified setting, but struggle with more complex tasks such as inter-sentence relations, and in few shot learning and open domain scenarios. despite recent works attempting to overcome these challenges @xcite @xcite they can still be considered largely unsolved. one challenge commonly faced in re research is acquiring high quality labelled data to train and evaluate supervised models on. state-of-the-art models require a large amount of adequately varied, labelled data in order to be successfully trained, however re is usually domain specific and so for each new task it can be difficult to acquire or create high quality gold standard training data. this is one of the main motivations for few-shot and crossdomain re @xcite , as they both focus on learning with small amounts of labelled data. @xcite discuss the complexities that must be considered when creating labelled data for re, and the difficulties that are encountered. they explore issues such as the high cognitive load that annotators can encounter with complex annotation schemes, and the resource intensive nature of annotating data with experts. in addition to few-shot learning and cross domain adaptation, there have been other methods attempting to mitigate the lack of data, such as initially training on coarse-grained labels (which are much quicker and cheaper to produce), then adopting a label-as-you-go approach whereby users create fine-grained labels as the system is used @xcite . distant supervision has been used to increase usable data, and is applied to re by using the basic assumption "if two entities participate in a relation, any sentence that contains those two entities might express that relation" @xcite to create weakly labelled data. this understandably leads to noisy data, with false positives and incomplete labels, however it has still shown to work reasonably well in practice @xcite . several approaches have attempted to address the shortcomings of distant supervision methods in re, such as revising the basic assumption (that all sentences containing two entities with a known relation express that relation) to reduce the noise of the data @xcite , creating undirected graphical models for distant supervision @xcite , allowing more than one relation to be predicted for two entities. these models have since been further developed to introduce negative examples (of unrelated entity pairs) with an additional layer that denotes whether a relation holds for given entities @xcite , leading to an increase in precision by between 1-4% @xcite . @xcite add a variable when aligning the text with the knowledge base, denoting whether a relation fact is present in the text. the model penalises disagreement between this variable and the variable that indicates whether the relation is present in the knowledge base. this allows us to encode certain intuitions about how likely certain relations are to be present or absent from the text, helping to reduce the issue of missing labels. despite the continued development of distantly supervised approaches, noisy and incomplete data remains a problem that limits the performance of such systems @xcite . current re methods work best for relations that can be construed as holding between two named entities in text, e.g. [margaret hamilton, worked at, nasa]. there has been less progress on relations that are inherently higher order, such as temporal relations which hold between events, e.g. before[[margaret hamilton, worked at, nasa], [margaret hamilton, founded, hamilton technologies]], or causal relations, which can hold between events, entities, and relations. both temporal and causal re are typically treated as separate tasks from general re, which means they cannot take advantage of the mutually constraining nature of the general re. a recent survey of temporal relation extraction @xcite shows that extracting and classifying time expressions e.g. 'last week' and events e.g. 'pain has increased' are close to solved problems, but there still is much room for improvement in extracting relations that involve such time expressions. causal relation extraction has seen much attention, with a recent focus on deep neural network approaches, however a recent survey shows that causal re is also an open problem @xcite . developing re methods to better extract higher-order relations such as temporal and causal relations is still an important challenge for research to address. from our review of the relation extraction literature, of which selected highlights were explored in this section, we have identified several key areas to address in our thesis. we will focus on the task of extracting a higher-order family of relations -specifically temporal, comparative, and causalnaturally construed as holding between other relations. we will develop theoretical underpinnings of and practical solutions for the challenges of (1) incorporating meta-relations into re task conceptualisations, and annotation schemes for lower-order relations and named entities, (2) obtaining annotations for them with tolerable cognitive load on annotators, (3) creating models capable of reliably extracting meta-relations, and related to that (4) how to overcome the limited-data problem exacerbated by the introduction of meta-relations into the learning task. ## meta-relations the focus of the proposed thesis is a class of relations that are inherently meta, because they hold between (structured) relations, rather than monadic entities. this class includes temporal relations such as before and after, comparative relations such as longer and shorter, and causal relations. as a starting point, we propose to incorporate meta-relations into the re processes as follows, aiming to explore both sequential and joint modelling of the subtasks: with typical re, this is where the process would end and the (unordered) extracted information would be: wellbutrin started being taken, wellbutrin stopped being taken, wellbutrin caused nausea, and zoloft started being taken. this information is useful but lacks important information such as the order of events and causality. each relation is treated separately, and from this relation information we cannot infer the order of events as these same relations could be present in another order of events. e.g. if the user was taking zoloft and wellbutrin together, then they stopped taking wellbutrin after feeling sick. this demonstrates that the temporal order of events is a crucial aspect to properly understanding the content in a sentence/document. it is crucial in a number of real world tasks such as automated yellow card filling, foot_1 for reporting unknown adverse side effects from drugs, where the temporal order of events -which drugs a person was taking, at what time, and in what order -must be reported as all of these factors may play a role in the adverse reaction @xcite . figure 1 shows how temporal meta-relations could be integrated with other extracted relations. in doing this we extract an unambiguous temporal order that shows wellbutrin was taken, then the user felt sick, then they stopped taking wellbutrin and started taking zoloft. temporal metarelations are only one example, there are other types of meta-relations that could provide additional key pieces of information such as causal meta-relations. in our example in figure 1 , we can see two event relations from blez et al.'s (2019) annotation scheme: drug modif[@xmath0. here we could use a causal meta-relation to show explicitly that starting to take the labelled drug (in this case wellbutrin) is what caused the labelled effect (nausea). ## conclusion in this paper we have provided the research context that our work will contribute to, identified various limitations in current work, including data collection/creation, cross domain adaptation, higherorder relation extraction, and distantly supervised relation extraction. we then proposed using metarelations to extract more knowledge from text and discussed our plans to conceptualise the task of meta-relation extraction, and to incorporate metarelations in an annotation scheme and dataset we will create. we also identified challenges we anticipate in addition to the data and methods we plan to use, and how we will utilise these methods in our work. focusing on extraction of temporal, comparative, and causal relations in user-generated texts in the health domain, the expected contributions of our thesis are as follows: