Modular Self-Supervision for Document-Level Relation Extraction (2021)

Abstract: Extracting relations across large text spans has been relatively underexplored in NLP, but it is particularly important for high-value domains such as biomedicine, where obtaining high recall of the latest findings is crucial for practical applications. Compared to conventional information extraction confined to short text spans, document-level relation extraction faces additional challenges in both inference and learning. Given longer text spans, state-of-the-art neural architectures are less effective and task-specific self-supervision such as distant supervision becomes very noisy. In this paper, we propose decomposing document-level relation extraction into relation detection and argument resolution, taking inspiration from Davidsonian semantics. This enables us to incorporate explicit discourse modeling and leverage modular self-supervision for each sub-problem, which is less noise-prone and can be further refined end-to-end via variational EM. We conduct a thorough evaluation in biomedical machine reading for precision oncology, where cross-paragraph relation mentions are prevalent. Our method outperforms prior state of the art, such as multi-scale learning and graph neural networks, by over 20 absolute F1 points. The gain is particularly pronounced among the most challenging relation instances whose arguments never co-occur in a paragraph.

## introduction prior work on information extraction tends to focus on binary relations within sentences. however, practical applications often require extracting complex relations across large text spans. this is especially important in high-value domains such as biomedicine, where obtaining high recall of the latest findings is crucial. for example, figure 1 shows a ternary (drug, gene, mutation) relation signifying that a tumor with map2k1 mutation k57t is sensitive to cobimetinib, yet the entities never co-occur in any single paragraph. such precision oncology knowledge is key for determining personalized treatment for cancer patients, but it is scattered among a vast biomedical literature of more than 30 million papers, with over 1 million being added each year foot_0 . recently, there has been increasing interest in cross-sentence relation extraction, but most existing work still focuses on short text spans. @xcite and @xcite restrict extraction to three consecutive sentences and @xcite to abstracts. docred @xcite , a popular document-level relation extraction dataset, consists of wikipedia introduction sections, each with only eight sentences on average. further, half of the relation instances reside in a single sentence, all effectively in a single paragraph. to the best of our knowledge, @xcite is the first to consider relation extraction in full-text articles, which is considerably more challenging. they use the ckb dataset @xcite for evaluation where each document contains, on "... the patient's peripheral blood indices are shown over time relative to the first dose of the mek inhibitor cobimetinib ..." (... 17 sentences spanning 2 paragraphs ...) "... map2k1 mutations appeared later with p.k57t expanding to become the dominant clone ..." (... 10 sentences spanning 3 paragraphs ...) "... expression of the map2k1 mutants in a lymphoid cell line showed that while each mutation was able to activate erk in the presence of vemurafenib, all mutations remained sensitive to mek inhibitors ..." document "... expression of the map2k1 mutants in a lymphoid cell line showed that while each mutation was able to activate erk in the presence of vemurafenib, all mutations remained sensitive to mek inhibitors ..." ## document-level relation extraction let e 1 , . . . , e n be entities that co-occur in a document d. relation extraction amounts to classifying whether a relation r holds for e 1 , . . . , e n in d. for example, in figure 2 , r is the relation of precision cancer drug response, and e 1 , e 2 , e 3 represent drug cobimetinib, gene map2k1, mutation k57t, respectively. the relation mention spans multiple paragraphs and dozens of sentences. direct extraction is challenging and ignores the elaborate underlying linguistic phenomena. the drug-response relation is explicitly mentioned in the last paragraph, though it is between "mek inhibitors" and "map2k1 mutations". meanwhile, the top paragraph states the isa relation between "cobimetinib" and "mek inhibitors", as apparent from the apposition. from the middle paragraph one can infer the isa relation between "k57t" and "map2k1 mutations". finally, "map2k1 mutants" in the last paragraph can be resolved with "map2k1 mutations" in the middle based on semantic similarity. combining these, we can conclude that the drug-response relation holds for (cobimetinib, map2k1, k57t) in this document (figure 2 ), even though cobimetinib never co-occurs with map2k1 or k57t in any paragraph. formally, we represent n-ary relation extraction by neo-davidsonian semantics @xcite : here, t is a text span in d, r is a reified event variable introduced to represent relation r, and the arguments are represented by binary relations with the event variable. the distributed nature of this representation makes it suitable for arbitrary n-ary relations and does not require drastic changes when arguments are missing or when new arguments are added. given this representation, document-level relation extraction is naturally decomposed into local relation detection (e.g., classifying if r t (r) holds for some paragraph t ) and global argument resolution (e.g., classifying a i (r, e i )). entity-level argument resolution can be reduced to mention-level argument resolution a(r, e) . = ∃e. [mention(e, e) ∧ a(r, e)], where mention(e, e) signifies that e is an entity mention of e. additionally, the transitivity rule applies: a(r, e) ∧ resolve(r, e, e ) =⇒ a(r, e ) here, resolve(r, e, e ) signifies that the mentions e, e are interchangeable in the context of relation mention r. for brevity, in this paper we drop the relation context r and simply consider resolve(e, e ). if e, e are coreferent or semantically equivalent, as in (map2k1 mutations, map2k1 mutants), resolve obviously holds. more generally, isa (e.g., k57t and map2k1 mutations) and partof (e.g., a mutation and a cell line containing it) may also signify resolution: coref(e, e ) =⇒ resolve(e, e ) isa(e, e ) =⇒ resolve(e, e ) partof(e, e ) =⇒ resolve(e, e ) also, transitivity generally holds for resolve: resolve(e, e ) ∧ resolve(e , e ) =⇒ resolve(e, e ) modular self-supervision relation detection distant supervision civc, gdkd, oncokb argument resolution data programming identical mentions, apposition joint inference transitivity ## modular self-supervision our problem formulation makes it natural to introduce modular self-supervision for relation detection and argument resolution ( relation detection the goal is to train a classifier for r t (r). in this paper, we consider paragraphs as candidates for t and use distant supervision @xcite for self-supervision. specifically, knowledge bases (kbs) with known relation instances for r are used to annotate examples from unlabeled text. co-occurring mention tuples of known relations are annotated as positive examples and those not known to have relations are sampled as negative examples. these examples are then used to train a paragraph-level relation classifier. here, we leverage the fact that paragraph-level distant supervision is much less noise-prone, but document-level relation mentions still observe similar textual patterns as paragraph-level ones, as can be seen in figure 2 . ## our model figure 3 shows our document-level relation extraction system, which uses deep probabilistic logic @xcite to incorporate modular self-supervision and joint inference. prediction module the prediction module comprises transformer-based neural models @xcite for local relation detection and global argument resolution. for relation detection, let (m 1 , • • • , m n ) be a candidate co-occurring mention tuple in a paragraph t . we input t to a transformer-based relation classifier, with mentions m i dummified. 2 for argument resolution, let (m, m ) be a candidate mention pair. we compute the contextual representation using a transformer model for both mentions and classify the pair using a comparison network. the input concatenates contextual representations of the entities as well as their element-wise multiplication. for the detailed neural architectures, see appendix a.