PARE : A Simple and Strong Baseline for Monolingual and Multilingual Distantly Supervised Relation Extraction (2022)

Abstract: Neural models for distantly supervised relation extraction (DS-RE) encode each sentence in an entity-pair bag separately. These are then aggregated for bag-level relation prediction. Since, at encoding time, these approaches do not allow information to flow from other sentences in the bag, we believe that they do not utilize the available bag data to the fullest. In response, we explore a simple baseline approach (PARE) in which all sentences of a bag are concatenated into a passage of sentences, and encoded jointly using BERT. The contextual embeddings of tokens are aggregated using attention with the candidate relation as query – this summary of whole passage predicts the candidate relation. We find that our simple baseline solution outperforms existing state-of-the-art DS-RE models in both monolingual and multilingual DS-RE datasets.

## introduction given some text (typically, a sentence) t mentioning an entity pair (e 1 , e 2 ), the goal of relation extraction (re) is to predict the relationships between e 1 and e 2 that can be inferred from t. let b(e 1 , e 2 ) denote the set of all sentences (bag) in the corpus mentioning e 1 and e 2 and let r(e 1 , e 2 ) denote all relations from e 1 to e 2 in a kb. distant supervision (ds) trains re models given b(e 1 , e 2 ) and r(e 1 , e 2 ), without sentence level annotation @xcite . most ds-re models use the "at-least one" assumption: ∀@xmath3, e 2 ), ∃t @xmath4, e 2 ) such that t r expresses (e 1 , r, e 2 ). recent neural approaches to ds-re encode each sentence @xmath5, e 2 ) and then aggregate sentence embeddings using an aggregation operator -the common operator being intra-bag attention @xcite . various models differ in their approach to encoding (e.g., pcnns, gcns, bert) * equal contribution and their loss functions (e.g., contrastive learning, mlm), but agree on the design choice of encoding each sentence independently of the others @xcite @xcite . we posit that this choice leads to a suboptimal usage of the available data -information from other sentences might help in better encoding a given sentence. we explore this hypothesis by developing a simple baseline solution. we first construct a passage p (e 1 , e 2 ) by concatenating all sentences in b(e 1 , e 2 ). we then encode the whole passage through bert @xcite (or mbert for multilingual setting). this produces a contextualized embedding of every token in the bag. to make these embeddings aware of the candidate relation, we take a (trained) relation query vector, r, to generate a relation-aware summary of the whole passage using attention. this is then used to predict whether (e 1 , r, e 2 ) is a valid prediction. despite its simplicity, our baseline has some conceptual advantages. first, each token is able to exchange information with other tokens from other sentences in the bag -so the embeddings are likely more informed. second, in principle, the model may be able to relax a part of the at-least-one assumption. for example, if no sentence individually expresses a relation, but if multiple facts in different sentences collectively predict the relation, our model may be able to learn to extract that. we name our baseline model passage-attended relation extraction, pare (mpare for multilingual ds-re). we experiment on four ds-re datasets -three in english, nyt-10d @xcite , @xcite , and one multilingual, dis-rex @xcite . we find that in all four datasets, our proposed baseline significantly outperforms existing state of the art, yielding up to 5 point auc gain. further attention analysis and ablations provide additional insight into model performance. we re- 𝐵(𝑒 ! , 𝑒 " ) 𝑡 ! : [𝑤 ! ! 𝑤 " ! … 𝑤 #! ! ] 𝑡 " : [𝑤 ! " 𝑤 " " … 𝑤 # " " ] 𝑡 @xmath0 𝑤 " @xmath1 ] [cls] 𝑤 ! ! 𝑤 " ! … 𝑤 # ! ! [sep] 𝑤 ! " 𝑤 " " … 𝑤 # " " [sep] 𝑤 ! @xmath2 … 𝑤 # # $ [sep][pad][pad][pad] passage construction 𝒛 𝑪𝑳𝑺 𝒛 𝟏 𝟏 𝒛 𝟐 𝟏 … 𝒛 𝒏𝟏 𝟏 𝒛 𝑺𝑬𝑷 𝟏 𝒛 𝟏 𝟐 𝒛 𝟐 𝟐 … 𝒛 𝒏 𝟐 𝟐 𝒛 𝑺𝑬𝑷 𝟐 𝒛 𝟏 𝟑 𝒛 𝟐 𝟑 … 𝒛 𝒏 𝟑 𝟑 𝒛 𝑺𝑬𝑷 𝟑 𝒛 𝑷𝑨𝑫 𝒛 𝑷𝑨𝑫 𝒛 𝑷𝑨𝑫 𝒛 𝑪𝑳𝑺 𝒛 𝟏 𝟏 𝒛 𝟐 𝟏 … 𝒛 𝒏 𝟏 𝟏 𝒛 𝑺𝑬𝑷 𝟏 𝒛 𝟏 𝟐 𝒛 𝟐 𝟐 … 𝒛 𝒏 𝟐 𝟐 𝒛 𝑺𝑬𝑷 𝟐 𝒛 𝟏 𝟑 𝒛 𝟐 𝟑 … 𝒛 𝒏𝟑 𝟑 𝒛 𝑺𝑬𝑷 𝟑 𝒛 𝑷𝑨𝑫 𝒛 𝑷𝑨𝑫 𝒛 𝑷𝑨𝑫 𝒓 𝒊 passage encoding passage summarization tuple classifier 𝒛 (𝒆 𝟏 ,𝒓 𝒊 ,𝒆 𝟐 ) 𝑝 " attention with relation query lease our code for reproducibility. 1 we believe that our work represents a simple but strong baseline that can form the basis for further ds-re research. to evaluate ds-re, at test time, the model makes a prediction for an unseen bag. unfortunately, most popular ds-re dataset (nyt-10d) has a noisy test set, as it is automatically annotated @xcite . recently @xcite has released nyt-10m and wiki-20m, which have manually annotated test sets. we use all three datasets in our work. ## passage attended relation extraction pare explores the value of cross-sentence attention during encoding time. it uses a sequence of three key steps: passage construction, encoding and summarization, followed by prediction. figure 1 illustrates these for a three-sentence bag. passage construction constructs a passage p (e 1 , e 2 ) from sentences @xmath6, e 2 ). the construction process uses a sequential sampling of sentences in the bag without replacement. it terminates if (a) adding any new sentence would exceed the maximum number of tokens allowed by the encoder (512 tokens for bert), or (b) all sentences from the bag have been sampled. passage encoding takes the constructed passage and sends it to an encoder (bert or mbert) to generate contextualized embeddings z j of every token w j in the passage. for this, it first creates an encoder input. the input starts with the [cls] token, followed by each passage sentence separated by @xcite , and pads all remaining tokens with [pad]. moreover, following best-practices in re @xcite , each mention of e 1 and e 2 in the passage are surrounded by special entity marker @xmath7,</e1>, @xmath8,</e2>, respectively. passage summarization maintains a (randomlyinitialized) query vector r i for every relation r i . it then computes α i j , the normalized attention of r i on each token w j , using dot-product attention. finally, it computes a relation-attended summary of the whole passage z (e 1 ,r i ,e 2 ) = @xmath9, where l is the input length. we note that this summation also aggregates embeddings of we compare pare and mpare against the state of the art models on the respective datasets. we also perform ablations and analyses to understand model behavior and reasons for its performance. ## experiments and analysis datasets and evaluation metrics: we evaluate pare on three english datasets: nyt-10d, nyt-10m, wiki-20m. mpare is compared using the dis-rex benchmark. data statistics are in ## conclusion and future work we introduce pare, a simple baseline for the task of distantly supervised relation extraction. our experiments demonstrate that this simple baseline produces very strong results for the task, and outperforms existing top models by varying margins across four datasets in monolingual and multilingual settings. several experiments for studying model behavior show its consistent performance that generalizes across settings. we posit that our framework would serve as a strong backbone for further research in the field of ds-re. there are several directions to develop the pare architecture further. e.g., pare initializes relation embeddings randomly and also constructs passage via random sampling. alternatively, one could make use of label descriptions and aliases from wikidata to initialize label query vectors; one could also use a sampling strategy to filter away noisy sentences (e.g. a sentence selector @xcite module integrated with pare). in the multilingual setting, contextualized embeddings of entity mentions in a passage may be aligned using constrained learning techniques @xcite to learn potentially better token embeddings. constraints can be imposed on the label hierarchy as well (e.g. presidentof ⇒ citizenof, etc.) since label query vectors operate independently of each other on the passage in pare. additionally, translation-based approaches at training or inference @xcite could improve mpare performance. recent ideas of joint entity and relation alignment in multilingual kbs @xcite may be combined along with mpare's relation extraction capabilities. yankai lin, zhiyuan liu, and maosong sun. 2017. neural relation extraction with multi-lingual attention. in proceedings of the 55th annual meeting of the association for computational linguistics (volume 1: long papers), pages 34-43, vancouver, canada. association for computational linguistics. yankai lin, shiqi shen, zhiyuan liu, huanbo luan, and maosong sun. 2016. neural relation extraction with selective attention over instances. in proceedings of the 54th annual meeting of the association for computational linguistics (volume 1: long papers), pages 2124-2133, berlin, germany. association for computational linguistics. ilya loshchilov and frank hutter. 2019. decoupled weight decay regularization. in international conference on learning representations. quinn mcnemar. 1947. note on the sampling error of the difference between correlated proportions or percentages. psychometrika, 12(2):153-157. sanket vaibhav mehta, jay yoon lee, and jaime g carbonell. 2018. towards semi-supervised learning for deep semantic role labeling. in proceedings of the 2018 conference on empirical methods in natural language processing, pages 4958-4963. mike mintz, steven bills, rion snow, and daniel jurafsky. 2009. distant supervision for relation extraction without labeled data. in proceedings of the joint conference of the 47th annual meeting of the acl and the 4th international joint conference on natural language processing of the afnlp, pages 1003-1011, suntec, singapore. association for computational linguistics.