Original: Hierarchical Relation-Guided Type-Sentence Alignment for Long-Tail Relation Extraction with Distant Supervision
Revised: Hierarchical Relation-Guided Type-Sentence Alignment for Long-Tail Relation Extraction with Distant Supervision

Original: Anonymous ACL submission

Original: Abstract
Revised: Abstract

Original: Distant supervision uses triple facts in knowledge graphs to label a corpus for relation extraction, leading to wrong labeling and longtail problems.
Revised: Distant supervision uses triple facts in knowledge graphs to label a corpus for relation extraction, leading to wrong labeling and longtail problems.

Original: Some works use the hierarchyof relations for knowledge transfer to longtail relations.
Revised: Some works use the hierarchy of relations for knowledge transfer to longtail relations.

Original: However, a coarse-grained relation often implies only an attribute (e.g., domain or topic) of the distant fact, making ithard to discriminate relations based solely onsentence semantics.
Revised: However, a coarse-grained relation often implies only an attribute (e.g., domain or topic) of the distant fact, making it hard to discriminate relations based solely on sentence semantics.

Original: One solution is resorting to entity types, but open questions remainabout how to fully leverage the information ofentity types and how to align multi-granularentity types with sentences.
Revised: One solution is resorting to entity types, but open questions remain about how to fully leverage the information of entity types and how to align multi-granular entity types with sentences.

Original: In this work,we propose a novel model to enrich distantlysupervised sentences with entity types.
Revised: In this work, we propose a novel model to enrich distantlysupervised sentences with entity types.

Original: It consists of (1) a pairwise type-enriched sentenceencoding module injecting both context-freeand -related backgrounds to alleviate sentencelevel wrong labeling, and (2) a hierarchicaltype-sentence alignment module enriching asentence with the triple fact’s basic attributesto support long-tail relations.
Revised: It consists of (1) a pairwise type-enriched sentence encoding module injecting both context-free and -related backgrounds to alleviate sentencelevel wrong labeling, and (2) a hierarchical type-sentence alignment module enriching a sentence with the triple fact’s basic attributes to support long-tail relations.

Original: Our modelachieves new state-of-the-art results in overalland long-tail performance on benchmarks.
Revised: Our model achieves new state-of-the-art results in overall and long-tail performance on benchmarks.

Original:  Introduction
Revised:  Introduction

Original: Human-curated knowledge graphs (KGs), play acritical role in many downstream tasks but sufferfrom the incompleteness (Xiong et al., 2018; Yaoet al., 2019).
Revised: Human-curated knowledge graphs (KGs), play a critical role in many downstream tasks but suffer from the incompleteness (Xiong et al., 2018; Yao et al., 2019).

Original: As a remedy, relation extraction is todistinguish the relation between two entities according to their semantics in text, but a major obstacle isa lack of sufﬁcient labeled corpus.
Revised: As a remedy, relation extraction is to distinguish the relation between two entities according to their semantics in text, but a major obstacle is a lack of sufﬁcient labeled corpus.

Original: Fortunately, distant supervision can be used to annotate a raw textcorpus via KGs for relation extraction,
Revised: Fortunately, distant supervision can be used to annotate a raw text corpus via KGs for relation extraction,

Original: a.k.a.
Revised: a.k.a.

Original: distantly supervised relation extraction (DSRE).
Revised: distantly supervised relation extraction (DSRE).

Original: Thisis based on a strong assumption that a sentencecontaining two entities will express the semanticsof their relation in a KG (Riedel et al., 2010).
Revised: This is based on a strong assumption that a sentence containing two entities will express the semantics of their relation in a KG (Riedel et al., 2010).

Original: The assumption cannot always hold, leading tothe wrong labeling problem.
Revised: The assumption cannot always hold, leading to the wrong labeling problem.

Original: For example, both
Revised: For example, both

Original: [Figure]
Revised: [Figure]

Original: “ Jobs founded Apple ” and “ Jobs ate Apple ” are[Figure] labeled with “/ BUSINESS / COMPANY / FOUNDERS ” [Figure]
Revised: “ Jobs founded Apple ” and “ Jobs ate Apple ” are labeled with “/ BUSINESS / COMPANY / FOUNDERS ” according to a KG triple fact ( Steven Jobs , / BUSINESS / COMPANY / FOUNDERS , Apple Inc ).

Original: according to a KG triple fact ( Steven Jobs , / BUSI -

Original: [Figure]

Original: NESS / COMPANY / FOUNDERS , Apple Inc ).

Original: A basic[Figure] technique for this problem is selective attention 
Revised: A basic technique for this problem is selective attention (Zeng et al., 2015;

Original: [Figure] (Zeng et al., 2015; Lin et al., 2016; Ji et al., 2017)[Figure] under multi-instance learning framework (Riedel 
Revised: Lin et al., 2016; Ji et al., 2017) under multi-instance learning framework (Riedel et al., 2010; Hoffmann et al., 2011).

Original: [Figure]

Original: et al., 2010; Hoffmann et al., 2011).

Original: Given a bag of sentences with the same entity pair, it learns to select correct one(s) by an end-to-end attention.
Revised: Given a bag of sentences with the same entity pair, it learns to select correct one(s) by an end-to-end attention.

Original: The other major challenge is known as the long-tail problem, caused by domain mismatching during distant supervision.
Revised: The other major challenge is known as the long-tail problem, caused by domain mismatching during distant supervision.

Original: That is, many relation labels correspond only to a limited number of training sentences in the corpus (Ye et al., 2019).
Revised: That is, many relation labels correspond only to a limited number of training sentences in the corpus (Ye et al., 2019).

Original: For example, in a DSRE benchmark, the distant supervision is an encyclopedic KG (i.e., Freebase (Bollacker et al., 2008)) while the corpus is news articles from the New York Times (NYT), so relations, like “/ PEOPLE / PERSON / RELIGION ”, scarcely appear.
Revised: For example, in a DSRE benchmark, the distant supervision is an encyclopedic KG (i.e., Freebase (Bollacker et al., 2008)) while the corpus is news articles from the New York Times (NYT), so relations, like “/ PEOPLE / PERSON / RELIGION ”, scarcely appear.

Original: As illustrated by Li et al. (2020b) and Zhang et al.
Revised: As illustrated by Li et al. (2020b) and Zhang et al.

Original: (2019), more than 70% of relation labels in
Revised: (2019), more than 70% of relation labels in

Original: NYT can be regarded as long-tail relations .
Revised: NYT can be regarded as long-tail relations .

Original: To mitigate the long-tail problem, some works
Revised: To mitigate the long-tail problem, some works

Original: (Han et al., 2018; Zhang et al., 2019; Li et al. ,2020b) resort to the hierarchy of relations forknowledge transfer from data-rich relations tothe long-tail ones since the relations have coarsegrained overlap.
Revised: (Han et al., 2018; Zhang et al., 2019; Li et al., 2020b) resort to the hierarchy of relations for knowledge transfer from data-rich relations to the long-tail ones since the relations have coarsegrained overlap.

Original: They focus on interactive operations between hierarchical relations and intra-bagsentences, including relation-to-sentence attention(Han et al., 2018) as a hierarchical extension of selective attention, and sentence-to-relation attention(Li et al., 2020b) enriching sentences with multigranular relations.
Revised: They focus on interactive operations between hierarchical relations and intra-bag sentences, including relation-to-sentence attention (Han et al., 2018) as a hierarchical extension of selective attention, and sentence-to-relation attention (Li et al., 2020b) enriching sentences with multigranular relations.

Original: As such, they achieve knowledge transfer by learning to distinguish coarsegrained relations for sentences with sufﬁcient data,which provides a latent constraint for the long-tailrelations.
Revised: As such, they achieve knowledge transfer by learning to distinguish coarsegrained relations for sentences with sufﬁcient data, which provides a latent constraint for the long-tail relations.

Original: However, a coarse-grained relation usually denotes the only basic attribute of the distantoracle triple fact in KG, so a sentence scarcelycontains its semantics and we can only imply therelation via background information.
Revised: However, a coarse-grained relation usually denotes the only basic attribute of the distant oracle triple fact in KG, so a sentence scarcely contains its semantics and we can only imply the relation via background information.

Original: Again, truelabeled “ Jobs founded Apple ”, does not explicitlycontain any semantics of its coarse-grained relation “/ BUSINESS / COMPANY ”, but we can directlyreason it from the predicate founded and type of
Revised: Again, truelabeled “ Jobs founded Apple ”, does not explicitly contain any semantics of its coarse-grained relation “/ BUSINESS / COMPANY ”, but we can directly reason it from the predicate founded and type of

Original: Apple .
Revised: Apple .

Original: Thus, it is a challenge for a hierarchical
Revised: Thus, it is a challenge for a hierarchical

Original: DSRE model to correctly imply coarse-grained relations based solely on sentences, not to mentionthe existence of the wrong labeling problem.
Revised: DSRE model to correctly imply coarse-grained relations based solely on sentences, not to mention the existence of the wrong labeling problem.

Original: A direct yet promising way to overcome thischallenge is to incorporate extra information forentities (Vashishth et al., 2018;
Revised: A direct yet promising way to overcome this challenge is to incorporate extra information for entities (Vashishth et al., 2018;

Original: Hu et al., 2019;
Revised: Hu et al., 2019;

Original: Chu et al., 2020).
Revised: Chu et al., 2020).

Original: One popular source is the entitytypes, i.e., an entity’s “ IS A” attributes in KG, whichcharacterizes the entity from multiple perspectives(Chen et al., 2020).
Revised: One popular source is the entity types, i.e., an entity’s “ IS A” attributes in KG, which characterizes the entity from multiple perspectives (Chen et al., 2020).

Original: As Figure 1 shows, althoughthe 1st sentence’s semantics is irrelevant to relation,the pairwise types people.deceased_person and location.location directly align with the ﬁne grainedrelation.
Revised: As Figure 1 shows, although the 1st sentence’s semantics is irrelevant to relation, the pairwise types people.deceased_person and location.location directly align with the ﬁne grained relation.

Original: However, existing works (Vashishth et al. ,2018; Chu et al., 2020) ignore this potential of explicit structured types information.
Revised: However, existing works (Vashishth et al., 2018; Chu et al., 2020) ignore this potential of explicit structured types information.

Original: In this work, we aim to improve DSRE by exploiting structured information in the entity typesfrom both pairwise and hierarchical perspectivesto alleviate the wrong labeling and the long-tailproblems respectively.
Revised: In this work, we aim to improve DSRE by exploiting structured information in the entity types from both pairwise and hierarchical perspectives to alleviate the wrong labeling and the long-tail problems respectively.

Original: To this end, we ﬁrst proposea context-free type-enriched embedding module togenerate word embeddings with pairwise types associated with the entity pair in a bag.
Revised: To this end, we ﬁrst propose a context-free type-enriched embedding module to generate word embeddings with pairwise types associated with the entity pair in a bag.

Original: As shown in
Revised: As shown in

Original: Figure 1, even without the corresponding semanticsupport, pairwise types can provide direct attributesof entities to align with the relation.
Revised: Figure 1, even without the corresponding semantic support, pairwise types can provide direct attributes of entities to align with the relation.

Original: Besides, wedevelop a context-related type-sentence alignment module to generate robust sentence representation with pairwise types.
Revised: Besides, we develop a context-related type-sentence alignment module to generate robust sentence representation with pairwise types.

Original: Since entities have speciﬁc characteristic in certain semantics, we leverage semantics to select proper pairwise types and then enrich sentence representation, as the 2nd sentence in Figure 1 shows.
Revised: Since entities have speciﬁc characteristic in certain semantics, we leverage semantics to select proper pairwise types and then enrich sentence representation, as the 2nd sentence in Figure 1 shows.

Original: Such an alignment is enhanced by a guidance from the relation to auto-seek for associations between pairwise types and sentences.
Revised: Such an alignment is enhanced by a guidance from the relation to auto-seek for associations between pairwise types and sentences.

Original: At the meantime, hierarchical information has been proven crucial in knowledge transfer for longtail relations (Han et al., 2018; Zhang et al., 2019;
Revised: At the meantime, hierarchical information has been proven crucial in knowledge transfer for longtail relations (Han et al., 2018; Zhang et al., 2019;

Original: Li et al., 2020b).
Revised: Li et al., 2020b).

Original: Thereby, we naturally extend the base alignment module into a hierarchy by proposing a hierarchical type-sentence alignment module.
Revised: Thereby, we naturally extend the base alignment module into a hierarchy by proposing a hierarchical type-sentence alignment module.

Original: An intuitive example in Figure 1 shows that different grained relations are pointed by various granular pairwise types.
Revised: An intuitive example in Figure 1 shows that different grained relations are pointed by various granular pairwise types.

Original: This indicates that these pairwise types contain hierarchical semantics, which makes it feasible to extend base alignment into hierarchy.
Revised: This indicates that these pairwise types contain hierarchical semantics, which makes it feasible to extend base alignment into hierarchy.

Original: Thus, the strong association between pairwise types and coarse-grained relations can improve knowledge transfer for long-tail relations.
Revised: Thus, the strong association between pairwise types and coarse-grained relations can improve knowledge transfer for long-tail relations.

Original: We conduct extensive experiments on two popu- [Figure] lar benchmarks, NYT-520k and NYT-570k, showing that our model achieves new state-of-the-art overall and long-tail performance.
Revised: We conduct extensive experiments on two popular benchmarks, NYT-520k and NYT-570k, showing that our model achieves new state-of-the-art overall and long-tail performance.

Original: Further analyses reveal insights into our model.
Revised: Further analyses reveal insights into our model.

Original:  Approach
Revised:  Approach

Original: Task Deﬁnition.
Revised: Task Deﬁnition.

Original: Given a bag of sentences B = { s 1 , . . .
Revised: Given a bag of sentences B = { s 1 , . . .

Original: , s N } containing a pair of subject e ( s ) and object e ( o ) entities, the distant supervision (Mintz et al., 2009) assigns the sentence bag with a relation label r according to KG triple fact.
Revised: , s N } containing a pair of subject e ( s ) and object e ( o ) entities, the distant supervision (Mintz et al., 2009) assigns the sentence bag with a relation label r according to KG triple fact.

Original: The goal of relation extraction is to predict the relation label ˆ r of an entity pair based on the corresponding sentences bag B .
Revised: The goal of relation extraction is to predict the relation label ˆ r of an entity pair based on the corresponding sentences bag B .

Original: Labels of coarse-grained relations, [ r (1) , . .
Revised: Labels of coarse-grained relations, [ r (1) , . .

Original: , r ( M ) ] , can be derived from the mention of r .
Revised: , r ( M ) ] , can be derived from the mention of r .

Original: For instance, when r = / BUSINESS / COMPANY / FOUNDERS , r (1) = / BUSINESS / COMPANY and r (2) = / BUSINESS .
Revised: For instance, when r = / BUSINESS / COMPANY / FOUNDERS , r (1) = / BUSINESS / COMPANY and r (2) = / BUSINESS .

Original: In the following, we will detail our approach, as illustrated in Figure 2.
Revised: In the following, we will detail our approach, as illustrated in Figure 2.

Original: 2. Context-Free Type-Enriched Word Emb
Revised: 2. Context-Free Type-Enriched Word Emb

Original: Following most previous DSRE works, we ﬁrst tokenize each sentence s j ∈ B and employ a word2vec method (Mikolov et al., 2013) to derive a sequence of word embeddings by looking up a learnable matrix W ( emb ) ∈ R d e ×| V | , i.e.,164 ˜ 
Revised: Following most previous DSRE works, we ﬁrst tokenize each sentence s j ∈ B and employ a word2vec method (Mikolov et al., 2013) to derive a sequence of word embeddings by looking up a learnable matrix W ( emb ) ∈ R d e ×| V | , i.e., ˜ X

Original: X j =
Revised: j =

Original: [˜ x j 1 , . . .
Revised: [˜ x j 1 , . . .

Original: , ˜ x jn ] ∈ R d e , where V denotes word
Revised: , ˜ x jn ] ∈ R d e , where V denotes word

Original: [Figure]
Revised: [Figure]

Original: Position Word Emb
Revised: Position Word Emb

Original: Subject Emb
Revised: Subject Emb

Original: Object Emb
Revised: Object Emb

Original: Subject Types & Object Types Emb
Revised: Subject Types & Object Types Emb

Original: Type-enriched Embedding
Revised: Type-enriched Embedding

Original: Context
Revised: Context

Original: [Figure]
Revised: [Figure]

Original: free Type-enriched Word Embedding
Revised: free Type-enriched Word Embedding

Original: Pairwise Type Embedding
Revised: Pairwise Type Embedding

Original: Context-Related Type-sentece Alignment
Revised: Context-Related Type-sentece Alignment

Original: Multi-Layer Perceptron Module
Revised: Multi-Layer Perceptron Module

Original: Relation Guided by Classifier Rel Guided
Revised: Relation Guided by Classifier Rel Guided

Original: A sentence bag w/ an entity pair and their type sets
Revised: A sentence bag w/ an entity pair and their type sets

Original: Pairwise Type
Revised: Pairwise Type

Original: Mul-Sel-Attn
Revised: Mul

Original: [Figure]
Revised: [Figure]

Original: Classifier
Revised: Sel-Attn

Original: [Figure]
Revised: [Figure]

Original: vocabulary.
Revised: vocabulary.

Original: j denotes the index of a sentence inthe bag and n denotes the sentence length.
Revised: j denotes the index of a sentence in the bag and n denotes the sentence length.

Original: In thesequel, we omit j if no confusion is caused.
Revised: In the sequel, we omit j if no confusion is caused.

Original: Then,as a common practice in DSRE (Zeng et al., 2014),a word’s relative distances to both the subject andobject entities (a.k.a relative positions) also playsigniﬁcant roles.
Revised: Then, as a common practice in DSRE (Zeng et al., 2014), a word’s relative distances to both the subject and object entities (a.k.a relative positions) also play signiﬁcant roles.

Original: The distances are ﬁrst denotedas two integers ( dist ( s ) and dist ( o ) ∈ Z ) and thenembedded into two learnable vectors ( ˜ x ( ds ) i and˜ x ( do ) i ∈ R d p ).
Revised: The distances are ﬁrst denoted as two integers ( dist ( s ) and dist ( o ) ∈ Z ) and then embedded into two learnable vectors ( ˜ x ( ds ) i and ˜ x ( do ) i ∈ R d p ).

Original: Therefore, the updated sequenceof word embeddings is X j =
Revised: Therefore, the updated sequence of word embeddings is X j =

Original: [ x 1 , . .
Revised: [ x 1 , . .

Original: , x n ] , wherex i =
Revised: , x n ] , where x i =

Original: [˜ x i ; ˜ x
Revised: [˜ x i ; ˜ x

Original: ( ds ) i ; ˜ x ( do ) i ] ∈ R d w , [; ] denotes vectorconcatenation, and d w :=
Revised: ( ds ) i ; ˜ x ( do ) i ] ∈ R d w , [; ] denotes vector concatenation, and d w :=

Original: d e + 2 d p .
Revised: d e + 2 d p .

Original: Previous works (Li et al., 2020a,b) also foundthat explicitly enriching each word with both entityembeddings (i.e., e ( s ) and e ( o ) ) in a context-freemanner is important to DSRE’s success.
Revised: Previous works (Li et al., 2020a,b) also found that explicitly enriching each word with both entity embeddings (i.e., e ( s ) and e ( o ) ) in a context-free manner is important to DSRE’s success.

Original: However,many entities scarcely appear in the raw corpus andhave multi-characteristics (e.g., Apple could be afruit or a company).
Revised: However, many entities scarcely appear in the raw corpus and have multi-characteristics (e.g., Apple could be a fruit or a company).

Original: Thus, the model is hard to distinguish the relations only via sentence semantics.
Revised: Thus, the model is hard to distinguish the relations only via sentence semantics.

Original: Therefore, we leverage entity types to characterize entities’ attributes.
Revised: Therefore, we leverage entity types to characterize entities’ attributes.

Original: That is, given an entity e ,its types are deﬁned as a set of type mentions, i.e.,
Revised: That is, given an entity e , its types are deﬁned as a set of type mentions, i.e.,

Original: T = { t 1 , t 2 , . . . } .
Revised: T = { t 1 , t 2 , . . . } .

Original: However, previous works (Chuet al., 2020) directly concatenate the entity types ofboth e ( s ) and e ( o ) , completely regardless of potentials of explicit structured information of types.
Revised: However, previous works (Chu et al., 2020) directly concatenate the entity types of both e ( s ) and e ( o ) , completely regardless of potentials of explicit structured information of types.

Original: Asdemonstrated by Krompaß et al.
Revised: As demonstrated by Krompaß et al.

Original: , a relationin KG is usually constrained by the entity types ofe ( s ) and e ( o ) simultaneously (i.e., pairwise types),instead of their individuals.
Revised: , a relation in KG is usually constrained by the entity types of e ( s ) and e ( o ) simultaneously (i.e., pairwise types), instead of their individuals.

Original: We thereby proposea pairwise type embedding module to enrich theword embedding X also in a context-free manner.
Revised: We thereby propose a pairwise type embedding module to enrich the word embedding X also in a context-free manner.

Original: Type and Pairwise Type Embedding.
Revised: Type and Pairwise Type Embedding.

Original: First, given an entity type set T = { t 1 , t 2 , . . . }
Revised: First, given an entity type set T = { t 1 , t 2 , . . . }

Original: (either
Revised: (either

Original: T ( s ) for subject or T ( o ) for object), we tokenize each type mention t j into a sequence of words, then embed the words by looking up W ( emb ) , and lastly derive the type embedding t j by applying a meanpooling to the word embeddings of the mention.
Revised: T ( s ) for subject or T ( o ) for object), we tokenize each type mention t j into a sequence of words, then embed the words by looking up W ( emb ) , and lastly derive the type embedding t j by applying a meanpooling to the word embeddings of the mention.

Original: The embedding of the entire type is
Revised: The embedding of the entire type is

Original: [Equation]
Revised: [Equation]

Original: As such, we subsequently deﬁne the embedding of the pairwise type by considering a combination of every subject ∀ t ( s ) l ∈ T ( s ) and object type ∀ t ( o ) k ∈ T ( o ) .
Revised: As such, we subsequently deﬁne the embedding of the pairwise type by considering a combination of every subject ∀ t ( s ) l ∈ T ( s ) and object type ∀ t ( o ) k ∈ T ( o ) .

Original: Instead of sole semantics via a vector concatenation, we take into account the prior structured information in each type pair by leveraging a translational scheme (Bordes et al., 2013).
Revised: Instead of sole semantics via a vector concatenation, we take into account the prior structured information in each type pair by leveraging a translational scheme (Bordes et al., 2013).

Original: Hence, we represent each type pair ( t ( s ) l , t ( o ) k ) asc l,k =
Revised: Hence, we represent each type pair ( t ( s ) l , t ( o ) k ) as

Original: [˜ c ( sem ) l,k ; ˜ c ( str ) l,k ] ∈
Revised: [Equation]

Original: R 4 d e , (2) where, ˜ c ( sem ) l,k = t ( s ) l (cid:12) W ( sem ) t ( o ) k , and ˜ c ( str ) l,k = t ( o )
Revised: , (2) where, ˜ c ( sem ) l,k = t ( s ) l (cid:12) W ( sem ) t ( o ) k , and ˜ c ( str ) l,k = t ( o )

Original: k − t ( s ) l .
Revised: k − t ( s ) l .

Original: Here, “ (cid:12) ” denotes Hadamard product, and W ( sem ) denotes a learnable projection.
Revised: Here, “ (cid:12) ” denotes Hadamard product, and W ( sem ) denotes a learnable projection.

Original: ˜ c ( sem ) l,k aims to capture the prior semantic relation in the pair (Nickel et al., 2011) since not all types combinations are valid in the whole dataset.
Revised: ˜ c ( sem ) l,k aims to capture the prior semantic relation in the pair (Nickel et al., 2011) since not all types combinations are valid in the whole dataset.

Original: ˜ c ( str ) l,k aims to measure its structured relation.
Revised: ˜ c ( str ) l,k aims to measure its structured relation.

Original: Lastly, we denote all the embeddings of pairwise types as
Revised: Lastly, we denote all the embeddings of pairwise types as

Original: [Equation]
Revised: [Equation]

Original: where C ∈ R 4 d e × m and m = |T ( s ) | · |T ( o ) | .
Revised: where C ∈ R 4 d e × m and m = |T ( s ) | · |T ( o ) | .

Original: Type-Enriched Word Embedding.
Revised: Type-Enriched Word Embedding.

Original: However,an open question still remains about how to operateon variable-length embeddings of pairwise types,
Revised: However, an open question still remains about how to operate on variable-length embeddings of pairwise types,

Original: C , to enrich each word embedding, x j ∈ X , in acontext-free manner.
Revised: C , to enrich each word embedding, x j ∈ X , in a context-free manner.

Original: Inspired by self-attentive sentence encoding (Lin et al., 2016), we present a baglevel type-attentive module, which compresses Cinto a single vector representation to facilitate typeenriching.
Revised: Inspired by self-attentive sentence encoding (Lin et al., 2016), we present a baglevel type-attentive module, which compresses C into a single vector representation to facilitate typeenriching.

Original: Intuitively, such self-attentive moduleis focused on the prior knowledge of the type pairin the corpus.
Revised: Intuitively, such self-attentive module is focused on the prior knowledge of the type pair in the corpus.

Original: Formally, we ﬁrst generate a globalquery (Lin et al., 2016) with structured informationof both entities and types to retrieve possible priorpairwise types, i.e.,˜ q ( f ) =[
Revised: Formally, we ﬁrst generate a global query (Lin et al., 2016) with structured information of both entities and types to retrieve possible prior pairwise types, i.e., ˜

Original: e ( o ) ;Pool( T ( o ) )]
Revised: [Equation] ;Pool( T [Equation]

Original:  − [ e ( s ) ;Pool( T ( s ) )] , (4)followed by a standard Bilinear-based attention,q ( f ) = 
Revised: ) )] − [ e ( s ) ;Pool( T ( s ) )] , (4) followed by a standard Bilinear-based attention, q ( f [Equation] = C [Equation]

Original: C · softmax( C T W ( sa ) q ( f ) )
Revised: softmax [Equation] W ( sa ) q ( f ) )

Original: ∈
Revised: [Equation]

Original: R 4 d e , (5)where “ · ” denotes matrix multiplication and W ( sa )is a learnable weight matrix.
Revised: 4 d e , (5) where “ · ” denotes matrix multiplication and W ( sa ) is a learnable weight matrix.

Original: Lastly, we use a gateas in (Li et al., 2020b) to derive the context-freetype-enriched word embedding, i.e.,g
Revised: Lastly, we use a gate as in (Li et al., 2020b) to derive the context-free type-enriched word embedding, i.e., g ( gf ) i = Sigmoid(MLP([ x i ; q ( f ) ]; θ ( gf 1) )) , (6) x (

Original: [Equation]
Revised: [Equation]

Original: gf ) i = 
Revised: i = MLP([

Original: [Equation]
Revised: [Equation]

Revised: ( f

Original: [Equation] 
Revised: [Equation] [Equation] [Equation]

Revised: (7) v i

Original: [Equation]
Revised: [Equation]

Revised: i

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where MLP denotes a multi-layer perceptron(MLP) module.
Revised: where MLP denotes a multi-layer perceptron (MLP) module.

Original: Hence, word embeddings for sare updated to V =
Revised: Hence, word embeddings for s are updated to V =

Original: [ v 1 , . .
Revised: [ v 1 , . .

Original: . , v n ] ∈ R d w × n .2.
Revised: . , v n ] ∈ R d w × n .

Original:  Context-Related Type-Sent Alignment
Revised: 2. Context-Related Type-Sent Alignment

Original: Sentence Encoding.
Revised: Sentence Encoding.

Original: In DSRE, piecewise convolutional neural network (PCNN) (Zeng et al., 2015)is used for sentence embedding.
Revised: In DSRE, piecewise convolutional neural network (PCNN) (Zeng et al., 2015) is used for sentence embedding.

Original: 1D-CNN (Kim,2014) is ﬁrst invoked over V for contextualizedrepresentations.
Revised: 1D-CNN (Kim, 2014) is ﬁrst invoked over V for contextualized representations.

Original: Then a piecewise max-pooling performs over the output sequence to obtain sentencelevel embedding with highlighted entity positions:
Revised: Then a piecewise max-pooling performs over the output sequence to obtain sentencelevel embedding with highlighted entity positions:

Original: H =
Revised: H =

Original: [ h 1 , . . .
Revised: [ h 1 , . . .

Original: , h n ] = 1D-CNN( V ; θ ( cnn ) ) ,
Revised: , h n ] = 1D-CNN( V ; θ ( cnn ) ) ,

Revised: [Equation]

Original: s =tanh([Pool( H (1) );
Revised: =tanh([Pool( H (1) )

Original: Pool( H (2) );Pool( H (3) )]) ,where H (1) , H (2) and H (3) are three consecutiveparts of H by dividing H w.r.t.
Revised: ;Pool( H (2) ); Pool( H (3) )]) , where H (1) , H (2) and H (3) are three consecutive parts of H by dividing H w.r.t.

Original: the indices ofsubject e ( s ) and object e ( o ) entities.
Revised: the indices of subject e ( s ) and object e ( o ) entities.

Original: Consequently,s ∈ R d h is the resulting sentence-level embedding.
Revised: Consequently, s ∈ R d h is the resulting sentence-level embedding.

Original: Type-Sentence Alignment.
Revised: Type-Sentence Alignment.

Original: Consider that types are not comprehensive enough to align with multigranular relations, we leverage semantic context to select valid pairwise types for generating robust sentence representation.
Revised: Consider that types are not comprehensive enough to align with multigranular relations, we leverage semantic context to select valid pairwise types for generating robust sentence representation.

Original: Hence, we ﬁrst calculate alignment scores between a sentence s ∈ R d h and the embeddings of pairwise types C ∈ R 4 d e × m by using a simple Bilinear layer, i.e.,
Revised: Hence, we ﬁrst calculate alignment scores between a sentence s ∈ R d h and the embeddings of pairwise types C ∈ R 4 d e × m by using a simple Bilinear layer, i.e.,

Original: [Equation]
Revised: [Equation]

Revised: p

Original: [Equation]
Revised: [Equation]

Original: Then, we enrich the sentence embedding with the aligned type pairs via another gating mechanism:
Revised: d

Original: [Equation]
Revised: [Equation]

Revised: m

Original: [Equation]
Revised: [Equation]

Revised: Then, we enrich the sentence embedding with the aligned type pairs via another gating mechanism:

Original: [Equation]
Revised: [Equation]

Original: Lastly, following previous success (Li et al., 2020b;
Revised: Lastly, following previous success (Li et al., 2020b;

Original: Devlin et al., 2019), we leverage a residual connection (He et al., 2016) with layer normalization (Ba et al., 2016) to derive the ﬁnal context-related type-enriched sentence embedding, i.e.,u = LayerNorm
Revised: Devlin et al., 2019), we leverage a residual connection (He et al., 2016) with layer normalization (Ba et al., 2016) to derive the ﬁnal context-related type-enriched sentence embedding, i.e.,

Original: ( s + ˜ u ; θ ( lm ) ) .
Revised: [Equation]

Original: Relation-Guided Alignment at the Sentence
Revised: Relation-Guided Alignment at the Sentence

Original: Level.
Revised: Level.

Original: Due to the severe wrong labeling problem at the sentence level, previous DSRE works usually skip over sentence-level relation supervisions.
Revised: Due to the severe wrong labeling problem at the sentence level, previous DSRE works usually skip over sentence-level relation supervisions.

Original: Fortunately, empowered by the proposed context-free type enrichment and context-related type-sentence alignment, we can utilize the sentence-level relation label even if the relation label is wrong.
Revised: Fortunately, empowered by the proposed context-free type enrichment and context-related type-sentence alignment, we can utilize the sentence-level relation label even if the relation label is wrong.

Original: The reason for this is that, a sentence has already been equipped with structured background to support sentence-level relation even if the sentence semantics cannot deliver the relation.
Revised: The reason for this is that, a sentence has already been equipped with structured background to support sentence-level relation even if the sentence semantics cannot deliver the relation.

Original: We applied an
Revised: We applied an

Original: MLP-based neural classiﬁer to the type-enriched sentence embedding, u , to determine the relation at the sentence level, i.e.,
Revised: MLP-based neural classiﬁer to the type-enriched sentence embedding, u , to determine the relation at the sentence level, i.e.,

Original: P ( sl ) (ˆ r | u ) = softmax(MLP( u ; θ ( sl ) )) , (15) where, P ( sl ) (ˆ r | u ) is a categorical distribution over all possible relations.
Revised: [Equation] where, P ( sl ) (ˆ r | u ) is a categorical distribution over all possible relations.

Original: Hence, the training objective is to minimize the cross-entropy loss,
Revised: Hence, the training objective is to minimize the cross-entropy loss,

Original: [Equation]
Revised: [Equation]

Original: D (cid:88)

Original: [Equation]

Original: 16) where D denotes a DSRE dataset consisting of sentence bags B .
Revised: where D denotes a DSRE dataset consisting of sentence bags B .

Original: The guidance from the sentence-level relation leads to strong type-sentence alignment(as illustrated in §3.1 and §3.2).
Revised: The guidance from the sentence-level relation leads to strong type-sentence alignment (as illustrated in §3.1 and §3.2).

Original: As a result, thesentence-level wrong labeling problem is alleviated.
Revised: As a result, the sentence-level wrong labeling problem is alleviated.

Original: In contrast, previous works w/ sentence-level relation supervisions (Li and Roth, 2002) suffer fromthe conﬁrmation bias problem (Chen et al., 2019)caused by the sentence-level wrong labeling.
Revised: In contrast, previous works w/ sentence-level relation supervisions (Li and Roth, 2002) suffer from the conﬁrmation bias problem (Chen et al., 2019) caused by the sentence-level wrong labeling.

Original: 2. Hierarchical Type-Sentence Alignment
Revised: 2. Hierarchical Type-Sentence Alignment

Original: Inspired by former works (Han et al., 2018; Zhanget al., 2019; Li et al., 2020b) for handling long-tailrelations, we also extend our basic model into hierarchy.
Revised: Inspired by former works (Han et al., 2018; Zhang et al., 2019; Li et al., 2020b) for handling long-tail relations, we also extend our basic model into hierarchy.

Original: However, the basic attributes contained bycoarse-grained relation are irrelevant to the semantics in sentences.
Revised: However, the basic attributes contained by coarse-grained relation are irrelevant to the semantics in sentences.

Original: Thus, instead of direct operatingon the hierarchy of relations (i.e., from ﬁne-grainedr to coarse-grained [ r (1) . . .
Revised: Thus, instead of direct operating on the hierarchy of relations (i.e., from ﬁne-grained r to coarse-grained [ r (1) . . .

Original: r ( M ) ] relations), weleverage coarse-grained entity types describing thedomain/type properties of the entities in the triplefacts to enrich each sentence via the guidance fromcoarse-grained relation.
Revised: r ( M ) ] relations), we leverage coarse-grained entity types describing the domain/type properties of the entities in the triple facts to enrich each sentence via the guidance from coarse-grained relation.

Original: Formally, we adapt the relation-guided typesentence alignment (§2.2) into hierarchy, whichshares a high-level inspiration with multi-head attention (Vaswani et al., 2017).
Revised: Formally, we adapt the relation-guided typesentence alignment (§2.2) into hierarchy, which shares a high-level inspiration with multi-head attention (Vaswani et al., 2017).

Original: First, we reuse thearchitecture from Eq.(9-14) by deﬁninga ( l ) , ˜ C ( l ) = TS-Align ( l ) ( s , C ) , ∀ l ∈ [1 , M ] ,u ( l ) =
Revised: First, we reuse the architecture from Eq.(9-14) by deﬁning a ( l ) , ˜ C ( l ) = TS-Align ( l ) ( s , C ) , ∀ l ∈ [1 , M ] , u ( l ) =

Original: TS-Integrate ( l ) ( a ( l ) , ˜ C ( l ) , s ) , (17)where TS-Align() denotes Eq.(9-10) to obtaintype-sentence alignment a ( l ) and TS-Integrate()denotes Eq.(11-14) to generate enriched sentencerepresentation u ( l ) at level l .
Revised: TS-Integrate ( l ) ( a ( l ) , ˜ C ( l ) , s ) , (17) where TS-Align() denotes Eq.(9-10) to obtain type-sentence alignment a ( l ) and TS-Integrate() denotes Eq.(11-14) to generate enriched sentence representation u ( l ) at level l .

Original: Note that, these modules are parameter-untied from each other.
Revised: Note that, these modules are parameter-untied from each other.

Original: Then,we update the sentence-level relation-guided lossin Eq.(16) to its hierarchical version, i.e.,
Revised: Then, we update the sentence-level relation-guided loss in Eq.(16) to its hierarchical version, i.e.,

Original: [Equation]
Revised: [Equation]

Original: Again, learnable parameters of the sentence-levelclassiﬁers across l are also untied.
Revised: Again, learnable parameters of the sentence-level classiﬁers across l are also untied.

Original: Lastly, we obtainthe hierarchical type-enriched representation, i.e.,u ( h ) = [ u ; u (1) ; . . . ; u ( M ) ] [Equation]
Revised: Lastly, we obtain the hierarchical type-enriched representation, i.e.,

Original: R

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Different to previous works (Han et al., 2018;
Revised: Different to previous works (Han et al., 2018;

Original: Zhang et al., 2019; Li et al., 2020b) focusing on hierarchical relation embeddings, our work exploresthe constraints by pairwise types for relations tomitigate sentence-level wrong labeling and usesthe hierarchy of entity types on par with that of therelation to improve long-tail performance.
Revised: Zhang et al., 2019; Li et al., 2020b) focusing on hierarchical relation embeddings, our work explores the constraints by pairwise types for relations to mitigate sentence-level wrong labeling and uses the hierarchy of entity types on par with that of the relation to improve long-tail performance.

Original: 2. Relation Classiﬁcation and Objectives
Revised: 2. Relation Classiﬁcation and Objectives

Original: Lastly, we put the sentences back into the bag and derive bag-level embedding for the ﬁnal relation classiﬁcation.
Revised: Lastly, we put the sentences back into the bag and derive bag-level embedding for the ﬁnal relation classiﬁcation.

Original: Hence, for a bag B =
Revised: Hence, for a bag B =

Original: [ s 1 , ...
Revised: [ s 1 , ...

Original: s N ] , we can obtain sentence embeddings of all the sentences U ( h ) =
Revised: s N ] , we can obtain sentence embeddings of all the sentences U ( h ) =

Original: [ u ( h ) 1 , . .
Revised: [ u ( h ) 1 , . .

Original: , u ( h ) N ] , where u ( h ) j is hierarchical type-enriched sentence encoding derived from Eq.(19).
Revised: , u ( h ) N ] , where u ( h ) j is hierarchical type-enriched sentence encoding derived from Eq.(19).

Original: To preserve the hierarchical information learned in u ( h ) j , we proposed to apply multiple selective modules to its different parts, i.e.,b =
Revised: To preserve the hierarchical information learned in u ( h ) j , we proposed to apply multiple selective modules to its different parts, i.e.,

Original: Mul-Sel-Attn( U ( h ) ) = [ b (0) ; b (1) ; . . . ; b ( M ) ] , b (
Revised: [Equation]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where, Selective-Attn() represents the selective attention among the sentences in each granular relation, and Mul-Sel-Attn() represents the selective attention among the multi-granular bag representations.
Revised: where, Sel-Attn() represents the selective attention among the sentences in each granular relation, and Mul-Sel-Attn() represents the selective attention among the multi-granular bag representations.

Original: For bag representation, b (0) denotes the ﬁnest grained and b ( l ) denotes coarser grained.
Revised: For bag representation, b (0) denotes the ﬁnest grained and b ( l ) denotes coarser grained.

Original: Lastly, we use an MLP-based classiﬁer upon b to derive a bag-level categorical distribution, i.e., 
Revised: Lastly, we use an MLP-based classiﬁer upon b to derive a bag-level categorical distribution, i.e., P

Original: [Equation]
Revised: [Equation]

Original: Meanwhile, the corresponding training loss is
Revised: Meanwhile, the corresponding training loss is

Original: [Equation]
Revised: [Equation]

Original: Therefore, the ﬁnal training objective is to minimize a linear combination of both sentence-level in
Revised: Therefore, the ﬁnal training objective is to minimize a linear combination of both sentence-level in

Original: Eq.(16) and bag-level (in Eq.(21)) losses, i.e.,
Revised: Eq.(16) and bag-level (in Eq.(21)) losses, i.e.,

Original: [Equation]
Revised: [Equation]

Original:  Experiments
Revised:  Experiments

Original: Datasets.
Revised: Datasets

Original: We evaluate our HiRAM on DSRE benchmarks, New York Times – NYT (Riedel et al., 2010), including NYT-520K and NYT-570K. NYT datasets have 53 distinct relations, including an
Revised: We evaluate our HiRAM on DSRE benchmarks, New York Times – NYT (Riedel et al., 2010), including NYT-520K and NYT-570K. NYT datasets have 53 distinct relations, including an

Original: NA class denoting the unavailable relation between entity pairs.
Revised: NA class denoting the unavailable relation between entity pairs.

Original: Each relation includes two coarsegrained relations (i.e., M = 2 ), and the number of relations from ﬁne to coarse are 53, 36 and 9.
Revised: Each relation includes two coarsegrained relations (i.e., M = 2 ), and the number of relations from ﬁne to coarse are 53, 36 and 9.

Original: NYT-520K and NYT-570K have the same testing set containing 172,488 sentences, with 96,678 entity pairs.
Revised: NYT-520K and NYT-570K have the same testing set containing 172,488 sentences, with 96,678 entity pairs.

Original: The only difference is that there is an overlap of 11,416 entity pairs between training and testing in NYT-570K. Thus, NYT-520K has severer wrong labeling and long-tail problems.
Revised: The only difference is that there is an overlap of 11,416 entity pairs between training and testing in NYT-570K. Thus, NYT-520K has severer wrong labeling and long-tail problems.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Evaluation Metrics.
Revised: Evaluation Metrics.

Original: Following previous works(Lin et al., 2016; Han et al., 2018; Zhang et al. ,2019; Li et al., 2020b; Chu et al., 2020), we usearea under precision-recall curve (AUC) and top-Nprecision (P@N) to measure models’ performancewith the disturbance of wrong labeling, and use
Revised: Following previous works (Lin et al., 2016; Han et al., 2018; Zhang et al., 2019; Li et al., 2020b; Chu et al., 2020), we use area under precision-recall curve (AUC) and top-N precision (P@N) to measure models’ performance with the disturbance of wrong labeling, and use

Original: Hits@K to measure the performance on long-tailrelations.
Revised: Hits@K to measure the performance on long-tail relations.

Original: AUC measures the ability of relation classiﬁcation, while P@N measures the precision ofhigh-conﬁdence predictions ranked by the model.
Revised: AUC measures the ability of relation classiﬁcation, while P@N measures the precision of high-conﬁdence predictions ranked by the model.

Original: Settings.
Revised: Settings.

Original: For both versions of NYT datasets, d e ,d p , d w , d h and M are 50, 5, 60, 690, and 2 respectively.
Revised: For both versions of NYT datasets, d e , d p , d w , d h and M are 50, 5, 60, 690, and 2 respectively.

Original: The types number of each entity is variousbut we set an upper limit and pad BLANK as achoice.
Revised: The types number of each entity is various but we set an upper limit and pad BLANK as a choice.

Original: We use AdaDelta (Zeiler, 2012) with 0 . 1learning rate.
Revised: We use AdaDelta (Zeiler, 2012) with 0 .learning rate.

Original: Batch size is 160 with 15 epochs and5-th is the best, dropout probability is 0.5, weightdecay of L2-reg is 10 − 5 .
Revised: Batch size is 160 with 15 epochs and 5-th is the best, dropout probability is 0.5, weight decay of L2-reg is 10 − 5 .

Original: We use random initialization or RoBERTa-base to initialize our models.
Revised: We use random initialization or RoBERTa-base to initialize our models.

Original: Comparative Approach.
Revised: Comparative Approach.

Original: We compare our HiRAM with many strong competitors, including(1) PCNN+ATT (Lin et al., 2016) proposes a selective attention to alleviate wrong labeling.
Revised: We compare our HiRAM with many strong competitors, including (1) PCNN+ATT (Lin et al., 2016) proposes a selective attention to alleviate wrong labeling.

Original: PCNN+HATT (Han et al., 2018) extends selective attention with hierarchical relations.
Revised: PCNN+HATT (Han et al., 2018) extends selective attention with hierarchical relations.

Original: (3) RESIDE (Vashishth et al., 2018) leverages side KGs’ information to improve DSRE.
Revised: (3) RESIDE (Vashishth et al., 2018) leverages side KGs’ information to improve DSRE.

Original: PCNN+BAGATT (Ye and Ling, 2019) proposes intra-bag and inter-bag attentions to handle the wrongly labeled sentences.
Revised: PCNN+BAGATT (Ye and Ling, 2019) proposes intra-bag and inter-bag attentions to handle the wrongly labeled sentences.

Original: (5) PCNN+KATT
Revised: (5) PCNN+KATT

Original: (Zhang et al., 2019) integrates externally pre-trained graph embeddings with relation hierarchies for long-tail relations.
Revised: (Zhang et al., 2019) integrates externally pre-trained graph embeddings with relation hierarchies for long-tail relations.

Original: SeG (Li et al., 2020a) focuses on one-sentence bags and proposes entity-aware embedding.
Revised: SeG (Li et al., 2020a) focuses on one-sentence bags and proposes entity-aware embedding.

Original: (7) CoRA (Li et al., 2020b) transfers multi-granular relations features into sentences in hierarchies for long-tail relations.
Revised: (7) CoRA (Li et al., 2020b) transfers multi-granular relations features into sentences in hierarchies for long-tail relations.

Original: (8) InSRL
Revised: (8) InSRL

Original: (Chu et al., 2020) integrates sentence, entity description and types together via intact space representation learning.
Revised: (Chu et al., 2020) integrates sentence, entity description and types together via intact space representation learning.

Original: 3. Overall Performance on Benchmarks
Revised: 3. Overall Performance on Benchmarks

Original: As shown in Tables 1 and 2, HiRAM outperforms former baselines on NYT-570K. Different from
Revised: As shown in Tables 1 and 2, HiRAM outperforms former baselines on NYT-570K. Different from

Original: CoRA’s poor performance on NYT-520K, HiRAM achieves a new state-of-the-art on both popular
Revised: CoRA’s poor performance on NYT-520K, HiRAM achieves a new state-of-the-art on both popular

Original: [Table] [Table]
Revised: [Table] [Table]

Original: benchmarks in P@N and AUC.
Revised: benchmarks in P@N and AUC.

Original: Compared with
Revised: Compared with

Original: InSRL integrating both clean entity types’ concatenation and accurate entity descriptions, HiRAMincreases the AUC score by nearly 7%, verifyingthe capability of our speciﬁc model designer.
Revised: InSRL integrating both clean entity types’ concatenation and accurate entity descriptions, HiRAM increases the AUC score by nearly 7%, verifying the capability of our speciﬁc model designer.

Original: 3. Ablation Study
Revised: 3. Ablation Study

Original: We conduct an ablation study on NYT-520K, asshown at the bottom of Table 1.
Revised: We conduct an ablation study on NYT-520K, as shown at the bottom of Table 1.

Original: Compared to
Revised: Compared to

Original: HiRAM, “HiRAM w/o Hierarchy” drops 6% in
Revised: HiRAM, “HiRAM w/o Hierarchy” drops 6% in

Original: AUC.
Revised: AUC.

Original: “HiRAM w/o Rel Guidance” performs wellon P@N and AUC but has huge gap in P@One,which represents that the relation-Guided alignment in hierarchy can empower sentence representation with less data in Multi-instance Learning.
Revised: “HiRAM w/o Rel Guidance” performs well on P@N and AUC but has huge gap in P@One, which represents that the relation-Guided alignment in hierarchy can empower sentence representation with less data in Multi-instance Learning.

Original: Meanwhile, top-n precision of “HiRAM w/o
Revised: Meanwhile, top-n precision of “HiRAM w/o

Original: CF” drops by nearly 10.5%.
Revised: CF” drops by nearly 10.5%.

Original: To prove the superiority of our speciﬁc design, we replace the pairwisetype in §2.1 with simple type concatenation.
Revised: To prove the superiority of our speciﬁc design, we replace the pairwise type in §2.1 with simple type concatenation.

Original: The
Revised: The

Original: AUC score of “HiRAM w/ TC” decreases by 4.5%and nearly 5.6% of top-n precision.
Revised: AUC score of “HiRAM w/ TC” decreases by 4.5% and nearly 5.6% of top-n precision.

Original: To furtheremphasize our word embedding §2.1 is moduleagnostic, we combine RoBERTa
Revised: To further emphasize our word embedding §2.1 is moduleagnostic, we combine RoBERTa

Original: (Liu et al., 2019)with our module respectively.
Revised: (Liu et al., 2019) with our module respectively.

Original: As the bottom panelshows, “RoBERTa w/ CF” makes great progress,and “RoBERTa w/ HiRARM” achieves the bestperformance among three RoBERTa-related experiments.
Revised: As the bottom panel shows, “RoBERTa w/ CF” makes great progress, and “RoBERTa w/ HiRARM” achieves the best performance among three RoBERTa-related experiments.

Original: However, due to the strong ability of
Revised: However, due to the strong ability of

Original: RoBERTa model, the wrong labeling problem hurt the performance severely, especially in P@N.
Revised: RoBERTa model, the wrong labeling problem hurt the performance severely, especially in P@N.

Original: 3. Performance on Long-Tail Relations
Revised: 3. Performance on Long-Tail Relations

Original: Since former baselines are mainly trained on NYT570K, we reproduce CoRA on NYT-520K for fair comparison as shown in Table 3.
Revised: Since former baselines are mainly trained on NYT570K, we reproduce CoRA on NYT-520K for fair comparison as shown in Table 3.

Original: HiRAM achieves a new state-of-the-art result in Hits@K with 20% superiority.
Revised: HiRAM achieves a new state-of-the-art result in Hits@K with 20% superiority.

Original: Removing hierarchy module in §2.3, the performance of “HiRAM w/o Hierarchy” decreases by nearly 30% on Hits@10 but is better than baselines in other settings, verifying the importance of hierarchical model for long-tail relations.
Revised: Removing hierarchy module in §2.3, the performance of “HiRAM w/o Hierarchy” decreases by nearly 30% on Hits@10 but is better than baselines in other settings, verifying the importance of hierarchical model for long-tail relations.

Original: The huge decline of “HiRAM w/o Rel Guidance” veriﬁes the necessity of relation guidance.
Revised: The huge decline of “HiRAM w/o Rel Guidance” veriﬁes the necessity of relation guidance.

Original: Due to lacks of plenty reliable training data, RoBERTa is hard to handle the long-tail problem but our speciﬁc modules further increase its performance.
Revised: Due to lacks of plenty reliable training data, RoBERTa is hard to handle the long-tail problem but our speciﬁc modules further increase its performance.

Original: 3. Case Study and Error Analysis
Revised: 3. Case Study and Error Analysis

Original: Firstly, we conduct a case study to qualitatively analyze the effect of our model in §2.3 The case study of two samples are shown in Table 4 and the type-sentence alignment distribution is shown in
Revised: Firstly, we conduct a case study to qualitatively analyze the effect of our model in §2.3 The case study of two samples are shown in Table 4 and the type-sentence alignment distribution is shown in

Original: [Figure]
Revised: [Figure]

Original: film.film religion.religion

Original: B L ANK p
Revised: L ANK p

Original: e op l e .
Revised: e op l e .

Original: t op i c p
Revised: t op i c p

Original: e op l e . nndb - p e r s on gov . po liti c i a n p e op l e .
Revised: e op l e . nndb [Figure] p e r s on [Figure]

Original: f a m il y_ m e m b e r gov . po litica l _ a ppo i n t e r m e d i c i  n  e .
Revised: po liti c i a n p e op l e [Figure] f a m il y [Figure]

Original: phy s i
Revised: e [Figure] e r [Figure]

Original: c i a  n 
Revised: po liti [Figure] l _ a ppo i n t e r [Figure]

Revised: e d i c i n e

Original: [Figure]
Revised: [Figure]

Revised: phy

Revised: [Figure]

Revised: n

Revised: [Figure]

Original: p e op l e .
Revised: p e op l e .

Original: p er s
Revised: p e

Original: [Figure]
Revised: [Figure]

Original: Figure 3.
Revised: Figure 3.

Original: Secondly, we investigate the possiblereasons for the misclassiﬁcations of HiRAM.
Revised: Secondly, we investigate the possible reasons for the misclassiﬁcations of HiRAM.

Original: Distribution of Type-Sentence Alignment.
Revised: Distribution of Type-Sentence Alignment.

Original: Forthe ﬁrst case, despite the failure in expressing thelong-tail relation “/ PEOPLE / PERSON / RELIGION ”,the selected pairwise types are sufﬁcient to predict this relation.
Revised: For the ﬁrst case, despite the failure in expressing the long-tail relation “/ PEOPLE / PERSON / RELIGION ”, the selected pairwise types are sufﬁcient to predict this relation.

Original: As the top row of Figure3 shows, people.person with BLANK helps toidentify the character of subject entity, and religion.religion with high alignment score can provide direct attributes.
Revised: As the top row of Figure 3 shows, people.person with BLANK helps to identify the character of subject entity, and religion.religion with high alignment score can provide direct attributes.

Original: For the second case, the semantics is implicitly related to its long-tail relation“/ BUSINESS / COMPANY / FOUNDER ”.
Revised: For the second case, the semantics is implicitly related to its long-tail relation “/ BUSINESS / COMPANY / FOUNDER ”.

Original: The properpairwise types are selected by coarser relation guidance, like ( organizer.organizer , organizer.founder ).
Revised: The proper pairwise types are selected by coarser relation guidance, like ( organizer.organizer , organizer.founder ).

Original: Error Analysis.
Revised: Error Analysis.

Original: To analyse the implicit reasonsfor wrong predictions, we have manually checkedseveral randomly-sampled error test examples.
Revised: To analyse the implicit reasons for wrong predictions, we have manually checked several randomly-sampled error test examples.

Original: Most of error cases are annotated as / PEOPLE /P ERSON /P
Revised: Most of error cases are annotated as / PEOPLE /P ERSON /P

Original: LACE _
Revised: LACE _

Original: O F _B IRTH because the semanticsand the relation may be completely irrelevant andthe types are hard to maintain people’s birth place.2)
Revised: O F _B IRTH because the semantics and the relation may be completely irrelevant and the types are hard to maintain people’s birth place.

Original:  Mean pooling in Eq.(4) might not be the mostoptimal way to replace entity itself when the entityhas too many characters.
Revised: 2) Mean pooling in Eq.(4) might not be the most optimal way to replace entity itself when the entity has too many characters.

Original:  Related Work
Revised:  Related Work

Original: Wrong Labeling Problem.
Revised: Wrong Labeling Problem.

Original: Many works (Liuet al., 2016; Ji et al., 2017; Ye and Ling, 2019; Liet al., 2020a) propose various extensions of vanillaselective attention (Lin et al., 2016).
Revised: Many works (Liu et al., 2016; Ji et al., 2017; Ye and Ling, 2019; Li et al., 2020a) propose various extensions of vanilla selective attention (Lin et al., 2016).

Original: Ye and Ling(2019) combine intra-/inter-bag level selective attention for DSRE.
Revised: Ye and Ling (2019) combine intra-/inter-bag level selective attention for DSRE.

Original: For one-sentence bags, Li et al.
Revised: For one-sentence bags, Li et al.

Original: (2020a) design the entity-aware embedding in a context-free manner with a gate mechanism.
Revised: (2020a) design the entity-aware embedding in a context-free manner with a gate mechanism.

Original: Long-tail Relations.
Revised: Long-tail Relations.

Original: Knowledge transfer via hierarchical relations is effective.
Revised: Knowledge transfer via hierarchical relations is effective.

Original: Han et al.
Revised: Han et al.

Original: (2018) design relation-to-sentence attention in hierarchies, and Li et al.
Revised: (2018) design relation-to-sentence attention in hierarchies, and Li et al.

Original: (2020b) modify it to sentence-torelation attention.
Revised: (2020b) modify it to sentence-torelation attention.

Original: Many works (Vashishth et al., 2018;
Revised: Many works (Vashishth et al., 2018;

Original: Hu et al., 2019; Chu et al., 2020) resort to extra knowledge, i.e., entity description and entity types.
Revised: Hu et al., 2019; Chu et al., 2020) resort to extra knowledge, i.e., entity description and entity types.

Original: Entity description (Hu et al., 2019; Chu et al., 2020) mainly stems from the Wikipedia page, which contains factual statements of the relation with other entities.
Revised: Entity description (Hu et al., 2019; Chu et al., 2020) mainly stems from the Wikipedia page, which contains factual statements of the relation with other entities.

Original: Such oracle knowledge can boost DSRE performance but is impractical.
Revised: Such oracle knowledge can boost DSRE performance but is impractical.

Original:  Conclusion
Revised:  Conclusion

Original: In this work, we propose a new model, HiRAM, training on a single Titan XP, except for RoBERTa w/ RTX 6000, to alleviate wrong labeling and longtail problems in DSRE.
Revised: In this work, we propose a new model, HiRAM, training on a single Titan XP, except for RoBERTa w/ RTX 6000, to alleviate wrong labeling and longtail problems in DSRE.

Original: For the wrong labeling problem, we propose a context-free type-enriched word embedding to enrich each word with prior knowledge and a context-related type-sentence alignment module to complement sentences with semantics-ﬁtted pairwise types.
Revised: For the wrong labeling problem, we propose a context-free type-enriched word embedding to enrich each word with prior knowledge and a context-related type-sentence alignment module to complement sentences with semantics-ﬁtted pairwise types.

Original: For the long-tail problem, we extend the base alignment into the hierarchy to utilize the multi-granular entity types.
Revised: For the long-tail problem, we extend the base alignment into the hierarchy to utilize the multi-granular entity types.

Original: The experiments with extensive analyses show the superiority of HiRAM.
Revised: The experiments with extensive analyses show the superiority of HiRAM.

Revised: Association for Computational Linguistics

Revised: Ashish Vaswani

Revised: Shazeer

Revised: Noam

Revised: Niki Parmar

Revised: Jakob

Revised: Uszkoreit

Revised: Llion

Revised: Polosukhin

Revised: Xiong

Revised: Duan

Revised: Emp

Revised: ER, MoBiD, MREBA, QMMQ, SCME, Xi’an, China

Revised: October 22-25, abs

Revised: Association

Revised: Computa

Revised: abs

Revised: 25th

Revised: International Conference on Computational Linguistics, Proceedings of the Conference:

Revised: Technical Papers, August 23-29,

Revised: Dublin

Revised: Ireland

Revised: ACL.

Revised: Ningyu Zhang, Shumin Deng, Zhanlin Sun, Guanying Wang, Xi Chen, Wei Zhang, and Huajun Chen.

Revised: graph embeddings and graph convolution networks.

Revised: In Proceedings of the 2019 Conference of the North

Revised: American Chapter of the Association for Computational Linguistics: Human Language Technologies,

Revised: NAACL-HLT 2019, Minneapolis, MN, USA, June 27, 2019, Volume 1 (Long and Short Papers) , pages 3016–3025.

Revised: Association for Computational Linguistics.