Original: T RANSFORMER -XH: M ULTI - HOP Q UESTION A NSWERING WITH E X TRA H OP A TTENTION
Revised: T RANSFORMER -XH: M ULTI - HOP Q UESTION A NSWERING WITH E X TRA H OP A TTENTION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Transformers have obtained signiﬁcant success modeling natural language as a sequence of text tokens.
Revised: Transformers have obtained signiﬁcant success modeling natural language as a sequence of text tokens.

Original: However, in many real world scenarios, textual data inherently exhibits structures beyond a linear sequence such as tree and graph; an important one being multi-hop question answering, where evidence required to answer questions are scattered across multiple related documents.
Revised: However, in many real world scenarios, textual data inherently exhibits structures beyond a linear sequence such as trees and graphs; an important one being multi-hop question answering, where evidence required to answer questions are scattered across multiple related documents.

Original: This paper presents Transformer-XH, which uses eXtra Hop attention to enable the intrinsic modeling of structured texts in a fully data-driven way.
Revised: This paper presents Transformer-XH, which uses eXtra Hop attention to enable the intrinsic modeling of structured texts in a fully data-driven way.

Original: Its new attention mechanism naturally “hops” across the connected text sequences in addition to attending over tokens within each sequence.
Revised: Its new attention mechanism naturally “hops” across the connected text sequences in addition to attending over tokens within each sequence.

Original: Thus, Transformer-XH better answers multi-hop questions by propagating information between multiple documents, constructing global contextualized representations, and jointly reasoning over multiple pieces of evidence.
Revised: Thus, Transformer-XH better answers multi-hop questions by propagating information between multiple documents, constructing global contextualized representations, and jointly reasoning over multiple pieces of evidence.

Original: This leads to a simpler multi-hop QA system which outperforms previous state-of-the-art on the HotpotQA FullWiki setting by large margins.
Revised: This leads to a simpler multi-hop QA system which outperforms previous state-of-the-art on the HotpotQA FullWiki setting by large margins.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Transformers effectively model natural language in sequential form (Vaswani et al., 2017; Dai et al., 2019;
Revised: Transformers effectively model natural language in sequential form (Vaswani et al., 2017; Dai et al., 2019;

Original: Devlin et al., 2019; Yang et al., 2019).
Revised: Devlin et al., 2019; Yang et al., 2019).

Original: Nevertheless, in many NLP tasks, text does not simply appear as a linear sequence of tokens but rather carries meaningful structure in the form of paragraphs, headings, and hyperlinks.
Revised: Nevertheless, in many NLP tasks, text does not simply appear as a linear sequence of tokens but rather carries meaningful structure in the form of paragraphs, headings, and hyperlinks.

Original: These structures can be abstracted into trees or graphs with nodes and edges.
Revised: These structures can be abstracted into trees or graphs with nodes and edges.

Original: Multi-hop question answering (Yang et al., 2018) is one such task in which structure plays an important role, since the evidence required to formulate the answer is scattered across multiple documents, requiring systems to jointly reason across links between them.
Revised: Multi-hop question answering (Yang et al., 2018) is one such task in which structure plays an important role, since the evidence required to formulate the answer is scattered across multiple documents, requiring systems to jointly reason across links between them.

Original: Recent approaches leverage pre-trained Transformers (e.g., BERT) for multi-hop question answering (QA) by converting the structural reasoning task into sub-tasks that model ﬂat sequences.
Revised: Recent approaches leverage pre-trained Transformers (e.g., BERT) for multi-hop question answering (QA) by converting the structural reasoning task into sub-tasks that model ﬂat sequences.

Original: For example, Min et al.
Revised: For example, Min et al.

Original: (2019b) decompose a multi-hop question into a series of single-hop questions;
Revised: (2019b) decompose a multi-hop question into a series of single-hop questions;

Original: Ding et al.
Revised: Ding et al.

Original: (2019) conduct several steps of single-hop reading comprehension to simulate the multihop reasoning.
Revised: (2019) conduct several steps of single-hop reading comprehension to simulate the multihop reasoning.

Original: The hope is that additional processing to fuse the outputs of the sub-models can recover all the necessary information from the original structure.
Revised: The hope is that additional processing to fuse the outputs of the sub-models can recover all the necessary information from the original structure.

Original: While pre-trained Transformer language models have shown improvements on multi-hop QA, manipulating the inherent structure of the problem to ﬁt the rigid requirements of out-of-the-box models can introduce problematic assumptions or information loss.
Revised: While pre-trained Transformer language models have shown improvements on multi-hop QA, manipulating the inherent structure of the problem to ﬁt the rigid requirements of out-of-the-box models can introduce problematic assumptions or information loss.

Original: This paper presents Transformer-XH (meaning eXtra Hop), which upgrades Transformers with the ability to natively represent structured texts.
Revised: This paper presents Transformer-XH (meaning eXtra Hop), which upgrades Transformers with the ability to natively represent structured texts.

Original: Transformer-XH introduces extra hop attention in its layers that connects different text pieces following their inherent structure while also maintaining the powerful pre-trained Transformer abilities over each textual piece individually.
Revised: Transformer-XH introduces extra hop attention in its layers that connects different text pieces following their inherent structure while also maintaining the powerful pre-trained Transformer abilities over each textual piece individually.

Original: Our extra hop attention enables 1) a more global representation of the evidence presented by each piece of text as it relates to the other evidence, and 2) a more natural way to jointly reason over an evidence graph by propagating information along edges necessary to complete the task at hand.
Revised: Our extra hop attention enables 1) a more global representation of the evidence presented by each piece of text as it relates to the other evidence, and 2) a more natural way to jointly reason over an evidence graph by propagating information along edges necessary to complete the task at hand.

Original: We apply Transformer-XH to HotpotQA, a challenging benchmark for the multi-hop question answering task (Yang et al., 2018).
Revised: We apply Transformer-XH to HotpotQA, a challenging benchmark for the multi-hop question answering task (Yang et al., 2018).

Original: Rather than decomposing the task into a series of sub-tasks to ﬁt the constraints of pre-trained Transformers, Transformer-XH is a solution that ﬁts the problem as it naturally occurs.
Revised: Rather than decomposing the task into a series of sub-tasks to ﬁt the constraints of pre-trained Transformers, Transformer-XH is a solution that ﬁts the problem as it naturally occurs.

Original: It is a single model that represents and combines evidence from multiple documents to construct the answer.
Revised: It is a single model that represents and combines evidence from multiple documents to construct the answer.

Original: [Figure]
Revised: [Figure]

Original: On HotpotQA’s FullWiki setting, which is a more realistic setup and requires stronger multihop reasoning ability (Min et al., 2019b; Jiang & Bansal, 2019), Transformer-XH outperforms
Revised: On HotpotQA’s FullWiki setting, which is a more realistic setup and requires stronger multihop reasoning ability (Min et al., 2019b; Jiang & Bansal, 2019), Transformer-XH outperforms

Original: CogQA (Ding et al., 2019), the previous start-of-the-art, by 12 points on answer F1.
Revised: CogQA (Ding et al., 2019), the previous start-of-the-art, by 12 points on answer F1.

Original: It also beats the contemporary BERT based pipeline SR-MRS (Nie et al., 2019), by 3 points.
Revised: It also beats the contemporary BERT based pipeline SR-MRS (Nie et al., 2019), by 3 points.

Original: The results follow from our simple yet effective design, with one uniﬁed model operating over the inherent structure of the task, rather than melding the outputs from disparate sub-tasks adapted to the sequential constraints of pre-trained Transformer.
Revised: The results follow from our simple yet effective design, with one uniﬁed model operating over the inherent structure of the task, rather than melding the outputs from disparate sub-tasks adapted to the sequential constraints of pre-trained Transformer.

Original: Our ablation analysis demonstrates Transformer-XH’s efﬁcacy on questions that are known to require multi-hop reasoning (Min et al., 2019b), and that the source of this success is due to the eXtra Hop attention mechanism’s ability to combines information from multiple documents.
Revised: Our ablation analysis demonstrates Transformer-XH’s efﬁcacy on questions that are known to require multi-hop reasoning (Min et al., 2019b), and that the source of this success is due to the eXtra Hop attention mechanism’s ability to combines information from multiple documents.

Original:  M ODEL
Revised:  M ODEL

Original: This section ﬁrst discusses preliminaries on sequential Transformers, then we show how we incorporate eXtra hop attention to create Transformer-XH.
Revised: This section ﬁrst discusses preliminaries on sequential Transformers, then we show how we incorporate eXtra hop attention to create Transformer-XH.

Original: 2. P RELIMINARIES
Revised: 2. P RELIMINARIES

Original: Transformers represent a sequence of input text tokens
Revised: Transformers represent a sequence of input text tokens

Original: X = { x 1 , ..., x i , ..., x n } into contextualized distributed representations H = { h 1 , ..., h i , ..., h n } (Vaswani et al., 2017).
Revised: X = { x 1 , ..., x i , ..., x n } into contextualized distributed representations H = { h 1 , ..., h i , ..., h n } (Vaswani et al., 2017).

Original: This process involves multiple stacked self-attention layers that converts the input X into { H 0 , H 1 , ..., H l , ...
Revised: This process involves multiple stacked self-attention layers that converts the input X into { H 0 , H 1 , ..., H l , ...

Original: H L } , starting from H 0 , the embeddings, to the ﬁnal layer of depth L .
Revised: H L } , starting from H 0 , the embeddings, to the ﬁnal layer of depth L .

Original: The key idea of Transformer is its attention mechanism, which calculates the l -th layer output H l using the input H l − 1 from the previous layer:
Revised: The key idea of Transformer is its attention mechanism, which calculates the l -th layer output H l using the input H l − 1 from the previous layer:

Original: [Equation]
Revised: [Equation]

Original: It includes three projections on the input H l − 1 : Query (Q), Key (K), and Value (V).
Revised: It includes three projections on the input H l − 1 : Query (Q), Key (K), and Value (V).

Original: Speciﬁcally, the slices of token h li in Eqn.(2) is:
Revised: Speciﬁcally, the slices of token h li in Eqn.(2) is:

Original: [Equation]
Revised: [Equation]

Original: which ﬁrst calculates its attention to all other tokens j in the sequence and then combines the token values v j into a new representation h li , using the normalized attention weights
Revised: which ﬁrst calculates its attention to all other tokens j in the sequence and then combines the token values v j into a new representation h li , using the normalized attention weights

Original: Multiple attentions can be used in one Transformer layer and concatenated as multi-head attention (Vaswani et al.,
Revised: Multiple attentions can be used in one Transformer layer and concatenated as multi-head attention (Vaswani et al.,

Original: The architecture is stacked to form rather deep networks, which leads to signiﬁcant success of large pre-trained Transformer models (Devlin et al., 2019; Liu et al., 2019).
Revised: The architecture is stacked to form rather deep networks, which leads to signiﬁcant success of large pre-trained Transformer models (Devlin et al., 2019; Liu et al., 2019a).

Original: A challenge of Transformer is that its attention is calculated over all token pairs (Eqn. 3), which is hard to scale to long text sequences.
Revised: A challenge of Transformer is that its attention is calculated over all token pairs (Eqn. 3), which is hard to scale to long text sequences.

Original: Transformer-XL (eXtra Long) addresses this challenge by breaking down longer texts, e.g., a multi-paragraph document, into a sequence of text segments: { X 1 , ..., X τ , ..., X ζ } , and propagates the information between adjacent text segments using the following attention:
Revised: Transformer-XL (eXtra Long) addresses this challenge by breaking down longer texts, e.g., a multi-paragraph document, into a sequence of text segments: { X 1 , ..., X τ , ..., X ζ } , and propagates the information between adjacent text segments using the following attention:

Original: [Equation]
Revised: [Equation]

Original: It concatenates ( ◦ ) the representation of the previous segment H l − 1 τ − 1 to the current segment as segment level recurrences.
Revised: It concatenates ( ◦ ) the representation of the previous segment H l − 1 τ − 1 to the current segment as segment level recurrences.

Original: The new representation ˜
Revised: The new representation ˜

Original: H l − 1 τ includes the information from previous segment and is integrated in the new attention mechanism:
Revised: H l − 1 τ includes the information from previous segment and is integrated in the new attention mechanism:

Original: [Equation]
Revised: [Equation]

Original: The attention over the previous segment allows Transformer-XL to effectively model long form text data recurrently as a sequence of text chunks (Dai et al., 2019).
Revised: The attention over the previous segment allows Transformer-XL to effectively model long form text data recurrently as a sequence of text chunks (Dai et al., 2019).

Original: Nevertheless, in many scenarios, the text segments are organized in nontrivial structures beyond a linear sequence.
Revised: Nevertheless, in many scenarios, the text segments are organized in nontrivial structures beyond a linear sequence.

Original: For example, documents are connected by hyperlinks in a graphical structure that does not readily simplify to form a linear sequence, prohibiting Transformer-XL’s recurrent approach.
Revised: For example, documents are connected by hyperlinks in a graphical structure that does not readily simplify to form a linear sequence, prohibiting Transformer-XL’s recurrent approach.

Original: 2. T RANSFORMER -XH WITH E X TRA H OP A TTENTION
Revised: 2. T RANSFORMER -XH WITH E X TRA H OP A TTENTION

Original: Transformer-XH models structured text sequence by linking them with eXtra Hop attention following their original structure.
Revised: Transformer-XH models structured text sequence by linking them with eXtra Hop attention following their original structure.

Original: As illustrated in Figure 1a, to model three connected documents d 2 → d 1 → d 3 , Transformer-XH uses eXtra Hop attention to propagate information along the graph edges, enabling information sharing between connected text sequence.
Revised: As illustrated in Figure 1a, to model three connected documents d 2 → d 1 → d 3 , Transformer-XH uses eXtra Hop attention to propagate information along the graph edges, enabling information sharing between connected text sequence.

Original: Formally, the structured text data includes a set of nodes, X =
Revised: Formally, the structured text data includes a set of nodes, X =

Original: { X 1 , ..., X τ , ...
Revised: { X 1 , ..., X τ , ...

Original: X ζ } , each corresponding to a text sequence, and an edge matrix E , which includes the connections (e.g., links) between them.
Revised: X ζ } , each corresponding to a text sequence, and an edge matrix E , which includes the connections (e.g., links) between them.

Original: The goal is to learn representations H = { ˜ H 1 , ..., ˜ H τ , ...
Revised: The goal is to learn representations H = { ˜ H 1 , ..., ˜ H τ , ...

Original: ˜
Revised: ˜

Original: H ζ } , that incorporate not only the local information in each sequence X , but also the global contexts on the entire structured text {X , E } .
Revised: H ζ } , that incorporate not only the local information in each sequence X , but also the global contexts on the entire structured text {X , E } .

Original: Transformer-XH achieves this by two attention mechanisms: in-sequence attention and eXtra Hop attention.
Revised: Transformer-XH achieves this by two attention mechanisms: in-sequence attention and eXtra Hop attention.

Original: The in-sequence attention is the same as vanilla Transformer: in layer l , token i gathers information from other tokens inside the same text piece τ :
Revised: The in-sequence attention is the same as vanilla Transformer: in layer l , token i gathers information from other tokens inside the same text piece τ :

Original: [Equation]
Revised: [Equation]

Original: The eXtra Hop attention uses the ﬁrst token in each sequence – the added special token “[CLS]” – as an “attention hub”, which attends on all other connected nodes’ hub token.
Revised: The eXtra Hop attention uses the ﬁrst token in each sequence – the added special token “[CLS]” – as an “attention hub”, which attends on all other connected nodes’ hub token.

Original: In layer l , the τ -th text sequence attends on other text sequence η if there is an edge between them ( e τη = 1 ):
Revised: In layer l , the τ -th text sequence attends on other text sequence η if there is an edge between them ( e τη = 1 ):

Original: [Equation]
Revised: [Equation]

Original: Node τ calculates the attention weight on its neighbor η using hop query ˆ q τ, 0 and key ˆ k η, 0 .
Revised: Node τ calculates the attention weight on its neighbor η using hop query ˆ q τ, 0 and key ˆ k η, 0 .

Original: Then it uses the weights to combine its neighbors’ value ˆ v η, 0 and forms a globalized representation ˆ h lτ, 0 .
Revised: Then it uses the weights to combine its neighbors’ value ˆ v η, 0 and forms a globalized representation ˆ h lτ, 0 .

Original: The two attention mechanism are combined to form the new representation of layer l :
Revised: The two attention mechanism are combined to form the new representation of layer l :

Original: [Equation]
Revised: [Equation]

Original:  Note that the non-hub tokens ( i (cid:54) = 0 ) still have access to the hop attention in the previous layer through Eqn.
Revised: [Equation] [Equation] Note that the non-hub tokens ( i (cid:54) = 0 ) still have access to the hop attention in the previous layer through Eqn.

Original: One layer of eXtra Hop attention can be viewed as single-step of information propagation along edges E .
Revised: One layer of eXtra Hop attention can be viewed as single-step of information propagation along edges E .

Original: For example, in Figure 1a, the document node d 3 updates its representation by gathering information from its neighbor d 1 using the hop attention d 1 → d 3 .
Revised: For example, in Figure 1a, the document node d 3 updates its representation by gathering information from its neighbor d 1 using the hop attention d 1 → d 3 .

Original: When multiple TransformerXH layers are stacked, this information in d 1 includes both d 1 ’s local contexts from its in-sequence attention, and cross-sequence information from the hop attention d 2 → d 1 of the l − 1 layer.
Revised: When multiple TransformerXH layers are stacked, this information in d 1 includes both d 1 ’s local contexts from its in-sequence attention, and cross-sequence information from the hop attention d 2 → d 1 of the l − 1 layer.

Original: Hence, an L-layer Transformer-XH can attend over information from up to L hops away.
Revised: Hence, an L-layer Transformer-XH can attend over information from up to L hops away.

Original: Together, three main properties equip Transformer-XH to effectively model raw structured text data: the propagation of information (values) along edges, the importance of that information (hop attention weights), and the balance of in-sequence and cross-sequence information (attention combination).
Revised: Together, three main properties equip Transformer-XH to effectively model raw structured text data: the propagation of information (values) along edges, the importance of that information (hop attention weights), and the balance of in-sequence and cross-sequence information (attention combination).

Original: The representations learned in H can innately express nuances in structured text that are required for complex chains of reasoning in tasks such as multi-hop QA.
Revised: The representations learned in H can innately express nuances in structured text that are required for complex chains of reasoning in tasks such as multi-hop QA.

Original:  A PPLICATION TO M ULTI -H OP Q UESTION A NSWERING
Revised:  A PPLICATION TO M ULTI -H OP Q UESTION A NSWERING

Original: This section describes how Transformer-XH applies to multi-hop QA.
Revised: This section describes how Transformer-XH applies to multi-hop QA.

Original: Given a question q , the task is to ﬁnd an answer span a in a large open-domain document corpus, e.g. the ﬁrst paragraph of all Wikipedia pages.
Revised: Given a question q , the task is to ﬁnd an answer span a in a large open-domain document corpus, e.g. the ﬁrst paragraph of all Wikipedia pages.

Original: By design, the questions are complex and often require information from multiple documents to answer.
Revised: By design, the questions are complex and often require information from multiple documents to answer.

Original: For example, in the case shown in Figure 1b, the correct answer “Cambridge” requires combining the information from both the Wikipedia pages “Facebook” and “Harvard University”.
Revised: For example, in the case shown in Figure 1b, the correct answer “Cambridge” requires combining the information from both the Wikipedia pages “Facebook” and “Harvard University”.

Original: To apply Transformer-XH in the open domain multi-hop QA task, we ﬁrst construct an evidence graph and then apply Transformer-XH on the graph to ﬁnd the answer.
Revised: To apply Transformer-XH in the open domain multi-hop QA task, we ﬁrst construct an evidence graph and then apply Transformer-XH on the graph to ﬁnd the answer.

Original: Evidence Graph Construction.
Revised: Evidence Graph Construction.

Original: The ﬁrst step is to ﬁnd the relevant candidate documents D for the question q and connect them with edges E to form the graph G .
Revised: The ﬁrst step is to ﬁnd the relevant candidate documents D for the question q and connect them with edges E to form the graph G .

Original: Our set D consists of three sources.
Revised: Our set D consists of three sources.

Original: The ﬁrst two sources are from canonical information retrieval and entity linking techniques:
Revised: The ﬁrst two sources are from canonical information retrieval and entity linking techniques:

Original: • D ir : the top  documents retrieved by  TF-IDFmatching on the question (e.g., “Facebook”)
Revised: D ir : the top 100 documents retrieved by DrQA’s TF-IDF on the question (Chen et al., 2017).

Original: (Chen et al., 2017), • D el : the  documents associated with the entities that appear in the question (e.g., “Mark Zuckerberg”), annotated by entity linking systems (Ferragina & Scaiella, 2010; Hasibi et al., 2017).
Revised: D el : the Wikipedia documents associated with the entities that appear in the question, annotated by entity linking systems: TagMe (Ferragina & Scaiella, 2010) and CMNS (Hasibi et al., 2017).

Original: For better retrieval quality, we use a BERT ranker (Nogueira & Cho, 2019) on the set D ir ∪ D el and keep  top K ranked ones.
Revised: For better retrieval quality, we use a BERT ranker (Nogueira & Cho, 2019) on the set D ir ∪ D el and keep the top two ranked ones in D ir and top one per question entity in D el .

Original: Then the third source D exp include all documents connected to or from any top ranked documents via Wikipedia hyperlinks (e.g., “Facebook” → “Harvard University”).
Revised: Then the third source D exp include all documents connected to or from any top ranked documents via Wikipedia hyperlinks (e.g., “Facebook” → “Harvard University”).

Original: The ﬁnal graph comprises all documents from the three sources as nodes X and the Wikipedia links between them to form edge matrix E , i.e. e ij = 1 if there is a hyperlink from document i to j .
Revised: The ﬁnal graph comprises all documents from the three sources as nodes X . The edge matrix E is ﬂexible. We experiment with various edge matrix settings, including directed edges along Wikipedia links, i.e. e ij = 1 if there is a hyperlink from document i to j , the bidirectional edges along Wiki links, and fully-connected graphs, which rely on Transformer-XH to learns the edge importance.

Original: Similar to previous work (Ding et al., 2019), the textual representation for each node in the graph is the [SEP]-delimited concatenation of the question, anchor text 1 and the paragraph itself.
Revised: Similar to previous work (Ding et al., 2019), the textual representation for each node in the graph is the [SEP]-delimited concatenation of the question, anchor text (the text in the hyperlink in parent nodes pointing to the child node), and the paragraph itself.

Original: More details and analysis of the evidence graph construction are in AppendixSection A.1.
Revised: More details on the evidence graph construction are in Appendix A.1.

Original: Transformer-XH on Evidence Graph.
Revised: Transformer-XH on Evidence Graph.

Original: Transformer-XH takes the input nodes X and edges E , and produces the global representation of all text sequences:
Revised: Transformer-XH takes the input nodes X and edges E , and produces the global representation of all text sequences:

Original: [Equation]
Revised: [Equation]

Original: Then we add two task-speciﬁc layers upon the last layer’s representation H L : one auxiliary layer to predict the relevance score of the evidence node, and one layer to extract the answer span within it:
Revised: Then we add two task-speciﬁc layers upon the last layer’s representation H L : one auxiliary layer to predict the relevance score of the evidence node, and one layer to extract the answer span within it:

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: The ﬁnal model is trained end-to-end with cross-entropy loss for both tasks in a multi-task setting.
Revised: The ﬁnal model is trained end-to-end with cross-entropy loss for both tasks in a multi-task setting.

Original: During inference, we ﬁrst select the document with the highest relevance score, and then the start and end positions of the answer within that document.
Revised: During inference, we ﬁrst select the document with the highest relevance score, and then the start and end positions of the answer within that document.

Original:  E XPERIMENTAL M ETHODOLOGIES
Revised:  E XPERIMENTAL M ETHODOLOGIES

Original: Dataset.
Revised: Dataset.

Original: We conduct our experiments on HotpotQA, the multi-hop question answering benchmark dataset (Yang et al., 2018).
Revised: We conduct our experiments on HotpotQA, the multi-hop question answering benchmark dataset (Yang et al., 2018).

Original: It includes 112 k crowd-sourced questions designed to require multiple pieces of textual evidence, which are the ﬁrst paragraphs of Wikipedia pages.
Revised: It includes 112 k crowd-sourced questions designed to require multiple pieces of textual evidence, which are the ﬁrst paragraphs of Wikipedia pages.

Original: It has two type of questions: bridge question require hopping via an outside entity, and comparison question compare a property of two entities.
Revised: It has two type of questions: bridge question require hopping via an outside entity, and comparison question compare a property of two entities.

Original: There are two settings in HotpotQA.
Revised: There are two settings in HotpotQA.

Original: The Distractor setting provides golden evidence paragraphs together with TF-IDF retrieved negatives.
Revised: The Distractor setting provides golden evidence paragraphs together with TF-IDF retrieved negatives.

Original: The FullWiki setting requires systems to retrieve evidence paragraphs from the full Wikipedia.
Revised: The FullWiki setting requires systems to retrieve evidence paragraphs from the full Wikipedia.

Original: We focus on FullWiki setting since previous research found that the negative documents in Distractor may be too weak and mitigate the needs of multi-hop reasoning (Min et al., 2019b).
Revised: We focus on FullWiki setting since previous research found that the negative documents in Distractor may be too weak and mitigate the needs of multi-hop reasoning (Min et al., 2019b).

Original: There are 90 k
Revised: There are 90 k

Original: Train, 7 k Dev and 7 k Test questions.
Revised: Train, 7 k Dev and 7 k Test questions.

Original: The ground truth answer and supporting evidence sentences in Train and Dev sets are provided.
Revised: The ground truth answer and supporting evidence sentences in Train and Dev sets are provided.

Original: Test labels are hidden; only one submission is allowed to the leaderboard per 30 days 2 .
Revised: Test labels are hidden; only one submission is allowed to the leaderboard per 30 days 1 .

Original: We evaluate our best model on Test and conduct ablations on Dev.
Revised: We evaluate our ﬁnal model on Test and conduct ablations on Dev.

Original: Metrics.
Revised: Metrics.

Original: We use ofﬁcial evaluation metrics of HotpotQA. They include exact match (EM) and F1 on answer (Ans), supporting facts (Supp), and the combination (Joint).
Revised: We use ofﬁcial evaluation metrics of HotpotQA: exact match (EM) and F1 on answer (Ans), supporting facts (Supp), and the combination (Joint).

Original: The supporting facts prediction is an auxiliary task that evaluates model’s ability to ﬁnd the evidence sentences.
Revised: The supporting facts prediction is an auxiliary task that evaluates model’s ability to ﬁnd the evidence sentences.

Original: Joint EM is the product of the two EM result.
Revised: Joint EM is the product of the two EM result.

Original: Joint F1 ﬁrst multiplies the precision and recall from Ans and Supp, then combines the Joint precision and recall to F1.
Revised: Joint F1 ﬁrst multiplies the precision and recall from Ans and Supp, then combines the Joint precision and recall to F1.

Original: Baseline.
Revised: Baseline.

Original: We compare with previous approaches on the FullWiki setting, including Ofﬁcial Baseline (Yang et al., 2018), MUPPET (Feldman & El-Yaniv, 2019), QFE (Nishida et al., 2019), DecompRC (Min et al., 2019a), Cognitive QA (CogQA, Ding et al.
Revised: The main baselines include Cognitive QA (CogQA, Ding et al.

Original: (2019)), and SemanticRetrieval
Revised: and Semantic

Original:  MRS (SR-MRS, Nie et al.
Revised: Retrieval MRS (SR-MRS, Nie et al.

Original: We also build our BERT Pipeline baseline, which decomposes the task into several sub components and uses BERT based models on every component.

Original: CogQA is our major baseline and was the previous public SOTA.

Original: It uses several ﬁne-tuned BERT machine reading comprehension (MRC) models to ﬁnd hop entities and candidate spans, and then uses  Graph Convolution Network to rank the candidate spans, which is on top of BERT.
Revised: CogQA uses several ﬁne-tuned BERT machine reading comprehension (MRC) models to ﬁnd hop entities and candidate spans, and then uses a BERT based Graph Convolution Network to rank the candidate spans.

Original: SR-MRS is a contemporary work3 and was the previous leaderboard rank one.
Revised: SR-MRS is a contemporary work and was the previous leaderboard rank one.

Original: It is a BERT based pipeline and uses ﬁne-tuned BERT models to ﬁrst rank the documents (twice), then to rank sentences to ﬁnd supporting facts, and ﬁnally conduct BERT MRC on the concatenated evidence sentences. 
Revised: It is a BERT based pipeline and uses ﬁne-tuned BERT models to ﬁrst rank the documents (twice), then to rank sentences to ﬁnd supporting facts, and ﬁnally conducts BERT MRC on the concatenated evidence sentences. We also re-implement CogQA and upgrade its IR with our BERT IR model (BERT on D ir ∪ D el , same as Transformer-XH), for fair comparisons.

Revised: We include other approaches on the FullWiki setting: Ofﬁcial Baseline (Yang et al., 2018), MUPPET (Feldman & El-Yaniv, 2019), QFE (Nishida et al., 2019), and DecompRC (Min et al., 2019a),

Original: Implementation Details.
Revised: Implementation Details.

Original: The in-sequence attention and other standard Transformer components in Transformer-XH are initialized by the pre-trained BERT base model (Devlin et al., 2019).
Revised: The in-sequence attention and other standard Transformer components in Transformer-XH are initialized by the pre-trained BERT base model (Devlin et al., 2019).

Original: The extra hop attention parameters are initialized randomly and trained from scratch.
Revised: The extra hop attention parameters are initialized randomly and trained from scratch.

Original: We set the max hops in the reasoning graph to three, following the nature of HotpotQA (Yang et al., 2018).
Revised: The ﬁnal model uses three hop steps. For bridge questions, we build the evidence graph described in Section 3.

Original: More details of Transformer-XH and our BERT Pipeline can be found in the Appendix A.2 and A.4.
Revised: And for comparison questions, we build the fully-connected graph on the set D ir ∪ D el and train Transformer-XH separately. We leave more implementation details in the Appendix.

Original:  E VALUATION R ESULTS
Revised:  E VALUATION R ESULTS

Original: This section presents the evaluation results on Hotpot QA and analyzes the inﬂuence of different design choices of Transformer-XH, and evidence graph structure.
Revised: This section presents the evaluation results, ablation studies, and analyses on Hotpot QA.

Original: 5. O VERALL R ESULT
Revised: 5. O VERALL R ESULT

Original: We present the overall results of HotpotQA FullWiki setting in Table 1.
Revised: Table 1 presents HotpotQA FullWiki results.

Original: Transformer-XH outperforms all previous methods by signiﬁcant margins.
Revised: Transformer-XH outperforms all previous methods by signiﬁcant margins.

Original: Compared with CogQA, our main baseline, the improvements are consistently more than 10 absolute points.

Original: The only exception is Supp F1, where SR-MRS performs slightly better, though with much lower Supp EM.
Revised: The only exception is Supp F1, where SR-MRS performs slightly better, perhaps due to our different ways of Supp extraction.

Original: We think the reason is mainly from the search space variation.

Original: After predicting the answer, we selectthe supporting facts along the inference path (i.e., from all the parents node) whichnaturally ﬁts the task purpose but would hurt the recall (moredetails in Appendix).The ArXiv version is released one week before the submission of this work [Table] [Table]
Revised: We only select supporting facts from the inference path to the answer, which ﬁts the task purpose but may hurt the recall (more in Appendix).

Original: Besides strong results, Transformer-XH’s ability to natively represent structured data leads to much simpler QA system.
Revised: Besides strong results, Transformer-XH’s ability to natively represent structured data leads to much simpler QA system.

Original: Previously, in order to utilize pre-trained BERT, Hotpot QA approaches adapted the multi-hop reasoning task to comprise multiple sub-tasks.
Revised: Previously, in order to utilize pre-trained BERT, Hotpot QA approaches adapted the multi-hop reasoning task to comprise multiple sub-tasks.

Original: For example, given the retrieved documents, our pipeline approach ﬁrst leverages one BERT MRC model to ﬁnd hop entities and then another BERT MRCmodel to ﬁnd candidate answer spans.
Revised: For example, given the retrieved documents, CogQA (w. BERT IR) ﬁrst leverages one BERT MRC model to ﬁnd hop entities and then another BERT MRC to ﬁnd candidate answer spans.

Original: After that, it ranks the candidateanswer spans  using a BERT initialized GAT, which is the only structure modeling step.
Revised: After that, it ranks the candidate spans [Table] [Table] using a BERT based GAT, which is the only structure modeling step.

Original: In comparison, Transformer-XH is a uniﬁed model which directly represents structured texts and incorporates pre-trained BERT weights.
Revised: In comparison, TransformerXH is a uniﬁed model which directly represents structured texts and integrates BERT weights.

Original: Table 2 further inspects model performance on the Dev set by question type (Comparison or Bridge) and reasoning type (Single-Hop v.s. Multi-Hop).
Revised: Table 2 further inspects model performances on the Dev set by question types and reasoning types.

Original: Compared with baselines, Transformer-XH consistently achieves stronger results on all categories.
Revised: Transformer-XH signiﬁcantly outperforms all baselines on bridge questions which require more multi-hop reasoning.

Original: And on multi-hop questions, Transformer-XH has more relative gains ( 39% over CogQA on EM) than single-hop question ( 27% ), which demonstrates its stronger multi-hop reasoning capability.
Revised: And on the “multi-hop” questions, Transformer-XH has higher relative gains ( 39% over CogQA on EM) than the “single-hop” questions ( 27% ), demonstrating its stronger multihop reasoning capability.

Original: We further study thisproperty in Section 5.3.
Revised: We further study this in Section 5.3.

Revised: The gains of SR-MRS on comparison questions are mainly from its better retrieval on Comparison questions.

Revised: For fair comparison of Transformer-XH’s reasoning ability, we use the top retrieved documents from SR-MRS to construct Transformer-XH’s evidence graph.

Revised: The result system, Transformer-XH (w. SR-MRS), outperforms SR-MRS’s multi-step BERT based reasoning on all metrics and question types.

Revised: Transformer-XH’s effectiveness is robust with multiple IR systems.

Original: 5. A BLATION S TUDIES
Revised: 5. A BLATION S TUDIES

Original: Model Variations.
Revised: Model Variations.

Original: We show the results of different model variations on the top left of Table 3.
Revised: We show the results of different model variations on the top left of Table 3.

Original: Single-Hop BERT uses BERT MRC model on each document individually, which signiﬁcantly decreases the accuracy, conﬁrming the importance of multi-hop reasoning in FullWiki setting (Min et al., 2019a).
Revised: Single-Hop BERT uses BERT MRC model on each document individually, which signiﬁcantly decreases the accuracy, conﬁrming the importance of multi-hop reasoning in FullWiki setting (Min et al., 2019a).

Original: GAT + BERT ﬁrst uses Graph Attention Network (Veliˇckovi´c et al., 2017) on the evidence graph to predict the best node; then it uses BERT MRC on the best document.
Revised: GAT + BERT ﬁrst uses Graph Attention Network (Veliˇckovi´c et al., 2017) on the evidence graph to predict the best node; then it uses BERT MRC on the best document.

Original: It is 10% worse than Transformer-XH since the MRC model has no access to the information from other documents.
Revised: It is 10% worse than Transformer-XH since the MRC model has no access to the information from other documents.

Original: No Node Prediction eliminates the node prediction task and only trains on span prediction task; the accuracy difference shows node prediction task helps the model training.
Revised: No Node Prediction eliminates the node prediction task and only trains on span prediction task; the accuracy difference shows node prediction task helps the model training.

Original: Graph Structures.
Revised: Graph Structures.

Original: We show Transformer-XH’s performance with different graph structures on the bottom left of Table 3.
Revised: We show Transformer-XH’s performance with different graph structures on the bottom left of Table 3.

Original: Bidirectional Edges adds reverse edges along the hyperlinks; Fully
Revised: Bidirectional Edges adds reverse edges along the hyperlinks; Fully

Original: Connected Graph connects all document pairs; Node Sequence randomly permutes the documents and connects them into a sequence to simulate the Transformer-XL setting.
Revised: Connected Graph connects all document pairs; Node Sequence randomly permutes the documents and connects them into a sequence to simulate the Transformer-XL setting.

Original: Both Bidirectional
Revised: Both Bidirectional

Original: Links and Fully Connected Graph have comparable performance with the original graph structure.
Revised: Links and Fully Connected Graph have comparable performance with the original graph structure.

Original: Transformer-XH is able to learn meaningful connections using its hop attentions and is less dependent on the pre-existing graph structural.
Revised: Transformer-XH is able to learn meaningful connections using its hop attentions and is less dependent on the pre-existing graph structural.

Original: The fully connected graph can be used if there is no strong
Revised: The fully connected graph can be used if there is no strong

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: edge patterns available in the task.
Revised: edge patterns available in the task.

Original: However, the performance drops signiﬁcantly on Node Sequence , showing that structured texts cannot be treated as a linear sequence which cuts off many connections.
Revised: However, the performance drops signiﬁcantly on Node Sequence , showing that structured texts cannot be treated as a linear sequence which cuts off many connections.

Original: Hop Steps.
Revised: Hop Steps.

Original: Recall that a Transformer-XH layer with extra hop attention corresponds to one information propagation (hop) step in the graph.
Revised: Recall that a Transformer-XH layer with extra hop attention corresponds to one information propagation (hop) step in the graph.

Original: Thus Transformer-XH with last K layers conducts
Revised: Thus Transformer-XH with last K layers conducts

Original: K-step attention hops in the graph.
Revised: K-step attention hops in the graph.

Original: We show results with different K on the right side of Table 3.
Revised: We show results with different K on the right side of Table 3.

Original: Transformer-XH reaches its peak performance with three hops (our full-model).
Revised: Transformer-XH reaches its peak performance with three hops (our full-model).

Original: This is expected as most Hotpot QA questions can be answered by two documents (Yang et al., 2018).
Revised: This is expected as most Hotpot QA questions can be answered by two documents (Yang et al., 2018).

Original: 5. H OP A TTENTION A NALYSIS
Revised: 5. H OP A TTENTION A NALYSIS

Original: This experiment analyzes the hop attentions using our full-model (three-hop) on the fully connected graph to study their behavior without pre-deﬁned structure.
Revised: This experiment analyzes the hop attentions using our full-model (three-hop) on the fully connected graph to study their behavior without pre-deﬁned structure.

Original: Figure 2 plots the distributions of the learned hop attentions on the Dev set.
Revised: Figure 2 plots the distributions of the learned hop attentions on the Dev set.

Original: It shows a strong shift away from the normal distribution with more hops.
Revised: It shows a strong shift away from the normal distribution with more hops.

Original: Transformer-XH learns to distinguish different nodes after multi-hop attention: the attention score becomes a bimodal distribution after three hops, ignoring some non-useful nodes.
Revised: Transformer-XH learns to distinguish different nodes after multi-hop attention: the attention score becomes a bimodal distribution after three hops, ignoring some non-useful nodes.

Original: Transformer-XH also learns to focus on meaningful edges: the score is higher on the path
Revised: Transformer-XH also learns to focus on meaningful edges: the score is higher on the path

Original: Supp → Ans than All → Ans.
Revised: Supp → Ans than All → Ans.

Original: And the margin is larger as the hop step increases from one to three.
Revised: And the margin is larger as the hop step increases from one to three.

Original: 5. C ASE S TUDY
Revised: 5. C ASE S TUDY

Original: Table 4 lists two examples from Transformer-XH and our BERT Pipeline.
Revised: Table 4 lists two examples from Transformer-XH and CogQA (w. BERT IR).

Original: The ﬁrst case has a clear evidence chain “2011/S/S” → “Winner” → “YG Entertainment”; both methods ﬁnd the correct answer.
Revised: The ﬁrst case has a clear evidence chain “2011/S/S” → “Winner” → “YG Entertainment”; both methods ﬁnd the correct answer.

Original: However, the second case has too many distractors in the ﬁrst document.
Revised: However, the second case has too many distractors in the ﬁrst document.

Original: Without additional clues from document 2, it is likely that the single-hop hop entity extraction component in our pipeline approach misses the correct answer document in its candidate sets; and the later structural reasoning component can not recover from this cascade error.
Revised: Without additional clues from document 2, it is likely that the single-hop hop entity extraction component in CogQA (w. BERT IR) misses the correct answer document in its candidate sets; and the later structural reasoning component can not recover from this cascade error.

Original: In comparison, Transformer-XH ﬁnds the correct answer by combining the evidence with the hop attentions between the two evidence pieces.
Revised: In comparison, Transformer-XH ﬁnds the correct answer by combining the evidence with the hop attentions between the two evidence pieces.

Original: We leave more positive and negative cases in Appendix A.5.
Revised: We leave more positive and negative cases in Appendix A.5.

Original: [Table]
Revised: [Table]

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Machine reading comprehension is an important task of natural language processing (Rajpurkar et al., 2016; Bajaj et al., 2016).
Revised: Machine reading comprehension is an important task of natural language processing (Rajpurkar et al., 2016; Bajaj et al., 2016).

Original: The initial MRC questions are mostly single-hop, which only require single evidence sentence to answer (Min et al., 2018).
Revised: The initial MRC questions are mostly single-hop, which only require single evidence sentence to answer (Min et al., 2018).

Original: One direction to make the task more realistic is open-domain question answering (QA), in which the evidence paragraph has to be retrieved from a large corpus (Chen et al., 2017).
Revised: One direction to make the task more realistic is open-domain question answering (QA), in which the evidence paragraph has to be retrieved from a large corpus (Chen et al., 2017).

Original: The other direction is multi-hop QA, where the questions require multiple pieces of evidence to answer (Welbl et al., 2018; Yang et al., 2018).
Revised: The other direction is multi-hop QA, where the questions require multiple pieces of evidence to answer (Welbl et al., 2018; Yang et al., 2018).

Original: The FullWiki settingof HotpotQA is a combination of open-domain QA and multi-hop QA (Yang et al., 2018): the questions are designed to require multiple pieces of evidence and these evidence pieces are documents to retrieve from the entire Wikipedia.
Revised: HotpotQA’s FullWiki setting is a combination of open-domain QA and multi-hop QA (Yang et al., 2018): the questions are designed to require multiple pieces of evidence and these evidence pieces are documents to retrieve from the entire Wikipedia.

Original: The combination makes this setting rather challenging. The retrieved documents are inevitably noisy and include much stronger distractors than the TF-IDF retrievednegative documents in the Distractor setting (Min et al., 2019a; Jiang & Bansal, 2019).
Revised: The combination makes this setting challenging: The retrieved documents are inevitably noisy and include much stronger distractors than the TF-IDF retrieved documents in the Distractor setting (Min et al., 2019a; Jiang & Bansal, 2019).

Original: Various solutions have been proposed for Hotpot QA (Min et al., 2019b; Feldman & El-Yaniv, 2019;
Revised: Various solutions have been proposed for Hotpot QA (Min et al., 2019b; Feldman & El-Yaniv, 2019;

Original: Nishida et al., 2019).
Revised: Nishida et al., 2019).

Original: These solutions often use complicated pipelines to adapt the multi-hop task into a combination of single-hop tasks, in order to leverage the advantage of pre-trained models.
Revised: These solutions often use complicated pipelines to adapt the multi-hop task into a combination of single-hop tasks, in order to leverage the advantage of pre-trained models.

Original: For example, CogQA (Ding et al., 2019) uses two BERT based MRC model to ﬁnd candidate spans and then another BERT initialized Graph Neural Network (GNN) to rank spans;
Revised: For example, CogQA (Ding et al., 2019) uses two BERT based MRC model to ﬁnd candidate spans and then another BERT initialized Graph Neural Network (GNN) to rank spans;

Original: SR-MRS (Nie et al., 2019) uses three BERT based rankers to ﬁnd supporting sentences, and then another BERT MRC model on the concatenated sentences to get the answer span.
Revised: SR-MRS (Nie et al., 2019) uses three BERT based rankers to ﬁnd supporting sentences, and then another BERT MRC model on the concatenated sentences to get the answer span.

Original: Transformer-XH is a simpler model that directly represents and reasons with multiple pieces of evidence using extra hop attentions.
Revised: Transformer-XH is a simpler model that directly represents and reasons with multiple pieces of evidence using extra hop attentions.

Original: In addition to Transformer-XL (Dai et al., 2019), Transformer-XH is also inspired by GNN (Kipf &
Revised: In addition to Transformer-XL (Dai et al., 2019), Transformer-XH is also inspired by GNN (Kipf &

Original: Welling, 2017; Schlichtkrull et al., 2017;
Revised: Welling, 2017; Schlichtkrull et al., 2017;

Original: Veli ˇ ckovi ´ c et al., 2017), which leverages neural networks to model graph structured data.
Revised: Veli ˇ ckovi ´ c et al., 2017), which leverages neural networks to model graph structured data.

Original: The key difference is that a “node” in Transformer-XH is a text sequence, and modeling of the structure is conducted jointly with the representation of the text.
Revised: The key difference is that a “node” in Transformer-XH is a text sequence, and modeling of the structure is conducted jointly with the representation of the text.

Original: Transformer-XH combines the advantages Transformer has in understanding text with the power that GNN has in modeling structure.
Revised: Transformer-XH combines the advantages Transformer has in understanding text with the power that GNN has in modeling structure.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: Transformer-XH and its eXtra Hop attention mechanism is a simple yet powerful adaptation of
Revised: Transformer-XH and its eXtra Hop attention mechanism is a simple yet powerful adaptation of

Original: Transformer to learn better representations of structured text data as it naturally occurs.
Revised: Transformer to learn better representations of structured text data as it naturally occurs.

Original: It innately integrates with pre-trained language models to allow for complex reasoning across many hops of a textual evidence graph, where clues to the correct answer of a question are split across multiple documents.
Revised: It innately integrates with pre-trained language models to allow for complex reasoning across many hops of a textual evidence graph, where clues to the correct answer of a question are split across multiple documents.

Original: When applied to HotpotQA, Transformer-XH signiﬁcantly shrinks the typical multi-hop
Revised: When applied to HotpotQA, Transformer-XH signiﬁcantly shrinks the typical multi-hop

Original: QA pipeline, eliminating many cascading errors that arise from the linear sequence input constraints of pre-trained Transformers.
Revised: QA pipeline, eliminating many cascading errors that arise from the linear sequence input constraints of pre-trained Transformers.

Original: Compared to previous SOTA baselines that use pipelines of BERT models to mimic multi-hop reasoning, one Transformer-XH model is all we need to obtain a much stronger answer accuracy.
Revised: Compared to previous SOTA baselines that use pipelines of BERT models to mimic multi-hop reasoning, one Transformer-XH model is all we need to obtain a much stronger answer accuracy.

Original: With its simplicity and efﬁcacy, we envision Transformer-XH will beneﬁt many applications in the near future.
Revised: With its simplicity and efﬁcacy, we envision Transformer-XH will beneﬁt many applications in the near future.

Original: A  PPENDIX
Revised: A A PPENDIX

Original: A.
Revised: The appendix includes details of the evidence graph construction for Hotpot QA, ablation studies in the BERT IR component, more details and results on Hotpot QA, and results on the FEVER fact veriﬁcation task.

Original:  H OTPOT QA E VIDENCE G RAPH C ONSTRUCTION D ETAILSA.1. G RAPH C ONSTRUCTION DETAILS
Revised: A. H OTPOT QA E VIDENCE G RAPH C ONSTRUCTION D ETAILS

Original: To construct the evidence graph, we start with extracting documents directly from question.
Revised: The evidence graph construction includes two stages. The ﬁrst stage is BERT IR, which extract related documents directly from question.

Original: This step contains both documents from information retrieval D ir and entity linking D el .
Revised: The second stage expands the related documents along Wikipedia links. The ﬁrst is applied on all questions while the second is only required by Bridge questions.

Original: The goal of this step is to have a reasonable size set with high recall of necessary information, i.e., supporting pages.
Revised: The ﬁrst stage uses two methods to ﬁnd documents. The ﬁrst method uses DrQA’s retrieval system (Chen et al., 2017), which is unsupervised TF-IDF.

Original: [Table] We use Top-100 ranked documents for D ir .
Revised: We keep the top 100 DrQA retrieved ones D ir for each question.

Original: For D el , to increase the recall of existing entity linking tool, we combine the output from two entity linking tools TagMe (Ferragina & Scaiella, 2010) and CMNS (Hasibi et al., 2017).
Revised: The second method uses TagMe (Ferragina & Scaiella, 2010) and CMNS (Hasibi et al., 2017), two commonly used entity linkers, to annotate questions. We keep the TagMe output entity and three highest scored entities per surface form (a phrase in the question linked with entities) from CMNS, and use its corresponding Wikipedia document as D el .

Original: We use BERT ranker (Nogueira & Cho, 2019) to re-rank the initial set.
Revised: We use BERT ranker (Nogueira & Cho, 2019) to re-rank the initial set D ir ∪ D el .

Original: The input to the BERT is the concatenation of question and ﬁrst paragraph of document, and it outputs the relevance score of the document.
Revised: The input to the BERT is the concatenation of question and ﬁrst paragraph of document: [Equation] Then a linear layer is added on the last layer’s

Original: To make sure the documents corresponding to question entities are extracted, we use SpaCy (Honnibal & Montani, 2017) to tag question named entities, and choose the topranked documents for each tagged entity from D el .
Revised: [CLS] representation to score the relevance of the document. We use BERT base and ﬁne-tune it using the relevance label (from supporting facts) with cross-entropy loss.

Original: For D ir, we choose the top-K documents. We experiment with different K in Sec A.1.2, and use K=2 as document set D for the next step.
Revised: The top two highest scored documents from D ir and the top one document per entity position (surface form) in D el are kept as the ﬁrst stage BERT IR documents.

Original: Then we use Wikipedia hyperlinksover top-ranked documents D to get the expanded document set
Revised: The second stage expands the ﬁrst stage BERT IR documents by Wikipedia hyperlinks to obtain

Original: D exp .
Revised: D exp .

Revised: A document is included if it is linked to or links to a document in the ﬁrst stage.

Original: We use the same BERT ranker to rank D exp for the top-K documents.
Revised: We use the same BERT ranker to rank D exp and keep the top 15 documents in D exp .

Original: This step is for the memory efﬁciency purpose and we choose a relatively large K = 15  . 
Revised: The ﬁnal evidence graph nodes per question includes the top two highest ranked documents in D ir , top one per entity name in D el , and top 15 from the expanded documents D exp . The comparison questions only require information from two question entities; thus when building the evidence graph we do not expand them (i.e. there is no D exp ). The retrieval pipeline is a multi-stage retrieval enhanced with entity linking.

Original: We report our ﬁnal graph recall in Sec A.1.2.
Revised: It is close to the retrieval system used in SR-MRS (Nie et al., 2019). When using SR-MRS retrieved documents for documents, we use top 10 documents on bridge questions and top two documents on comparison questions. In the next section, we show that Transformer-XH is robust to different number of documents kept in the evidence graph and performs similarly using the documents retrieved from SR-MRS.

Original: A.1. R ETRIEVALR ESULT
Revised: A. A BLATION S TUDIES O N D OCUMENT R ETRIEVAL

Original: Table 5 compares the coverage of supporting facts between CogQA and our approach. Our two-step ranking is helpful, which increases the coverage from 70 .
Revised: This experiment studies the effectiveness and inﬂuence of different retrieval settings. We use different numbers of top K ranked documents from the BERT ranker, run Transformer-XH in the corresponding evidence graph, and evaluate its performance on Bridge questions in the Dev set. We also evaluate the Supporting facts Recall and Answer Recall.

Original: 8% to 78 .
Revised: Supp Recall evaluates whether the document with the supporting fact is included in the ﬁrst stage retrieved documents. Ans Recall evaluates whether their exists a document in the evidence graph that includes the ground truth answer. The results are in Table 5. Our BERT IR system performs better than CogQA’s TF-IDF and on par with SR-MRS, as expected.

Original: Also the high answer recall of evidence graph provides a good basis for Transformer-XH modeling.
Revised: The latter uses a similar retrieval pipeline with our BERT IR system; Transformer-XH is robust on different retrieval settings and keeps its effectiveness when applied on Top 10 documents from SR-MRS (including both stages). [Table]

Original: A. O THER H OTPOT QA COMPONENTS
Revised: A. O THER H OTPOT QA COMPONENTS

Original: This section describes the other components for HotpotQA dataset.
Revised: This section describes the other components for HotpotQA dataset.

Original: The whole QA system starts with question  classiﬁcation.
Revised: The whole QA system starts with question type classiﬁcation.

Original: For bridge question, we use Transformer-XH over  evidence graph to get the answer.
Revised: We train transformer-XH separately on each type of questions over their evidence graph.

Original: For comparison question, we use BERT based model to predict the span or yes/no as answer.

Original: Besides answer prediction, we also adopt BERT based model for predicting supporting sentences.
Revised: Besides answer prediction, we also adopt BERT based model for predicting supporting sentences.

Original: A.2. QUESTION CLASSIFICATION
Revised: A.3. QUESTION CLASSIFICATION

Original: The ﬁrst component of our system is to classify the question to bridge and comparison types.
Revised: The ﬁrst component of our system is to classify the question to bridge and comparison types.

Original: We adopt BERT classiﬁcation ﬁne-tuning setting on HotpotQA questions with 99.1 % accuracy on the dev set.
Revised: We adopt BERT classiﬁcation ﬁne-tuning setting on HotpotQA questions using the question type labels provided in HotpotQA. The classiﬁer achieves 99.1 % accuracy on the dev set.

Revised: We use the classiﬁer to split the questions into Comparison and Bridge.

Original: A.2. S UPPORTING FACTS CLASSIFICATION
Revised: A.3. S UPPORTING FACTS CLASSIFICATION

Original: The supporting facts prediction task is to extract all sentences that help get the answer.
Revised: The supporting facts prediction task is to extract all sentences that help get the answer.

Original: For bridge question, these sentences usually cover different pieces of questions.
Revised: For bridge question, these sentences usually cover different pieces of questions.

Original: And for comparison questions, the supporting facts are the properties of two question entities.
Revised: And for comparison questions, the supporting facts are the properties of two question entities.

Original: We design one model architecture for this task, but we train two model on each type to reﬂect the inherent difference.
Revised: We design one model architecture for this task, but we train two models on each type to reﬂect the inherent difference.

Original: We use BERT as our base model and on top of BERT, we conduct multi-task learning scheme.
Revised: We use BERT as our base model and on top of BERT, we conduct multi-task learning scheme.

Original: The ﬁrst task is document relevance prediction, similar as Transformer-XH, we add a linear layer on the [CLS] token of BERT to predict the relevance score.
Revised: The ﬁrst task is document relevance prediction, similar as Transformer-XH, we add a linear layer on the [CLS] token of BERT to predict the relevance score.

Original: The other task is sentence binary classiﬁcation, we concatenate the ﬁrst and last token representation of each sentence in the document through a linear layer, the binary output decides whether this sentence is supporting sentence.
Revised: The other task is sentence binary classiﬁcation, we concatenate the ﬁrst and last token representation of each sentence in the document through a linear layer, the binary output decides whether this sentence is supporting sentence.

Original: Bridge question supporting facts prediction
Revised: Bridge question supporting facts prediction

Original: For bridge questions, we predict supporting facts after answer prediction from Transformer-XH to resume the inference chain.
Revised: For bridge questions, we predict supporting facts after answer prediction from Transformer-XH to resume the inference chain.

Original: We start by predicting supporting facts in the answer document.
Revised: We start by predicting supporting facts in the answer document.

Original: The other document is chosen from the parents of the answer document in the evidence graph.
Revised: The other document is chosen from the parents of the answer document in the evidence graph.

Original: Compare with the contemporary model Nie et al.
Revised: Compare with the contemporary model Nie et al.

Original: (2019), which does not limit the search space along the inference chain (i.e., the answer document may not be relevant to the other supporting page), our method more naturally ﬁts the task purpose.
Revised: (2019), which does not limit the search space along the inference chain (i.e., the answer document may not be relevant to the other supporting page), our method more naturally ﬁts the task purpose.

Original: Comparison question supporting facts prediction
Revised: Comparison question supporting facts prediction

Original: For comparison questions, after extracting the ﬁrst step documents D , we simply run this supporting facts prediction model to select the topdocuments, and predict the corresponding supporting facts.
Revised: For comparison questions, after extracting the ﬁrst step documents D , we simply run this supporting facts prediction model to select the topdocuments, and predict the corresponding supporting facts.

Original: A.2. A NSWER C OMPARISON QUESTIONS
Revised: A.3. T RAINING DETAILS

Original: After predicting supporting facts, we concatenate the sentences and follow Min et al. (2019b) to run a BERT MRC model to predict either span or yes/
Revised: We use DGL (Wang et al., 2019) for implementing Transformer-XH and CogQA (w. BERT IR) with batch size 1 (i.e., one graph for each batch), and keep the other parameters same as default BERT setting.

Original: no as the answer.
Revised: We train Transformer-XH separately on two different types of questions, following previous research (Ding et al., 2019). We train Transformer-XH and the GNN of CogQA (w. BERT IR) forepochs.

Original: As shown in Table 2, this straightforward approach achieves strong result.
Revised: All other BERT based models use the default BERT parameters and train the model forepoch.

Original: A. H OTPOT QA  BERT P IPELINE A PPROACH
Revised: A. I MPLEMENTATION D ETAILS OF C OG QA (W. BERT IR)

Original: This section discusses our implementation of pipeline approach for HotpotQA bridge question.
Revised: This section discusses our implementation of CogQA (W. BERT IR).

Original: We start with  same methods for extracting the document set D .
Revised: We start with the same documents from BERT IR, the one used by Transformer-XH, and then implement the following steps:

Original: A.3. H OP ENTITY EXTRACTION
Revised: A.4. H OP ENTITY EXTRACTION

Original: For each document from the previous step, we run BERT MRC model(Ding et al., 2019) and limit the span candidates as hyperlinked entities for hop entity extraction (e.g., in Figure 1, “Harvard University” is a hop entity).
Revised: For each document from the previous step, we run BERT MRC model and limit the span candidates as hyperlinked entities for hop entity extraction (e.g., in Figure 1, “Harvard University” is a hop entity).

Original: Follow Ding et al.
Revised: Following Ding et al.

Original: (2019), we predict the top-3 entities that above the relative threshold that is the start span probability of [CLS] position.
Revised: (2019), we predict the top three entities that above the relative threshold that is the start span probability of [CLS] position.

Original: A.3. A NSWER SPAN E XTRACTION
Revised: A.4. A NSWER SPAN E XTRACTION

Original: For each document (add the hop entity document), follow Ding et al.
Revised: For each document (add the hop entity document), following Ding et al.

Original: (2019), we run BERT MRC modelDing et al. (2019) to extract spans (e.g., “Combridge” in Figure 1.).
Revised: (2019), we run BERT MRC model to extract spans (e.g., “Combridge” in Figure 1.).

Original: We predict the top-1 span that above the threshold that is the start span probability of [CLS] position.
Revised: We predict the top one span that above the threshold that is the start span probability of [CLS] position.

Original: We train both hop entity extraction and span extraction tasks with same BERT model but different prediction layers.
Revised: We train both hop entity extraction and span extraction tasks with same BERT model but different prediction layers.

Original: For each training example, we extract the link between two given supporting pages.
Revised: For each training example, we extract the link between two given supporting pages.

Original: The page includes the link  is the supporting page for hop entity extraction, while the other page is the answer page  for answer span extraction.
Revised: The page includes the link (e.g., “Harvard University” in Figure 1.) is the supporting page for hop entity extraction, while the other page is the answer page (e.g., “Combridge” in Figure 1.) for answer span extraction.

Original: A.3. GAT MODELING
Revised: A.4. GAT MODELING

Original: All the entities and answer spans form the ﬁnal graph.
Revised: All the entities and answer spans form the ﬁnal graph.

Original: The nodes are the entities and spans, and edges are the connections from the entities to the extracted hop entities or spans.
Revised: The nodes are the entities and spans, and edges are the connections from the entities to the extracted hop entities or spans.

Original: We use BERT for each node representation with question, anchor sentences and context, following Ding et al.
Revised: We use BERT for each node representation with question, anchor sentences and context, following Ding et al.

Original: We runstandard GAT (Veliˇckovi´c et al., 2017) on top of BERT to predict the correct answer span node.
Revised: We run GAT (Veliˇckovi´c et al., 2017) on top of BERT to predict the correct answer span node

Original: A. T RAINING DETAILS
Revised: . A.4. C OMPARISON Q UESTIONS

Original: We use DGL (Wang et al., 2019) for implementing Transformer-XH and pipeline GAT model with batch size 1 (i.e., one graph for each batch), and keep the other parameters same as default BERT setting.
Revised: After predicting supporting facts, we concatenate the sentences and follow Min et al. (2019b) to run a BERT MRC model to predict either span or yes/ no as the answer. A. A DDITIONAL CASE STUDY

Original: We train the model on bridge questions only for 2 epochs.
Revised: We provide addition case studies in Table 6.

Original: For all other BERT based models, we use the default BERT parameters and train the model forepoch
Revised: This section evaluates Transformer-XH in the FEVER Fact Extraction and Veriﬁcation Task (Thorne et al., 2018), the other task SR-MRS was evaluated on (Nie et al., 2019)

Original: . A. A DDITIONAL CASE STUDY
Revised: . A.6. FEVER T ASK

Revised: Dataset and Metrics.

Original:  [Table]
Revised: The FEVER task provides a claim sentence and requires the system to classify it into three categories: SUPPORTS, REFUTES, and NOT ENOUGH INFO, using the Wikipedia corpus as the evidence source. It provides 185,455 claims with manual labels and uses the Wikipedia dump in June 2017 which includes 5.4 million documents. There are two ofﬁcial evaluation metrics in FEVER: Label Accuracy (LA), which evaluates the classiﬁcation accuracy of the veriﬁcation labels, and FEVER Score, which evaluates both the correctness [Table]

Revised: of the evidence sentences used in veriﬁcation and the LA.

Revised: The latter is close to Joint EM in Hotpot

Revised: QA and is the main metric.

Revised: We use the ofﬁcial evaluation scripts 3 from FEVER task and we refer to

Revised: Thorne et al.

Revised: (2018) for more details of this task.

Revised: Evidence Graph Construction.

Revised: Most top performance FEVER systems on the leaderboard ﬁrst retrieve the evidence sentences for the claim and then reason upon the evidence sentences to verify the claim (Nie et al., 2019; Zhou et al., 2019).

Revised: The ﬁrst step is similar as the retrieval stage in Hotpot

Revised: QA.

Revised: And the second step is a multi-evidence reasoning task, where Transformer-XH is applied.

Revised: We keep the evidence sentence retrieval step consistent with previous methods.

Revised: The sentence retrieval results of SR-MRS is not yet released at the time of our experiments, thus we instead use the BERT-based retrieval results from another contemporary work (Liu et al., 2019b) in the FEVER leaderboard 4 .

Revised: We construct the evidence graph using the top ﬁve sentences from Liu et al.

Revised: (2019b)

Revised: as the nodes X and fully connected edges E .

Revised: Following Liu et al.

Revised: (2019b), the representation of each node is the concatenation of the claim, the Wiki title (entity name) of the document which includes the sentence, and the evidence sentence.

Revised: A.6. T RANSFORMER -XH ON FEVER

Revised: Transformer-XH takes the evidence graph { X, E } and learns to verify the claim to three categories: y ∈ { SUPPORT, REFUSE, and NOT ENOUGH EVIDENCE } .

Revised: Similar to the application in Hotpot 3 https://github.com/sheffieldnlp/fever-scorerhttps://competitions.codalab.org/competitions/18814#results

Revised: [Table]

Revised: QA, it ﬁrst produces the global representation of the graph:

Revised: [Equation]

Revised: [Equation]

Revised: Then two task-speciﬁc layers are added upon the last layer.

Revised: The ﬁrst layer conducts the fact prediction per node using the “[CLS]” token:

Revised: [Equation]

Revised: [Equation]

Revised: The second layer learns to measure the importance of each node in the graph:

Revised: [Equation]

Revised: The node level predictions and node importance are combined to the ﬁnal prediction for the claim:

Revised: [Equation]

Revised: Similar as in Hotpot QA, we use multi-task learning that combines the node prediction task and the claim veriﬁcation task.

Revised: The ﬁrst uses the evidence sentence label provided by FEVER and crossentropy loss on Eqn. 15.

Revised: The second is the ﬁnal veriﬁcation label with cross-entropy loss on Eqn. 16.

Revised: A.6. E XPERIMENTS ON FEVER

Revised: Experimental Setups.

Revised: We follow the experiment settings used by previous research in FEVER 1.

Revised: shared task, i.e. Nie et al.

Revised: Zhou et al.

Revised: (2019), and Liu et al. (2019b).

Revised: Similar as Liu et al.

Revised: (2019b), we also split the data into single and multi evidence categories and evaluate TransformerXH on the two splits.

Revised: Baselines.

Revised: The baselines include GEAR (Zhou et al., 2019) and two contemporary work, SRMRS (Nie et al., 2019) and KGAT (Liu et al., 2019b).

Revised: SR-MRS uses similar adaptations as

Revised: Transformer-XH from Hotpot QA to FEVER.

Revised: GEAR is a graph attention network based approach specially designed for fact veriﬁcation.

Revised: KGAT further improves GEAR’s GAT by adding the kernel information, and is the best method on the current leaderboard with BERT base or smaller parameter sizes.

Revised: (The better entries use Bert large or XL-NET.)

Revised: We also include the BERT Concat baseline listed in Liu et al.

Revised: (2019b) which concatenates the evidence sentences to a text sequence and applies

Revised: BERT on it.

Revised: Evaluation Results.

Revised: The results on FEVER are shown in Table 7.

Revised: Transformer-XH outperforms SRMRS by 4 FEVER score on Dev and 1.8 on Test.

Revised: It performs on par with KGAT.

Revised: The effectiveness of Transformer-XH is also more observed on claims that require multiple evidence to verify: it outperforms the contemporary work KGAT by 20 FEVER scores on the multi-evidence claims, a 49% relative improvement.

Revised: Compared to KGAT, Transformer-XH mainly loses on the ”not enough evidence” category which is not single nor multi evidence.

Revised: This is a unique property of the fact veriﬁcation task which our system is not speciﬁcally designed for.

Revised: This result further demonstrates Transformer-XH’s effectiveness on tasks with multiple text inputs not in sequential formats.

Revised: We simply adapt Transformer-XH to the FEVER dataset by only changing the last (linear) task speciﬁc layer; it leads to similar or better performances over contemporary approaches that were speciﬁcally designed for the fact veriﬁcation task.