Original: T RANSFORMER -XH: M ULTI - HOP Q UESTION A NSWERING WITH E X TRA H OP A TTENTION
Revised: T RANSFORMER -XH: M ULTI - HOP Q UESTION A NSWERING WITH E X TRA H OP A TTENTION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Transformers have obtained signiﬁcant success modeling natural language as a sequence of text tokens.
Revised: Transformers have obtained signiﬁcant success modeling natural language as a sequence of text tokens.

Original: However, in many real world scenarios, textual data inherently exhibits structures beyond a linear sequence such as trees and graphs; an important one being multi-hop question answering, where evidence required to answer questions are scattered across multiple related documents.
Revised: However, in many real world scenarios, textual data inherently exhibits structures beyond a linear sequence such as trees and graphs; an important one being multi-hop question answering, where evidence required to answer questions are scattered across multiple related documents.

Original: This paper presents Transformer-XH, which uses eXtra Hop attention to enable the intrinsic modeling of structured texts in a fully data-driven way.
Revised: This paper presents Transformer-XH, which uses eXtra Hop attention to enable the intrinsic modeling of structured texts in a fully data-driven way.

Original: Its new attention mechanism naturally “hops” across the connected text sequences in addition to attending over tokens within each sequence.
Revised: Its new attention mechanism naturally “hops” across the connected text sequences in addition to attending over tokens within each sequence.

Original: Thus, Transformer-XH better answers multi-hop questions by propagating information between multiple documents, constructing global contextualized representations, and jointly reasoning over multiple pieces of evidence.
Revised: Thus, Transformer-XH better answers multi-hop questions by propagating information between multiple documents, constructing global contextualized representations, and jointly reasoning over multiple pieces of evidence.

Original: This leads to a simpler multi-hop QA system which outperforms previous state-of-the-art on the HotpotQA FullWiki setting by large margins.
Revised: This leads to a simpler multi-hop QA system which outperforms previous state-of-the-art on the HotpotQA FullWiki setting by large margins.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Transformers effectively model natural language in sequential form (Vaswani et al., 2017; Dai et al., 2019;
Revised: Transformers effectively model natural language in sequential form (Vaswani et al., 2017; Dai et al., 2019;

Original: Devlin et al., 2019; Yang et al., 2019).
Revised: Devlin et al., 2019; Yang et al., 2019).

Original: Nevertheless, in many NLP tasks, text does not simply appear as a linear sequence of tokens but rather carries meaningful structure in the form of paragraphs, headings, and hyperlinks.
Revised: Nevertheless, in many NLP tasks, text does not simply appear as a linear sequence of tokens but rather carries meaningful structure in the form of paragraphs, headings, and hyperlinks.

Original: These structures can be abstracted into trees or graphs with nodes and edges.
Revised: These structures can be abstracted into trees or graphs with nodes and edges.

Original: Multi-hop question answering (Yang et al., 2018) is one such task in which structure plays an important role, since the evidence required to formulate the answer is scattered across multiple documents, requiring systems to jointly reason across links between them.
Revised: Multi-hop question answering (Yang et al., 2018) is one such task in which structure plays an important role, since the evidence required to formulate the answer is scattered across multiple documents, requiring systems to jointly reason across links between them.

Original: Recent approaches leverage pre-trained Transformers (e.g., BERT) for multi-hop question answering (QA) by converting the structural reasoning task into sub-tasks that model ﬂat sequences.
Revised: Recent approaches leverage pre-trained Transformers (e.g., BERT) for multi-hop question answering (QA) by converting the structural reasoning task into sub-tasks that model ﬂat sequences.

Original: For example, Min et al.
Revised: For example, Min et al.

Original: (2019b) decompose a multi-hop question into a series of single-hop questions;
Revised: (2019b) decompose a multi-hop question into a series of single-hop questions;

Original: Ding et al.
Revised: Ding et al.

Original: (2019) conduct several steps of single-hop reading comprehension to simulate the multihop reasoning.
Revised: (2019) conduct several steps of single-hop reading comprehension to simulate the multihop reasoning.

Original: The hope is that additional processing to fuse the outputs of the sub-models can recover all the necessary information from the original structure.
Revised: The hope is that additional processing to fuse the outputs of the sub-models can recover all the necessary information from the original structure.

Original: While pre-trained Transformer language models have shown improvements on multi-hop QA, manipulating the inherent structure of the problem to ﬁt the rigid requirements of out-of-the-box models can introduce problematic assumptions or information loss.
Revised: While pre-trained Transformer language models have shown improvements on multi-hop QA, manipulating the inherent structure of the problem to ﬁt the rigid requirements of out-of-the-box models can introduce problematic assumptions or information loss.

Original: This paper presents Transformer-XH (meaning eXtra Hop), which upgrades Transformers with the ability to natively represent structured texts.
Revised: This paper presents Transformer-XH (meaning eXtra Hop), which upgrades Transformers with the ability to natively represent structured texts.

Original: Transformer-XH introduces extra hop attention in its layers that connects different text pieces following their inherent structure while also maintaining the powerful pre-trained Transformer abilities over each textual piece individually.
Revised: Transformer-XH introduces extra hop attention in its layers that connects different text pieces following their inherent structure while also maintaining the powerful pre-trained Transformer abilities over each textual piece individually.

Original: Our extra hop attention enables 1) a more global representation of the evidence presented by each piece of text as it relates to the other evidence, and 2) a more natural way to jointly reason over an evidence graph by propagating information along edges necessary to complete the task at hand.
Revised: Our extra hop attention enables 1) a more global representation of the evidence presented by each piece of text as it relates to the other evidence, and 2) a more natural way to jointly reason over an evidence graph by propagating information along edges necessary to complete the task at hand.

Original: We apply Transformer-XH to HotpotQA, a challenging benchmark for the multi-hop question answering task (Yang et al., 2018).
Revised: We apply Transformer-XH to HotpotQA, a challenging benchmark for the multi-hop question answering task (Yang et al., 2018).

Original: Rather than decomposing the task into a series of sub-tasks to ﬁt the constraints of pre-trained Transformers, Transformer-XH is a solution that ﬁts the problem as it naturally occurs.
Revised: Rather than decomposing the task into a series of sub-tasks to ﬁt the constraints of pre-trained Transformers, Transformer-XH is a solution that ﬁts the problem as it naturally occurs.

Original: It is a single model that represents and combines evidence from multiple documents to construct the answer.
Revised: It is a single model that represents and combines evidence from multiple documents to construct the answer.

Original: [Figure]
Revised: [Figure]

Original: On HotpotQA’s FullWiki setting, which is a more realistic setup and requires stronger multihop reasoning ability (Min et al., 2019b; Jiang & Bansal, 2019), Transformer-XH outperforms
Revised: On HotpotQA’s FullWiki setting, which is a more realistic setup and requires stronger multihop reasoning ability (Min et al., 2019b; Jiang & Bansal, 2019), Transformer-XH outperforms

Original: CogQA (Ding et al., 2019), the previous start-of-the-art, by 12 points on answer F1.
Revised: CogQA (Ding et al., 2019), the previous start-of-the-art, by 12 points on answer F1.

Original: It also beats the contemporary BERT based pipeline SR-MRS (Nie et al., 2019), by 3 points.
Revised: It also beats the contemporary BERT based pipeline SR-MRS (Nie et al., 2019), by 3 points.

Original: The results follow from our simple yet effective design, with one uniﬁed model operating over the inherent structure of the task, rather than melding the outputs from disparate sub-tasks adapted to the sequential constraints of pre-trained Transformer.
Revised: The results follow from our simple yet effective design, with one uniﬁed model operating over the inherent structure of the task, rather than melding the outputs from disparate sub-tasks adapted to the sequential constraints of pre-trained Transformer.

Original: Our ablation analysis demonstrates Transformer-XH’s efﬁcacy on questions that are known to require multi-hop reasoning (Min et al., 2019b), and that the source of this success is due to the eXtra Hop attention mechanism’s ability to combines information from multiple documents.
Revised: Our ablation analysis demonstrates Transformer-XH’s efﬁcacy on questions that are known to require multi-hop reasoning (Min et al., 2019b), and that the source of this success is due to the eXtra Hop attention mechanism’s ability to combines information from multiple documents.

Original:  M ODEL
Revised:  M ODEL

Original: This section ﬁrst discusses preliminaries on sequential Transformers, then we show how we incorporate eXtra hop attention to create Transformer-XH.
Revised: This section ﬁrst discusses preliminaries on sequential Transformers, then we show how we incorporate eXtra hop attention to create Transformer-XH.

Original: 2. P RELIMINARIES
Revised: 2. P RELIMINARIES

Original: Transformers represent a sequence of input text tokens
Revised: Transformers represent a sequence of input text tokens

Original: X = { x 1 , ..., x i , ..., x n } into contextualized distributed representations H = { h 1 , ..., h i , ..., h n } (Vaswani et al., 2017).
Revised: X = { x 1 , ..., x i , ..., x n } into contextualized distributed representations H = { h 1 , ..., h i , ..., h n } (Vaswani et al., 2017).

Original: This process involves multiple stacked self-attention layers that converts the input X into { H 0 , H 1 , ..., H l , ...
Revised: This process involves multiple stacked self-attention layers that converts the input X into { H 0 , H 1 , ..., H l , ...

Original: H L } , starting from H 0 , the embeddings, to the ﬁnal layer of depth L .
Revised: H L } , starting from H 0 , the embeddings, to the ﬁnal layer of depth L .

Original: The key idea of Transformer is its attention mechanism, which calculates the l -th layer output H l using the input H l − 1 from the previous layer:
Revised: The key idea of Transformer is its attention mechanism, which calculates the l -th layer output H l using the input H l − 1 from the previous layer:

Original: [Equation]
Revised: [Equation]

Original: It includes three projections on the input H l − 1 : Query (Q), Key (K), and Value (V).
Revised: It includes three projections on the input H l − 1 : Query (Q), Key (K), and Value (V).

Original: Speciﬁcally, the slices of token h li in Eqn.(2) is:
Revised: Speciﬁcally, the slices of token h li in Eqn.(2) is:

Original: [Equation]
Revised: [Equation]

Original: which ﬁrst calculates its attention to all other tokens j in the sequence and then combines the token values v j into a new representation h li , using the normalized attention weights
Revised: which ﬁrst calculates its attention to all other tokens j in the sequence and then combines the token values v j into a new representation h li , using the normalized attention weights

Original: Multiple attentions can be used in one Transformer layer and concatenated as multi-head attention (Vaswani et al.,
Revised: Multiple attentions can be used in one Transformer layer and concatenated as multi-head attention (Vaswani et al.,

Original: The architecture is stacked to form rather deep networks, which leads to signiﬁcant success of large pre-trained Transformer models (Devlin et al., 2019; Liu et al., 2019a).
Revised: The architecture is stacked to form rather deep networks, which leads to signiﬁcant success of large pre-trained Transformer models (Devlin et al., 2019; Liu et al., 2019a).

Original: A challenge of Transformer is that its attention is calculated over all token pairs (Eqn. 3), which is hard to scale to long text sequences.
Revised: A challenge of Transformer is that its attention is calculated over all token pairs (Eqn. 3), which is hard to scale to long text sequences.

Original: Transformer-XL (eXtra Long) addresses this challenge by breaking down longer texts, e.g., a multi-paragraph document, into a sequence of text segments: { X 1 , ..., X τ , ..., X ζ } , and propagates the information between adjacent text segments using the following attention:
Revised: Transformer-XL (eXtra Long) addresses this challenge by breaking down longer texts, e.g., a multi-paragraph document, into a sequence of text segments: { X 1 , ..., X τ , ..., X ζ } , and propagates the information between adjacent text segments using the following attention:

Original: [Equation]
Revised: [Equation]

Original: It concatenates ( ◦ ) the representation of the previous segment H l − 1 τ − 1 to the current segment as segment level recurrences.
Revised: It concatenates ( ◦ ) the representation of the previous segment H l − 1 τ − 1 to the current segment as segment level recurrences.

Original: The new representation ˜
Revised: The new representation ˜

Original: H l − 1 τ includes the information from previous segment and is integrated in the new attention mechanism:
Revised: H l − 1 τ includes the information from previous segment and is integrated in the new attention mechanism:

Original: [Equation]
Revised: [Equation]

Original: The attention over the previous segment allows Transformer-XL to effectively model long form text data recurrently as a sequence of text chunks (Dai et al., 2019).
Revised: The attention over the previous segment allows Transformer-XL to effectively model long form text data recurrently as a sequence of text chunks (Dai et al., 2019).

Original: Nevertheless, in many scenarios, the text segments are organized in nontrivial structures beyond a linear sequence.
Revised: Nevertheless, in many scenarios, the text segments are organized in nontrivial structures beyond a linear sequence.

Original: For example, documents are connected by hyperlinks in a graphical structure that does not readily simplify to form a linear sequence, prohibiting Transformer-XL’s recurrent approach.
Revised: For example, documents are connected by hyperlinks in a graphical structure that does not readily simplify to form a linear sequence, prohibiting Transformer-XL’s recurrent approach.

Original: 2. T RANSFORMER -XH WITH E X TRA H OP A TTENTION
Revised: 2. T RANSFORMER -XH WITH E X TRA H OP A TTENTION

Original: Transformer-XH models structured text sequence by linking them with eXtra Hop attention following their original structure.
Revised: Transformer-XH models structured text sequence by linking them with eXtra Hop attention following their original structure.

Original: As illustrated in Figure 1a, to model three connected documents d 2 → d 1 → d 3 , Transformer-XH uses eXtra Hop attention to propagate information along the graph edges, enabling information sharing between connected text sequence.
Revised: As illustrated in Figure 1a, to model three connected documents d 2 → d 1 → d 3 , Transformer-XH uses eXtra Hop attention to propagate information along the graph edges, enabling information sharing between connected text sequence.

Original: Formally, the structured text data includes a set of nodes, X =
Revised: Formally, the structured text data includes a set of nodes, X =

Original: { X 1 , ..., X τ , ...
Revised: { X 1 , ..., X τ , ...

Original: X ζ } , each corresponding to a text sequence, and an edge matrix E , which includes the connections (e.g., links) between them.
Revised: X ζ } , each corresponding to a text sequence, and an edge matrix E , which includes the connections (e.g., links) between them.

Original: The goal is to learn representations H = { ˜ H 1 , ..., ˜ H τ , ...
Revised: The goal is to learn representations H = { ˜ H 1 , ..., ˜ H τ , ...

Original: ˜
Revised: ˜

Original: H ζ } , that incorporate not only the local information in each sequence X , but also the global contexts on the entire structured text {X , E } .
Revised: H ζ } , that incorporate not only the local information in each sequence X , but also the global contexts on the entire structured text {X , E } .

Original: Transformer-XH achieves this by two attention mechanisms: in-sequence attention and eXtra Hop attention.
Revised: Transformer-XH achieves this by two attention mechanisms: in-sequence attention and eXtra Hop attention.

Original: The in-sequence attention is the same as vanilla Transformer: in layer l , token i gathers information from other tokens inside the same text piece τ :
Revised: The in-sequence attention is the same as vanilla Transformer: in layer l , token i gathers information from other tokens inside the same text piece τ :

Original: [Equation]
Revised: [Equation]

Original: The eXtra Hop attention uses the ﬁrst token in each sequence – the added special token “[CLS]” – as an “attention hub”, which attends on all other connected nodes’ hub token.
Revised: The eXtra Hop attention uses the ﬁrst token in each sequence – the added special token “[CLS]” – as an “attention hub”, which attends on all other connected nodes’ hub token.

Original: In layer l , the τ -th text sequence attends on other text sequence η if there is an edge between them ( e τη = 1 ):
Revised: In layer l , the τ -th text sequence attends on other text sequence η if there is an edge between them ( e τη = 1 ):

Original: [Equation]
Revised: [Equation]

Original: Node τ calculates the attention weight on its neighbor η using hop query ˆ q τ, 0 and key ˆ k η, 0 .
Revised: Node τ calculates the attention weight on its neighbor η using hop query ˆ q τ, 0 and key ˆ k η, 0 .

Original: Then it uses the weights to combine its neighbors’ value ˆ v η, 0 and forms a globalized representation ˆ h lτ, 0 .
Revised: Then it uses the weights to combine its neighbors’ value ˆ v η, 0 and forms a globalized representation ˆ h lτ, 0 .

Original: The two attention mechanism are combined to form the new representation of layer l :
Revised: The two attention mechanism are combined to form the new representation of layer l :

Original: [Equation]
Revised: [Equation]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Note that the non-hub tokens ( i (cid:54) = 0 ) still have access to the hop attention in the previous layer through Eqn.
Revised: Note that the non-hub tokens ( i (cid:54) = 0 ) still have access to the hop attention in the previous layer through Eqn.

Original: One layer of eXtra Hop attention can be viewed as single-step of information propagation along edges E .
Revised: One layer of eXtra Hop attention can be viewed as single-step of information propagation along edges E .

Original: For example, in Figure 1a, the document node d 3 updates its representation by gathering information from its neighbor d 1 using the hop attention d 1 → d 3 .
Revised: For example, in Figure 1a, the document node d 3 updates its representation by gathering information from its neighbor d 1 using the hop attention d 1 → d 3 .

Original: When multiple TransformerXH layers are stacked, this information in d 1 includes both d 1 ’s local contexts from its in-sequence attention, and cross-sequence information from the hop attention d 2 → d 1 of the l − 1 layer.
Revised: When multiple TransformerXH layers are stacked, this information in d 1 includes both d 1 ’s local contexts from its in-sequence attention, and cross-sequence information from the hop attention d 2 → d 1 of the l − 1 layer.

Original: Hence, an L-layer Transformer-XH can attend over information from up to L hops away.
Revised: Hence, an L-layer Transformer-XH can attend over information from up to L hops away.

Original: Together, three main properties equip Transformer-XH to effectively model raw structured text data: the propagation of information (values) along edges, the importance of that information (hop attention weights), and the balance of in-sequence and cross-sequence information (attention combination).
Revised: Together, three main properties equip Transformer-XH to effectively model raw structured text data: the propagation of information (values) along edges, the importance of that information (hop attention weights), and the balance of in-sequence and cross-sequence information (attention combination).

Original: The representations learned in H can innately express nuances in structured text that are required for complex chains of reasoning in tasks such as multi-hop QA.
Revised: The representations learned in H can innately express nuances in structured text that are required for complex chains of reasoning in tasks such as multi-hop QA.

Original:  A PPLICATION TO M ULTI -H OP Q UESTION A NSWERING
Revised:  A PPLICATION TO M ULTI -H OP Q UESTION A NSWERING

Original: This section describes how Transformer-XH applies to multi-hop QA.
Revised: This section describes how Transformer-XH applies to multi-hop QA.

Original: Given a question q , the task is to ﬁnd an answer span a in a large open-domain document corpus, e.g. the ﬁrst paragraph of all Wikipedia pages.
Revised: Given a question q , the task is to ﬁnd an answer span a in a large open-domain document corpus, e.g. the ﬁrst paragraph of all Wikipedia pages.

Original: By design, the questions are complex and often require information from multiple documents to answer.
Revised: By design, the questions are complex and often require information from multiple documents to answer.

Original: For example, in the case shown in Figure 1b, the correct answer “Cambridge” requires combining the information from both the Wikipedia pages “Facebook” and “Harvard University”.
Revised: For example, in the case shown in Figure 1b, the correct answer “Cambridge” requires combining the information from both the Wikipedia pages “Facebook” and “Harvard University”.

Original: To apply Transformer-XH in the open domain multi-hop QA task, we ﬁrst construct an evidence graph and then apply Transformer-XH on the graph to ﬁnd the answer.
Revised: To apply Transformer-XH in the open domain multi-hop QA task, we ﬁrst construct an evidence graph and then apply Transformer-XH on the graph to ﬁnd the answer.

Original: Evidence Graph Construction.
Revised: Evidence Graph Construction.

Original: The ﬁrst step is to ﬁnd the relevant candidate documents D for the question q and connect them with edges E to form the graph G .
Revised: The ﬁrst step is to ﬁnd the relevant candidate documents D for the question q and connect them with edges E to form the graph G .

Original: Our set D consists of three sources.
Revised: Our set D consists of three sources.

Original: The ﬁrst two sources are from canonical information retrieval and entity linking techniques:
Revised: The ﬁrst two sources are from canonical information retrieval and entity linking techniques:

Original: D ir : the top 100 documents retrieved by DrQA’s TF-IDF on the question (Chen et al., 2017).
Revised: D ir : the top 100 documents retrieved by DrQA’s TF-IDF on the question (Chen et al., 2017).

Original: D el : the Wikipedia documents associated with the entities that appear in the question, annotated by entity linking systems: TagMe (Ferragina & Scaiella, 2010) and CMNS (Hasibi et al., 2017).
Revised: D el : the Wikipedia documents associated with the entities that appear in the question, annotated by entity linking systems: TagMe (Ferragina & Scaiella, 2010) and CMNS (Hasibi et al., 2017).

Original: For better retrieval quality, we use a BERT ranker (Nogueira & Cho, 2019) on the set D ir ∪ D el and keep the top two ranked ones in D ir and top one per question entity in D el .
Revised: For better retrieval quality, we use a BERT ranker (Nogueira & Cho, 2019) on the set D ir ∪ D el and keep the top two ranked ones in D ir and top one per question entity in D el .

Original: Then the third source D exp include all documents connected to or from any top ranked documents via Wikipedia hyperlinks (e.g., “Facebook” → “Harvard University”).
Revised: Then the third source D exp include all documents connected to or from any top ranked documents via Wikipedia hyperlinks (e.g., “Facebook” → “Harvard University”).

Original: The ﬁnal graph comprises all documents from the three sources as nodes X .
Revised: The ﬁnal graph comprises all documents from the three sources as nodes X .

Original: The edge matrix E is ﬂexible.
Revised: The edge matrix E is ﬂexible.

Original: We experiment with various edge matrix settings, including directed edges along Wikipedia links, i.e. e ij = 1 if there is a hyperlink from document i to j , the bidirectional edges along Wiki links, and fully-connected graphs, which rely on Transformer-XH to learns the edge importance.
Revised: We experiment with various edge matrix settings, including directed edges along Wikipedia links, i.e. e ij = 1 if there is a hyperlink from document i to j , the bidirectional edges along Wiki links, and fully-connected graphs, which rely on Transformer-XH to learns the edge importance.

Original: Similar to previous work (Ding et al., 2019), the textual representation for each node in the graph is the [SEP]-delimited concatenation of the question, anchor text (the text in the hyperlink in parent nodes pointing to the child node), and the paragraph itself.
Revised: Similar to previous work (Ding et al., 2019), the textual representation for each node in the graph is the [SEP]-delimited concatenation of the question, anchor text (the text in the hyperlink in parent nodes pointing to the child node), and the paragraph itself.

Original: More details on the evidence graph construction are in Appendix A.1.
Revised: More details on the evidence graph construction are in Appendix A.1.

Original: Transformer-XH on Evidence Graph.
Revised: Transformer-XH on Evidence Graph.

Original: Transformer-XH takes the input nodes X and edges E , and produces the global representation of all text sequences:
Revised: Transformer-XH takes the input nodes X and edges E , and produces the global representation of all text sequences:

Original: [Equation]
Revised: [Equation]

Original: Then we add two task-speciﬁc layers upon the last layer’s representation H L : one auxiliary layer to predict the relevance score of the evidence node, and one layer to extract the answer span within it:
Revised: Then we add two task-speciﬁc layers upon the last layer’s representation H L : one auxiliary layer to predict the relevance score of the evidence node, and one layer to extract the answer span within it:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: The ﬁnal model is trained end-to-end with cross-entropy loss for both tasks in a multi-task setting.
Revised: The ﬁnal model is trained end-to-end with cross-entropy loss for both tasks in a multi-task setting.

Original: During inference, we ﬁrst select the document with the highest relevance score, and then the start and end positions of the answer within that document.
Revised: During inference, we ﬁrst select the document with the highest relevance score, and then the start and end positions of the answer within that document.

Original:  E XPERIMENTAL M ETHODOLOGIES
Revised:  E XPERIMENTAL M ETHODOLOGIES

Original: Dataset.
Revised: Dataset.

Original: We conduct our experiments on HotpotQA, the multi-hop question answering benchmark dataset (Yang et al., 2018).
Revised: We conduct our experiments on HotpotQA, the multi-hop question answering benchmark dataset (Yang et al., 2018).

Original: It includes 112 k crowd-sourced questions designed to require multiple pieces of textual evidence, which are the ﬁrst paragraphs of Wikipedia pages.
Revised: It includes 112 k crowd-sourced questions designed to require multiple pieces of textual evidence, which are the ﬁrst paragraphs of Wikipedia pages.

Original: It has two type of questions: bridge question require hopping via an outside entity, and comparison question compare a property of two entities.
Revised: It has two type of questions: bridge question require hopping via an outside entity, and comparison question compare a property of two entities.

Original: There are two settings in HotpotQA.
Revised: There are two settings in HotpotQA.

Original: The Distractor setting provides golden evidence paragraphs together with TF-IDF retrieved negatives.
Revised: The Distractor setting provides golden evidence paragraphs together with TF-IDF retrieved negatives.

Original: The FullWiki setting requires systems to retrieve evidence paragraphs from the full Wikipedia.
Revised: The FullWiki setting requires systems to retrieve evidence paragraphs from the full Wikipedia.

Original: We focus on FullWiki setting since previous research found that the negative documents in Distractor may be too weak and mitigate the needs of multi-hop reasoning (Min et al., 2019b).
Revised: We focus on FullWiki setting since previous research found that the negative documents in Distractor may be too weak and mitigate the needs of multi-hop reasoning (Min et al., 2019b).

Original: There are 90 k
Revised: There are 90 k

Original: Train, 7 k Dev and 7 k Test questions.
Revised: Train, 7 k Dev and 7 k Test questions.

Original: The ground truth answer and supporting evidence sentences in Train and Dev sets are provided.
Revised: The ground truth answer and supporting evidence sentences in Train and Dev sets are provided.

Original: Test labels are hidden; only one submission is allowed to the leaderboard per 30 days 1 .
Revised: Test labels are hidden; only one submission is allowed to the leaderboard per 30 days 1 .

Original: We evaluate our ﬁnal model on Test and conduct ablations on Dev.
Revised: We evaluate our ﬁnal model on Test and conduct ablations on Dev.

Original: Metrics.
Revised: Metrics.

Original: We use ofﬁcial evaluation metrics of HotpotQA: exact match (EM) and F1 on answer (Ans), supporting facts (Supp), and the combination (Joint).
Revised: We use ofﬁcial evaluation metrics of HotpotQA: exact match (EM) and F1 on answer (Ans), supporting facts (Supp), and the combination (Joint).

Original: The supporting facts prediction is an auxiliary task that evaluates model’s ability to ﬁnd the evidence sentences.
Revised: The supporting facts prediction is an auxiliary task that evaluates model’s ability to ﬁnd the evidence sentences.

Original: Joint EM is the product of the two EM result.
Revised: Joint EM is the product of the two EM result.

Original: Joint F1 ﬁrst multiplies the precision and recall from Ans and Supp, then combines the Joint precision and recall to F1.
Revised: Joint F1 ﬁrst multiplies the precision and recall from Ans and Supp, then combines the Joint precision and recall to F1.

Original: Baseline.
Revised: Baseline.

Original: The main baselines include Cognitive QA (CogQA, Ding et al.
Revised: The main baselines include Cognitive QA (CogQA, Ding et al.

Original: and Semantic
Revised: and Semantic

Original: Retrieval MRS (SR-MRS, Nie et al.
Revised: Retrieval MRS (SR-MRS, Nie et al.

Original: CogQA uses several ﬁne-tuned BERT machine reading comprehension (MRC) models to ﬁnd hop entities and candidate spans, and then uses a BERT based
Revised: CogQA uses several ﬁne-tuned BERT machine reading comprehension (MRC) models to ﬁnd hop entities and candidate spans, and then uses a BERT based

Original: Graph Convolution Network to rank the candidate spans.
Revised: Graph Convolution Network to rank the candidate spans.

Original: SR-MRS is a contemporary work and was the previous leaderboard rank one.
Revised: SR-MRS is a contemporary work and was the previous leaderboard rank one.

Original: It is a BERT based pipeline and uses ﬁne-tuned BERT models to ﬁrst rank the documents (twice), then to rank sentences to ﬁnd supporting facts, and ﬁnally conducts
Revised: It is a BERT based pipeline and uses ﬁne-tuned BERT models to ﬁrst rank the documents (twice), then to rank sentences to ﬁnd supporting facts, and ﬁnally conducts

Original: BERT MRC on the concatenated evidence sentences.
Revised: BERT MRC on the concatenated evidence sentences.

Original: We also re-implement CogQA and upgrade its IR with our BERT IR model (BERT on D ir ∪ D el , same as Transformer-XH), for fair comparisons.
Revised: We also re-implement CogQA and upgrade its IR with our BERT IR model (BERT on D ir ∪ D el , same as Transformer-XH), for fair comparisons.

Original: We include other approaches on the FullWiki setting: Ofﬁcial Baseline (Yang et al., 2018), MUPPET (Feldman & El-Yaniv, 2019), QFE (Nishida et al., 2019), and DecompRC (Min et al., 2019a),
Revised: We include other approaches on the FullWiki setting: Ofﬁcial Baseline (Yang et al., 2018), MUPPET (Feldman & El-Yaniv, 2019), QFE (Nishida et al., 2019), and DecompRC (Min et al., 2019a),

Original: Implementation Details.
Revised: Implementation Details.

Original: The in-sequence attention and other standard Transformer components in Transformer-XH are initialized by the pre-trained BERT base model (Devlin et al., 2019).
Revised: The in-sequence attention and other standard Transformer components in Transformer-XH are initialized by the pre-trained BERT base model (Devlin et al., 2019).

Original: The extra hop attention parameters are initialized randomly and trained from scratch.
Revised: The extra hop attention parameters are initialized randomly and trained from scratch.

Original: The ﬁnal model uses three hop steps.
Revised: The ﬁnal model uses three hop steps.

Original: For bridge questions, we build the evidence graph described in Section 3.
Revised: For bridge questions, we build the evidence graph described in Section 3.

Original: And for comparison questions, we build the fully-connected graph on the set D ir ∪ D el and train
Revised: And for comparison questions, we build the fully-connected graph on the set D ir ∪ D el and train

Original: Transformer-XH
Revised: Transformer-XH

Original: separately.
Revised: separately.

Original: We leave more implementation details in the Appendix.
Revised: We leave more implementation details in the Appendix.

Original:  E VALUATION R ESULTS
Revised:  E VALUATION R ESULTS

Original: This section presents the evaluation results, ablation studies, and analyses on Hotpot QA.
Revised: This section presents the evaluation results, ablation studies, and analyses on Hotpot QA.

Original: 5. O VERALL R ESULT
Revised: 5. O VERALL R ESULT

Original: Table 1 presents HotpotQA FullWiki results.
Revised: Table 1 presents HotpotQA FullWiki results.

Original: Transformer-XH outperforms all previous methods by signiﬁcant margins.
Revised: Transformer-XH outperforms all previous methods by signiﬁcant margins.

Original: The only exception is Supp F1, where SR-MRS performs slightly better, perhaps due to our different ways of Supp extraction.
Revised: The only exception is Supp F1, where SR-MRS performs slightly better, perhaps due to our different ways of Supp extraction.

Original: We only select supporting facts from the inference path to the answer, which ﬁts the task purpose but may hurt the recall (more in Appendix).
Revised: We only select supporting facts from the inference path to the answer, which ﬁts the task purpose but may hurt the recall (more in Appendix).

Original: Besides strong results, Transformer-XH’s ability to natively represent structured data leads to much simpler QA system.
Revised: Besides strong results, Transformer-XH’s ability to natively represent structured data leads to much simpler QA system.

Original: Previously, in order to utilize pre-trained BERT, Hotpot QA approaches adapted the multi-hop reasoning task to comprise multiple sub-tasks.
Revised: Previously, in order to utilize pre-trained BERT, Hotpot QA approaches adapted the multi-hop reasoning task to comprise multiple sub-tasks.

Original: For example, given the retrieved documents, CogQA (w. BERT IR)
Revised: For example, given the retrieved documents, CogQA (w. BERT IR)

Original: ﬁrst leverages one BERT MRC model to ﬁnd hop entities and then another BERT MRC to ﬁnd candidate answer spans.
Revised: ﬁrst leverages one BERT MRC model to ﬁnd hop entities and then another BERT MRC to ﬁnd candidate answer spans.

Original: After that, it ranks the candidate spans
Revised: After that, it ranks the candidate spans

Original: [Table] [Table]
Revised: [Table] [Table]

Original: using a BERT based GAT, which is the only structure modeling step.
Revised: using a BERT based GAT, which is the only structure modeling step.

Original: In comparison, TransformerXH is a uniﬁed model which directly represents structured texts and integrates BERT weights.
Revised: In comparison, TransformerXH is a uniﬁed model which directly represents structured texts and integrates BERT weights.

Original: Table 2 further inspects model performances on the Dev set by question types and reasoning types.
Revised: Table 2 further inspects model performances on the Dev set by question types and reasoning types.

Original: Transformer-XH signiﬁcantly outperforms all baselines on bridge questions which require more multi-hop reasoning.
Revised: Transformer-XH signiﬁcantly outperforms all baselines on bridge questions which require more multi-hop reasoning.

Original: And on the “multi-hop” questions, Transformer-XH has higher relative gains ( 39% over CogQA on EM) than the “single-hop” questions ( 27% ), demonstrating its stronger multihop reasoning capability.
Revised: And on the “multi-hop” questions, Transformer-XH has higher relative gains ( 39% over CogQA on EM) than the “single-hop” questions ( 27% ), demonstrating its stronger multihop reasoning capability.

Original: We further study this in Section 5.3.
Revised: We further study this in Section 5.3.

Original: The gains of SR-MRS on comparison questions are mainly from its better retrieval on Comparison questions. For fair comparison of Transformer-XH’s reasoning ability, we use the top retrieved documents from SR-MRS to construct Transformer-XH’s evidence graph.
Revised: To further investigate the reasoning ability of Transformer-XH, we replace our retrieval pipeline with the top retrieved documents from the SR-MRS pipeline. More speciﬁcally, we use the top retrieved documents from SR-MRS to construct Transformer-XH’s evidence graph while keeping all else constant.

Original: The result system, Transformer-XH (w. SR-MRS), outperforms SR-MRS’s multi-step BERT based reasoning on all metrics and question types.
Revised: The result system, Transformer-XH (w. SR-MRS), outperforms SR-MRS’s multistep BERT based reasoning on all metrics and question types.

Original: Transformer-XH’s effectiveness is robust with multiple IR systems.
Revised: Transformer-XH’s effectiveness is robust with multiple IR systems.

Original: 5. A BLATION S TUDIES
Revised: 5. A BLATION S TUDIES

Original: Model Variations.
Revised: Model Variations.

Original: We show the results of different model variations on the top left of Table 3.
Revised: We show the results of different model variations on the top left of Table 3.

Original: Single-Hop BERT uses BERT MRC model on each document individually, which signiﬁcantly decreases the accuracy, conﬁrming the importance of multi-hop reasoning in FullWiki setting (Min et al., 2019a).
Revised: Single-Hop BERT uses BERT MRC model on each document individually, which signiﬁcantly decreases the accuracy, conﬁrming the importance of multi-hop reasoning in FullWiki setting (Min et al., 2019a).

Original: GAT + BERT ﬁrst uses Graph Attention Network (Veliˇckovi´c et al., 2017) on the evidence graph to predict the best node; then it uses BERT MRC on the best document.
Revised: GAT + BERT ﬁrst uses Graph Attention Network (Veliˇckovi´c et al., 2017) on the evidence graph to predict the best node; then it uses BERT MRC on the best document.

Original: It is 10% worse than Transformer-XH since the MRC model has no access to the information from other documents.
Revised: It is 10% worse than Transformer-XH since the MRC model has no access to the information from other documents.

Original: No Node Prediction eliminates the node prediction task and only trains on span prediction task; the accuracy difference shows node prediction task helps the model training.
Revised: No Node Prediction eliminates the node prediction task and only trains on span prediction task; the accuracy difference shows node prediction task helps the model training.

Original: Graph Structures.
Revised: Graph Structures.

Original: We show Transformer-XH’s performance with different graph structures on the bottom left of Table 3.
Revised: We show Transformer-XH’s performance with different graph structures on the bottom left of Table 3.

Original: Bidirectional Edges adds reverse edges along the hyperlinks; Fully
Revised: Bidirectional Edges adds reverse edges along the hyperlinks; Fully

Original: Connected Graph connects all document pairs; Node Sequence randomly permutes the documents and connects them into a sequence to simulate the Transformer-XL setting.
Revised: Connected Graph connects all document pairs; Node Sequence randomly permutes the documents and connects them into a sequence to simulate the Transformer-XL setting.

Original: Both Bidirectional
Revised: Both Bidirectional

Original: Links and Fully Connected Graph have comparable performance with the original graph structure.
Revised: Links and Fully Connected Graph have comparable performance with the original graph structure.

Original: Transformer-XH is able to learn meaningful connections using its hop attentions and is less dependent on the pre-existing graph structural. The fully connected graph can be used if there is no strong
Revised: Transformer-XH is able to learn meaningful connections using its hop attentions and is less depen-

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original:  edge patterns available in the task.
Revised: dent on the pre-existing graph structural. The fully connected graph can be used if there is no strong edge patterns available in the task.

Original: However, the performance drops signiﬁcantly on Node Sequence , showing that structured texts cannot be treated as a linear sequence which cuts off many connections.
Revised: However, the performance drops signiﬁcantly on Node Sequence , showing that structured texts cannot be treated as a linear sequence which cuts off many connections.

Original: Hop Steps.
Revised: Hop Steps.

Original: Recall that a Transformer-XH layer with extra hop attention corresponds to one information propagation (hop) step in the graph.
Revised: Recall that a Transformer-XH layer with extra hop attention corresponds to one information propagation (hop) step in the graph.

Original: Thus Transformer-XH with last K layers conducts
Revised: Thus Transformer-XH with last K layers conducts

Original: K-step attention hops in the graph.
Revised: K-step attention hops in the graph.

Original: We show results with different K on the right side of Table 3.
Revised: We show results with different K on the right side of Table 3.

Original: Transformer-XH reaches its peak performance with three hops (our full-model).
Revised: Transformer-XH reaches its peak performance with three hops (our full-model).

Original: This is expected as most Hotpot QA questions can be answered by two documents (Yang et al., 2018).
Revised: This is expected as most Hotpot QA questions can be answered by two documents (Yang et al., 2018).

Original: 5. H OP A TTENTION A NALYSIS
Revised: 5. H OP A TTENTION A NALYSIS

Original: This experiment analyzes the hop attentions using our full-model (three-hop) on the fully connected graph to study their behavior without pre-deﬁned structure.
Revised: This experiment analyzes the hop attentions using our full-model (three-hop) on the fully connected graph to study their behavior without pre-deﬁned structure.

Original: Figure 2 plots the distributions of the learned hop attentions on the Dev set.
Revised: Figure 2 plots the distributions of the learned hop attentions on the Dev set.

Original: It shows a strong shift away from the normal distribution with more hops.
Revised: It shows a strong shift away from the normal distribution with more hops.

Original: Transformer-XH learns to distinguish different nodes after multi-hop attention: the attention score becomes a bimodal distribution after three hops, ignoring some non-useful nodes.
Revised: Transformer-XH learns to distinguish different nodes after multi-hop attention: the attention score becomes a bimodal distribution after three hops, ignoring some non-useful nodes.

Original: Transformer-XH also learns to focus on meaningful edges: the score is higher on the path
Revised: Transformer-XH also learns to focus on meaningful edges: the score is higher on the path

Original: Supp → Ans than All → Ans.
Revised: Supp → Ans than All → Ans.

Original: And the margin is larger as the hop step increases from one to three.
Revised: And the margin is larger as the hop step increases from one to three.

Original: 5. C ASE S TUDY
Revised: 5. C ASE S TUDY

Original: Table 4 lists two examples from Transformer-XH and CogQA (w. BERT IR).
Revised: Table 4 lists two examples from Transformer-XH and CogQA (w. BERT IR).

Original: The ﬁrst case has a clear evidence chain “2011/S/S” → “Winner” → “YG Entertainment”; both methods ﬁnd the correct answer.
Revised: The ﬁrst case has a clear evidence chain “2011/S/S” → “Winner” → “YG Entertainment”; both methods ﬁnd the correct answer.

Original: However, the second case has too many distractors in the ﬁrst document.
Revised: However, the second case has too many distractors in the ﬁrst document.

Original: Without additional clues from document 2, it is likely that the single-hop hop entity extraction component in CogQA (w. BERT IR) misses the correct answer document in its candidate sets; and the later structural reasoning component can not recover from this cascade error.
Revised: Without additional clues from document 2, it is likely that the single-hop hop entity extraction component in CogQA (w. BERT IR) misses the correct answer document in its candidate sets; and the later structural reasoning component can not recover from this cascade error.

Original: In comparison, Transformer-XH ﬁnds the correct answer by combining the evidence with the hop attentions between the two evidence pieces.
Revised: In comparison, Transformer-XH ﬁnds the correct answer by combining the evidence with the hop attentions between the two evidence pieces.

Original: We leave more positive and negative cases in Appendix A.5.
Revised: We leave more positive and negative cases in Appendix A.5.

Original: [Table]
Revised: [Table]

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Machine reading comprehension is an important task of natural language processing (Rajpurkar et al., 2016; Bajaj et al., 2016).
Revised: Machine reading comprehension is an important task of natural language processing (Rajpurkar et al., 2016; Bajaj et al., 2016).

Original: The initial MRC questions are mostly single-hop, which only require single evidence sentence to answer (Min et al., 2018).
Revised: The initial MRC questions are mostly single-hop, which only require single evidence sentence to answer (Min et al., 2018).

Original: One direction to make the task more realistic is open-domain question answering (QA), in which the evidence paragraph has to be retrieved from a large corpus (Chen et al., 2017).
Revised: One direction to make the task more realistic is open-domain question answering (QA), in which the evidence paragraph has to be retrieved from a large corpus (Chen et al., 2017).

Original: The other direction is multi-hop QA, where the questions require multiple pieces of evidence to answer (Welbl et al., 2018; Yang et al., 2018).
Revised: The other direction is multi-hop QA, where the questions require multiple pieces of evidence to answer (Welbl et al., 2018; Yang et al., 2018).

Original: HotpotQA’s FullWiki setting is a combination of open-domain QA and multi-hop QA (Yang et al., 2018): the questions are designed to require multiple pieces of evidence and these evidence pieces are documents to retrieve from the entire Wikipedia.
Revised: HotpotQA’s FullWiki setting is a combination of open-domain QA and multi-hop QA (Yang et al., 2018): the questions are designed to require multiple pieces of evidence and these evidence pieces are documents to retrieve from the entire Wikipedia.

Original: The combination makes this setting challenging: The retrieved documents are inevitably noisy and include much stronger distractors than the
Revised: The combination makes this setting challenging: The retrieved documents are inevitably noisy and include much stronger distractors than the

Original: TF-IDF retrieved documents in the Distractor setting (Min et al., 2019a; Jiang & Bansal, 2019).
Revised: TF-IDF retrieved documents in the Distractor setting (Min et al., 2019a; Jiang & Bansal, 2019).

Original: Various solutions have been proposed for Hotpot QA (Min et al., 2019b; Feldman & El-Yaniv, 2019;
Revised: Various solutions have been proposed for Hotpot QA (Min et al., 2019b; Feldman & El-Yaniv, 2019;

Original: Nishida et al., 2019).
Revised: Nishida et al., 2019).

Original: These solutions often use complicated pipelines to adapt the multi-hop task into a combination of single-hop tasks, in order to leverage the advantage of pre-trained models.
Revised: These solutions often use complicated pipelines to adapt the multi-hop task into a combination of single-hop tasks, in order to leverage the advantage of pre-trained models.

Original: For example, CogQA (Ding et al., 2019) uses two BERT based MRC model to ﬁnd candidate spans and then another BERT initialized Graph Neural Network (GNN) to rank spans;
Revised: For example, CogQA (Ding et al., 2019) uses two BERT based MRC model to ﬁnd candidate spans and then another BERT initialized Graph Neural Network (GNN) to rank spans;

Original: SR-MRS (Nie et al., 2019) uses three BERT based rankers to ﬁnd supporting sentences, and then another BERT MRC model on the concatenated sentences to get the answer span.
Revised: SR-MRS (Nie et al., 2019) uses three BERT based rankers to ﬁnd supporting sentences, and then another BERT MRC model on the concatenated sentences to get the answer span.

Original: Transformer-XH is a simpler model that directly represents and reasons with multiple pieces of evidence using extra hop attentions.
Revised: Transformer-XH is a simpler model that directly represents and reasons with multiple pieces of evidence using extra hop attentions.

Original: In addition to Transformer-XL (Dai et al., 2019), Transformer-XH is also inspired by GNN (Kipf &
Revised: In addition to Transformer-XL (Dai et al., 2019), Transformer-XH is also inspired by GNN (Kipf &

Original: Welling, 2017; Schlichtkrull et al., 2017;
Revised: Welling, 2017; Schlichtkrull et al., 2017;

Original: Veli ˇ ckovi ´ c et al., 2017), which leverages neural networks to model graph structured data.
Revised: Veli ˇ ckovi ´ c et al., 2017), which leverages neural networks to model graph structured data.

Original: The key difference is that a “node” in Transformer-XH is a text sequence, and modeling of the structure is conducted jointly with the representation of the text.
Revised: The key difference is that a “node” in Transformer-XH is a text sequence, and modeling of the structure is conducted jointly with the representation of the text.

Original: Transformer-XH combines the advantages Transformer has in understanding text with the power that GNN has in modeling structure.
Revised: Transformer-XH combines the advantages Transformer has in understanding text with the power that GNN has in modeling structure.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: Transformer-XH and its eXtra Hop attention mechanism is a simple yet powerful adaptation of
Revised: Transformer-XH and its eXtra Hop attention mechanism is a simple yet powerful adaptation of

Original: Transformer to learn better representations of structured text data as it naturally occurs.
Revised: Transformer to learn better representations of structured text data as it naturally occurs.

Original: It innately integrates with pre-trained language models to allow for complex reasoning across many hops of a textual evidence graph, where clues to the correct answer of a question are split across multiple documents.
Revised: It innately integrates with pre-trained language models to allow for complex reasoning across many hops of a textual evidence graph, where clues to the correct answer of a question are split across multiple documents.

Original: When applied to HotpotQA, Transformer-XH signiﬁcantly shrinks the typical multi-hop
Revised: When applied to HotpotQA, Transformer-XH signiﬁcantly shrinks the typical multi-hop

Original: QA pipeline, eliminating many cascading errors that arise from the linear sequence input constraints of pre-trained Transformers.
Revised: QA pipeline, eliminating many cascading errors that arise from the linear sequence input constraints of pre-trained Transformers.

Original: Compared to previous SOTA baselines that use pipelines of BERT models to mimic multi-hop reasoning, one Transformer-XH model is all we need to obtain a much stronger answer accuracy.
Revised: Compared to previous SOTA baselines that use pipelines of BERT models to mimic multi-hop reasoning, one Transformer-XH model is all we need to obtain a much stronger answer accuracy.

Original: With its simplicity and efﬁcacy, we envision Transformer-XH will beneﬁt many applications in the near future.
Revised: With its simplicity and efﬁcacy, we envision Transformer-XH will beneﬁt many applications in the near future.

Original: A A PPENDIX
Revised: A A PPENDIX

Original: The appendix includes details of the evidence graph construction for Hotpot QA, ablation studies in the BERT IR component, more details and results on Hotpot QA, and results on the FEVER fact veriﬁcation task.
Revised: The appendix includes details of the evidence graph construction for Hotpot QA, ablation studies in the BERT IR component, more details and results on Hotpot QA, and results on the FEVER fact veriﬁcation task.

Original: A. H OTPOT QA E VIDENCE G RAPH C ONSTRUCTION D ETAILS
Revised: A. H OTPOT QA E VIDENCE G RAPH C ONSTRUCTION D ETAILS

Original: The evidence graph construction includes two stages.
Revised: The evidence graph construction includes two stages.

Original: The ﬁrst stage is BERT IR, which extract related documents directly from question.
Revised: The ﬁrst stage is BERT IR, which extract related documents directly from question.

Original: The second stage expands the related documents along
Revised: The second stage expands the related documents along

Original: Wikipedia links.
Revised: Wikipedia links.

Original: The ﬁrst is applied on all questions while the second is only required by Bridge questions.
Revised: The ﬁrst is applied on all questions while the second is only required by Bridge questions.

Original: The ﬁrst stage uses two methods to ﬁnd documents.
Revised: The ﬁrst stage uses two methods to ﬁnd documents.

Original: The ﬁrst method uses DrQA’s retrieval system (Chen et al., 2017), which is unsupervised TF-IDF.
Revised: The ﬁrst method uses DrQA’s retrieval system (Chen et al., 2017), which is unsupervised TF-IDF.

Original: We keep the top 100 DrQA retrieved ones D ir for each question.
Revised: We keep the top 100 DrQA retrieved ones D ir for each question.

Original: The second method uses TagMe (Ferragina & Scaiella, 2010) and
Revised: The second method uses TagMe (Ferragina & Scaiella, 2010) and

Original: CMNS (Hasibi et al., 2017), two commonly used entity linkers, to annotate questions.
Revised: CMNS (Hasibi et al., 2017), two commonly used entity linkers, to annotate questions.

Original: We keep the TagMe output entity and three highest scored entities per surface form (a phrase in the question linked with entities) from CMNS, and use its corresponding Wikipedia document as D el .
Revised: We keep the TagMe output entity and three highest scored entities per surface form (a phrase in the question linked with entities) from CMNS, and use its corresponding Wikipedia document as D el .

Original: We use BERT ranker (Nogueira & Cho, 2019) to re-rank the initial set D ir ∪ D el .
Revised: We use BERT ranker (Nogueira & Cho, 2019) to re-rank the initial set D ir ∪ D el .

Original: The input to the
Revised: The input to the

Original: BERT is the concatenation of question and ﬁrst paragraph of document:
Revised: BERT is the concatenation of question and ﬁrst paragraph of document:

Original: [Equation]
Revised: [Equation]

Original: Then a linear layer is added on the last layer’s
Revised: Then a linear layer is added on the last layer’s

Original: [CLS] representation to score the relevance of the document.
Revised: [CLS] representation to score the relevance of the document.

Original: We use BERT base and ﬁne-tune it using the relevance label (from supporting facts) with cross-entropy loss.
Revised: We use BERT base and ﬁne-tune it using the relevance label (from supporting facts) with cross-entropy loss.

Original: The top two highest scored documents from D ir and the top one document per entity position (surface form) in D el are kept as the ﬁrst stage BERT IR documents.
Revised: The top two highest scored documents from D ir and the top one document per entity position (surface form) in D el are kept as the ﬁrst stage BERT IR documents.

Original: The second stage expands the ﬁrst stage BERT IR documents by Wikipedia hyperlinks to obtain
Revised: The second stage expands the ﬁrst stage BERT IR documents by Wikipedia hyperlinks to obtain

Original: D exp .
Revised: D exp .

Original: A document is included if it is linked to or links to a document in the ﬁrst stage.
Revised: A document is included if it is linked to or links to a document in the ﬁrst stage.

Original: We use the same BERT ranker to rank D exp and keep the top 15 documents in D exp .
Revised: We use the same BERT ranker to rank D exp and keep the top 15 documents in D exp .

Original: The ﬁnal evidence graph nodes per question includes the top two highest ranked documents in D ir , top one per entity name in D el , and top 15 from the expanded documents D exp .
Revised: The ﬁnal evidence graph nodes per question includes the top two highest ranked documents in D ir , top one per entity name in D el , and top 15 from the expanded documents D exp .

Original: The comparison questions only require information from two question entities; thus when building the evidence graph we do not expand them (i.e. there is no D exp ).
Revised: The comparison questions only require information from two question entities; thus when building the evidence graph we do not expand them (i.e. there is no D exp ).

Original: The retrieval pipeline is a multi-stage retrieval enhanced with entity linking.
Revised: The retrieval pipeline is a multi-stage retrieval enhanced with entity linking.

Original: It is close to the retrieval system used in SR-MRS (Nie et al., 2019).
Revised: It is close to the retrieval system used in SR-MRS (Nie et al., 2019).

Original: When using SR-MRS retrieved documents for documents, we use top 10 documents on bridge questions and top two documents on comparison questions.
Revised: When using SR-MRS retrieved documents for documents, we use top 10 documents on bridge questions and top two documents on comparison questions.

Original: In the next section, we show that Transformer-XH is robust to different number of documents kept in the evidence graph and performs similarly using the documents retrieved from SR-MRS.
Revised: In the next section, we show that Transformer-XH is robust to different number of documents kept in the evidence graph and performs similarly using the documents retrieved from SR-MRS.

Original: A. A BLATION S TUDIES O N D OCUMENT R ETRIEVAL
Revised: A. A BLATION S TUDIES O N D OCUMENT R ETRIEVAL

Original: This experiment studies the effectiveness and inﬂuence of different retrieval settings.
Revised: This experiment studies the effectiveness and inﬂuence of different retrieval settings.

Original: We use different numbers of top K ranked documents from the BERT ranker, run Transformer-XH in the corresponding evidence graph, and evaluate its performance on Bridge questions in the Dev set.
Revised: We use different numbers of top K ranked documents from the BERT ranker, run Transformer-XH in the corresponding evidence graph, and evaluate its performance on Bridge questions in the Dev set.

Original: We also evaluate the Supporting facts Recall and Answer Recall.
Revised: We also evaluate the Supporting facts Recall and Answer Recall.

Original: Supp Recall evaluates whether the document with the supporting fact is included in the ﬁrst stage retrieved documents.
Revised: Supp Recall evaluates whether the document with the supporting fact is included in the ﬁrst stage retrieved documents.

Original: Ans Recall evaluates whether their exists a document in the evidence graph that includes the ground truth answer.
Revised: Ans Recall evaluates whether their exists a document in the evidence graph that includes the ground truth answer.

Original: The results are in Table 5.
Revised: The results are in Table 5.

Original: Our BERT IR system performs better than CogQA’s TF-IDF and on par with SR-MRS, as expected.
Revised: Our BERT IR system performs better than CogQA’s TF-IDF and on par with SR-MRS, as expected.

Original: The latter uses a similar retrieval pipeline with our BERT IR system; Transformer-XH is robust on different retrieval settings and keeps its effectiveness when applied on Top 10 documents from
Revised: The latter uses a similar retrieval pipeline with our BERT IR system; Transformer-XH is robust on different retrieval settings and keeps its effectiveness when applied on Top 10 documents from

Original: SR-MRS (including both stages).
Revised: SR-MRS (including both stages).

Original: [Table]
Revised: [Table]

Original: A. O THER H OTPOT QA COMPONENTS
Revised: A. O THER H OTPOT QA COMPONENTS

Original: This section describes the other components for HotpotQA dataset.
Revised: This section describes the other components for HotpotQA dataset.

Original: The whole QA system starts with question type classiﬁcation.
Revised: The whole QA system starts with question type classiﬁcation.

Original: We train transformer-XH separately on each type of questions over their evidence graph.
Revised: We train transformer-XH separately on each type of questions over their evidence graph.

Original: Besides answer prediction, we also adopt BERT based model for predicting supporting sentences.
Revised: Besides answer prediction, we also adopt BERT based model for predicting supporting sentences.

Original: A.3. QUESTION CLASSIFICATION
Revised: A.3. QUESTION CLASSIFICATION

Original: The ﬁrst component of our system is to classify the question to bridge and comparison types.
Revised: The ﬁrst component of our system is to classify the question to bridge and comparison types.

Original: We adopt BERT classiﬁcation ﬁne-tuning setting on HotpotQA questions using the question type labels provided in HotpotQA.
Revised: We adopt BERT classiﬁcation ﬁne-tuning setting on HotpotQA questions using the question type labels provided in HotpotQA.

Original: The classiﬁer achieves 99.1 % accuracy on the dev set.
Revised: The classiﬁer achieves 99.1 % accuracy on the dev set.

Original: We use the classiﬁer to split the questions into Comparison and Bridge.
Revised: We use the classiﬁer to split the questions into Comparison and Bridge.

Original: A.3. S UPPORTING FACTS CLASSIFICATION
Revised: A.3. S UPPORTING FACTS CLASSIFICATION

Original: The supporting facts prediction task is to extract all sentences that help get the answer.
Revised: The supporting facts prediction task is to extract all sentences that help get the answer.

Original: For bridge question, these sentences usually cover different pieces of questions.
Revised: For bridge question, these sentences usually cover different pieces of questions.

Original: And for comparison questions, the supporting facts are the properties of two question entities.
Revised: And for comparison questions, the supporting facts are the properties of two question entities.

Original: We design one model architecture for this task, but we train two models on each type to reﬂect the inherent difference.
Revised: We design one model architecture for this task, but we train two models on each type to reﬂect the inherent difference.

Original: We use BERT as our base model and on top of BERT, we conduct multi-task learning scheme.
Revised: We use BERT as our base model and on top of BERT, we conduct multi-task learning scheme.

Original: The ﬁrst task is document relevance prediction, similar as Transformer-XH, we add a linear layer on the [CLS] token of BERT to predict the relevance score.
Revised: The ﬁrst task is document relevance prediction, similar as Transformer-XH, we add a linear layer on the [CLS] token of BERT to predict the relevance score.

Original: The other task is sentence binary classiﬁcation, we concatenate the ﬁrst and last token representation of each sentence in the document through a linear layer, the binary output decides whether this sentence is supporting sentence.
Revised: The other task is sentence binary classiﬁcation, we concatenate the ﬁrst and last token representation of each sentence in the document through a linear layer, the binary output decides whether this sentence is supporting sentence.

Original: Bridge question supporting facts prediction
Revised: Bridge question supporting facts prediction

Original: For bridge questions, we predict supporting facts after answer prediction from Transformer-XH to resume the inference chain.
Revised: For bridge questions, we predict supporting facts after answer prediction from Transformer-XH to resume the inference chain.

Original: We start by predicting supporting facts in the answer document.
Revised: We start by predicting supporting facts in the answer document.

Original: The other document is chosen from the parents of the answer document in the evidence graph.
Revised: The other document is chosen from the parents of the answer document in the evidence graph.

Original: Compare with the contemporary model Nie et al.
Revised: Compare with the contemporary model Nie et al.

Original: (2019), which does not limit the search space along the inference chain (i.e., the answer document may not be relevant to the other supporting page), our method more naturally ﬁts the task purpose.
Revised: (2019), which does not limit the search space along the inference chain (i.e., the answer document may not be relevant to the other supporting page), our method more naturally ﬁts the task purpose.

Original: Comparison question supporting facts prediction
Revised: Comparison question supporting facts prediction

Original: For comparison questions, after extracting the ﬁrst step documents D , we simply run this supporting facts prediction model to select the topdocuments, and predict the corresponding supporting facts.
Revised: For comparison questions, after extracting the ﬁrst step documents D , we simply run this supporting facts prediction model to select the topdocuments, and predict the corresponding supporting facts.

Original: A.3. T RAINING DETAILS
Revised: A.3. T RAINING DETAILS

Original: We use DGL (Wang et al., 2019) for implementing Transformer-XH and CogQA (w. BERT IR) with batch size 1 (i.e., one graph for each batch), and keep the other parameters same as default BERT setting.
Revised: We use DGL (Wang et al., 2019) for implementing Transformer-XH and CogQA (w. BERT IR) with batch size 1 (i.e., one graph for each batch), and keep the other parameters same as default BERT setting.

Original: We train Transformer-XH separately on two different types of questions, following previous research (Ding et al., 2019).
Revised: We train Transformer-XH separately on two different types of questions, following previous research (Ding et al., 2019).

Original: We train Transformer-XH and the GNN of CogQA (w. BERT IR) forepochs.
Revised: We train Transformer-XH and the GNN of CogQA (w. BERT IR) forepochs.

Original: All other BERT based models use the default BERT parameters and train the model forepoch.
Revised: All other BERT based models use the default BERT parameters and train the model forepoch.

Original: A. I MPLEMENTATION D ETAILS OF C OG QA (W. BERT IR)
Revised: A. I MPLEMENTATION D ETAILS OF C OG QA (W. BERT IR)

Original: This section discusses our implementation of CogQA (W. BERT IR).
Revised: This section discusses our implementation of CogQA (W. BERT IR).

Original: We start with the same documents from BERT IR, the one used by Transformer-XH, and then implement the following steps:
Revised: We start with the same documents from BERT IR, the one used by Transformer-XH, and then implement the following steps:

Original: A.4. H OP ENTITY EXTRACTION
Revised: A.4. H OP ENTITY EXTRACTION

Original: For each document from the previous step, we run BERT MRC model and limit the span candidates as hyperlinked entities for hop entity extraction (e.g., in Figure 1, “Harvard University” is a hop entity).
Revised: For each document from the previous step, we run BERT MRC model and limit the span candidates as hyperlinked entities for hop entity extraction (e.g., in Figure 1, “Harvard University” is a hop entity).

Original: Following Ding et al.
Revised: Following Ding et al.

Original: (2019), we predict the top three entities that above the relative threshold that is the start span probability of [CLS] position.
Revised: (2019), we predict the top three entities that above the relative threshold that is the start span probability of [CLS] position.

Original: A.4. A NSWER SPAN E XTRACTION
Revised: A.4. A NSWER SPAN E XTRACTION

Original: For each document (add the hop entity document), following Ding et al.
Revised: For each document (add the hop entity document), following Ding et al.

Original: (2019), we run BERT MRC model to extract spans (e.g., “Combridge” in Figure 1.).
Revised: (2019), we run BERT MRC model to extract spans (e.g., “Combridge” in Figure 1.).

Original: We predict the top one span that above the threshold that is the start span probability of [CLS] position.
Revised: We predict the top one span that above the threshold that is the start span probability of [CLS] position.

Original: We train both hop entity extraction and span extraction tasks with same BERT model but different prediction layers.
Revised: We train both hop entity extraction and span extraction tasks with same BERT model but different prediction layers.

Original: For each training example, we extract the link between two given supporting pages.
Revised: For each training example, we extract the link between two given supporting pages.

Original: The page includes the link (e.g., “Harvard University” in Figure 1.) is the supporting page for hop entity extraction, while the other page is the answer page (e.g., “Combridge” in Figure 1.)
Revised: The page includes the link (e.g., “Harvard University” in Figure 1.) is the supporting page for hop entity extraction, while the other page is the answer page (e.g., “Combridge” in Figure 1.)

Original: for answer span extraction.
Revised: for answer span extraction.

Original: A.4. GAT MODELING
Revised: A.4. GAT MODELING

Original: All the entities and answer spans form the ﬁnal graph.
Revised: All the entities and answer spans form the ﬁnal graph.

Original: The nodes are the entities and spans, and edges are the connections from the entities to the extracted hop entities or spans.
Revised: The nodes are the entities and spans, and edges are the connections from the entities to the extracted hop entities or spans.

Original: We use BERT for each node representation with question, anchor sentences and context, following Ding et al.
Revised: We use BERT for each node representation with question, anchor sentences and context, following Ding et al.

Original: We run GAT (Veliˇckovi´c et al., 2017) on top of BERT to predict the correct answer span node
Revised: We run GAT (Veliˇckovi´c et al., 2017) on top of BERT to predict the correct answer span node

Original: . A.4. C OMPARISON Q UESTIONS
Revised: . A.4. C OMPARISON Q UESTIONS

Original: After predicting supporting facts, we concatenate the sentences and follow Min et al.
Revised: After predicting supporting facts, we concatenate the sentences and follow Min et al.

Original: (2019b) to run a BERT MRC model to predict either span or yes/
Revised: (2019b) to run a BERT MRC model to predict either span or yes/

Original: no as the answer.
Revised: no as the answer.

Original: A. A DDITIONAL CASE STUDY
Revised: A. A DDITIONAL CASE STUDY

Original: We provide addition case studies in Table 6.
Revised: We provide addition case studies in Table 6.

Original: A. E VALUATION ON FEVER F ACT E XTRACTION AND V ERIFICATION T ASK
Revised: A. E VALUATION ON FEVER F ACT E XTRACTION AND V ERIFICATION T ASK

Original: This section evaluates Transformer-XH in the FEVER Fact Extraction and Veriﬁcation Task (Thorne et al., 2018), the other task SR-MRS was evaluated on (Nie et al., 2019)
Revised: This section evaluates Transformer-XH in the FEVER Fact Extraction and Veriﬁcation Task (Thorne et al., 2018), the other task SR-MRS was evaluated on (Nie et al., 2019)

Original: . A.6. FEVER T ASK
Revised: . A.6. FEVER T ASK

Original: Dataset and Metrics.
Revised: Dataset and Metrics.

Original: The FEVER task provides a claim sentence and requires the system to classify it into three categories: SUPPORTS, REFUTES, and NOT ENOUGH INFO, using the Wikipedia corpus as the evidence source.
Revised: The FEVER task provides a claim sentence and requires the system to classify it into three categories: SUPPORTS, REFUTES, and NOT ENOUGH INFO, using the Wikipedia corpus as the evidence source.

Original: It provides 185,455 claims with manual labels and uses the Wikipedia dump in June 2017 which includes 5.4 million documents.
Revised: It provides 185,455 claims with manual labels and uses the Wikipedia dump in June 2017 which includes 5.4 million documents.

Original: There are two ofﬁcial evaluation metrics in FEVER: Label Accuracy (LA), which evaluates the classiﬁcation accuracy of the veriﬁcation labels, and FEVER Score, which evaluates both the correctness
Revised: There are two ofﬁcial evaluation metrics in FEVER: Label Accuracy (LA), which evaluates the classiﬁcation accuracy of the veriﬁcation labels, and FEVER Score, which evaluates both the correctness

Original: [Table]
Revised: [Table]

Original: of the evidence sentences used in veriﬁcation and the LA.
Revised: of the evidence sentences used in veriﬁcation and the LA.

Original: The latter is close to Joint EM in Hotpot
Revised: The latter is close to Joint EM in Hotpot

Original: QA and is the main metric.
Revised: QA and is the main metric.

Original: We use the ofﬁcial evaluation scripts 3 from FEVER task and we refer to
Revised: We use the ofﬁcial evaluation scripts 3 from FEVER task and we refer to

Original: Thorne et al.
Revised: Thorne et al.

Original: (2018) for more details of this task.
Revised: (2018) for more details of this task.

Original: Evidence Graph Construction.
Revised: Evidence Graph Construction.

Original: Most top performance FEVER systems on the leaderboard ﬁrst retrieve the evidence sentences for the claim and then reason upon the evidence sentences to verify the claim (Nie et al., 2019; Zhou et al., 2019).
Revised: Most top performance FEVER systems on the leaderboard ﬁrst retrieve the evidence sentences for the claim and then reason upon the evidence sentences to verify the claim (Nie et al., 2019; Zhou et al., 2019).

Original: The ﬁrst step is similar as the retrieval stage in Hotpot
Revised: The ﬁrst step is similar as the retrieval stage in Hotpot

Original: QA.
Revised: QA.

Original: And the second step is a multi-evidence reasoning task, where Transformer-XH is applied.
Revised: And the second step is a multi-evidence reasoning task, where Transformer-XH is applied.

Original: We keep the evidence sentence retrieval step consistent with previous methods.
Revised: We keep the evidence sentence retrieval step consistent with previous methods.

Original: The sentence retrieval results of SR-MRS is not yet released at the time of our experiments, thus we instead use the BERT-based retrieval results from another contemporary work (Liu et al., 2019b) in the FEVER leaderboard 4 .
Revised: The sentence retrieval results of SR-MRS is not yet released at the time of our experiments, thus we instead use the BERT-based retrieval results from another contemporary work (Liu et al., 2019b) in the FEVER leaderboard 4 .

Original: We construct the evidence graph using the top ﬁve sentences from Liu et al.
Revised: We construct the evidence graph using the top ﬁve sentences from Liu et al.

Original: (2019b)
Revised: (2019b)

Original: as the nodes X and fully connected edges E .
Revised: as the nodes X and fully connected edges E .

Original: Following Liu et al.
Revised: Following Liu et al.

Original: (2019b), the representation of each node is the concatenation of the claim, the Wiki title (entity name) of the document which includes the sentence, and the evidence sentence.
Revised: (2019b), the representation of each node is the concatenation of the claim, the Wiki title (entity name) of the document which includes the sentence, and the evidence sentence.

Original: A.6. T RANSFORMER -XH ON FEVER
Revised: A.6. T RANSFORMER -XH ON FEVER

Original: Transformer-XH takes the evidence graph { X, E } and learns to verify the claim to three categories: y ∈ { SUPPORT, REFUSE, and NOT ENOUGH EVIDENCE } .
Revised: Transformer-XH takes the evidence graph { X, E } and learns to verify the claim to three categories: y ∈ { SUPPORT, REFUSE, and NOT ENOUGH EVIDENCE } .

Original: Similar to the application in Hotpot 3 https://github.com/sheffieldnlp/fever-scorerhttps://competitions.codalab.org/competitions/18814#results
Revised: Similar to the application in Hotpot 3 https://github.com/sheffieldnlp/fever-scorerhttps://competitions.codalab.org/competitions/18814#results

Original: [Table]
Revised: [Table]

Original: QA, it ﬁrst produces the global representation of the graph:
Revised: QA, it ﬁrst produces the global representation of the graph:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Then two task-speciﬁc layers are added upon the last layer.
Revised: Then two task-speciﬁc layers are added upon the last layer.

Original: The ﬁrst layer conducts the fact prediction per node using the “[CLS]” token:
Revised: The ﬁrst layer conducts the fact prediction per node using the “[CLS]” token:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: The second layer learns to measure the importance of each node in the graph:
Revised: The second layer learns to measure the importance of each node in the graph:

Original: [Equation]
Revised: [Equation]

Original: The node level predictions and node importance are combined to the ﬁnal prediction for the claim:
Revised: The node level predictions and node importance are combined to the ﬁnal prediction for the claim:

Original: [Equation]
Revised: [Equation]

Original: Similar as in Hotpot QA, we use multi-task learning that combines the node prediction task and the claim veriﬁcation task.
Revised: Similar as in Hotpot QA, we use multi-task learning that combines the node prediction task and the claim veriﬁcation task.

Original: The ﬁrst uses the evidence sentence label provided by FEVER and crossentropy loss on Eqn. 15.
Revised: The ﬁrst uses the evidence sentence label provided by FEVER and crossentropy loss on Eqn. 15.

Original: The second is the ﬁnal veriﬁcation label with cross-entropy loss on Eqn. 16.
Revised: The second is the ﬁnal veriﬁcation label with cross-entropy loss on Eqn. 16.

Original: A.6. E XPERIMENTS ON FEVER
Revised: A.6. E XPERIMENTS ON FEVER

Original: Experimental Setups.
Revised: Experimental Setups.

Original: We follow the experiment settings used by previous research in FEVER 1.
Revised: We follow the experiment settings used by previous research in FEVER 1.

Original: shared task, i.e. Nie et al.
Revised: shared task, i.e. Nie et al.

Original: Zhou et al.
Revised: Zhou et al.

Original: (2019), and Liu et al. (2019b).
Revised: (2019), and Liu et al. (2019b).

Original: Similar as Liu et al.
Revised: Similar as Liu et al.

Original: (2019b), we also split the data into single and multi evidence categories and evaluate TransformerXH on the two splits.
Revised: (2019b), we also split the data into single and multi evidence categories and evaluate TransformerXH on the two splits.

Original: Baselines.
Revised: Baselines.

Original: The baselines include GEAR (Zhou et al., 2019) and two contemporary work, SRMRS (Nie et al., 2019) and KGAT (Liu et al., 2019b).
Revised: The baselines include GEAR (Zhou et al., 2019) and two contemporary work, SRMRS (Nie et al., 2019) and KGAT (Liu et al., 2019b).

Original: SR-MRS uses similar adaptations as
Revised: SR-MRS uses similar adaptations as

Original: Transformer-XH from Hotpot QA to FEVER.
Revised: Transformer-XH from Hotpot QA to FEVER.

Original: GEAR is a graph attention network based approach specially designed for fact veriﬁcation.
Revised: GEAR is a graph attention network based approach specially designed for fact veriﬁcation.

Original: KGAT further improves GEAR’s GAT by adding the kernel information, and is the best method on the current leaderboard with BERT base or smaller parameter sizes.
Revised: KGAT further improves GEAR’s GAT by adding the kernel information, and is the best method on the current leaderboard with BERT base or smaller parameter sizes.

Original: (The better entries use Bert large or XL-NET.)
Revised: (The better entries use Bert large or XL-NET.)

Original: We also include the BERT Concat baseline listed in Liu et al.
Revised: We also include the BERT Concat baseline listed in Liu et al.

Original: (2019b) which concatenates the evidence sentences to a text sequence and applies
Revised: (2019b) which concatenates the evidence sentences to a text sequence and applies

Original: BERT on it.
Revised: BERT on it.

Original: Evaluation Results.
Revised: Evaluation Results.

Original: The results on FEVER are shown in Table 7.
Revised: The results on FEVER are shown in Table 7.

Original: Transformer-XH outperforms SRMRS by 4 FEVER score on Dev and 1.8 on Test.
Revised: Transformer-XH outperforms SRMRS by 4 FEVER score on Dev and 1.8 on Test.

Original: It performs on par with KGAT.
Revised: It performs on par with KGAT.

Original: The effectiveness of Transformer-XH is also more observed on claims that require multiple evidence to verify: it outperforms the contemporary work KGAT by 20 FEVER scores on the multi-evidence claims, a 49% relative improvement.
Revised: The effectiveness of Transformer-XH is also more observed on claims that require multiple evidence to verify: it outperforms the contemporary work KGAT by 20 FEVER scores on the multi-evidence claims, a 49% relative improvement.

Original: Compared to KGAT, Transformer-XH mainly loses on the ”not enough evidence” category which is not single nor multi evidence.
Revised: Compared to KGAT, Transformer-XH mainly loses on the ”not enough evidence” category which is not single nor multi evidence.

Original: This is a unique property of the fact veriﬁcation task which our system is not speciﬁcally designed for.
Revised: This is a unique property of the fact veriﬁcation task which our system is not speciﬁcally designed for.

Original: This result further demonstrates Transformer-XH’s effectiveness on tasks with multiple text inputs not in sequential formats.
Revised: This result further demonstrates Transformer-XH’s effectiveness on tasks with multiple text inputs not in sequential formats.

Original: We simply adapt Transformer-XH to the FEVER dataset by only changing the last (linear) task speciﬁc layer; it leads to similar or better performances over contemporary approaches that were speciﬁcally designed for the fact veriﬁcation task.
Revised: We simply adapt Transformer-XH to the FEVER dataset by only changing the last (linear) task speciﬁc layer; it leads to similar or better performances over contemporary approaches that were speciﬁcally designed for the fact veriﬁcation task.