Original: Learning to Navigate Wikipedia by Taking Random Walks
Revised: Learning to Navigate Wikipedia by Taking Random Walks

Original: Abstract
Revised: Abstract

Original: A fundamental ability of an intelligent web-based agent is seeking out and acquiring new information.
Revised: A fundamental ability of an intelligent web-based agent is seeking out and acquiring new information.

Original: Internet search engines reliably find the correct vicinity but the top results may be a few links away from the desired target.
Revised: Internet search engines reliably find the correct vicinity but the top results may be a few links away from the desired target.

Original: A complementary approach is navigation via hyperlinks, employing a policy that comprehends local content and selects a link that moves it closer to the target.
Revised: A complementary approach is navigation via hyperlinks, employing a policy that comprehends local content and selects a link that moves it closer to the target.

Original: In this paper, we show that behavioral cloning of randomly sampled trajectories is sufficient to learn an effective link selection policy.
Revised: In this paper, we show that behavioral cloning of randomly sampled trajectories is sufficient to learn an effective link selection policy.

Original: We demonstrate the approach on a graph version of Wikipedia with 38 M nodes and 387 M edges.
Revised: We demonstrate the approach on a graph version of Wikipedia with 38 M nodes and 387 M edges.

Original: The model is able to efficiently navigate between nodes 5 and 20 steps apart 96% and 92% of the time, respectively.
Revised: The model is able to efficiently navigate between nodes 5 and 20 steps apart 96% and 92% of the time, respectively.

Original: We then use the resulting embeddings and policy in downstream fact verification and question answering tasks where, in combination with basic TF-IDF search and ranking methods, they are competitive results to the state-of-the-art methods.
Revised: We then use the resulting embeddings and policy in downstream fact verification and question answering tasks where, in combination with basic TF-IDF search and ranking methods, they are competitive results to the state-of-the-art methods.

Original:  Introduction
Revised:  Introduction

Original: The ability to gather new knowledge about the world is a fundamental aspect of intelligence.
Revised: The ability to gather new knowledge about the world is a fundamental aspect of intelligence.

Original: Based only on a few words, a fact, question, or even vague idea, humans have the ability to use the internet to find extremely specific information about the world.
Revised: Based only on a few words, a fact, question, or even vague idea, humans have the ability to use the internet to find extremely specific information about the world.

Original: From the important (how to administer CPR) to the trivial (the LA Raiders won the 1983 SuperBowl), virtually any knowledge is a click away.
Revised: From the important (how to administer CPR) to the trivial (the LA Raiders won the 1983 SuperBowl), virtually any knowledge is a click away.

Original: In this work we focus in one particular aspect of this ability: web navigation.
Revised: In this work we focus in one particular aspect of this ability: web navigation.

Original: In general, navigation is a key component of an embodied agent: the ability to move efficiently toward a target.
Revised: In general, navigation is a key component of an embodied agent: the ability to move efficiently toward a target.

Original: In known environments, where map information is available, shortest-path algorithms provide a viable solution.
Revised: In known environments, where map information is available, shortest-path algorithms provide a viable solution.

Original: However, in novel settings, the agent lacks such global information and must instead navigate to the target, using its understanding of the local environment to select actions
Revised: However, in novel settings, the agent lacks such global information and must instead navigate to the target, using its understanding of the local environment to select actions

Original: Other approaches to web agents have focused mostly on retrieval or search engines.
Revised: Other approaches to web agents have focused mostly on retrieval or search engines.

Original: These, however, only provide a partial solution, typically getting close to a desired target but often not exactly to the right page.
Revised: These, however, only provide a partial solution, typically getting close to a desired target but often not exactly to the right page.

Original: In this work, we present an approach for navigation on graph-structured web data that uses hyperlinks within articles to navigate toward a target.
Revised: In this work, we present an approach for navigation on graph-structured web data that uses hyperlinks within articles to navigate toward a target.

Original: It complements search engines, using them to provide a sensible starting point for a local search for specific information.
Revised: It complements search engines, using them to provide a sensible starting point for a local search for specific information.

Original: To be able to navigate on the web effectively, we must read the text of the page, see how concepts in the current page are related to a possible next page, and use our understanding to learn a policy to make the right navigation decision.
Revised: To be able to navigate on the web effectively, we must read the text of the page, see how concepts in the current page are related to a possible next page, and use our understanding to learn a policy to make the right navigation decision.

Original: Consider the scenario shown in Figure 1.
Revised: Consider the scenario shown in Figure 1.

Original: The agent is trying to navigate to the target node N ORTH A MERICA while currently at the node P RESIDENTS OF THE
Revised: The agent is trying to navigate to the target node N ORTH A MERICA while currently at the node P RESIDENTS OF THE

Original: U NITED S TATES containing two hyperlinks: B ARACK O BAMA and USA.
Revised: U NITED S TATES containing two hyperlinks: B ARACK O BAMA and USA.

Original: Our approach learns an embedding from the text of the current page and the text for each hyperlink node and passes these
Revised: Our approach learns an embedding from the text of the current page and the text for each hyperlink node and passes these

Original: Barack Obama
Revised: Barack Obama

Original: USA
Revised: USA

Original: Current Node
Revised: Current Node

Original: Presidents of the United States
Revised: Presidents of the United States

Original: Node
Revised: Node

Original: Node Text “Barack Obama was the President of the United States…”
Revised: Node Text “Barack Obama was the President of the United States…”

Original: “Barack Obama is an American Politician…” “The USA is a nation in North America…” ϕ
Revised: “Barack Obama is an American Politician…” “The USA is a nation in North America…” ϕ

Original: Transformer s t a“The USA is a nation in North America…”
Revised: Transformer s t a“The USA is a nation in North America…”

Original: a
Revised: a

Original: North America “North America is a continent on the…” s g Goal Node
Revised: North America “North America is a continent on the…” s g Goal Node

Original: MLP
Revised: MLP

Original: Action probability pp
Revised: Action probability pp

Original: Node
Revised: Node

Original: Nodeϕ ϕ ϕ
Revised: Nodeϕ ϕ ϕ

Original: θ θ to a goal-conditioned policy that selects the best possible action.
Revised: θ θ to a goal-conditioned policy that selects the best possible action.

Original: To make the correct choice in this example, the agent should associate USA with N ORTH A MERICA .
Revised: To make the correct choice in this example, the agent should associate USA with N ORTH A MERICA .

Original: The particular web environment we consider here is Wikipedia, converted into a graph form.
Revised: The particular web environment we consider here is Wikipedia, converted into a graph form.

Original: Each of the 38M paragraphs is represented by a node; edges are links within and between articles.
Revised: Each of the 38M paragraphs is represented by a node; edges are links within and between articles.

Original: We introduce an approach for navigation on this graph based on behavioral cloning of automatically generated trajectories.
Revised: We introduce an approach for navigation on this graph based on behavioral cloning of automatically generated trajectories.

Original: This allows for unsupervised pre-training of both web page embeddings and navigation policy.
Revised: This allows for unsupervised pre-training of both web page embeddings and navigation policy.

Original: We explore different trajectory distributions for training the model, the best practical choice being equivalent to random walks on the graph.
Revised: We explore different trajectory distributions for training the model, the best practical choice being equivalent to random walks on the graph.

Original: Contributions: Our work presents the first viable solution to the problem of web navigation 1 , as demonstrated by reliable (>90% success) navigation on a graph of size 10 7 .
Revised: Contributions: Our work presents the first viable solution to the problem of web navigation 1 , as demonstrated by reliable (>90% success) navigation on a graph of size 10 7 .

Original: Previous attempts such as
Revised: Previous attempts such as

Original: Nogueira and Cho [2016] show a far inferior performance, well below that needed for practical utility.
Revised: Nogueira and Cho [2016] show a far inferior performance, well below that needed for practical utility.

Original: Furthermore, as there is nothing Wikipedia-specific about our approach and it scales gracefully, the method is applicable to general web settings.
Revised: Furthermore, as there is nothing Wikipedia-specific about our approach and it scales gracefully, the method is applicable to general web settings.

Original: We also demonstrate the relevance of the pre-train + fine-tune paradigm to the web domain, showing that pre-training for navigation is a suitable objective to aid downstream tasks with limited data such as Q&A.
Revised: We also demonstrate the relevance of the pre-train + fine-tune paradigm to the web domain, showing that pre-training for navigation is a suitable objective to aid downstream tasks with limited data such as Q&A.

Original: We demonstrate this by applying our navigation approach on two challenging tasks: fact verification (FEVER; Thorne et al.
Revised: We demonstrate this by applying our navigation approach on two challenging tasks: fact verification (FEVER; Thorne et al.

Original: [2018]) and
Revised: [2018]) and

Original: Q&A (Natural Questions; Lee et al.
Revised: Q&A (Natural Questions; Lee et al.

Original: [2019]) for evidence gathering.
Revised: [2019]) for evidence gathering.

Original: In both settings, we show a jump in performance over existing search-based approaches that rely on retrieval followed by re-ranking, matching state-of-the-art methods despite using far simpler retrieval and re-ranking mechanisms.
Revised: In both settings, we show a jump in performance over existing search-based approaches that rely on retrieval followed by re-ranking, matching state-of-the-art methods despite using far simpler retrieval and re-ranking mechanisms.

Original:  Background
Revised:  Background

Original: Wikipedia as a navigation environment: Human behavior on Wikipedia [West and Leskovec, 2012a] was collected via the Wikispeedia game [West et al., 2009], which asked the players to move between a pair of arbitrarily chosen articles in the fewest steps.
Revised: Wikipedia as a navigation environment: Human behavior on Wikipedia [West and Leskovec, 2012a] was collected via the Wikispeedia game [West et al., 2009], which asked the players to move between a pair of arbitrarily chosen articles in the fewest steps.

Original: This rule allows the emergence of a semantic distance between concepts that facilitates retrieval.
Revised: This rule allows the emergence of a semantic distance between concepts that facilitates retrieval.

Original: West and Leskovec [2012b] conducted initial studies on this dataset for navigation, using supervised learning and reinforcement learning methods on a small subset of articles ( 4 . 6 K) with simple TF-IDF features.
Revised: West and Leskovec [2012b] conducted initial studies on this dataset for navigation, using supervised learning and reinforcement learning methods on a small subset of articles ( 4 . 6 K) with simple TF-IDF features.

Original: Most relevantly, Nogueira and Cho [2016] explore the same problem of navigation on a large
Revised: Most relevantly, Nogueira and Cho [2016] explore the same problem of navigation on a large

Original: Wikipedia graph ( 12 M nodes).
Revised: Wikipedia graph ( 12 M nodes).

Original: They train LSTM agents to perform navigation on the graph, using
Revised: They train LSTM agents to perform navigation on the graph, using

Original: Bag-of-Word features for each article.
Revised: Bag-of-Word features for each article.

Original: The trained agent is then applied to a Jeopardy Q&A task.
Revised: The trained agent is then applied to a Jeopardy Q&A task.

Original: Our approach revisits this framing but with a number of important differences: (i) training on a significantly larger graph ( 38 M vs 12 M nodes, 51 .
Revised: Our approach revisits this framing but with a number of important differences: (i) training on a significantly larger graph ( 38 M vs 12 M nodes, 51 .

Original: 5 M vs 380 M edges); (ii) using Transformer-based architectures for paragraph encodings and for the navigation policy; (iii) they use a single fixed start node, whereas ours is able to navigate from any start node; (iv) removal of beam search, or other complex search mechanisms at evaluation time.
Revised: 5 M vs 380 M edges); (ii) using Transformer-based architectures for paragraph encodings and for the navigation policy; (iii) they use a single fixed start node, whereas ours is able to navigate from any start node; (iv) removal of beam search, or other complex search mechanisms at evaluation time.

Original: Collectively, our approach boosts the navigation success rate from 12 .
Revised: Collectively, our approach boosts the navigation success rate from 12 .

Original: 5% in [Nogueira and Cho, 2016] to over 90% , 2 despite a much larger graph.
Revised: 5% in [Nogueira and Cho, 2016] to over 90% , 2 despite a much larger graph.

Original: Fact verification and Q&A using Wikipedia:
Revised: Fact verification and Q&A using Wikipedia:

Original: Wikipedia is a rich resource for evaluating language understanding through Q&A tasks.
Revised: Wikipedia is a rich resource for evaluating language understanding through Q&A tasks.

Original: Chen et al.
Revised: Chen et al.

Original: [2017] combine TF-IDF retrieval with an LSTM document reader to find answers to free-form questions within Wikipedia documents.
Revised: [2017] combine TF-IDF retrieval with an LSTM document reader to find answers to free-form questions within Wikipedia documents.

Original: Contemporary Q&A systems [Semnani and Pandey, 2020, Qu et al., 2020] extend this general approach to more complex pipelines, combining traditional and neural methods with inverted index search/retrieval schemes, followed by document readers that perform re-ranking.
Revised: Contemporary Q&A systems [Semnani and Pandey, 2020, Qu et al., 2020] extend this general approach to more complex pipelines, combining traditional and neural methods with inverted index search/retrieval schemes, followed by document readers that perform re-ranking.

Original: We demonstrate our navigation approach in conjunction with very basic retrieval and re-ranking approaches, so as to clearly demonstrate the value of navigation.
Revised: We demonstrate our navigation approach in conjunction with very basic retrieval and re-ranking approaches, so as to clearly demonstrate the value of navigation.

Original: In common with our approach, Wang et al.
Revised: In common with our approach, Wang et al.

Original: [2021] also use a graph version of
Revised: [2021] also use a graph version of

Original: Wikipedia, but with the goal of aligning it to a knowledge base, rather than navigation.
Revised: Wikipedia, but with the goal of aligning it to a knowledge base, rather than navigation.

Original: Stammbach and Neumann [2019] propose a scheme for finding evidence for the FEVERThorne et al.
Revised: Stammbach and Neumann [2019] propose a scheme for finding evidence for the FEVER

Original: [2018] benchmark that explores all links one step away from an initial retrieval set.
Revised: [Thorne et al., 2018] benchmark that explores all links one step away from an initial retrieval set.

Original: This can be viewed as a 1-step navigation operation, in contrast to our scheme that permits an arbitrary number of steps.
Revised: This can be viewed as a 1-step navigation operation, in contrast to our scheme that permits an arbitrary number of steps.

Original: Navigation in knowledge bases: Graph navigation is a key element in utilizing knowledge bases (KB), where facts are represented as tuples of entity nodes connected by a labelled edge corresponding to relation between the two nodes.
Revised: Navigation in knowledge bases: Graph navigation is a key element in utilizing knowledge bases (KB), where facts are represented as tuples of entity nodes connected by a labelled edge corresponding to relation between the two nodes.

Original: For example, Das et al.
Revised: For example, Das et al.

Original: [2017] answer complex queries by navigating the graph, inferring missing edges along the way, and finding a valid path to the answer node.
Revised: [2017] answer complex queries by navigating the graph, inferring missing edges along the way, and finding a valid path to the answer node.

Original: In our case, the graph is fixed but consists of natural language text and is partially observed.
Revised: In our case, the graph is fixed but consists of natural language text and is partially observed.

Original: This makes the task much harder on two fronts.
Revised: This makes the task much harder on two fronts.

Original: First, the policy must comprehend the natural language actions, without access to crisply labelled relation actions.
Revised: First, the policy must comprehend the natural language actions, without access to crisply labelled relation actions.

Original: Second, the action space is open and varying in size, unlike in a KB which typically has a fixed schema, i.e. relations/actions come from a fixed size set.
Revised: Second, the action space is open and varying in size, unlike in a KB which typically has a fixed schema, i.e. relations/actions come from a fixed size set.

Original: Perhaps the closest work involving natural language actions, Fu et al.
Revised: Perhaps the closest work involving natural language actions, Fu et al.

Original: [2019] augment a KB with textual nodes.
Revised: [2019] augment a KB with textual nodes.

Original: However, it still operates under the fixed KB schema.
Revised: However, it still operates under the fixed KB schema.

Original: Further improvements on navigation in KB have been shown by incorporating tricks like reward-shaping [Xiong et al., 2017] and action dropout
Revised: Further improvements on navigation in KB have been shown by incorporating tricks like reward-shaping [Xiong et al., 2017] and action dropout

Original: [Lin et al., 2018].
Revised: [Lin et al., 2018].

Original: We adopted the action dropout where some outgoing edges are masked during training to enable more effective path exploration.
Revised: We adopted the action dropout where some outgoing edges are masked during training to enable more effective path exploration.

Original: Pushing more on exploration, inspired by AlphaGo, Shen et al.
Revised: Pushing more on exploration, inspired by AlphaGo, Shen et al.

Original: He et al.
Revised: He et al.

Original: [2022] combine the policy with Monte-Carlo tree search to navigate KB.
Revised: [2022] combine the policy with Monte-Carlo tree search to navigate KB.

Original: We find that no search method is necessary for our approach at evaluation time, with good performance obtained from the policy alone.
Revised: We find that no search method is necessary for our approach at evaluation time, with good performance obtained from the policy alone.

Original: Internet powered Q&A:
Revised: Internet powered Q&A:

Original: Several recent works have attempted complex question answering using the internet.
Revised: Several recent works have attempted complex question answering using the internet.

Original: Talmor and Berant [2018] retrieve snippets of information from the web with a search engine to answer a structured decomposition of the question with a simpler Q&A model.
Revised: Talmor and Berant [2018] retrieve snippets of information from the web with a search engine to answer a structured decomposition of the question with a simpler Q&A model.

Original: The recentWebGPT Nakano et al.
Revised: The recent

Original: [2021] system takes a broadly similar approach to long-form question answering.
Revised: WebGPT [Nakano et al., 2021] system takes a broadly similar approach to long-form question answering.

Original: WebGPT accures information from the Bing search API as well as some navigation, and passes it to an LM to generate the answer text.
Revised: WebGPT accures information from the Bing search API as well as some navigation, and passes it to an LM to generate the answer text.

Original: The complexity of the system necessitates the use of human demonstrations for training.
Revised: The complexity of the system necessitates the use of human demonstrations for training.

Original: Related work from Lazaridou et al.
Revised: Related work from Lazaridou et al.

Original: [2022] utilizes a few-shot supervision of an LM to perform Google queries, which are then used by the LM to generate an answer.
Revised: [2022] utilizes a few-shot supervision of an LM to perform Google queries, which are then used by the LM to generate an answer.

Original: They show that by incorporating information from Google queries, the answers are more factual than those generated directly from LMs.
Revised: They show that by incorporating information from Google queries, the answers are more factual than those generated directly from LMs.

Original: Another line of work, has leveraged link structure of webpages found on the internet to improve Q&A.
Revised: Another line of work, has leveraged link structure of webpages found on the internet to improve Q&A.

Original: In LinkBERT [Yasunaga et al., 2022] is pretrained to capture dependencies between documents and results in improvement for for multi-hop reasoning and few-shot reading comprehension.
Revised: In LinkBERT [Yasunaga et al., 2022] is pretrained to capture dependencies between documents and results in improvement for for multi-hop reasoning and few-shot reading comprehension.

Original: Asai et al.
Revised: Asai et al.

Original: [2020] use the link structure to enhance open domain
Revised: [2020] use the link structure to enhance open domain

Original: Q&A by learning to retrieve according to reasoning paths.
Revised: Q&A by learning to retrieve according to reasoning paths.

Original: In contrast to these works, we focus on mastering navigation rather than any combination with search/retrieval, albeit in more limited domain (Wikipedia vs the internet).
Revised: In contrast to these works, we focus on mastering navigation rather than any combination with search/retrieval, albeit in more limited domain (Wikipedia vs the internet).

Original: However, our approach could potentially act as a core component in a general web navigation, being more versatile on account of not requiring supervision.
Revised: However, our approach could potentially act as a core component in a general web navigation, being more versatile on account of not requiring supervision.

Original:  Approach
Revised:  Approach

Original: We consider navigation in a web environment W = { N , L } , where N := { n i } Ni =1 is the set of nodes, each n i representing a web page with content s i .
Revised: We consider navigation in a web environment W = { N , L } , where N := { n i } Ni =1 is the set of nodes, each n i representing a web page with content s i .

Original: The links between pages are represented by L := { l i } Ni =1 , where l i = { l 1 i , . . .
Revised: The links between pages are represented by L := { l i } Ni =1 , where l i = { l 1 i , . . .

Original: , l ji , . . .
Revised: , l ji , . . .

Original: , l M i i } are the set of M i (directed) out-links from node i .
Revised: , l M i i } are the set of M i (directed) out-links from node i .

Original: We define NH ( n i ) as the set of nodes reached by following all links l i from node n i , where the j th neighbor is given by l ji ( n i ) .
Revised: We define NH ( n i ) as the set of nodes reached by following all links l i from node n i , where the j th neighbor is given by l j i ( n i ) .

Original: We assume some target node n g and some starting node n 0 are given, and wish to train an agent that can navigate from n 0 to n g by walking along the edges of the graph within T steps.
Revised: We assume some target node n g and some starting node n 0 are given, and wish to train an agent that can navigate from n 0 to n g by walking along the edges of the graph within T steps.

Original: To accomplish this, we construct a parametric policy p θ ( n t +1 | n t , n g ) which, at time step t , parameterizes a distribution over n t +1 ∈ NH ( n t ) .
Revised: To accomplish this, we construct a parametric policy p θ ( n t +1 | n t , n g ) which, at time step t , parameterizes a distribution over n t +1 ∈ NH ( n t ) .

Original: We train this model via behavioral cloning to maximize
Revised: We train this model via behavioral cloning to maximize

Original: [Equation]
Revised: [Equation]

Original: t

Original: [Equation]

Original: log [Equation] (cid:35) (1) where E ( n 0 , . . . , n T ) is our trajectory distribution which generates trajectories ( n 0 , . .
Revised: where E ( n 0 , . . . , n T ) is our trajectory distribution which generates trajectories ( n 0 , . .

Original: . , n T ) that we would like our model to replicate (i.e. setting n g = n T ).
Revised: . , n T ) that we would like our model to replicate (i.e. setting n g = n T ).

Original: 3.1 Trajectory distribution
Revised: 3.1 Trajectory distribution

Original: This framework opens up many choices for E .
Revised: This framework opens up many choices for E .

Original: Clearly some will result in ineffective navigation strategies while others may enable us to train a policy which is capable of generalizing to new graphs and downstream applications.
Revised: Clearly some will result in ineffective navigation strategies while others may enable us to train a policy which is capable of generalizing to new graphs and downstream applications.

Original: We elaborate on some choices below.
Revised: We elaborate on some choices below.

Original: Reverse trajectories We define E ( n T ) =
Revised: Reverse trajectories: We define E ( n T ) =

Original: Uniform ( N ) (the uniform distribution over nodes in the graph).
Revised: Uniform ( N ) (the uniform distribution over nodes in the graph).

Original: Now we can define
Revised: Now we can define

Original: [Equation]
Revised: [Equation]

Original: | [Equation] or, a distribution which uniformly walks the reverse graph (with transposed adjacency matrix).
Revised: or, a distribution which uniformly walks the reverse graph (with transposed adjacency matrix).

Original: We begin with some uniformly chosen target node and walk backward randomly to n 0 in such a way that we can reach the target by walking forward from n 0 .
Revised: We begin with some uniformly chosen target node and walk backward randomly to n 0 in such a way that we can reach the target by walking forward from n 0 .

Original: Intuitively, training with the reverse trajectories defined above should allow our model to be able to reach any node in the graph from any node it is connected to.
Revised: Intuitively, training with the reverse trajectories defined above should allow our model to be able to reach any node in the graph from any node it is connected to.

Original: Unfortunately, in realistic web graph data, there exist many “dead-ends” or nodes with no in-going edges.
Revised: Unfortunately, in realistic web graph data, there exist many “dead-ends” or nodes with no in-going edges.

Original: The trajectories get stuck at these nodes leading to low-diversity training data.
Revised: The trajectories get stuck at these nodes leading to low-diversity training data.

Original: Random forward trajectories In our real-world knowledge graph data, there are far fewer nodes which have no out-going edges.
Revised: Random forward trajectories: In our real-world knowledge graph data, there are far fewer nodes which have no out-going edges.

Original: Thus we propose an alternative method to generate target nodes based on randomly walking forward.
Revised: Thus we propose an alternative method to generate target nodes based on randomly walking forward.

Original: We define E ( n 0 ) to be the Uniform ( N ) and
Revised: We define E ( n 0 ) to be the Uniform ( N ) and

Original: E ( n t +1 | n t ) = | NH ( n t ) | if n t +1 ∈ NH ( n t ) , else 0 .
Revised: [Equation]

Original: (3) or, a distribution which uniformly picks a starting node and randomly walks the forward graph for
Revised: or, a distribution which uniformly picks a starting node and randomly walks the forward graph for

Original: T steps.
Revised: T steps.

Original: Training in this way should encourage our model to be able to reach any node which is reachable from some uniformly chosen start.
Revised: Training in this way should encourage our model to be able to reach any node which is reachable from some uniformly chosen start.

Original: We note that the target node distribution E ( n T ) will not be uniform and instead be a function of the graph structure.
Revised: We note that the target node distribution E ( n T ) will not be uniform and instead be a function of the graph structure.

Original: We also note that while this generates a
Revised: We also note that while this generates a

Original: T -step trajectory, because it is generated by random walk, the shortest path distance between start and target nodes may be smaller.
Revised: T -step trajectory, because it is generated by random walk, the shortest path distance between start and target nodes may be smaller.

Original: Random shortest paths We define E ( n T ) =
Revised: Random shortest paths: We define E ( n T ) =

Original: Uniform ( N ) and let E ( n 0 | n T ) be the uniform distribution of nodes whose shortest path length to n T is T steps.
Revised: Uniform ( N ) and let E ( n 0 | n T ) be the uniform distribution of nodes whose shortest path length to n T is T steps.

Original: We then define E ( n t | n 0 , n T ) to be the delta distribution that n t is the t -th node of this shortest path.
Revised: We then define E ( n t | n 0 , n T ) to be the delta distribution that n t is the t -th node of this shortest path.

Original: While it can be challenging to compute the probabilities of this distribution efficiently we can draw samples from it using a shortest path finding algorithm such as Dijkstra’s algorithm.
Revised: While it can be challenging to compute the probabilities of this distribution efficiently we can draw samples from it using a shortest path finding algorithm such as Dijkstra’s algorithm.

Original: 3.2 Model parameterization
Revised: 3.2 Model parameterization

Original: From above, we can see the core of our model is the p θ ( n t +1 | n t , n T ) distribution.
Revised: From above, we can see the core of our model is the p θ ( n t +1 | n t , n T ) distribution.

Original: Given a sampled trajectory that begins at n 0 , this is the distribution which attempts to choose the correct action to take at timestep t + 1 starting at node n t in order to get to target node n T .
Revised: Given a sampled trajectory that begins at n 0 , this is the distribution which attempts to choose the correct action to take at timestep t + 1 starting at node n t in order to get to target node n T .

Original: We parameterize this distribution as a function of the text available at node n t .
Revised: We parameterize this distribution as a function of the text available at node n t .

Original: [Equation]
Revised: [Equation]

Original: where f θ embeds the text s g , s t and s t +1 of the target, current and destination nodes, respectively.
Revised: where f θ embeds the text s g , s t and s t +1 of the target, current and destination nodes, respectively.

Original: Further details on how we apply this general model to Wikipedia are given in Section 4.2 and an alternative interpretation of this approach based on variational inference in a latent-variable model is presented in Appendix D.
Revised: Further details on how we apply this general model to Wikipedia are given in Section 4.2 and an alternative interpretation of this approach based on variational inference in a latent-variable model is presented in Appendix D.

Original:  Application to Wikipedia 4.1 Navigation and sentence search tasks
Revised:  Application to Wikipedia 4.1 Navigation and sentence search tasks

Original: Data processing: We convert a snapshot of English Wikipedia into a graph but split each article ( ¯ µ = 1000 words) into paragraph-sized blocks ( ¯ µ = 100 words), which form the nodes .
Revised: Data processing: We convert a snapshot of English Wikipedia into a graph but split each article ( ¯ µ = 1000 words) into paragraph-sized blocks ( ¯ µ = 100 words), which form the nodes .

Original: Edges in the graph are the union of (i) organic hyperlinks, (ii) additional entity linking and (iii) next/previous paragraph links.
Revised: Edges in the graph are the union of (i) organic hyperlinks, (ii) additional entity linking and (iii) next/previous paragraph links.

Original: The full graph has ∼ 38 M nodes and 387 M edges.
Revised: We consider two snapshots from year 2017 and 2018, which produces large full graphs with ∼ 37 M nodes and ∼ 370 M edges.

Original: For initial experiments, we also use a smaller subsampled version of 200 k nodes. For this we sub-sample disjoint sets of 200k train / 200k evaluation Wikipedia nodes.
Revised: For initial experiments, we also use a smaller subsampled graph, for which we sub-sample disjoint sets of 200k train / 200k evaluation nodes from the 2018 graph.

Original: See Section A.1 for full details.
Revised: All the obtained graphs are well connected with median path length of 15. Further details have been relegated to the Appendix (full graphs in Appendix A.1, the 200k graph in Appendix A.2, and graph statistics in Appendix A.3).

Original: T-step navigation: The most basic task is T -step navigation.
Revised: T-step navigation: The most basic task is T -step navigation.

Original: At the start of the episode, we randomly sample a start node and generate a T -step random walk trajectory.
Revised: At the start of the episode, we randomly sample a start node and generate a T -step random walk trajectory.

Original: We then take the last node of this trajectory as the target n g .
Revised: We then take the last node of this trajectory as the target n g .

Original: The agent is then given a time budget B (in most experiments, we fix max steps to B = 100 ), and succeeds if it reaches the target node within B steps.
Revised: The agent is then given a time budget B (in most experiments, we fix max steps to B = 100 ), and succeeds if it reaches the target node within B steps.

Original: In our experiments, we show results on various graphs and splits with T = { 5 , 10 , 20 } -step navigation.
Revised: In our experiments, we show results on various graphs and splits with T = { 5 , 10 , 20 } -step navigation.

Original: 1-T Multistep navigation: The Multistep task is much like the previous setting, except that instead of generating fixed T -step trajectories, we sample from Uniform (1 , T ) , T = 20 .
Revised: 1-T Multistep navigation: The Multistep task is much like the previous setting, except that instead of generating fixed T -step trajectories, we sample from Uniform (1 , T ) , T = 20 .

Original: T-step sentence search: This is the same setup as T -step navigation, except that rather than provide the full text for the target node n g , only one sentence within that text is selected at random to compute the target embeddings from.
Revised: T-step sentence search: This is the same setup as T -step navigation, except that rather than provide the full text for the target node n g , only one sentence within that text is selected at random to compute the target embeddings from.

Original: Thus agents have the more challenging task of locating the correct node, given only a small snippet of its text (see Sec. 4.2 for details on target representation).
Revised: Thus agents have the more challenging task of locating the correct node, given only a small snippet of its text (see Section 4.2 for details on target representation).

Original: We include this task because one of the difficulties of web navigation is establishing which page should be the target in the first place.
Revised: We include this task because one of the difficulties of web navigation is establishing which page should be the target in the first place.

Original: It also has direct relevance to our downstream fact verification task.
Revised: It also has direct relevance to our downstream fact verification task.

Original: 4.2 Model architecture
Revised: 4.2 Model architecture

Original: State representation:
Revised: State representation:

Original: For each node n i we embed its text
Revised: For each node n i we embed its text

Original: s i using a Transformer ϕ ( . ) to produce ϕ ( s i ) .
Revised: s i using a Transformer ϕ ( . ) to produce ϕ ( s i ) .

Original: By basing the state representation on semantic content, rather than i.e. node index, the agent is able to learn about the relationships between entities in a manner that allows for generalization to new graphs at evaluation time, instead of memorizing the structure of the training graph.
Revised: By basing the state representation on semantic content, rather than i.e. node index, the agent is able to learn about the relationships between entities in a manner that allows for generalization to new graphs at evaluation time, instead of memorizing the structure of the training graph.

Original: Following Karpukhin et al.
Revised: Following Karpukhin et al.

Original: [2020], we append to the node text the title of the Wikipedia article to which it belongs.
Revised: [2020], we append to the node text the title of the Wikipedia article to which it belongs.

Original: This often provides context to the node that might otherwise be missing – in many articles, the subject of the article will be referred to indirectly (e.g. it might state that “she” was born ininstead of repeating the subject’s name).
Revised: This often provides context to the node that might otherwise be missing – in many articles, the subject of the article will be referred to indirectly (e.g. it might state that “she” was born ininstead of repeating the subject’s name).

Original: For those experiments where the node embeddings are kept fixed, ϕ ( . ) is a pre-trained RoBERTA
Revised: For those experiments where the node embeddings are kept fixed, ϕ ( . ) is a pre-trained RoBERTA

Original: [Liu et al., 2019] model that encodes the node text & title.
Revised: [Liu et al., 2019] model that encodes the node text & title.

Original: However, in our large-scale experiments, we pre-train the Transformer ϕ ( . ) directly using the text & title (see Appendix B.1).
Revised: However, in our large-scale experiments, we pre-train the Transformer ϕ ( . ) directly using the text & title (see Appendix B.1).

Original: In both cases, we apply the Transformer to the tokenized text, take the mean over the input tokens, and apply a tanh nonlinearity to produce ϕ ( s i ) .
Revised: In both cases, we apply the Transformer to the tokenized text, take the mean over the input tokens, and apply a tanh nonlinearity to produce ϕ ( s i ) .

Original: Target representation: Similarly, we use the article text s g of the target node as input to the
Revised: Target representation: Similarly, we use the article text s g of the target node as input to the

Original: Transformer model ϕ to compute its representation ϕ ( s g ) .
Revised: Transformer model ϕ to compute its representation ϕ ( s g ) .

Original: For some tasks, such as sentence search or fact verification, the target representation instead relies on a single sentence from each node or other text specific to the task and a separate transformer model ϕ target is trained to embed the target.
Revised: For some tasks, such as sentence search or fact verification, the target representation instead relies on a single sentence from each node or other text specific to the task and a separate transformer model ϕ target is trained to embed the target.

Original: Navigation policy network: The network operates on the target node n g and the current node n t , and outputs a distribution over the possible actions that can be taken at n t .
Revised: Navigation policy network: The network operates on the target node n g and the current node n t , and outputs a distribution over the possible actions that can be taken at n t .

Original: As the nodes in the graph have a variable number of outgoing edges, the policy must produce a distribution with a variable number of outcomes.
Revised: As the nodes in the graph have a variable number of outgoing edges, the policy must produce a distribution with a variable number of outcomes.

Original: To do this, we first concatenate the embeddings of s g and s t , and pass them through a 1-layer feed-forward network to produce a combined embedding e tg =
Revised: To do this, we first concatenate the embeddings of s g and s t , and pass them through a 1-layer feed-forward network to produce a combined embedding e tg =

Original: FF [ ϕ ( s t ) , ϕ ( s g )] .
Revised: FF [ ϕ ( s t ) , ϕ ( s g )] .

Original: Next we embed each possible action a i .
Revised: Next we embed each possible action a i .

Original: When we use a fixed ϕ , we directly use a i = ϕ
Revised: When we use a fixed ϕ , we directly use a i = ϕ

Original: ( s i ) for n i ∈
Revised: ( s i ) for n i ∈

Original: NH ( n t ) .
Revised: NH ( n t ) .

Original: However this is not feasible when pre-training our embedding model due to memory concerns.
Revised: However this is not feasible when pre-training our embedding model due to memory concerns.

Original: Instead, we compute a i = ϕ
Revised: Instead, we compute a i = ϕ

Original: ( s t [ l it ]) where s [ l i ] selects the words in s which belong to hyperlink i – see Appendix B.1 for more details.
Revised: ( s t [ l it ]) where s [ l i ] selects the words in s which belong to hyperlink i – see Appendix B.1 for more details.

Original: We then define the probability of moving to n i from n t with target n g as p ( n i | n t , n g )
Revised: We then define the probability of moving to n i from n t with target n g as p ( n i | n t , n g )

Original: ∝ exp ( e tg · a i ) which is normalized to produce a distribution over the | NH ( n t ) | actions.
Revised: ∝ exp ( e tg · a i ) which is normalized to produce a distribution over the | NH ( n t ) | actions.

Original: In our larger-scale model, we also include the trajectory leading up to the current node ( n 0 , . . . , n t ) .
Revised: In our larger-scale model, we also include the trajectory leading up to the current node ( n 0 , . . . , n t ) .

Original: We pass their embeddings, combined with the target embedding, [ ϕ ( s 0 ) , . . .
Revised: We pass their embeddings, combined with the target embedding, [ ϕ ( s 0 ) , . . .

Original: , ϕ ( s t ) , ϕ ( s g )] , through a 4-layer transformer and use the output as e tg when computing the transition probabilities above.
Revised: , ϕ ( s t ) , ϕ ( s g )] , through a 4-layer transformer and use the output as e tg when computing the transition probabilities above.

Original: We explore these two policy architecture choices, feed-forward and Transformer in Table 4.
Revised: We explore these two policy architecture choices, feed-forward and Transformer in Table 4.

Original: 4.3 Application to downstream tasks
Revised: 4.3 Downstream tasks

Original: Finally, we use our navigation approach to gather information for the tasks of fact verification on the
Revised: Finally, we use our navigation approach to gather information for the tasks of fact verification on the

Original: FEVER benchmark [Thorne et al., 2018] and question answering on Natural Questions (NQ)
Revised: FEVER benchmark [Thorne et al., 2018] and question answering on Natural Questions (NQ)

Original: [Lee et al., 2019] which also use Wikipedia as a knowledge base.
Revised: [Lee et al., 2019] which also use Wikipedia as a knowledge base.

Original: One important difference from our earlier navigation task is that the target node n g is not specified at evaluation time.
Revised: One important difference from our earlier navigation task is that the target node n g is not specified at evaluation time.

Original: Instead, we are given a claim whose veracity must be established or a question that must be answered by finding information in a particular node in the graph.
Revised: Instead, we are given a claim whose veracity must be established or a question that must be answered by finding information in a particular node in the graph.

Original: To determine which node this is, we use a target encoder which maps the claim or question to an embedding vector that can be used in place of ϕ ( s g ) in Section 4.2.
Revised: To determine which node this is, we use a target encoder which maps the claim or question to an embedding vector that can be used in place of ϕ ( s g ) in Section 4.2.

Original: Due to the limited size of the downstream training sets, we pre-train our graph embeddings, navigation policy and target encoder on the similar sentence search task.
Revised: Due to the limited size of the downstream training sets, we pre-train our graph embeddings, navigation policy and target encoder on the similar sentence search task.

Original: We then freeze the embeddings and policy and fine-tune the target encoder on ( n g , s g ) pairs from the downstream task’s training set.
Revised: We then freeze the embeddings and policy and fine-tune the target encoder on ( n g , s g ) pairs from the downstream task’s training set.

Original: To avoid over-fitting, we add an auxiliary loss which enforces the similarity between our target embedding and the embedding of the ground truth node containing the sentence needed to resolve the claim or question.
Revised: To avoid over-fitting, we add an auxiliary loss which enforces the similarity between our target embedding and the embedding of the ground truth node containing the sentence needed to resolve the claim or question.

Original: Training and evaluation on FEVER  required alignment between our version of Wikipedia and the one used to compile the benchmark and details of this can be found inAppendix
Revised: Training and evaluation on FEVER and NQ required alignment between our version of Wikipedia and the one used to compile the benchmark and details of this can be found in

Original: A.4.
Revised: Appendix C.1 and C.2.

Original: Next, our navigation method requires a starting node.
Revised: Next, our navigation method requires a starting node.

Original: To put our agent close to the articles we need in the downstream tasks, we run a popular variant of TF-IDF, called BM25 [Robertson and Zaragoza, 2009].
Revised: To put our agent close to the articles we need in the downstream tasks, we run a popular variant of TF-IDF, called BM25 [Robertson and Zaragoza, 2009].

Original: We first create an index over all of the nodes in the graph, then take the top-5 BM25 matches as the starting nodes, and run navigation for 20 steps on each.
Revised: We first create an index over all of the nodes in the graph, then take the top-5 BM25 matches as the starting nodes, and run navigation for 20 steps on each.

Original: Note that BM25  does not find the exact text (i.e. the hits@1 is notvery high) but matches are in the right vicinity, usuallyin the same or similar article.
Revised: Note that BM25 often does not find the exact text (i.e. the hits@1 is not high) but matches are in the right vicinity, usually the same or similar article.

Original: Thus the agent only has to navigate a few steps to reach the target: for the FEVER dataset, the mean length of shortest path between start and target node is 4 . 3 steps.
Revised: Thus the agent only has to navigate a few steps to reach the target: in FEVER and NQ datasets, the mean length of shortest path between start and target node is 4 . 3 and 5 . 1 steps respectively.

Original: The final component is a re-ranker that takes all the sentences from all nodes visited by the agent and assigns a match score between each one and the claim.
Revised: The final component is a re-ranker that takes all the sentences from all nodes visited by the agent and assigns a match score between each one and the claim.

Original: The ranked list of sentences can then be used for benchmark evaluation.
Revised: The ranked list of sentences can then be used for benchmark evaluation.

Original: We explore two types of ranker: (a) basic TF-IDF and (b) a  BERT model, fine tuned using hard negatives mined from the TF-IDF ranker.
Revised: We explore two types of ranker: (a) basic TF-IDF and (b) a transformer based model like BERT or BigBird, fine tuned using hard negatives mined from the TF-IDF ranker.

Original: See Appendix A.4 for more details.
Revised: See Appendix C.1 and C.2 for more details.

Original: Evaluation of our approach thus consists of: (i) given a claim/question, use BM-25 to return a list of promising start locations; (ii) from the top N = 10 of these, run our navigation model for 20 steps, using the target encoder to embed the claim/question, (iii) use the re-ranker to score sentences visited against the claim/question and (iv) select the top 5 for computing the  F1 score, for comparison on the benchmarks.
Revised: Evaluation of our approach thus consists of: (i) given a claim/question, use BM-25 to return a list of promising start locations; (ii) from the top N = 10 of these, run our navigation model for 20 steps, using the target encoder to embed the claim/question, (iii) use the re-ranker to score sentences visited against the claim/question and (iv) select the top 5 for computing the recall, precision, or F1 score, for comparison on the benchmarks.

Original:  Experiments 5.1 Investigation of training trajectory distribution
Revised:  Experiments 5.1 Investigation of training trajectory distribution

Original: On the smaller 200k node graph, we investigate different choices for the trajectory policy distributions
Revised: On the smaller 200k node graph, we investigate different choices for the trajectory policy distributions

Original: E described in Section 3: (i) Random Forward Trajectories , where we randomly sample start nodes and do a random walk to get a target, (ii) Reverse Trajectories , where we randomly sample a target node and and do a random walk on the reverse graph to get a start node, and (iii) Random Shortest
Revised: E described in Section 3: (i) Random Forward Trajectories , where we randomly sample start nodes and do a random walk to get a target, (ii) Reverse Trajectories , where we randomly sample a target node and and do a random walk on the reverse graph to get a start node, and (iii) Random Shortest

Original: Paths where we randomly sample a start node and take a random walk and then compute the shortest path between the start and target nodes.
Revised: Paths where we randomly sample a start node and take a random walk and then compute the shortest path between the start and target nodes.

Original: For each of these we trained policies on 5 , 10 , and 20 step navigation tasks.
Revised: For each of these we trained policies on 5 , 10 , and 20 step navigation tasks.

Original: We evaluate these in two different ways: (i) forward sampling of source and target in the disjoint evaluation graph, (ii) reverse sampling of source and target.
Revised: We evaluate these in two different ways: (i) forward sampling of source and target in the disjoint evaluation graph, (ii) reverse sampling of source and target.

Original: The results of this experiment are shown in Table 1.
Revised: The results of this experiment are shown in Table 1.

Original: Unsurprisingly, we can see that forward and reverse trained policies both do better when evaluated in the way that matches training.
Revised: Unsurprisingly, we can see that forward and reverse trained policies both do better when evaluated in the way that matches training.

Original: However, forward trajectory policies  generalize much better to reverse navigation.
Revised: However, forward trajectory policies [Table] generalize much better to reverse navigation.

Original: When we analyze the traces of these experiments, we see that in the reverse sampling, many target nodes are in “dead-ends” where many target nodes do not have long random trajectories to sample.
Revised: When we analyze the traces of these experiments, we see that in the reverse sampling, many target nodes are in “dead-ends” where many target nodes do not have long random trajectories to sample.

Original: This results in easier navigation (as we can see from higher accuracy in reverse trajectory trained policies on reverse navigation) and more extreme overfitting, leading to poor performance of reverse trajectory on forward navigation.
Revised: This results in easier navigation (as we can see from higher accuracy in reverse trajectory trained policies on reverse navigation) and more extreme overfitting, leading to poor performance of reverse trajectory on forward navigation.

Original: Not surprisingly, we also see that training on a shortest path trajectories leads to better performance both in the forward and reverse sampling.
Revised: Not surprisingly, we also see that training on a shortest path trajectories leads to better performance both in the forward and reverse sampling.

Original: This advantage is especially seen in the 20 -step navigation task.
Revised: This advantage is especially seen in the 20 -step navigation task.

Original: Ideally, we would always train on shortest paths.
Revised: Ideally, we would always train on shortest paths.

Original: However, the time complexity of pre-computing all shortest paths in a graph using Dijkstra’s algorithm is O ( V ( E + V logV )) for the number of nodes
Revised: However, the time complexity of pre-computing all shortest paths in a graph using Dijkstra’s algorithm is O ( V ( E + V logV )) for the number of nodes

Original: V and number of edges E in the graph.
Revised: V and number of edges E in the graph.

Original: On the full 38 M Wikipedia graph, this computation would be completely intractable.
Revised: On the full 38 M Wikipedia graph, this computation would be completely intractable.

Original: Luckily, the random forward trajectories perform almost as well and can be computed in constant time with respect to the size of the graph and only linear time with respect to the trajectory length.
Revised: Luckily, the random forward trajectories perform almost as well and can be computed in constant time with respect to the size of the graph and only linear time with respect to the trajectory length.

Original: For all subsequent experiments, we use the Random Forward Trajectories for our trajectory policy and evaluate on the forward version of the navigation tasks, where we call our approach Random Forward
Revised: For all subsequent experiments, we use the Random Forward Trajectories for our trajectory policy and evaluate on the forward version of the navigation tasks, where we call our approach Random Forward

Original: Behavioral Cloning (RFBC) .
Revised: Behavioral Cloning (RFBC) .

Original: [Table]

Original: 5.2 Navigation
Revised: 5.2 Navigation

Original: We next do a more thorough set of experiments on the smaller 200 k Wikipedia graph.
Revised: We next do a more thorough set of experiments on the smaller 200 k Wikipedia graph.

Original: Because the graph is much smaller than the full Wikipedia and thus prone to overfitting, for all of our methods and baselines we use the fixed RoBERTA embeddings ϕ ( s i ) , the simplest 1-layer feed-forward network, and MiniBERT Turc et al.
Revised: Because the graph is much smaller than the full Wikipedia and thus prone to overfitting, for all of our methods and baselines we use the fixed RoBERTA embeddings ϕ ( s i ) , the simplest 1-layer feed-forward network, and MiniBERT [Turc et al.,

Original: [2019] for target sentence embedding ϕ
Revised: 2019] for target sentence embedding ϕ

Original: ( s g ) .
Revised: ( s g ) .

Original: Alongside our approach, we employ the following baselines/ablations:
Revised: Alongside our approach, we employ the following baselines/ablations:

Original: • RFBC:
Revised: • RFBC:

Original: Random-forward behavioral cloning (our approach).
Revised: Random-forward behavioral cloning (our approach).

Original: • RL: Navigation using a policy gradients RL agent with same policy architecture as our model.

Original: See Sec.

Original: A.5 for more details. • RFBC + RF: RFBC but with random features sampled from the unit sphere instead of ϕ • RL + RF: RL with random features.
Revised: • RFBC + RF: RFBC but with random features sampled from the unit sphere instead of ϕ • RL + RF: RL with random features.

Original: • Random: policy that chooses random action (out-link) at each timestep.
Revised: • Random: policy that chooses random action (out-link) at each timestep.

Original: • Greedy: selects the action embedding which has the smallest cosine distance with the target.
Revised: • Greedy: selects the action embedding which has the smallest cosine distance with the target.

Original: • Random DFS: DFS of depth equal to number of steps, choosing actions at random.
Revised: • Random DFS: DFS of depth equal to number of steps, choosing actions at random.

Original: • Greedy DFS: DFS but select action with smallest cosine distance to the target.
Revised: • Greedy DFS: DFS but select action with smallest cosine distance to the target.

Revised: More training details are in Appendix B.2 and computational cost of each in Appendix E.

Original: The results evaluated on the held-out graph are shown in columns 1-4 of Table 2.
Revised: The results evaluated on the held-out graph are shown in columns 1-4 of Table 2.

Original: Our approach (RFBC) performs well, achieving 77% in the multistep case.
Revised: Our approach (RFBC) performs well, achieving 77% in the multistep case.

Original: Performance also drops as the distance to the target increases.
Revised: Performance also drops as the distance to the target increases.

Original: In contrast, the RL agent performs poorly (around 40 % on navigation tasks).
Revised: In contrast, the RL agent performs poorly (around 40 % on navigation tasks).

Original: Because we generate virtually infinite expert trajectories with RFBC, RL performs worse as it is trained with essentially the same amount of exploration, but with a more sparse signal than behavioral cloning.
Revised: Because we generate virtually infinite expert trajectories with RFBC, RL performs worse as it is trained with essentially the same amount of exploration, but with a more sparse signal than behavioral cloning.

Original: Using random features in place of ϕ ( . ) caused the performance to drop to near chance, showing that the models are utilizing the semantics at each node and are not relying on a general search strategy of some kind.
Revised: Using random features in place of ϕ ( . ) caused the performance to drop to near chance, showing that the models are utilizing the semantics at each node and are not relying on a general search strategy of some kind.

Original: Random methods also do quite poorly, even when adding a DFS.
Revised: Random methods also do quite poorly, even when adding a DFS.

Original: Greedy methods do a bit better than random, especially with DFS, but still well below our method.
Revised: Greedy methods do a bit better than random, especially with DFS, but still well below our method.

Original: We also consider the more challenging sentence search task described in Section 4.1.
Revised: We also consider the more challenging sentence search task described in Section 4.1.

Original: This task requires not only finding a known target node, but learning to find a target node given only a single random sentence from that node, more closely matching our downstream task and the way humans tend to approach finding information on the web.
Revised: This task requires not only finding a known target node, but learning to find a target node given only a single random sentence from that node, more closely matching our downstream task and the way humans tend to approach finding information on the web.

Original: Columns 5-7 of Table 2 show performance on this task for our approach and RL.
Revised: Columns 5-7 of Table 2 show performance on this task for our approach and RL.

Original: Our method has reduced performance on the task due to the extra difficulty of learning the target embedding, but still performs reasonably.
Revised: Our method has reduced performance on the task due to the extra difficulty of learning the target embedding, but still performs reasonably.

Original: RL, however, degrades to near chance performance.
Revised: RL, however, degrades to near chance performance.

Revised: [Table]

Revised: [Table]

Original: 5.3 Navigation on full 38M node Wikipedia graph:
Revised: 5.3 Navigation on full 38M node Wikipedia graph:

Original: Our method scales naturally to much larger graphs, which we demonstrate by training on the entire
Revised: Our method scales naturally to much larger graphs, which we demonstrate by training on the entire

Original: Wikipedia 2017 graph.
Revised: Wikipedia 2017 graph.

Original: We evaluate on the entire Wikipedia 2018 graph.
Revised: We evaluate on the entire Wikipedia 2018 graph.

Original: Each graph is ∼ 38 M nodes and ∼ 380 M edges (see Table 3 for details).
Revised: Each graph is ∼ 37 M nodes and ∼ 370 M edges (see Table 3 for details).

Original: [Table] [Table]
Revised: We would like to point out there is significant evolution (difference) between the 2017 and 2018 graph as analysed in Appendix A.3 along with further statistics.

Original: We explore both architectures described in Section 4.2 for the policy network: (i) the single feedforward layer (as used in the smaller graph) and (ii) the 4-layer Transformer model.
Revised: We explore both architectures described in Section 4.2 for the policy network: (i) the single feedforward layer (as used in the smaller graph) and (ii) the 4-layer Transformer model.

Original: We also compare using a fixed RoBERTA model for ϕ versus pre-training a text transformer directly on the navigation task (see Appendix B.1).
Revised: We also compare using a fixed RoBERTA model for ϕ versus pre-training a text transformer directly on the navigation task (see Appendix B.3).

Original: DistillBERT Sanh et al. [2019] is used for ϕ ( n g ) .
Revised: DistillBERT [Sanh et al., 2019] is used for ϕ ( n g ) .

Original: Table 4 shows the results on the full graph, evaluating on { 5 , 10 , 20 } step tasks, for both navigation and sentence search.
Revised: Table 4 shows the results on the full graph, evaluating on { 5 , 10 , 20 } step tasks, for both navigation and sentence search.

Original: Figure 2 shows example trajectories of the trained agent.
Revised: Figure 2 shows example trajectories of the trained agent.

Original: The first thing to note is that the overall navigation and sentence search performance of our methods is high.
Revised: The first thing to note is that the overall navigation and sentence search performance of our methods is high.

Original: The
Revised: The

Original: Wikipedia graph we use for evaluation contains over 5 million articles, over 38 million nodes and 387 million edges.
Revised: Wikipedia graph we use for evaluation contains over 5 million articles, over 38 million nodes and 387 million edges.

Original: On this graph (which we generalize to from the previous year’s graph) our best method can find an article 20 steps away 92 .
Revised: On this graph (which we generalize to from the previous year’s graph) our best method can find an article 20 steps away 92 .

Original: 2% percent of the time and can locate it given just a single sentence of that article 90 .
Revised: 2% percent of the time and can locate it given just a single sentence of that article 90 .

Original: 2% of the time.
Revised: 2% of the time.

Original: This compares favorably with a similar evaluation performed in Nogueira and Cho [2016], where their WebNav approach achieved a 12 . 5% success rate for 16 step tasks on a 12M node graph.
Revised: This compares favorably with a similar evaluation performed in Nogueira and Cho [2016], where their WebNav approach achieved a 12 . 5% success rate for 16 step tasks on a 12M node graph.

Original: Compared to the performance of our method on the smaller 200 k graph, we see that performance has greatly improved, most likely due to the addition of orders of magnitude more training data.
Revised: Compared to the performance of our method on the smaller 200 k graph, we see that performance has greatly improved, most likely due to the addition of orders of magnitude more training data.

Original: In particular we see that sentence search performance on 20 -step improves greatly from 34 .
Revised: In particular we see that sentence search performance on 20 -step improves greatly from 34 .

Original: 6% to a best performance of 90 . 2% .
Revised: 6% to a best performance of 90 . 2% .

Original: Additionally, in all cases, our learned embeddings perform better than using the fixed features, derived from large language models.
Revised: Additionally, in all cases, our learned embeddings perform better than using the fixed features, derived from large language models.

Original: In general, feed-forward policies perform best, except forstep navigation tasks.
Revised: In general, feed-forward policies perform best, except forstep navigation tasks.

Original: For longer trajectories, the Transformer policy network achieves slightly better performance, while on sentence search, the feed-forward model is consistently superior. 
Revised: For longer trajectories, the Transformer policy network achieves slightly better performance, while on sentence search, the feed-forward model is consistently superior. [Table]

Revised: Embed train (ours) + Transformer 93.

Original: 5.4 Application to fact verification
Revised: 5.4 Application to fact verification

Original: We now evaluate the ability of our navigation approach to select evidence on the FEVER development set Thorne et al. [2018] with results shown in Table 5.
Revised: We now evaluate the ability of our navigation approach to select evidence on the FEVER development set [Thorne et al., 2018] with results shown in Table 5.

Original: The primary metric used is F1@top5 but we also give precision and recall.
Revised: The primary metric used is F1@top5 but we also give precision and recall.

Original: Paired with a basic retrieval (BM25) and re-ranker (TF-IDF), the approach obtains a respectable F1 of 0.46.
Revised: Paired with a basic retrieval (BM25) and re-ranker (TF-IDF), the approach obtains a respectable F1 of 0.46.

Original: If we remove our navigation component then this drops substantially to 0.36, demonstrating its utility.
Revised: If we remove our navigation component then this drops substantially to 0.36, demonstrating its utility.

Original: Swapping to a more powerful re-ranker (BigBird) 
Revised: Swapping to a more powerful re-ranker (BigBird) boosts the F1 to 0.731, which is competitive with the state-of-the-art (FEVER leaderboard [2022] rank #6; rank #1 F1 is 0.799). We also compare to a leading approach [Stammbach, 2021] that similarly relies on the BigBird re-ranker, with our approach having a significantly better F1 score.

Original: [Table]
Revised: [Table]

Original: Embed train (ours) + Feed-forward 96. Embed train (ours) + Transformer 93.
Revised: (cid:19)(cid:29)(cid:3)(cid:57)(cid:79)(cid:68)(cid:71)(cid:76)(cid:80)(cid:76)(cid:85)(cid:3)(cid:51)(cid:68)(cid:71)(cid:90)(cid:68)(cid:3)(cid:11)(cid:24)(cid:12)(cid:20)(cid:29)(cid:3)(cid:51)(cid:79)(cid:68)(cid:81)(cid:21)(cid:29)(cid:3)(cid:37)(cid:88)(cid:86)(cid:76)(cid:81)(cid:72)(cid:86)(cid:86)(cid:22)(cid:29)(cid:3)(cid:51)(cid:85)(cid:82)(cid:71)(cid:88)(cid:70)(cid:87)(cid:3)(cid:11)(cid:69)(cid:88)(cid:86)(cid:76)(cid:81)(cid:72)(cid:86)(cid:86)(cid:12)(cid:23)(cid:29)(cid:3)(cid:54)(cid:92)(cid:86)(cid:87)(cid:72)(cid:80)(cid:24)(cid:29)(cid:3)(cid:54)(cid:92)(cid:86)(cid:87)(cid:72)(cid:80)(cid:3)(cid:11)(cid:20)(cid:12)(cid:25)(cid:29)(cid:3)(cid:54)(cid:92)(cid:86)(cid:87)(cid:72)(cid:80)(cid:3)(cid:11)(cid:21)(cid:12)(cid:26)(cid:29)(cid:3)(cid:37)(cid:76)(cid:82)(cid:79)(cid:82)(cid:74)(cid:76)(cid:86)(cid:87)(cid:27)(cid:29)(cid:3)(cid:47)(cid:76)(cid:73)(cid:72)(cid:28)(cid:29)(cid:3)(cid:38)(cid:72)(cid:79)(cid:79)(cid:3)(cid:86)(cid:76)(cid:74)(cid:81)(cid:68)(cid:79)(cid:76)(cid:81)(cid:74)(cid:20)(cid:19)(cid:29)(cid:3)(cid:39)(cid:76)(cid:86)(cid:72)(cid:68)(cid:86)(cid:72)(cid:20)(cid:20)(cid:29)(cid:3)(cid:41)(cid:88)(cid:81)(cid:70)(cid:87)(cid:76)(cid:82)(cid:81)(cid:3)(cid:11)(cid:69)(cid:76)(cid:82)(cid:79)(cid:82)(cid:74)(cid:92)(cid:12)(cid:20)(cid:21)(cid:29)(cid:3)(cid:41)(cid:76)(cid:87)(cid:81)(cid:72)(cid:86)(cid:86)(cid:3)(cid:11)(cid:69)(cid:76)(cid:82)(cid:79)(cid:82)(cid:74)(cid:92)(cid:12)(cid:20)(cid:22)(cid:29)(cid:3)(cid:54)(cid:72)(cid:91)(cid:88)(cid:68)(cid:79)(cid:3)(cid:86)(cid:72)(cid:79)(cid:72)(cid:70)(cid:87)(cid:76)(cid:82)(cid:81)(cid:20)(cid:23)(cid:29)(cid:3)(cid:53)(cid:72)(cid:83)(cid:85)(cid:82)(cid:71)(cid:88)(cid:70)(cid:87)(cid:76)(cid:89)(cid:72)(cid:3)(cid:86)(cid:88)(cid:70)(cid:70)(cid:72)(cid:86)(cid:86)(cid:20)(cid:24)(cid:29)(cid:3)(cid:42)(cid:72)(cid:81)(cid:82)(cid:87)(cid:92)(cid:83)(cid:72)(cid:20)(cid:25)(cid:29)(cid:3)(cid:42)(cid:72)(cid:81)(cid:82)(cid:87)(cid:92)(cid:83)(cid:72)(cid:20)(cid:26)(cid:29)(cid:3)(cid:42)(cid:72)(cid:81)(cid:72)(cid:20)(cid:27)(cid:29)(cid:3)(cid:37)(cid:76)(cid:82)(cid:79)(cid:82)(cid:74)(cid:92)(cid:20)(cid:28)(cid:29)(cid:3)(cid:38)(cid:72)(cid:79)(cid:79)(cid:3)(cid:11)(cid:69)(cid:76)(cid:82)(cid:79)(cid:82)(cid:74)(cid:92)(cid:12)(cid:21)(cid:19)(cid:29)(cid:3)(cid:40)(cid:80)(cid:69)(cid:85)(cid:92)(cid:82) (cid:19)(cid:29)(cid:3)(cid:24)(cid:25)(cid:87)(cid:75)(cid:3)(cid:55)(cid:85)(cid:68)(cid:76)(cid:81)(cid:76)(cid:81)(cid:74)(cid:3)(cid:54)(cid:84)(cid:88)(cid:68)(cid:71)(cid:85)(cid:82)(cid:81)(cid:3)(cid:11)(cid:20)(cid:20)(cid:12)(cid:20)(cid:29)(cid:3)(cid:24)(cid:25)(cid:87)(cid:75)(cid:3)(cid:55)(cid:85)(cid:68)(cid:76)(cid:81)(cid:76)(cid:81)(cid:74)(cid:3)(cid:54)(cid:84)(cid:88)(cid:68)(cid:71)(cid:85)(cid:82)(cid:81)(cid:3)(cid:11)(cid:20)(cid:20)(cid:12)(cid:21)(cid:29)(cid:3)(cid:24)(cid:25)(cid:87)(cid:75)(cid:3)(cid:55)(cid:85)(cid:68)(cid:76)(cid:81)(cid:76)(cid:81)(cid:74)(cid:3)(cid:54)(cid:84)(cid:88)(cid:68)(cid:71)(cid:85)(cid:82)(cid:81)(cid:3)(cid:11)(cid:20)(cid:20)(cid:12)(cid:22)(cid:29)(cid:3)(cid:24)(cid:25)(cid:87)(cid:75)(cid:3)(cid:55)(cid:85)(cid:68)(cid:76)(cid:81)(cid:76)(cid:81)(cid:74)(cid:3)(cid:54)(cid:84)(cid:88)(cid:68)(cid:71)(cid:85)(cid:82)(cid:81)(cid:3)(cid:11)(cid:20)(cid:20)(cid:12)(cid:23)(cid:29)(cid:3)(cid:24)(cid:25)(cid:87)(cid:75)(cid:3)(cid:55)(cid:85)(cid:68)(cid:76)(cid:81)(cid:76)(cid:81)(cid:74)(cid:3)(cid:54)(cid:84)(cid:88)(cid:68)(cid:71)(cid:85)(cid:82)(cid:81)(cid:3)(cid:11)(cid:20)(cid:19)(cid:12)(cid:24)(cid:29)(cid:3)(cid:54)(cid:84)(cid:88)(cid:68)(cid:71)(cid:85)(cid:82)(cid:81)(cid:3)(cid:11)(cid:81)(cid:68)(cid:89)(cid:68)(cid:79)(cid:12)(cid:25)(cid:29)(cid:3)(cid:58)(cid:68)(cid:85)(cid:86)(cid:75)(cid:76)(cid:83)(cid:26)(cid:29)(cid:3)(cid:49)(cid:68)(cid:89)(cid:92)(cid:27)(cid:29)(cid:3)(cid:51)(cid:82)(cid:90)(cid:72)(cid:85)(cid:3)(cid:83)(cid:85)(cid:82)(cid:77)(cid:72)(cid:70)(cid:87)(cid:76)(cid:82)(cid:81)(cid:28)(cid:29)(cid:3)(cid:51)(cid:82)(cid:90)(cid:72)(cid:85)(cid:3)(cid:83)(cid:85)(cid:82)(cid:77)(cid:72)(cid:70)(cid:87)(cid:76)(cid:82)(cid:81)(cid:20)(cid:19)(cid:29)(cid:3)(cid:54)(cid:82)(cid:89)(cid:72)(cid:85)(cid:72)(cid:76)(cid:74)(cid:81)(cid:3)(cid:86)(cid:87)(cid:68)(cid:87)(cid:72)(cid:20)(cid:20)(cid:29)(cid:3)(cid:38)(cid:72)(cid:81)(cid:87)(cid:85)(cid:68)(cid:79)(cid:76)(cid:86)(cid:68)(cid:87)(cid:76)(cid:82)(cid:81)(cid:20)(cid:21)(cid:29)(cid:3)(cid:41)(cid:82)(cid:85)(cid:80)(cid:86)(cid:3)(cid:82)(cid:73)(cid:3)(cid:74)(cid:82)(cid:89)(cid:72)(cid:85)(cid:81)(cid:80)(cid:72)(cid:81)(cid:87)(cid:20)(cid:22)(cid:29)(cid:3)(cid:39)(cid:72)(cid:73)(cid:76)(cid:81)(cid:76)(cid:87)(cid:76)(cid:82)(cid:81)

Original: [Table]
Revised: [Table]

Original: boosts the F1 to 0.731, which is competitive with the state-of-the-art (FEVER leaderboard [2022] rank #6; rank #1 F1 is 0.799). We also compare to a leading approach Stammbach [2021] that similarly relies on the BigBird re-ranker, with our approach having a significantly better F1 score.
Revised: [Figure] [Figure]

Original:  A possible explanation is that their approach overwhelms the re-ranker with all possible hyperlinks whereas, navigation being inherently sparse, our model presents a more refined set for re-ranking.
Revised: [Table] A possible explanation is that their approach overwhelms the re-ranker with all possible hyperlinks whereas, navigation being inherently sparse, our model presents a more refined set for re-ranking.

Original: We note that our approach: (a) is significantly simpler than many top ranked approaches and (b) selects evidence over a much larger set than the curated version of Wikipedia used in FEVER (38M vs 5M).
Revised: We note that our approach: (a) is significantly simpler than many top ranked approaches and (b) selects evidence over a much larger set than the curated version of Wikipedia used in FEVER (38M vs 5M).

Original: Examples of successful navigation traces are shown in Figure 3.
Revised: Examples of successful navigation traces are shown in Figure 3.

Original: [Table] [Figure] [Figure] [Table]
Revised: 5.5 Application to question answering

Original: (cid:19)(cid:29)(cid:3)(cid:55)(cid:75)(cid:72)(cid:3)(cid:59)(cid:16)(cid:41)(cid:76)(cid:79)(cid:72)(cid:86)(cid:3)(cid:11)(cid:27)(cid:23)(cid:12)(cid:20)(cid:29)(cid:3)(cid:43)(cid:88)(cid:80)(cid:69)(cid:88)(cid:74)(cid:3)(cid:11)(cid:55)(cid:75)(cid:72)(cid:3)(cid:59)(cid:16)(cid:41)(cid:76)(cid:79)(cid:72)(cid:86)(cid:12)(cid:21)(cid:29)(cid:3)(cid:43)(cid:88)(cid:80)(cid:69)(cid:88)(cid:74)(cid:3)(cid:11)(cid:55)(cid:75)(cid:72)(cid:3)(cid:59)(cid:16)(cid:41)(cid:76)(cid:79)(cid:72)(cid:86)(cid:12)
Revised: Finally

Original: [Table]
Revised: [Table]

Original: 5.5 Application to Question Answering

Original: Finally, we present results on another downstream task where navigation helps.
Revised: we present results on another downstream task where navigation helps.

Original: In particular, we consider the task of finding correct evidence passage for open domain question answering.
Revised: In particular, we consider the task of finding correct evidence passage for open domain question answering.

Original: We use the Natural Question (NQ) open-domain dataset presented in Lee et al.
Revised: We use the Natural Question (NQ) open-domain dataset presented in Lee et al.

Original: [2019] which has been aligned to Wikipedia dump of December 20, 2018 by [Karpukhin et al., 2020].

Original: Most prior works in literature utilize the processed collection of 21M passages provided by by [Karpukhin et al., 2020], but unfortunately we cannot use it as it has no hyperlink information which is crucial for our graph building.

Original: So we re-align the questions in NQ to our 2018 graph with 38M passages (i.e. nodes).

Original: We compute the shortest-path distance between top-1 retrieval of BM25 and the target node in this graph in Figure 10.

Original: As can be seen, most path lengths are relatively short, which implies that BM25 lands us in the right vicinity and by a small amount of navigation around we will be able to find the right evidence passage.

Original: Since we target navigating to the exact evidence passage required to answer the question, we use recall@{1,2,3,4,5} for finding the gold evidence passage as our metric.
Revised: Since we target navigating to the exact evidence passage required to answer the question, we use recall@{1,2,3,4,5} for finding the gold evidence passage as our metric.

Original: More commonly the metric marks a retrieved passage to be correct if it contains the answer string, but this causes a lot of false positives, e.g. the answer string appears in a totally irrelevant context.
Revised: More commonly the metric marks a retrieved passage to be correct if it contains the answer string, but this causes a lot of false positives, e.g. the answer string appears in a totally irrelevant context.

Original: Our method is similar as to our approach to FEVER.

Original: For each question, we first run BM25 over all nodes in the graph and take the top-5 matches as starting nodes.

Original: From each starting node, the agent then navigates for 20 steps.

Original: All the 100 nodes visited by the agent is then ranked by a simple cross-attention model.

Original: For the cross-attention model, we follow Hofstätter et al.

Original: [2020] to train 6-layer

Original: BigBird [Zaheer et al., 2020].

Original: [Table]
Revised: [Table]

Original: The results are tabulated in Table 6.
Revised: The results are tabulated in Table 6.

Original: Paired with a basic retrieval (BM25) and re-ranker (BigBird), the approach obtains a respectable Recall@1 of 31.6.
Revised: Paired with a basic retrieval (BM25) and re-ranker (BigBird), the approach obtains a respectable Recall@1 of 31.6.

Original: If we remove our navigation component then this drops substantially to 22.2, demonstrating its utility.
Revised: If we remove our navigation component then this drops substantially to 22.2, demonstrating its utility.

Original: As a reference, we also ran a state-of-the-art system RocketQA
Revised: As a reference, we also ran a state-of-the-art system RocketQA

Original: [Qu et al., 2021] on our setup and evaluated it using our harder metric.
Revised: [Qu et al., 2021] on our setup and evaluated it using our harder metric.

Original: It is worthwhile to note that our approach (a) is significantly simpler than state-of-the-art approaches like RocketQA which employ 4 stages of dual-encoder and cross-attention models [Table] and (b) selects evidence over a larger set than the commonly used preprocess Wikipedia passages (38M vs 21M).
Revised: It is worthwhile to note that our approach (a) is significantly simpler than state-of-the-art approaches like RocketQA which employ 4 stages of dual-encoder and cross-attention models, and (b) selects evidence over a larger set than the commonly used preprocess Wikipedia passages (38M vs 21M).

Original:  Discussion
Revised:  Discussion

Original: We have presented a simple and effective scheme for navigating a large Wikipedia graph that is applicable to more general web navigation problems.
Revised: We have presented a simple and effective scheme for navigating a large Wikipedia graph that is applicable to more general web navigation problems.

Original: We show that behavioral cloning of random trajectories is a viable approach to learning both entity embeddings and a navigation policy.
Revised: We show that behavioral cloning of random trajectories is a viable approach to learning both entity embeddings and a navigation policy.

Original: When applied to the FEVER verification task and the Open-Ended Natural Questions dataset, they offer highly competitive performance, whilst being complementary to existing approaches.
Revised: When applied to the FEVER verification task and the Open-Ended Natural Questions dataset, they offer highly competitive performance, whilst being complementary to existing approaches.

Original: One limitation of our approach when we move from Wikipedia to the wider Internet is that our scheme relies on a good target encoder.
Revised: One limitation of our approach when we move from Wikipedia to the wider Internet is that our scheme relies on a good target encoder.

Original: For navigation and our downstream tasks, there was a clear ground-truth target node available training, but in other settings this might not be the case.
Revised: For navigation and our downstream tasks, there was a clear ground-truth target node available training, but in other settings this might not be the case.

Original: Another limitation is that we require a re-ranker to decide on the final retrieved sentence, but ideally the agent would decide for itself when it has reached the correct node.
Revised: Another limitation is that we require a re-ranker to decide on the final retrieved sentence, but ideally the agent would decide for itself when it has reached the correct node.

Original: Checklist

Original: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?

Original: [Yes] (b) Did you describe the limitations of your work?

Original: [Yes] (c) Did you discuss any potential negative societal impacts of your work?

Original: [N/A] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?

Original: [Yes] 2.

Original: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?

Original: [N/A] (b) Did you include complete proofs of all theoretical results?

Original: [N/A] 3.

Original: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Original: [Yes] The paper and supplemental provide full details on how to reproduce the simple approach.

Original: The datasets used are all public.

Original: (b) Did you specify all the training details (e

Original: g., data splits, hyperparameters, how they were chosen)?

Original: [Yes] In appendix.

Original: (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Original: [N/A] Our experiments are on large datasets and thus did not require multiple runs.

Original: (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?

Original: [No] The experiments were run on a proprietary infrastructure on which it is hard to estimate the total amount of compute.

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?

Original: [Yes] (b) Did you mention the license of the assets?

Original: [Yes] See Thorne et al.

Original: reference.

Original: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [No] (d) Did you discuss whether

Original: and how consent was obtained from people whose data you’re using/curating?

Original: [N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?

Original: [N/A] 5.

Original: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?

Original: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?

Original: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?

Original: [N/A]

Original: A Dataset / Environment Details A.1 Wikipedia data processing
Revised: A Dataset / Environment details A.1 Wikipedia data processing

Original: We start by downloading English Wikipedia snapshot (the pages-articles.xml.bz2 file) from Wikimedia 3 or Internet Archive.
Revised: We start by downloading English Wikipedia snapshot (the pages-articles.xml.bz2 file) from Wikimedia 3 or Internet Archive.

Original: 4 We extract text from English Wikipedia for a given snapshot using
Revised: 4 We extract text from English Wikipedia for a given snapshot using

Original: Gensim’s WikiCorpus class.
Revised: Gensim’s WikiCorpus class.

Original: For each page, this tool extracts the plain text and hyperlinks, and strips out all structured data such as lists and figures.
Revised: For each page, this tool extracts the plain text and hyperlinks, and strips out all structured data such as lists and figures.

Original: To avoid pages without sufficient textual content, we filter out categories, listical, disambiguation pages and any other page with less than 200 characters.
Revised: To avoid pages without sufficient textual content, we filter out categories, listical, disambiguation pages and any other page with less than 200 characters.

Original: Each article ( µ = 1000 words) is split into paragraph-sized blocks ( µ = 100 words), which form the nodes in our Wikipedia Graph.
Revised: Each article ( µ = 1000 words) is split into paragraph-sized blocks ( µ = 100 words), which form the nodes in our Wikipedia Graph.

Original: This was done both for conceptual reasons (humans don’t read an entire article at once but look at bits and pieces of it at one go) and for modeling reasons (most language models have an upper input token length that is smaller than the average Wikipedia page).
Revised: This was done both for conceptual reasons (humans don’t read an entire article at once but look at bits and pieces of it at one go) and for modeling reasons (most language models have an upper input token length that is smaller than the average Wikipedia page).

Original: The edges in the graph are formed in three ways:
Revised: The edges in the graph are formed in three ways:

Original: 1. for nodes in the same article, we add “previous node” and “next node” links to connect them in a chain; 2.
Revised: 1. for nodes in the same article, we add “previous node” and “next node” links to connect them in a chain; 2.

Original: organic hyperlinks – the internal links connecting Wikipedia pages to each other; and 3.
Revised: organic hyperlinks – the internal links connecting Wikipedia pages to each other; and 3.

Original: we additionally run entity linking 6 because repeat mentions of an entity in Wikipedia lack hyperlinks, which is problematic when articles are split up.
Revised: we additionally run entity linking 6 because repeat mentions of an entity in Wikipedia lack hyperlinks, which is problematic when articles are split up.

Original: This constitutes our Wikipedia graph construction, where the text blocks act as vertices and the organic hyperlinks along with entity links are the edges.
Revised: This constitutes our Wikipedia graph construction, where the text blocks act as vertices and the organic hyperlinks along with entity links are the edges.

Original: We store the graph as an adjacency list with metadata, using a memory mapped key-value data-structure to enable fast random access during navigation.
Revised: We store the graph as an adjacency list with metadata, using a memory mapped key-value data-structure to enable fast random access during navigation.

Original: We run the above graph generation pipeline for two different snapshots of Wikipedia:
Revised: We run the above graph generation pipeline for two different snapshots of Wikipedia:

Original: June 01 2017 7 : This corresponds to FEVER fact verification dataset. 2.
Revised: June 01 2017 7 : This corresponds to FEVER fact verification dataset. 2.

Original: December 2018 8 : This corresponds to Natural Questions open dataset.
Revised: December 2018 8 : This corresponds to Natural Questions open dataset.

Original: A.2 Smaller 200k node graph construction
Revised: A.2 Smaller 200k node graph construction

Original: We construct smaller train and eval graphs with 200k nodes each for the following reasons:
Revised: We construct smaller train and eval graphs with 200k nodes each for the following reasons:

Original: Checking strict generalization by making train and eval graphs to be totally disjoint, i.e. no common nodes or edges.
Revised: Checking strict generalization by making train and eval graphs to be totally disjoint, i.e. no common nodes or edges.

Original: The full 2017 and 2018 are sufficiently different but not entirely disjoint.
Revised: The full 2017 and 2018 are sufficiently different but not entirely disjoint.

Original: Running experiments on full graphs are expensive, so for more thorough evaluation across multiple baselines we constructed the smaller graphs.
Revised: Running experiments on full graphs are expensive, so for more thorough evaluation across multiple baselines we constructed the smaller graphs.

Original: To construct these two smaller graphs, we start with the full 2018 graph.
Revised: To construct these two smaller graphs, we start with the full 2018 graph.

Original: We sort the nodes by their in-degrees.
Revised: We sort the nodes by their in-degrees.

Original: We mark nodes with odd ranks to be in train and even ranks to be in eval, which ensures complete separation between the two.
Revised: We mark nodes with odd ranks to be in train and even ranks to be in eval, which ensures complete separation between the two.

Original: For the train graph we start with rank 1 node (and for eval with rank 2 node) and then select all its neighbors which have odd rank (even rank).
Revised: For the train graph we start with rank 1 node (and for eval with rank 2 node) and then select all its neighbors which have odd rank (even rank).

Original: Then all the odd (even) nodes connected to the selected ones are picked.
Revised: Then all the odd (even) nodes connected to the selected ones are picked.

Original: This process is continued until the desired number of nodes are selected.
Revised: This process is continued until the desired number of nodes are selected.

Original: A.3 Dataset statistics
Revised: A.3 Dataset statistics

Original: We construct two versions of the full Wikipedia graph following the method outlined in Sec. A.1.
Revised: We construct two versions of the full Wikipedia graph following the method outlined in Appendix A.1.

Original: The main summary statistics (number of articles, nodes, edges, words per node, median path length) of the resulting graphs are listed in Table 3.
Revised: The main summary statistics (number of articles, nodes, edges, words per node, median path length) of the resulting graphs are listed in Table 3.

Original: A more detailed picture of the graph can be obtained by looking at its degree distribution, which is presented in Figure 4.
Revised: A more detailed picture of the graph can be obtained by looking at its degree distribution, which is presented in Figure 4.

Original: It shows the graph has tell-tale sign of web-graph: it has a power-law distribution with a few hub nodes
Revised: It shows the graph has tell-tale sign of web-graph: it has a power-law distribution with a few hub nodes

Original: [Figure]
Revised: [Figure]

Original: We try to estimate how far are two nodes in the graph.
Revised: We try to estimate how far are two nodes in the graph.

Original: Calculating the entire distribution of shortest path length is prohibitively expensive for such large graph.
Revised: Calculating the entire distribution of shortest path length is prohibitively expensive for such large graph.

Original: We follow Ye et al.
Revised: We follow Ye et al.

Original: [2010] method to get an approximation.
Revised: [2010] method to get an approximation.

Original: We start by selecting 250k nodes having highest in-degree, which is a good approximation of the giant component in a web graph as required by Ye et al.
Revised: We start by selecting 250k nodes having highest in-degree, which is a good approximation of the giant component in a web graph as required by Ye et al.

Original: We then perform a single source to all shortest paths on each of these 250k nodes using Dijkstra’s algorithm for sparse graphs.
Revised: We then perform a single source to all shortest paths on each of these 250k nodes using Dijkstra’s algorithm for sparse graphs.

Original: This resulted in computing approximately 1 trillion shortest paths based on which the estimated distribution of shortest path length is shown in Figure
Revised: This resulted in computing approximately 1 trillion shortest paths based on which the estimated distribution of shortest path length is shown in Figure

Original: [Figure]
Revised: [Figure]

Original: c t i o n o f N o d e s
Revised: t i o n o f N o d e s

Original: Finally, we would like to point out that there is significant evolution (difference) between the 2017 and 2018 graph.
Revised: Finally, we would like to point out that there is significant evolution (difference) between the 2017 and 2018 graph.

Original: Some important statistics reflecting the changes across the two graphs are:
Revised: Some important statistics reflecting the changes across the two graphs are:

Original: • New articles: 404,773 (7.5%) • New nodes: 3,871,578 (10%) •
Revised: • New articles: 404,773 (7.5%) • New nodes: 3,871,578 (10%) •

Original: Deleted articles: 58,476 (1.1%) • Deleted nodes: 874,372 (2.3%)
Revised: Deleted articles: 58,476 (1.1%) • Deleted nodes: 874,372 (2.3%)

Original: Further in Figure 6, we show histogram of changes in edges among the common nodes between the two graphs.
Revised: Further in Figure 6, we show histogram of changes in edges among the common nodes between the two graphs.

Original: Apart from addition/deletion of organic hyperlinks, just modifications of text would yield to differences as the chunking will be different.
Revised: Apart from addition/deletion of organic hyperlinks, just modifications of text would yield to differences as the chunking will be different.

Original: This shows significant generalization is needed to successfully navigate across the two graphs, simply memorization will not work.
Revised: This shows significant generalization is needed to successfully navigate across the two graphs, simply memorization will not work.

Original: [Figure]
Revised: [Figure]

Original: A.4 FEVER data processing

Original: FEVER Thorne et al.

Original: [2018] is a dataset containing 185 , 445 claims labeled as S UPPORTED , R EFUTED , or N OT

Original: E NOUGH I NFO .

Original: Verifiable claims (i.e. S UPPORTED or R EFUTED ) are annotated with evidence sentences supporting this classification, which are drawn from a preprocessed version of the

Original: June 2017 Wikipedia snapshot.

Original: Notably, only the article introductions are retained, and claims are generated from a curated set of approximately 50 , 000 popular pages.

Original: To make an aligned navigation graph for FEVER, we start with the June 01 2017 snapshot of

Original: Wikipedia and build the graph using the procedure described in the previous section.

Original: Because of differences in text preprocessing, FEVER’s version of Wikipedia does not precisely align to our own.

Original: We then try to match FEVER’s Wikipedia articles to our graph, relying on Wikipedia URL redirections for handling disambiguations.

Original: This resulted in 99 .

Original: 5% matching of articles between our navigation graph and the FEVER version of Wikipedia.

Original: Second, we align the evidence sentences in

Original: FEVER to sentences from our text blocks using a fuzzy string match.

Original: Specifically, we use the token set ratio to score the similarity between an evidence sentence and the text in a graph node.

Original: If the score between a sentence and a node is > = 80 , the node is added as an evidence node for the given claim.

Original: This threshold is sufficiently high to minimize the chances of a false positive, while also matching a high percentage of evidence sentences.

Original: Some sentences are difficult to match, particularly those that are split between two text blocks.

Original: Ultimately, 93 .

Original: 1% of all evidence sentences were matched to a node.

Original: This gives us an augmented FEVER dataset, where for each claim we have a corresponding set of evidence nodes in the graph.

Original: These nodes are then used as navigation targets for fine-tuning during the training phase.

Original: We generate trajectories for BC by running BM25 over all nodes in the graph, taking the top 10 matches as starting nodes, and finding shortest paths to evidence nodes.

Original: We compute the shortest-path distance between top-1 retrieval of BM25 and target node in this graph in Figure 7.

Original: As can be seen that most path lengths are relatively short, which implies that BM25 lands us in the right vicinity and by a small amount of navigation around we will be able to find the right evidence passage.

Original: [Figure]

Original: A.5 RL Baseline

Original: For the purposes of the RL baseline, we can view graph navigation as a goal-conditioned MDP.

Original: 9 Our observation is the same as the inputs to the RFBC models.

Original: The reward function R for current node n and goal node n g we can write simply as:

Original: [Equation]

Original: (cid:26)

Original: [Equation]

Original: g 0 otherwise (5)

Original: See Appendix B.2 for model and training details of RL baselines.

Original: B Experimental details B.1 Embedding pretraining
Revised: B Experimental details B.1 Embedding pretraining

Original: This section details our embedding pretraining procedure, as introduced in Section 3.
Revised: This section details our embedding pretraining procedure, as introduced in Section 3.

Original: Model details
Revised: Model details

Original: The transformer is fine-tuned from a pre-trained RoBERTa
Revised: The transformer is fine-tuned from a pre-trained RoBERTa

Original: [Liu et al., 2019] model.
Revised: [Liu et al., 2019] model.

Original: As in Equation 1 our model is trained to optimize
Revised: As in Equation 1 our model is trained to optimize

Original: [Equation]
Revised: [Equation]

Original: 6) where we set n g = n T .
Revised: (5) where we set n g = n T .

Original: We parameterize this distribution as
Revised: We parameterize this distribution as

Original: [Equation]
Revised: [Equation]

Original:  where s ϕ ( · ) and a ϕ ( · , · ) are functions which extract embeddings for an entire node and a node-action combination, respectively and f θ ( · , · , · ) is a function which combines these embeddings to produce action probabilities.
Revised: (6) where s ϕ ( · ) and a ϕ ( · , · ) are functions which extract embeddings for an entire node and a node-action combination, respectively and f θ ( · , · , · ) is a function which combines these embeddings to produce action probabilities.

Original: At their core, s ϕ and a ϕ are based on the same transformer model.
Revised: At their core, s ϕ and a ϕ are based on the same transformer model.

Original: We tokenize the text at node n to produce L tokens and pass these tokens through the transformer to produce L embeddings.
Revised: We tokenize the text at node n to produce L tokens and pass these tokens through the transformer to produce L embeddings.

Original: The state representation s ϕ ( n ) is produced by taking the mean of these embeddings and passing this vector through a tanh nonlinearity.
Revised: The state representation s ϕ ( n ) is produced by taking the mean of these embeddings and passing this vector through a tanh nonlinearity.

Original: The node-action embedding a ϕ ( n, n ′ ) embeds the action of moving from node n to its neighbor n ′ .
Revised: The node-action embedding a ϕ ( n, n ′ ) embeds the action of moving from node n to its neighbor n ′ .

Original: Let us assume the text at node n is “Barack Obama was the President of the United States during...” where the substrings “Barack Obama” and “President of the United States” correspond to hyperlinks to other neighboring articles n ′ and n ′′ .
Revised: Let us assume the text at node n is “Barack Obama was the President of the United States during...” where the substrings “Barack Obama” and “President of the United States” correspond to hyperlinks to other neighboring articles n ′ and n ′′ .

Original: Then we construct the node-action embedding a ϕ ( n, n ′ ) by passing the tokenized text of node n through our transformer, again producing L transformed token embeddings.
Revised: Then we construct the node-action embedding a ϕ ( n, n ′ ) by passing the tokenized text of node n through our transformer, again producing L transformed token embeddings.

Original: Then we take the transformed tokens which correspond to the text “Barack Obama” and take their mean and pass this through the tanh nonlinearity.
Revised: Then we take the transformed tokens which correspond to the text “Barack Obama” and take their mean and pass this through the tanh nonlinearity.

Original: If we instead wanted to produce a ϕ ( n, n ′′ ) we would take the transformed tokens which correspond to the text “President of the United States” instead, take the mean and apply the nonlinearity.
Revised: If we instead wanted to produce a ϕ ( n, n ′′ ) we would take the transformed tokens which correspond to the text “President of the United States” instead, take the mean and apply the nonlinearity.

Original: This is visualized in Figure 8.
Revised: This is visualized in Figure 7.

Original: The function f θ which combines these embeddings first concatenates them and passes the combined input through a simple neural network with a single hidden layer and 1 output 
Revised: The function f θ which combines these embeddings first concatenates them and passes the combined input through a simple neural network with a single hidden layer and 1 output neuron.

Original: Training details We train these models to optimize Equation 6.
Revised: Training details We train these models to optimize Equation 5.

Original: The objective is optimized with the Adam [Kingma and Ba, 2014] optimizer using an initial learning rate of 1 e − 5 with a one-cycle cosine decay schedule and linear warm up of 10 K steps.
Revised: The objective is optimized with the Adam [Kingma and Ba, 2014] optimizer using an initial learning rate of 10 − 5 with a one-cycle cosine decay schedule and linear warm up of 10 K steps.

Original: The training batch size is 512 and the model is trained over 3 M iterations.
Revised: The training batch size is 512 and the model is trained over 3 M iterations.

Original: Node text is clipped (or padded) to 200 tokens.
Revised: Node text is clipped (or padded) to 200 tokens.

Original: The weights of the transformer are initialized from a pretrained RoBERTa
Revised: The weights of the transformer are initialized from a pretrained RoBERTa

Original: [Liu et al., 2019] masked language model.
Revised: [Liu et al., 2019] masked language model.

Original: B.2 Small graph experiments
Revised: B.2 Small graph experiments

Original: Model details In the small graph experiments, we use the simple policy network FF as described in Section 4.2.
Revised: Model details In the small graph experiments, we use the simple policy network FF as described in Section 4.2.

Original: As discussed, we use a 1-layer MLP (meaning a single learned linear layer with no activation function).
Revised: As discussed, we use a 1-layer MLP (meaning a single learned linear layer with no activation function).

Original: We use a hidden size of 768 to match the size of ϕ .
Revised: We use a hidden size of 768 to match the size of ϕ .

Original: Because the final logits are the result of an inner product between each state and action embedding of size 768 , to aid network training, we add a normalization layer (jax.nn.normalize) before the inner product.
Revised: Because the final logits are the result of an inner product between each state and action embedding of size 768 , to aid network training, we add a normalization layer (jax.nn.normalize) before the inner product.

Original: For the action embedding a i = ϕ
Revised: For the action embedding a i = ϕ

Original: ( s i ) we also concatenate a 1-hot vector that represents the action type (e.g. whether it was a link action, or a next action, or a prev action).
Revised: ( s i ) we also concatenate a 1-hot vector that represents the action type (e.g. whether it was a link action, or a next action, or a prev action).

Original: We also concatenate another bit that indicates whether the state s i of the particular action has been visited before.
Revised: We also concatenate another bit that indicates whether the state s i of the particular action has been visited before.

Original: For the purposes of the RL baseline, we can view graph navigation as a goal-conditioned MDP.
Revised: For the purposes of the RL baseline, we can view graph navigation as a goal-conditioned MDP.

Original: 10 Our observation is the same as the inputs to the RFBC models.
Revised: 9 Our observation is the same as the inputs to the RFBC models.

Original: The reward function R for current node n and goal node n g we can write simply as:
Revised: The reward function R for current node n and goal node n g we can write simply as:

Original: [Equation]
Revised: [Equation]

Original: For the RL baseline, we used IMPALA [Espeholt et al., 2018], which is a kind of policy gradient approach, and thus has a policy network π and a “baseline” or value function network.
Revised: For the RL baseline, we used IMPALA [Espeholt et al., 2018], which is a kind of policy gradient approach, and thus has a policy network π and a “baseline” or value function network.

Original: Since the policy network has the same output space as our RFBC models, we use an identical architecture for this.
Revised: Since the policy network has the same output space as our RFBC models, we use an identical architecture for this.

Original: For the value network, we use a very similar architecture.
Revised: For the value network, we use a very similar architecture.

Original: We again use ϕ as our encoding of the current node and the target node, again concatenate them together.
Revised: We again use ϕ as our encoding of the current node and the target node, again concatenate them together.

Original: We then feed this through a 1-layer MLP to compute the value.
Revised: We then feed this through a 1-layer MLP to compute the value.

Original: Training details The procedure for generating the trajectories we use for the BC loss is described in
Revised: Training details The procedure for generating the trajectories we use for the BC loss is described in

Original: Section 3.1.
Revised: Section 3.1.

Original: Except for our experiment in Section 5.1, we always use the random forward trajectories.
Revised: Except for our experiment in Section 5.1, we always use the random forward trajectories.

Original: Following Lin et al.
Revised: Following Lin et al.

Original: [2018], in the navigation experiments, we randomly drop out edges in the training graph with probability 0 .
Revised: [2018], in the navigation experiments, we randomly drop out edges in the training graph with probability 0 .

Original: 5 to reduce over-fitting.
Revised: 5 to reduce over-fitting.

Original: As stated in Section 4.1, we set max steps to B = 100 .
Revised: As stated in Section 4.1, we set max steps to B = 100 .

Original: For RFBC training on the small graph, we use RMSProp with a learning rate of 0 .
Revised: For RFBC training on the small graph, we use RMSProp with a learning rate of 0 .

Original: 01 , a decay of 0 .
Revised: 01 , a decay of 0 .

Original: 9 an epsilon of 1 e − 10 .
Revised: 9 an epsilon of 10 − 10 .

Original: For the sentence search tasks, our models train a separate ϕ for the target embedding.
Revised: For the sentence search tasks, our models train a separate ϕ for the target embedding.

Original: As we stated, we use MiniBERT for this embedding ϕ target and train it with the same optimization settings, except that we reduce the learning rate for these weights to 1 e − 4 .
Revised: As we stated, we use MiniBERT for this embedding ϕ target and train it with the same optimization settings, except that we reduce the learning rate for these weights to 10 − 4 .

Original: We use a batch size of 512 and we train for 50 , 000 network update steps.
Revised: We use a batch size of 512 and we train for 50 , 000 network update steps.

Original: For RL training, we use VTRACE [Espeholt et al., 2018] loss, using the reward above.
Revised: For RL training, we use VTRACE [Espeholt et al., 2018] loss, using the reward above.

Original: We again use RMSProp with a learning rate of 0 .
Revised: We again use RMSProp with a learning rate of 0 .

Original: 01 , a decay of 0 .
Revised: 01 , a decay of 0 .

Original: 9 an epsilon of 1 e − 10 .
Revised: 9 an epsilon of 10 − 10 .

Original: We set the baseline cost to 0 .
Revised: We set the baseline cost to 0 .

Original: 5 , trajectory length (the number of timesteps the RL loss backprops through) to 100 , batch size again to 512 and max update steps to 50 , 000 .
Revised: 5 , trajectory length (the number of timesteps the RL loss backprops through) to 100 , batch size again to 512 and max update steps to 50 , 000 .

Original: We did a parameter sweep for entropy costs of 0 . 1 , 0 . 01 , 0 . 001 and gamma of 0 . 8 , 0 . 9 .
Revised: We did a parameter sweep for entropy costs of 0 . 1 , 0 . 01 , 0 . 001 and gamma of 0 . 8 , 0 . 9 .

Original: To give RL the best chance possible, we choose the maximum over this sweep for each experiment (but still none of these match RFBC performance).
Revised: To give RL the best chance possible, we choose the maximum over this sweep for each experiment (but still none of these match RFBC performance).

Original: B.3 Full Wikipedia navigation experiments
Revised: B.3 Full Wikipedia navigation experiments

Original: On the full Wikipedia training experiments in Section 5.3, we evaluate on navigation and sentence search using either our feed-forward or transformer model and using either RoBERTA fixed embeddings, or our trained embeddings described in Appendix B.1.
Revised: On the full Wikipedia training experiments in Section 5.3, we evaluate on navigation and sentence search using either our feed-forward or transformer model and using either RoBERTA fixed embeddings, or our trained embeddings described in Appendix B.1.

Original: The feed-forward model is identical to the one described in Appendix B.2.
Revised: The feed-forward model is identical to the one described in Appendix B.2.

Original: The transformer model is the standard transformer model from Vaswani et al.
Revised: The transformer model is the standard transformer model from Vaswani et al.

Original: [2017] with 4 layers, an attention size of 64 ,heads, and mlp hidden size of 3072 and dropout rate of 0 .
Revised: [2017] with 4 layers, an attention size of 64 ,heads, and mlp hidden size of 3072 and dropout rate of 0 .

Original: DistillBERT Sanh et al. [2019] is used for ϕ ( n g ) for the sentence search experiments.
Revised: DistillBERT [Sanh et al., 2019] is used for ϕ ( n g ) for the sentence search experiments.

Original: The learning parameters and all other relevant training parameters for all models are identical to those in Appendix B.2 except that the learning rate for the transformer model is lower ( 1 e − 4 ) as the earlier learning rate was too high for transformer models. 
Revised: The learning parameters and all other relevant training parameters for all models are identical to those in Appendix B.2 except that the learning rate for the transformer model is lower ( 10 − 4 ) as the earlier learning rate was too high for transformer models. B. 4 RFBC + RL

Revised: We additionally try finetuning an RL policy starting from our RFBC-trained navigation policies.

Original: B.4 Downstream experiments
Revised: We use the same network and learning settings as we did in the other RL experiments. We train for an additional 10,000 network updates (512M environment steps). We add the final numbers in Tableand show the training curve in Figure 8. We can see from the training curve that RL training accuracy fluctuates around the point where RFBC training left off.

Revised: From the results in Table 2 we see little statistically significant difference (about 1 point in either direction).

Revised: C Downstream task details C.1 FEVER experiments

Revised: FEVER

Revised: [Thorne et al., 2018] is a dataset containing 185 , 445 claims labeled as S UPPORTED , R EFUTED , or N OT

Revised: E NOUGH I NFO .

Revised: Verifiable claims (i.e. S UPPORTED or R EFUTED ) are annotated with

Revised: [Figure]

Revised: evidence sentences supporting this classification, which are drawn from a preprocessed version of the

Revised: June 2017 Wikipedia snapshot.

Revised: Notably, only the article introductions are retained, and claims are generated from a curated set of approximately 50 , 000 popular pages.

Revised: Data processing To make an aligned navigation graph for FEVER, we start with the June 01snapshot of Wikipedia and build the graph using the procedure described in Section A.1.

Revised: Because of differences in text preprocessing, FEVER’s version of Wikipedia does not precisely align to our own.

Revised: We then try to match FEVER’s Wikipedia articles to our graph, relying on Wikipedia URL redirections for handling disambiguations.

Revised: This resulted in 99 .

Original: Training details For the FEVER benchmark, our model uses learned embeddings for ϕ with a single feed-forward layer for the policy network.
Revised: 5% matching of articles between our navigation graph and the FEVER version of Wikipedia. Second, we align the evidence sentences in FEVER to sentences from our text blocks using a fuzzy string match. Specifically, we use the token set ratio to score the similarity between an evidence sentence and the text in a graph node.

Revised: If the score between a sentence and a node is ≥ 80 , the node is added as an evidence node for the given claim.

Revised: This threshold is sufficiently high to minimize the chances of a false positive, while also matching a high percentage of evidence sentences.

Original: The model is pretrained on the 5 -step sentence search task, and finetuned on trajectories generated from the augmented FEVER dataset described in 
Revised: Some sentences are difficult to match, particularly those that are split between two text blocks. Ultimately, 93 . 1% of all evidence sentences were matched to a node. This gives us an augmented FEVER dataset, where for each claim we have a corresponding set of evidence nodes in the graph.

Original: A.4 using the  the following loss function:
Revised: These nodes are then used as navigation targets for fine-tuning during the training phase. We generate trajectories for BC by running BM25 over all nodes in the graph, taking the top 10 matches as starting nodes, and finding shortest paths to evidence nodes. Data statistics We compute the shortest-path distance between top-1 retrieval of BM25 and target node in this graph in Figure 9.

Original: [Equation]
Revised: As can be seen that most path lengths are relatively short, which implies that BM25 lands us in the right vicinity and by a small amount of navigation around we will be able to find the right evidence passage. Training details For the FEVER benchmark, our model uses learned embeddings for ϕ with a single feed-forward layer for the policy network. The model is pretrained on the 5 -step sentence search task, and finetuned on trajectories generated from the augmented FEVER dataset using the the following loss function:

Original: [Equation]
Revised: [Equation]

Original: where L BC is the normal BC loss we use in our other experiments.
Revised: where L BC is the normal BC loss we use in our other experiments.

Original: Only the ϕ target weights are finetuned for this task.
Revised: Only the ϕ target weights are finetuned for this task.

Original: We use AdamW with a constant learning rate of 1e-6, b1=0.9, b2=0.999 , and eps=1e-6 .
Revised: We use AdamW with a constant learning rate of 10 − 6 , β 1 = 0 . 9 , β 2 = 0 . 999 , and ϵ = 10 − 6 .

Original: The model is finetuned for 100 , 000 update steps with a batch size of 512 .
Revised: The model is finetuned for 100 , 000 update steps with a batch size of 512 .

Original: No edge dropout is used during finetuning.
Revised: No edge dropout is used during finetuning.

Original: Evaluation details For each claim in the FEVER development dataset, we first run BM25 over all nodes in the graph and take the top 5 matches as starting nodes.
Revised: Evaluation details For each claim in the FEVER development dataset, we first run BM25 over all nodes in the graph and take the top 5 matches as starting nodes.

Original: From each node, the finetuned model then navigates for 20 steps.
Revised: From each node, the finetuned model then navigates for 20 steps.

Original: We collect all sentences in all nodes visited by the agent over the 100 total navigation steps, and match them to FEVER evidence sentences using the WRatio function in the fuzzywuzzy package.
Revised: We collect all sentences in all nodes visited by the agent over the 100 total navigation steps, and match them to FEVER evidence sentences using the WRatio function in the fuzzywuzzy package.

Original:  Two sentence ranking methods are explored: Gensim’s TF-IDF implementation, 12 and the BigBird re-ranker of Stammbach [2021], using their open source model and weights.
Revised: 10 Two sentence ranking methods are explored: Gensim’s TF-IDF implementation, 11 and the BigBird re-ranker of Stammbach [2021], using their open source model and weights.

Original: Finally, the top-5 evidence sentences are then submitted to the official FEVER scorerto compute the accuracy, recall, and F1 scores.
Revised: Finally, the top-5 evidence sentences are then submitted to the official FEVER scorerto compute the accuracy, recall, and F1 scores.

Original: C Further experimentsC.1 RFBC + RL
Revised: C.2 NQ experiments

Original: We additionally try finetuning an RL policy starting from our RFBC-trained navigation policies. We use the same network and learning settings as we did in the other RL experiments. We train for an additional 10,000 network updates (512M environment steps).

Original: We add the final numbers in Tableand show the training curve in Figure 9. We can see from  the training curve that RL training accuracy fluctuates around the point where RFBC training left off. From the results in Table 2 we see little statistically significant difference (about 1 point in either direction). C.2 Question Answer Downstream Task
Revised: Natural Questions (NQ) is a dataset collected from real users asking questions on the Google search engine which are answerable using Wikipedia.

Original: Finally, we present results on another downstream task where navigation helps.

Original: In particular, we consider the task of finding correct evidence passage for open domain question answering.

Original: We use the Natural Question(NQ) open-domain dataset presented in Lee et al.
Revised: We use the Natural Question open-domain subset presented in Lee et al.

Original: [2019] which has been aligned to Wikipedia dump of December 20, 2018 by [Karpukhin et al., 2020].
Revised: [2019] which has been aligned to Wikipedia dump of December 20,by Karpukhin et al.

Revised: In this split it has 58,880 questions for training and another 8, questions as development set for evaluation.

Revised: (The test set is not aligned to Wikipedia passages so we do not evaluate on it.)

Original:  Most prior works in literature utilize the processed collection of 21M passages provided by by [Karpukhin et al., 2020], but unfortunately we cannot use it as it has no hyperlink information which is crucial for our graph building.
Revised: Data processing Most prior works in literature utilize the processed collection of 21M passages provided by Karpukhin et al. [2020], but unfortunately we cannot use it as it has no hyperlink information which is crucial for our graph building.

Original: So we re-align the questions in NQ to our 2018 graph with 38M passages (i.e. nodes).
Revised: So we re-align the questions in NQ to ourgraph with 38M passages (i.e. nodes) which has different text blocks than Karpukhin et al. As the article title corresponding to the passage was given as part of the dataset, the alignment task was local to the document. We could find all the article titles, i.e. 100% match in locating the document in our dump. As before, to align the evidence passage in NQ to our text blocks, we first used a fuzzy string match.

Original: We compute the shortest-path distance between top-1 retrieval of BM25 and target node in this graph in
Revised: Then in the ranked list of fuzzy string matches, we searched for exact answer string. The highest ranked node with answer string was labelled as the ground truth. Some answer strings were difficult to match, particularly those that are split between two text blocks. Ultimately, 99 .

Revised: 3% of all evidence passages were matched to a node.

Original:  Figure 10.
Revised: This gives the re-aligned NQ dataset, where for each question we have a corresponding set of evidence nodes in our graph. These nodes are then used as navigation targets for fine-tuning during the training phase. We generate trajectories for BC by running BM25 over all nodes in the graph, taking the top 10 matches as starting nodes, and finding shortest paths to evidence nodes. Data statistics We compute the shortest-path distance between top-1 retrieval of BM25 and target node in this graph in Figure 10.

Original: As can be seen that most path lengths are relatively short, which implies that BM25 lands us in the right vicinity and by a small amount of navigation around we will be able to find the right evidence passage.
Revised: As can be seen that most path lengths are relatively short, which implies that BM25 lands us in the right vicinity and by a small amount of navigation around we will be able to find the right evidence passage.

Original: Since we target navigating to the exact evidence passage required to answer the question, we use recall@{1,2,3,4,5} forfinding the gold evidence passage as our metric.
Revised: Training details For the NQ benchmark, our model uses learned embeddings for ϕ with a single feed-forward layer for the policy network. The model is pretrained on the 5 -step sentence search task, and finetuned on trajectories generated from the NQ dataset using the the following loss function: [Equation] [Equation]

Original: More commonly the metric marks a retrieved passage to be correct if it contains the answer string, but this causes a lot of false positives, e.g. the answer string appears in a totally irrelevant context.
Revised: where L BC is the normal BC loss we use in our other experiments. Only the ϕ target weights are finetuned for this task. We use AdamW with a constant learning rate of 10 − 6 , β 1 = 0 . 9 , β 2 = 0 .

Revised: 999 , and ϵ = 10 − 6 .

Revised: The model is finetuned for 100 , 000 update steps with a batch size of 512 .

Original: Our method: For each question, we first run BM25 over all nodes in the graph and take the topmatches as starting nodes.
Revised: No edge dropout is used during finetuning. [Figure] node computed by BM25 to target node on Natural Questions dataset on 2018 graph. Evaluation details For each question, we first run BM25 over all nodes in the graph and take the top-5 matches as starting nodes.

Original: From each starting node, the agent then navigates for 20 steps.
Revised: From each starting node, the agent then navigates for 20 steps.

Original: All thenodes visited by the agent is then ranked by a simple cross-attention model.
Revised: All the 100 nodes visited by the agent is then ranked by a simple cross-attention model.

Original: For the cross-attention model, we follow Hofstätter et al.
Revised: For the cross-attention model, we follow Hofstätter et al.

Original: [2020] to train 6-layer BigBird [Zaheer et al., 2020].
Revised: [2020] to train 6-layer BigBird [Zaheer et al., 2020].

Original: The results are tabulated in Table 6. Paired with a basic retrieval (BM25) and re-ranker (BigBird), the approach obtains a respectable Recall@1 of 31.6. If we remove our navigation component then this drops substantially to 22.2, demonstrating its utility. As a reference, we also ran a state-of-the-art system RocketQA
Revised: Since we target navigating to the exact evidence passage required to answer the question, we use recall@{1,2,3,4,5} for finding the gold evidence passage as our metric.

Original: [Qu et al., 2021] on our setup and evaluated it using our harder metric. It is worthwhile to note that our approach (a) is significantly simpler than state-of-the-art approaches like RocketQA which employ 4 stages of dual-encoder and cross-attention models, and (b) selects evidence over a larger set than the commonly used preprocess Wikipedia passages (38M vs 21M). [Figure]
Revised: More commonly the metric marks a retrieved passage to be correct if it contains the answer string, but this causes a lot of false positives, e.g. the answer string appears in a totally irrelevant context.

Original: D Variational Interpretation of our Approach
Revised: D Variational Interpretation of our Approach

Original: We present an alternative motivation of our method based on variational inference in a latent variable model of states.
Revised: We present an alternative motivation of our method based on variational inference in a latent variable model of states.

Original: Assume we are given some goal node n g .
Revised: Assume we are given some goal node n g .

Original: We would like to parameterize a model p θ ( n T | n g ) which will generate a trajectory of states n 0 , n 1 , . . .
Revised: We would like to parameterize a model p θ ( n T | n g ) which will generate a trajectory of states n 0 , n 1 , . . .

Original: , n T such that n T = n g .
Revised: , n T such that n T = n g .

Original: We define our model as a latent variable model
Revised: We define our model as a latent variable model

Original: [Equation]
Revised: [Equation]

Original: i.e an autoregressive model which samples an initial state from n 0 ∼ p ( n 0 ) and then samples subsequent states n t ∼ p θ ( n t | n t − 1 , n g ) .
Revised: i.e an autoregressive model which samples an initial state from n 0 ∼ p ( n 0 ) and then samples subsequent states n t ∼ p θ ( n t | n t − 1 , n g ) .

Original: The transition distribution samples the next node n t from the neighbors of n t − 1 in our graph.
Revised: The transition distribution samples the next node n t from the neighbors of n t − 1 in our graph.

Original: We would like the probability that n T = n g to be large, thus we will train our model to maximize log p θ ( n T = n g | n g ) .
Revised: We would like the probability that n T = n g to be large, thus we will train our model to maximize log p θ ( n T = n g | n g ) .

Original: For latent-variable models such as this, we can rewrite the marginal likelihood as
Revised: For latent-variable models such as this, we can rewrite the marginal likelihood as

Original: [Equation]
Revised: [Equation]

Original: and can obtain a lower-bound on this quantity by replacing the intractable posterior p θ ( n 0 , . .
Revised: and can obtain a lower-bound on this quantity by replacing the intractable posterior p θ ( n 0 , . .

Original: . , n T − 1 | n T , n g ) with a variational approximation q ( n 0 , . .
Revised: . , n T − 1 | n T , n g ) with a variational approximation q ( n 0 , . .

Original: . , n T − 1 | n T , n g ) , i.e
Revised: . , n T − 1 | n T , n g ) , i.e

Original: L θ ( n T , n g ; q ) :
Revised: L θ ( n T , n g ; q ) :

Original: = log p θ ( n T | n g )
Revised: = log p θ ( n T | n g )

Original: [Equation]
Revised: [Equation]

Original: The above bound becomes tight when q ( n 0 , . .
Revised: The above bound becomes tight when q ( n 0 , . .

Original: . , n T − 1 | n T , n g ) =
Revised: . , n T − 1 | n T , n g ) =

Original: p θ ( n 0 , . . .
Revised: p θ ( n 0 , . . .

Original: , n T − 1 | n T , n g ) .
Revised: , n T − 1 | n T , n g ) .

Original: Thus, we can optimize our model parameters θ to maximize L θ ( n T , n g ; q ) .
Revised: Thus, we can optimize our model parameters θ to maximize L θ ( n T , n g ; q ) .

Original: This approach has had a long history of successfully training latent-variable generative models [Kingma and Welling, 2013, Ho et al., 2020].
Revised: This approach has had a long history of successfully training latent-variable generative models [Kingma and Welling, 2013, Ho et al., 2020].

Original: In the context of this work, we can view q ( n 0 , . . .
Revised: In the context of this work, we can view q ( n 0 , . . .

Original: , n T − 1 | n T , n g ) as a distribution over trajectories of nodes which end at our goal node.
Revised: , n T − 1 | n T , n g ) as a distribution over trajectories of nodes which end at our goal node.

Original: We can interpret the various trajectory generation methods introduced in Section 3.1 as different variational approximations q .
Revised: We can interpret the various trajectory generation methods introduced in Section 3.1 as different variational approximations q .

Original: Ideally, the option which most closely approximates the true posterior p θ ( n 0 , . .
Revised: Ideally, the option which most closely approximates the true posterior p θ ( n 0 , . .

Original: . , n T − 1 | n T , n g ) would be the most desirable but in general this distribution is intractable.
Revised: . , n T − 1 | n T , n g ) would be the most desirable but in general this distribution is intractable.

Original: We find that using simple random trajectories provides a good-enough approximation to the posterior to enable us to train a model which reliably finds the goal state.
Revised: We find that using simple random trajectories provides a good-enough approximation to the posterior to enable us to train a model which reliably finds the goal state.

Original: This result is not completely surprising in context of prior work [Dai et al., 2020] which successfully trains latent-variable models using random trajectories as an inference model.
Revised: This result is not completely surprising in context of prior work [Dai et al., 2020] which successfully trains latent-variable models using random trajectories as an inference model.

Original: E Efficiency Analysis
Revised: E Efficiency Analysis

Original: In this section we analyze the asmpytotic runtime of our method and baselines.
Revised: In this section we analyze the asmpytotic runtime of our method and baselines.

Original: In Table 7 we show the asymptotic runtime and whether or not it is scalable for each method.
Revised: In Table 7 we show the asymptotic runtime and whether or not it is scalable for each method.

Original: We also show the 5-step accuracy from Table 2 for quick reference.
Revised: We also show the 5-step accuracy from Table 2 for quick reference.

Original: T is the trajectory length (either the length of the trajectory for BC methods or the maximum allowed trajectory for other methods.
Revised: T is the trajectory length (either the length of the trajectory for BC methods or the maximum allowed trajectory for other methods.

Original: For BC, T is guaranteed to be shorter than the max, but this is a constant factor difference.
Revised: For BC, T is guaranteed to be shorter than the max, but this is a constant factor difference.

Original: E out is the average number of outgoing edges (i.e. actions) at a node.
Revised: E out is the average number of outgoing edges (i.e. actions) at a node.

Original: (small - 10 1 )
Revised: (small - 10 1 )

Original: E is the total edges in the graph (very large - 10 8 ).
Revised: E is the total edges in the graph (very large - 10 8 ).

Original: V is the number of nodes in the graph (large10 7 ).
Revised: V is the number of nodes in the graph (large10 7 ).

Original: [Table]
Revised: [Table]

Original: For most methods, the runtime is O ( E out T ) because the method does some constant amount of work for each possible actions E o ut for O ( T ) steps.
Revised: For most methods, the runtime is O ( E out T ) because the method does some constant amount of work for each possible actions E o ut for O ( T ) steps.

Original: For the random methods it is O ( T ) because the work is constant per step (choose randomly).
Revised: For the random methods it is O ( T ) because the work is constant per step (choose randomly).

Original: Shortest path distance has the worst runtime.
Revised: Shortest path distance has the worst runtime.

Original: The O ( ElogV ) term is the runtime of finding a shortest path (Dijkstra’s algorithm with Fibonacci heap).
Revised: The O ( ElogV ) term is the runtime of finding a shortest path (Dijkstra’s algorithm with Fibonacci heap).

Original: As the graph gets larger, this term because infeasible (we could not even run this on the full Wikipedia graph).
Revised: As the graph gets larger, this term because infeasible (we could not even run this on the full Wikipedia graph).

Original: DFS might seem like it should be larger than O ( E out T ) or O ( T ) , but because we restrict the maximum allowable steps, it is only proportional to T and (for greedy) E out .
Revised: DFS might seem like it should be larger than O ( E out T ) or O ( T ) , but because we restrict the maximum allowable steps, it is only proportional to T and (for greedy) E out .

Original: From this Table, we see that all methods (except shortest path) have scalable runtimes.
Revised: From this Table, we see that all methods (except shortest path) have scalable runtimes.

Original: The dominant factor is E out , which theoretically grows slowly with graph size, but in practice, there is a practical maximum number of links you have in a paragraph of text, and the runtime difference in our experiments going from 200k nodes to 38M nodes is only 1.5x despite the 200-fold increasein graph size.
Revised: The dominant factor is E out , which theoretically grows slowly with graph size. Moreover, in practice, there is a natural upper bound on the number of links one can have in a paragraph of text, and the runtime difference in our experiments going from 200k nodes to 38M nodes is only 5x despite the 200-fold increase size.