Original: Learning to Navigate Wikipedia by Taking Random Walks
Revised: Learning to Navigate Wikipedia by Taking Random Walks

Original: Anonymous Author(s)

Original: AffiliationAddress email

Original: Abstract
Revised: Abstract

Original: A fundamental ability of an intelligent web-based agent is seeking out and acquiringnew information.
Revised: A fundamental ability of an intelligent web-based agent is seeking out and acquiring new information.

Original: Internet search engines reliably find the correct vicinity but thetop results may be a few links away from the desired target.
Revised: Internet search engines reliably find the correct vicinity but the top results may be a few links away from the desired target.

Original: A complementaryapproach is navigation via hyperlinks, employing a policy that comprehends localcontent and selects a link that moves it closer to the target.
Revised: A complementary approach is navigation via hyperlinks, employing a policy that comprehends local content and selects a link that moves it closer to the target.

Original: In this paper we showthat using behavioral cloning of randomly sampled trajectories is sufficient to learnan effective link selection policy.
Revised: In this paper we show that using behavioral cloning of randomly sampled trajectories is sufficient to learn an effective link selection policy.

Original: We demonstrate the approach on a graph versionof Wikipedia with 38 M nodes and 387 M edges.
Revised: We demonstrate the approach on a graph version of Wikipedia with 38 M nodes and 387 M edges.

Original: The model is able to efficientlynavigate between nodes 5 and 20 steps apart 96% and 92% of the time, respectively.
Revised: The model is able to efficiently navigate between nodes 5 and 20 steps apart 96% and 92% of the time, respectively.

Original: We then use the resulting embeddings and policy ina downstream fact verificationtask where, in combination with basic TF-IDF search and ranking methods, theyare able to obtain competitive results to state-of-the-art methods
Revised: We then use the resulting embeddings and policy in downstream fact verification and question answering tasks where, in combination with basic TF-IDF search and ranking methods, they are able to obtain competitive results to state-of-the-art methods.

Original:  Introduction
Revised:  Introduction

Original: The ability to gather new knowledge about the world is a fundamental aspect of intelligence.
Revised: The ability to gather new knowledge about the world is a fundamental aspect of intelligence.

Original: Basedonly on a few words, a fact, question, or even vague idea, humans have the ability to use the internetto find extremely specific information about the world.
Revised: Based only on a few words, a fact, question, or even vague idea, humans have the ability to use the internet to find extremely specific information about the world.

Original: From the important (how to administer CPR)to the trivial (the LA Raiders won the 1983 SuperBowl), virtually any knowledge is a click away.
Revised: From the important (how to administer CPR) to the trivial (the LA Raiders won the 1983 SuperBowl), virtually any knowledge is a click away.

Original: In this work, we focus in one particular aspect of this ability: web navigation.
Revised: In this work we focus in one particular aspect of this ability: web navigation.

Original: In general, navigationis a key component of an embodied agent: the ability to move efficiently toward a target.
Revised: In general, navigation is a key component of an embodied agent: the ability to move efficiently toward a target.

Original: In knownenvironments, where map information is available, shortest-path algorithms provide a viable solution.
Revised: In known environments, where map information is available, shortest-path algorithms provide a viable solution.

Original: However, in novel settings, the agent lacks such global information and must instead navigate to thetarget, using its understanding of the local environment to select actions.
Revised: However, in novel settings, the agent lacks such global information and must instead navigate to the target, using its understanding of the local environment to select actions

Original: Other approaches to web agents have focused mostly on retrieval or search engines.
Revised: Other approaches to web agents have focused mostly on retrieval or search engines.

Original: These, however,only provide a partial solution, typically getting close to a desired target but often not exactly to theright page.
Revised: These, however, only provide a partial solution, typically getting close to a desired target but often not exactly to the right page.

Original: In this work, we present an approach for navigation on graph-structured web data thatuses hyperlinks within articles to navigate toward a target.
Revised: In this work, we present an approach for navigation on graph-structured web data that uses hyperlinks within articles to navigate toward a target.

Original: It complements search engines, using them 
Revised: It complements search engines, using them to provide a sensible starting point for a local search for specific information.

Original: To be able to navigate on the web effectively, we must read the text of the page, see how concepts inthe current page are related to a possible next page, and use our understanding to learn a policy tomake the right navigation decision.
Revised: To be able to navigate on the web effectively, we must read the text of the page, see how concepts in the current page are related to a possible next page, and use our understanding to learn a policy to make the right navigation decision.

Original: Consider the scenario shown in Figure 1.
Revised: Consider the scenario shown in Figure 1.

Original: The agent is tryingto navigate to the target node N ORTH A MERICA while currently at the node P RESIDENTS OF THE
Revised: The agent is trying to navigate to the target node N ORTH A MERICA while currently at the node P RESIDENTS OF THE

Original: U NITED S TATES containing two hyperlinks: B ARACK O BAMA and USA.
Revised: U NITED S TATES containing two hyperlinks: B ARACK O BAMA and USA.

Original: Our approach learns anembedding from the text of the current page and the text for each hyperlink node and passes theseto a goal-conditioned policy that selects the best possible action.
Revised: Our approach learns an embedding from the text of the current page and the text for each hyperlink node and passes these

Original: To make the correct choice in thisexample, the agent should associate USA with N ORTH A MERICA

Original: Barack Obama
Revised: Barack Obama

Original: USA
Revised: USA

Original: Current Node
Revised: Current Node

Original: Presidents of the United States
Revised: Presidents of the United States

Original: Node
Revised: Node

Original: Node Text “Barack Obama was the President of the United States…”
Revised: Node Text “Barack Obama was the President of the United States…”

Original: “Barack Obama is an American Politician…” “The USA is a nation in North America…” ϕ
Revised: “Barack Obama is an American Politician…” “The USA is a nation in North America…” ϕ

Original: Transformer s t a“The USA is a nation in North America…”
Revised: Transformer s t a“The USA is a nation in North America…”

Original: a
Revised: a

Original: North America “North America is a continent on the…” s g [Figure]
Revised: North America “North America is a continent on the…” s g Goal Node

Original: MLP
Revised: MLP

Original: Action probability pp
Revised: Action probability pp

Original: Node
Revised: Node

Original: Nodeϕ ϕ ϕθ θ
Revised: Nodeϕ ϕ ϕ

Revised: θ θ to a goal-conditioned policy that selects the best possible action.

Revised: To make the correct choice in this example, the agent should associate USA with N ORTH A MERICA .

Original: The particular web environment we consider here is Wikipedia, converted into a graph form.
Revised: The particular web environment we consider here is Wikipedia, converted into a graph form.

Original: Eachof the 38M paragraphs is represented by a node; edges are links within and between articles.
Revised: Each of the 38M paragraphs is represented by a node; edges are links within and between articles.

Original: Weintroduce an approach for navigation on this graph based on behavioral cloning of automaticallygenerated trajectories.
Revised: We introduce an approach for navigation on this graph based on behavioral cloning of automatically generated trajectories.

Original: This allows for unsupervised pre-training of both web page embeddingsand navigation policy.
Revised: This allows for unsupervised pre-training of both web page embeddings and navigation policy.

Original: We explore different trajectory distributions for training the model, the bestpractical choice being equivalent to random walks on the graph.
Revised: We explore different trajectory distributions for training the model, the best practical choice being equivalent to random walks on the graph.

Original: Contributions: Our work presents the first viable solution to the problem of web navigation, asdemonstrated by reliable (>90% success) navigation on a graph of size 10 7 .
Revised: Contributions: Our work presents the first viable solution to the problem of web navigation, as demonstrated by reliable (>90% success) navigation on a graph of size 10 7 .

Original: Previous attemptssuch as Nogueira and Cho [2016] show a far inferior performance, well below that needed forpractical utility.
Revised: Previous attempts such as Nogueira and Cho [2016] show a far inferior performance, well below that needed for practical utility.

Original: Furthermore, as there is nothing Wikipedia-specific about our approach and it scalesgracefully, the method is applicable to general web settings.
Revised: Furthermore, as there is nothing Wikipedia-specific about our approach and it scales gracefully, the method is applicable to general web settings.

Original: We also demonstrate the relevance ofthe pre-train + fine-tune paradigm to the web domain, showing that pre-training for navigation is asuitable objective to aid downstream tasks with limited data such as Q&A.
Revised: We also demonstrate the relevance of the pre-train + fine-tune paradigm to the web domain, showing that pre-training for navigation is a suitable objective to aid downstream tasks with limited data such as Q&A.

Original: We demonstrate this byapplying our navigation approach on two challenging Q&A tasks (FEVER Thorne et al.
Revised: We demonstrate this by applying our navigation approach on two challenging Q&A tasks (FEVER Thorne et al.

Original: [2018] and
Revised: [2018] and

Original: Natural Questions Lee et al.
Revised: Natural Questions Lee et al.

Original: In both settings, we show a jump in performance over existingsearch-based approaches that rely on retrieval followed by re-ranking, matching state-of-the-artmethods despite using far simpler retrieval and re-ranking mechanisms.
Revised: In both settings, we show a jump in performance over existing search-based approaches that rely on retrieval followed by re-ranking, matching state-of-the-art methods despite using far simpler retrieval and re-ranking mechanisms.

Original:  Background
Revised:  Background

Original: Wikipedia as a navigation environment: Human behavior on Wikipedia [West and Leskovec,2012a] was collected via the Wikispeedia game [West et al., 2009], which asked the players to movebetween a pair of arbitrarily chosen articles in the fewest steps.
Revised: Wikipedia as a navigation environment: Human behavior on Wikipedia [West and Leskovec, 2012a] was collected via the Wikispeedia game [West et al., 2009], which asked the players to move between a pair of arbitrarily chosen articles in the fewest steps.

Original: This rule allows the emergence of asemantic distance between concepts that facilitates retrieval.
Revised: This rule allows the emergence of a semantic distance between concepts that facilitates retrieval.

Original: West and Leskovec [2012b] conductedinitial studies on this dataset for navigation, using supervised learning and reinforcement learningmethods on a small subset of articles ( 4 . 6 K) with simple TF-IDF features.
Revised: West and Leskovec [2012b] conducted initial studies on this dataset for navigation, using supervised learning and reinforcement learning methods on a small subset of articles ( 4 . 6 K) with simple TF-IDF features.

Original: Most relevantly, Nogueira and Cho [2016] explore the same problem of navigation on a large
Revised: Most relevantly, Nogueira and Cho [2016] explore the same problem of navigation on a large

Original: Wikipedia graph ( 12 M nodes).
Revised: Wikipedia graph ( 12 M nodes).

Original: They train LSTM agents to perform navigation on the graph, using
Revised: They train LSTM agents to perform navigation on the graph, using

Original: Bag-of-Word features for each article.
Revised: Bag-of-Word features for each article.

Original: The trained agent is then applied to a Jeopardy Q&A task.
Revised: The trained agent is then applied to a Jeopardy Q&A task.

Original: Our approach revisits this framing but with a number of important differences: (i) training on asignificantly larger graph ( 38 M vs 12 M nodes, 51 .
Revised: Our approach revisits this framing but with a number of important differences: (i) training on a significantly larger graph ( 38 M vs 12 M nodes, 51 .

Original: 5 M vs 380 M edges); (ii) using Transformer-basedarchitectures for paragraph encodings and for the navigation policy; (iii) they use a single fixed startnode, whereas ours is able to navigate from any start node; (iv) removal of beam search, or othercomplex search mechanisms at evaluation time.
Revised: 5 M vs 380 M edges); (ii) using Transformer-based architectures for paragraph encodings and for the navigation policy; (iii) they use a single fixed start node, whereas ours is able to navigate from any start node; (iv) removal of beam search, or other complex search mechanisms at evaluation time.

Original: Collectively, our approach boosts the navigationsuccess rate from 12 .
Revised: Collectively, our approach boosts the navigation success rate from 12 .

Original: 5% in [Nogueira and Cho, 2016] to over 90% , 1 despite a much larger graph.
Revised: 5% in [Nogueira and Cho, 2016] to over 90% , 1 despite a much larger graph.

Original: Fact verification and Q&A using Wikipedia:
Revised: Fact verification and Q&A using Wikipedia:

Original: Wikipedia is a rich resource for evaluating languageunderstanding through Q&A tasks.
Revised: Wikipedia is a rich resource for evaluating language understanding through Q&A tasks.

Original: Chen et al.
Revised: Chen et al.

Original: [2017] combine TF-IDF retrieval with an LSTM document reader to find answers to free-form questions within Wikipedia documents.
Revised: [2017] combine TF-IDF retrieval with an LSTM docu- ment reader to find answers to free-form questions within Wikipedia documents.

Original: Contemporary Q&Asystems [Semnani and Pandey, 2020, Qu et al., 2020] extend this general approach to more complexpipelines, combining traditional and neural methods with inverted index search/retrieval schemes,followed by document readers that perform re-ranking.
Revised: Contemporary Q&A systems [Semnani and Pandey, 2020, Qu et al., 2020] extend this general approach to more complex pipelines, combining traditional and neural methods with inverted index search/retrieval schemes, followed by document readers that perform re-ranking.

Original: We demonstrate our navigation approach in conjunction with very basic retrieval and re-ranking approaches, so as to clearly demonstrate thevalue of navigation.
Revised: We demonstrate our navigation approach in conjunction with very basic retrieval and re-ranking approaches, so as to clearly demonstrate the value of navigation.

Original: In common with our approach, Wang et al.
Revised: In common with our approach, Wang et al.

Original: [2021] also use a graph version of
Revised: [2021] also use a graph version of

Original: Wikipedia, but with the goal of aligning it to a knowledge base, rather than navigation.
Revised: Wikipedia, but with the goal of aligning it to a knowledge base, rather than navigation.

Original: Stammbachand Neumann [2019] propose a scheme for finding evidence for the FEVER Thorne et al.
Revised: Stammbach and Neumann [2019] propose a scheme for finding evidence for the FEVER Thorne et al.

Original: [2018]benchmark that explores all links one step away from an initial retrieval set.
Revised: [2018] benchmark that explores all links one step away from an initial retrieval set.

Original: This can be viewed as a1-step navigation operation, in contrast to our scheme that permits an arbitrary number of steps.
Revised: This can be viewed as a 1-step navigation operation, in contrast to our scheme that permits an arbitrary number of steps.

Original: Navigation in knowledge bases: Graph navigation is a key element in utilizing knowledge bases(KB), where facts are represented as tuples of entity nodes connected by a labelled edge correspondingto relation between the two nodes.
Revised: Navigation in knowledge bases: Graph navigation is a key element in utilizing knowledge bases (KB), where facts are represented as tuples of entity nodes connected by a labelled edge corresponding to relation between the two nodes.

Original: For example, Das et al.
Revised: For example, Das et al.

Original: [2017] answer complex queries bynavigating the graph, inferring missing edges along the way, and finding a valid path to the answernode.
Revised: [2017] answer complex queries by navigating the graph, inferring missing edges along the way, and finding a valid path to the answer node.

Original: In our case, the graph is fixed but consists of natural language text and is partially observed.
Revised: In our case, the graph is fixed but consists of natural language text and is partially observed.

Original: This makes the task much harder on two fronts.
Revised: This makes the task much harder on two fronts.

Original: First, the policy must comprehend the naturallanguage actions, without access to crisply labelled relation actions.
Revised: First, the policy must comprehend the natural language actions, without access to crisply labelled relation actions.

Original: Second, the action space isopen and varying in size, unlike in a KB which typically has a fixed schema, i.e. relations/actionscome from a fixed size set.
Revised: Second, the action space is open and varying in size, unlike in a KB which typically has a fixed schema, i.e. relations/actions come from a fixed size set.

Original: Perhaps the closest work involving natural language actions, Fu et al.[2019] augment a KB with textual nodes.
Revised: Perhaps the closest work involving natural language actions, Fu et al. [2019] augment a KB with textual nodes.

Original: However, it still operates under the fixed KB schema.
Revised: However, it still operates under the fixed KB schema.

Original: Further improvements on navigation in KB have been shown by incorporating tricks like rewardshaping [Xiong et al., 2017] and action dropout [Lin et al., 2018].
Revised: Further improvements on navigation in KB have been shown by incorporating tricks like rewardshaping [Xiong et al., 2017] and action dropout [Lin et al., 2018].

Original: We adopted the action dropoutwhere some outgoing edges are masked during training to enable more effective path exploration.
Revised: We adopted the action dropout where some outgoing edges are masked during training to enable more effective path exploration.

Original: Pushing more on exploration, inspired by AlphaGo, Shen et al.
Revised: Pushing more on exploration, inspired by AlphaGo, Shen et al.

Original: He et al.
Revised: He et al.

Original: [2022] combine thepolicy with Monte-Carlo tree search to navigate KB.
Revised: [2022] combine the policy with Monte-Carlo tree search to navigate KB.

Original: We find that no search method is necessary forour approach at evaluation time, with good performance obtained from the policy alone.
Revised: We find that no search method is necessary for our approach at evaluation time, with good performance obtained from the policy alone.

Original: Internet powered Q&A:
Revised: Internet powered Q&A:

Original: Several recent works have attempted complex question answering usingthe internet.
Revised: Several recent works have attempted complex question answering using the internet.

Original: Talmor and Berant [2018] retrieve snippets of information from the web with a searchengine to answer a structured decomposition of the question with a simpler Q&A model.
Revised: Talmor and Berant [2018] retrieve snippets of information from the web with a search engine to answer a structured decomposition of the question with a simpler Q&A model.

Original: Therecent WebGPT Nakano et al.
Revised: The recent WebGPT Nakano et al.

Original: [2021] system takes a broadly similar approach to long-form questionanswering.
Revised: [2021] system takes a broadly similar approach to long-form question answering.

Original: WebGPT accures information from the Bing search API as well as some navigation, andpasses it to an LM to generate the answer text.
Revised: WebGPT accures information from the Bing search API as well as some navigation, and passes it to an LM to generate the answer text.

Original: The complexity of the system necessitates the use ofhuman demonstrations for training.
Revised: The complexity of the system necessitates the use of human demonstrations for training.

Original: Related work from Lazaridou et al.
Revised: Related work from Lazaridou et al.

Original: [2022] utilizes a few-shotsupervision of an LM to perform Google queries, which are then used by the LM to generate ananswer.
Revised: [2022] utilizes a few-shot supervision of an LM to perform Google queries, which are then used by the LM to generate an answer.

Original: They show that by incorporating information from Google queries, the answers are morefactual than those generated directly from LMs.
Revised: They show that by incorporating information from Google queries, the answers are more factual than those generated directly from LMs.

Original: Another line of work, has leveraged link structure ofwebpages found on the internet to improve Q&A.
Revised: Another line of work, has leveraged link structure of webpages found on the internet to improve Q&A.

Original: In LinkBERT [Yasunaga et al., 2022] is pretrainedto capture dependencies between documents and results in improvement for for multi-hop reasoningand few-shot reading comprehension.
Revised: In LinkBERT [Yasunaga et al., 2022] is pretrained to capture dependencies between documents and results in improvement for for multi-hop reasoning and few-shot reading comprehension.

Original: Asai et al.
Revised: Asai et al.

Original: [2020] use the link structure to enhance open domain
Revised: [2020] use the link structure to enhance open domain

Original: Q&A by learning to retrieve according to reasoning paths.
Revised: Q&A by learning to retrieve according to reasoning paths.

Original: In contrast to these works, we focus onmastering navigation rather than any combination with search/retrieval, albeit in more limited domain(Wikipedia vs the internet).
Revised: In contrast to these works, we focus on mastering navigation rather than any combination with search/retrieval, albeit in more limited domain (Wikipedia vs the internet).

Original: However, our approach could potentially act as a core component in ageneral web navigation, being more versatile on account of not requiring supervision.
Revised: However, our approach could potentially act as a core component in a general web navigation, being more versatile on account of not requiring supervision.

Original:  Approach
Revised:  Approach

Original: We consider navigation in a web environment W = { N , L } , where N := { n i } Ni =1 is the set ofnodes, each n i representing a web page with content s i .
Revised: We consider navigation in a web environment W = { N , L } , where N := { n i } Ni =1 is the set of nodes, each n i representing a web page with content s i .

Original: The links between pages are representedby L := { l i } Ni =1 , where l i = { l 1 i , . . .
Revised: The links between pages are represented by L := { l i } Ni =1 , where l i = { l 1 i , . . .

Original: , l ji , . . .
Revised: , l ji , . . .

Original: , l M i i } are the set of M i (directed) out-links from node i .
Revised: , l M i i } are the set of M i (directed) out-links from node i .

Original: We define NH ( n i ) as the set of nodes reached by following all links l i from node n i , where the j thneighbor is given by l ji ( n i ) .
Revised: We define NH ( n i ) as the set of nodes reached by following all links l i from node n i , where the j th neighbor is given by l ji ( n i ) .

Original: We assume some target node n g and some starting node n 0 are given, and wish to train an agent thatcan navigate from n 0 to n g by walking along the edges of the graph within T steps.
Revised: We assume some target node n g and some starting node n 0 are given, and wish to train an agent that can navigate from n 0 to n g by walking along the edges of the graph within T steps.

Original: To accomplish this,we construct a parametric policy p θ ( n t +1 | n t , n g ) which, at time step t , parameterizes a distributionover n t +1 ∈ NH ( n t ) .
Revised: To accomplish this, we construct a parametric policy p θ ( n t +1 | n t , n g ) which, at time step t , parameterizes a distribution over n t +1 ∈ NH ( n t ) .

Original: We train this model via behavioral cloning to maximize
Revised: We train this model via behavioral cloning to maximize

Original: [Equation]
Revised: [Equation]

Revised: t

Original:  where E ( n 0 , . . . , n T ) is our trajectory distribution which generates trajectories ( n 0 , . .
Revised: [Equation] [Equation] (cid:35) (1) where E ( n 0 , . . . , n T ) is our trajectory distribution which generates trajectories ( n 0 , . .

Original: . , n T ) that wewould like our model to replicate (i.e. setting n g = n T ).
Revised: . , n T ) that we would like our model to replicate (i.e. setting n g = n T ).

Original: 3.1 Trajectory distribution
Revised: 3.1 Trajectory distribution

Original: This framework opens up many choices for E .
Revised: This framework opens up many choices for E .

Original: Clearly some will result in ineffective navigationstrategies while others may enable us to train a policy which is capable of generalizing to new graphsand downstream applications.
Revised: Clearly some will result in ineffective navigation strategies while others may enable us to train a policy which is capable of generalizing to new graphs and downstream applications.

Original: We elaborate on some choices below.
Revised: We elaborate on some choices below.

Original: Reverse trajectories We define E ( n T ) =
Revised: Reverse trajectories We define E ( n T ) =

Original: Uniform ( N ) (the uniform distribution over nodes in thegraph).
Revised: Uniform ( N ) (the uniform distribution over nodes in the graph).

Original: Now we can define
Revised: Now we can define

Original: E (

Original: [Equation]

Original: =

Original: [Equation]

Original: { n

Original: [Equation]
Revised: [Equation]

Original: n i ∈

Original: NH

Original: [Equation] n t ) } [Equation] if n t +1 ∈ NH ( n t ) , else(2) or, a distribution which uniformly walks the reverse graph (with transposed adjacency matrix).
Revised: or, a distribution which uniformly walks the reverse graph (with transposed adjacency matrix).

Original: Webegin with some uniformly chosen target node and walk backward randomly to n 0 in such a way thatwe can reach the target by walking forward from n 0 .
Revised: We begin with some uniformly chosen target node and walk backward randomly to n 0 in such a way that we can reach the target by walking forward from n 0 .

Original: Intuitively, training with the reverse trajectories defined above should allow our model to be able toreach any node in the graph from any node it is connected to.
Revised: Intuitively, training with the reverse trajectories defined above should allow our model to be able to reach any node in the graph from any node it is connected to.

Original: Unfortunately, in realistic web graphdata, there exist many “dead-ends” or nodes with no in-going edges.
Revised: Unfortunately, in realistic web graph data, there exist many “dead-ends” or nodes with no in-going edges.

Original: The trajectories get stuck atthese nodes leading to low-diversity training data.
Revised: The trajectories get stuck at these nodes leading to low-diversity training data.

Original: Random forward trajectories In our real-world knowledge graph data, there are far fewer nodeswhich have no out-going edges.
Revised: Random forward trajectories In our real-world knowledge graph data, there are far fewer nodes which have no out-going edges.

Original: Thus we propose an alternative method to generate target nodesbased on randomly walking forward.
Revised: Thus we propose an alternative method to generate target nodes based on randomly walking forward.

Original: We define E ( n 0 ) to be the Uniform ( N ) and
Revised: We define E ( n 0 ) to be the Uniform ( N ) and

Original: E ( n t +1 | n t ) =
Revised: E ( n t +1 | n t ) =

Original: | NH ( n t ) | if n t +1 ∈ NH ( n t ) , else 0 .
Revised: | NH ( n t ) | if n t +1 ∈ NH ( n t ) , else 0 .

Original: (3) or, a distribution which uniformly picks a starting node and randomly walks the forward graph for
Revised: (3) or, a distribution which uniformly picks a starting node and randomly walks the forward graph for

Original: T steps.
Revised: T steps.

Original: Training in this way should encourage our model to be able to reach any node which isreachable from some uniformly chosen start.
Revised: Training in this way should encourage our model to be able to reach any node which is reachable from some uniformly chosen start.

Original: We note that the target node distribution E ( n T ) will notbe uniform and instead be a function of the graph structure.
Revised: We note that the target node distribution E ( n T ) will not be uniform and instead be a function of the graph structure.

Original: We also note that while this generates a
Revised: We also note that while this generates a

Original: T -step trajectory, because it is generated by random walk, the shortest path distance between startand target nodes may be smaller.
Revised: T -step trajectory, because it is generated by random walk, the shortest path distance between start and target nodes may be smaller.

Original: Random shortest paths We define E ( n T ) =
Revised: Random shortest paths We define E ( n T ) =

Original: Uniform ( N ) and let E ( n 0 | n T ) be the uniformdistribution of nodes whose shortest path length to n T is T steps.
Revised: Uniform ( N ) and let E ( n 0 | n T ) be the uniform distribution of nodes whose shortest path length to n T is T steps.

Original: We then define E ( n t | n 0 , n T ) tobe the delta distribution that n t is the t -th node of this shortest path.
Revised: We then define E ( n t | n 0 , n T ) to be the delta distribution that n t is the t -th node of this shortest path.

Original: While it can be challenging tocompute the probabilities of this distribution efficiently we can draw samples from it using a shortestpath finding algorithm such as Dijkstra’s algorithm.
Revised: While it can be challenging to compute the probabilities of this distribution efficiently we can draw samples from it using a shortest path finding algorithm such as Dijkstra’s algorithm.

Original: 3.2 Model parameterization
Revised: 3.2 Model parameterization

Original: From above, we can see the core of our model is the p θ ( n t +1 | n t , n T ) distribution.
Revised: From above, we can see the core of our model is the p θ ( n t +1 | n t , n T ) distribution.

Original: Given a sampledtrajectory that begins at n 0 , this is the distribution which attempts to choose the correct action totake at timestep t + 1 starting at node n t in order to get to target node n T .
Revised: Given a sampled trajectory that begins at n 0 , this is the distribution which attempts to choose the correct action to take at timestep t + 1 starting at node n t in order to get to target node n T .

Original: We parameterize thisdistribution as a function of the text available at node n t .
Revised: We parameterize this distribution as a function of the text available at node n t .

Original: [Equation]
Revised: [Equation]

Original: where f θ embeds the text s g , s t and s t +1 of the target, current and destination nodes, respectively.
Revised: where f θ embeds the text s g , s t and s t +1 of the target, current and destination nodes, respectively.

Original: Further details on how we apply this general model to Wikipedia are given in Section 4.2 and analternative interpretation of this approach based on variational inference in a latent-variable model ispresented in Appendix D.
Revised: Further details on how we apply this general model to Wikipedia are given in Section 4.2 and an alternative interpretation of this approach based on variational inference in a latent-variable model is presented in Appendix D.

Original:  Application to Wikipedia 
Revised:  Application to Wikipedia 4.1 Navigation and sentence search tasks

Original: 4.1 Navigation and sentence search tasks

Original: Data processing: We convert a snapshot of English Wikipedia into a graph but split each article( ¯ µ = 1000 words) into paragraph-sized blocks ( ¯ µ = 100 words), which form the nodes .
Revised: Data processing: We convert a snapshot of English Wikipedia into a graph but split each article ( ¯ µ = 1000 words) into paragraph-sized blocks ( ¯ µ = 100 words), which form the nodes .

Original: Edges inthe graph are the union of (i) organic hyperlinks, (ii) additional entity linking and (iii) next/previousparagraph links.
Revised: Edges in the graph are the union of (i) organic hyperlinks, (ii) additional entity linking and (iii) next/previous paragraph links.

Original: The full graph has ∼ 38 M nodes and 387 M edges.
Revised: The full graph has ∼ 38 M nodes and 387 M edges.

Original: For initial experiments, we also use a smaller subsampled version of 200 k nodes.
Revised: For initial experiments, we also use a smaller subsampled version of 200 k nodes.

Original: For this we sub-sample disjoint sets of 200k train /200k evaluation Wikipedia nodes.
Revised: For this we sub-sample disjoint sets of 200k train / 200k evaluation Wikipedia nodes.

Original: See Section A.1 for full details.
Revised: See Section A.1 for full details.

Original: T-step navigation: The most basic task is T -step navigation.
Revised: T-step navigation: The most basic task is T -step navigation.

Original: At the start of the episode, we randomlysample a start node and generate a T -step random walk trajectory.
Revised: At the start of the episode, we randomly sample a start node and generate a T -step random walk trajectory.

Original: We then take the last node of thistrajectory as the target n g .
Revised: We then take the last node of this trajectory as the target n g .

Original: The agent is then given a time budget B (in most experiments, we fix maxsteps to B = 100 ), and succeeds if it reaches the target node within B steps.
Revised: The agent is then given a time budget B (in most experiments, we fix max steps to B = 100 ), and succeeds if it reaches the target node within B steps.

Original: In our experiments, weshow results on various graphs and splits with T = { 5 , 10 , 20 } -step navigation.1-T Multistep navigation: The Multistep task is much like the previous setting, except that insteadof generating fixed T -step trajectories, we sample from Uniform (1 , T ) , T = 20 .
Revised: In our experiments, we show results on various graphs and splits with T = { 5 , 10 , 20 } -step navigation. 1-T Multistep navigation: The Multistep task is much like the previous setting, except that instead of generating fixed T -step trajectories, we sample from Uniform (1 , T ) , T = 20 .

Original: T-step sentence search: This is the same setup as T -step navigation, except that rather than providethe full text for the target node n g , only one sentence within that text is selected at random to computethe target embeddings from.
Revised: T-step sentence search: This is the same setup as T -step navigation, except that rather than provide the full text for the target node n g , only one sentence within that text is selected at random to compute the target embeddings from.

Original: Thus agents have the more challenging task of locating the correct node,given only a small snippet of its text (see Sec.
Revised: Thus agents have the more challenging task of locating the correct node, given only a small snippet of its text (see Sec.

Original: 4.2 for details on target representation).
Revised: 4.2 for details on target representation).

Original: We includethis task because one of the difficulties of web navigation is establishing which page should be thetarget in the first place.
Revised: We include this task because one of the difficulties of web navigation is establishing which page should be the target in the first place.

Original: It also has direct relevance to our downstream fact verification task.
Revised: It also has direct relevance to our downstream fact verification task.

Original: 4.2 Model architecture
Revised: 4.2 Model architecture

Original: State representation:
Revised: State representation:

Original: For each node n i we embed its text
Revised: For each node n i we embed its text

Original: s i using a Transformer ϕ ( . ) to produceϕ ( s i ) .
Revised: s i using a Transformer ϕ ( . ) to produce ϕ ( s i ) .

Original: By basing the state representation on semantic content, rather than i.e. node index, the agentis able to learn about the relationships between entities in a manner that allows for generalizationto new graphs at evaluation time, instead of memorizing the structure of the training graph.
Revised: By basing the state representation on semantic content, rather than i.e. node index, the agent is able to learn about the relationships between entities in a manner that allows for generalization to new graphs at evaluation time, instead of memorizing the structure of the training graph.

Original: Following Karpukhin et al.
Revised: Following Karpukhin et al.

Original: [2020], we append to the node text the title of the Wikipedia article to which itbelongs.
Revised: [2020], we append to the node text the title of the Wikipedia article to which it belongs.

Original: This often provides context to the node that might otherwise be missing – in many articles,the subject of the article will be referred to indirectly (e.g. it might state that “she” was born in 1945instead of repeating the subject’s name).
Revised: This often provides context to the node that might otherwise be missing – in many articles, the subject of the article will be referred to indirectly (e.g. it might state that “she” was born ininstead of repeating the subject’s name).

Original: For those experiments where the node embeddings are kept fixed, ϕ ( . ) is a pre-trained RoBERTA[Liu et al., 2019] model that encodes the node text & title.
Revised: For those experiments where the node embeddings are kept fixed, ϕ ( . ) is a pre-trained RoBERTA [Liu et al., 2019] model that encodes the node text & title.

Original: However, in our large-scale experiments,we pre-train the Transformer ϕ ( . ) directly using the text & title (see Appendix B.1).
Revised: However, in our large-scale experiments, we pre-train the Transformer ϕ ( . ) directly using the text & title (see Appendix B.1).

Original: In both cases, weapply the Transformer to the tokenized text, take the mean over the input tokens, and apply a tanhnonlinearity to produce ϕ ( s i ) .
Revised: In both cases, we apply the Transformer to the tokenized text, take the mean over the input tokens, and apply a tanh nonlinearity to produce ϕ ( s i ) .

Original: Target representation: Similarly, we use the article text s g of the target node as input to the
Revised: Target representation: Similarly, we use the article text s g of the target node as input to the

Original: Transformer model ϕ to compute its representation ϕ ( s g ) .
Revised: Transformer model ϕ to compute its representation ϕ ( s g ) .

Original: For some tasks, such as sentence search orfact verification, the target representation instead relies on a single sentence from each node or othertext specific to the task and a separate transformer model ϕ target is trained to embed the target.
Revised: For some tasks, such as sentence search or fact verification, the target representation instead relies on a single sentence from each node or other text specific to the task and a separate transformer model ϕ target is trained to embed the target.

Original: Navigation policy network: The network operates on the target node n g and the current node n t ,and outputs a distribution over the possible actions that can be taken at n t .
Revised: Navigation policy network: The network operates on the target node n g and the current node n t , and outputs a distribution over the possible actions that can be taken at n t .

Original: As the nodes in the graphhave a variable number of outgoing edges, the policy must produce a distribution with a variablenumber of outcomes.
Revised: As the nodes in the graph have a variable number of outgoing edges, the policy must produce a distribution with a variable number of outcomes.

Original: To do this, we first concatenate the embeddings of s g and s t , and pass themthrough a 1-layer feed-forward network to produce a combined embedding e tg =
Revised: To do this, we first concatenate the embeddings of s g and s t , and pass them through a 1-layer feed-forward network to produce a combined embedding e tg =

Original: FF [ ϕ ( s t ) , ϕ ( s g )] .
Revised: FF [ ϕ ( s t ) , ϕ ( s g )] .

Original: Next we embed each possible action a i .
Revised: Next we embed each possible action a i .

Original: When we use a fixed ϕ , we directly use a i = ϕ
Revised: When we use a fixed ϕ , we directly use a i = ϕ

Original: ( s i ) forn i ∈ NH ( n t ) .
Revised: ( s i ) for n i ∈ NH ( n t ) .

Original: However this is not feasible when pre-training our embedding model due to memoryconcerns.
Revised: However this is not feasible when pre-training our embedding model due to memory concerns.

Original: Instead, we compute a i = ϕ
Revised: Instead, we compute a i = ϕ

Original: ( s t [ l it ]) where s [ l i ] selects the words in s which belong tohyperlink i – see Appendix B.1 for more details.
Revised: ( s t [ l it ]) where s [ l i ] selects the words in s which belong to hyperlink i – see Appendix B.1 for more details.

Original: We then define the probability of moving to n i fromn t with target n g as p ( n i | n t , n g )
Revised: We then define the probability of moving to n i from n t with target n g as p ( n i | n t , n g )

Original: ∝ exp ( e tg · a i ) which is normalized to produce a distribution overthe | NH ( n t ) | actions.
Revised: ∝ exp ( e tg · a i ) which is normalized to produce a distribution over the | NH ( n t ) | actions.

Original: In our larger-scale model, we also include the trajectory leading up to the current node ( n 0 , . . . , n t ) .
Revised: In our larger-scale model, we also include the trajectory leading up to the current node ( n 0 , . . . , n t ) .

Original: We pass their embeddings, combined with the target embedding, [ ϕ ( s 0 ) , . . .
Revised: We pass their embeddings, combined with the target embedding, [ ϕ ( s 0 ) , . . .

Original: , ϕ ( s t ) , ϕ ( s g )] , through a4-layer transformer and use the output as e tg when computing the transition probabilities above.
Revised: , ϕ ( s t ) , ϕ ( s g )] , through a 4-layer transformer and use the output as e tg when computing the transition probabilities above.

Original: Weexplore these two policy architecture choices, feed-forward and Transformer in Table 4.
Revised: We explore these two policy architecture choices, feed-forward and Transformer in Table 4.

Original: 4.3 Application to fact verification
Revised: 4.3 Application to downstream tasks

Original: Finally, we use our navigation approach to gather information for the task of fact 
Revised: Finally, we use our navigation approach to gather information for the tasks of fact verification on the

Original: verification onthe FEVER benchmark Thorne et al.
Revised: FEVER benchmark [Thorne et al., 2018] and question answering on Natural Questions (NQ)

Original: [2018] which also uses Wikipedia as a knowledge base.
Revised: [Lee et al., 2019] which also use Wikipedia as a knowledge base.

Original: Oneimportant difference from our earlier navigation task is that the target node n g is not specifiedat evaluation time.
Revised: One important difference from our earlier navigation task is that the target node n g is not specified at evaluation time.

Original: Instead, we are given a claim whose veracity must be established  by finding information in a particular node in the graph.
Revised: Instead, we are given a claim whose veracity must be established or a question that must be answered by finding information in a particular node in the graph.

Original: To determine which node this is, we use a target encoderwhich maps the claim  to an embedding vector that can be used in place of ϕ ( s g ) in Section 4.2.
Revised: To determine which node this is, we use a target encoder which maps the claim or question to an embedding vector that can be used in place of ϕ ( s g ) in Section 4.2.

Original: Due to the limited size of the FEVER training set, we pre-train our graph embeddings, navigationpolicy and target encoder on the similar sentence search task.
Revised: Due to the limited size of the downstream training sets, we pre-train our graph embeddings, navigation policy and target encoder on the similar sentence search task.

Original: We then freeze the embeddings andpolicy and fine-tune the target encoder on ( n g , s g ) pairs from the FEVER training set.
Revised: We then freeze the embeddings and policy and fine-tune the target encoder on ( n g , s g ) pairs from the downstream task’s training set.

Original: To avoidover-fitting, we add an auxiliary loss which enforces the similarity between our target embedding andthe embedding of the ground truth node containing the sentence needed to resolve the claim.
Revised: To avoid over-fitting, we add an auxiliary loss which enforces the similarity between our target embedding and the embedding of the ground truth node containing the sentence needed to resolve the claim or question.

Original: Trainingand evaluation on FEVER required alignment between our version of Wikipedia and the one used tocompile the benchmark and details of this can be found in Appendix A.4.
Revised: Training and evaluation on FEVER required alignment between our version of Wikipedia and the one used to compile the benchmark and details of this can be found in Appendix A.4.

Original: Next, our navigation method requires a starting node.
Revised: Next, our navigation method requires a starting node.

Original: To put our agent close to the articles we needin the downstream task, we run a popular variant of TF-IDF, called BM25 [Robertson and Zaragoza,2009].
Revised: To put our agent close to the articles we need in the downstream tasks, we run a popular variant of TF-IDF, called BM25 [Robertson and Zaragoza, 2009].

Original: We first create an index over all of the nodes in the graph, then take the top-5 BM25 matchesas the starting nodes, and run navigation for 20 steps on each.
Revised: We first create an index over all of the nodes in the graph, then take the top-5 BM25 matches as the starting nodes, and run navigation for 20 steps on each.

Original: Note that BM25 does not find the exacttext (i.e. the hits@1 is not very high) but matches are in the right vicinity, usually in the same orsimilar article.
Revised: Note that BM25 does not find the exact text (i.e. the hits@1 is not very high) but matches are in the right vicinity, usually in the same or similar article.

Original: Thus the agent only has to navigate a few steps to reach the target: for the FEVERdataset, the mean length of shortest path between start and target node is 4 .
Revised: Thus the agent only has to navigate a few steps to reach the target: for the FEVER dataset, the mean length of shortest path between start and target node is 4 .

Original: 3 steps.
Revised: 3 steps.

Original: The final component is a re-ranker that takes all the sentences from all nodes visited by the agent andassigns a match score between each one and the claim.
Revised: The final component is a re-ranker that takes all the sentences from all nodes visited by the agent and assigns a match score between each one and the claim.

Original: The ranked list of sentences can then be usedfor benchmark evaluation.
Revised: The ranked list of sentences can then be used for benchmark evaluation.

Original: We explore two types of ranker: (a) basic TF-IDF and (b) a BERT model,fine tuned using hard negatives mined from the TF-IDF ranker.
Revised: We explore two types of ranker: (a) basic TF-IDF and (b) a BERT model, fine tuned using hard negatives mined from the TF-IDF ranker.

Original: See Appendix A.4 for more details.
Revised: See Appendix A.4 for more details.

Original: Evaluation of our approach thus consists of: (i) given a claim, use BM-25 to return a list of promisingstart locations; (ii) from the top N = 10 of these, run our navigation model for 20 steps, using thetarget encoder to embed the claim, (iii) use the re-ranker to score sentences visited against the claimand (iv) select the top 5 for computing the F1 score, for comparison on the FEVER benchmark.
Revised: Evaluation of our approach thus consists of: (i) given a claim/question, use BM-25 to return a list of promising start locations; (ii) from the top N = 10 of these, run our navigation model for 20 steps, using the target encoder to embed the claim/question, (iii) use the re-ranker to score sentences visited against the claim/question and (iv) select the top 5 for computing the F1 score, for comparison on the benchmarks.

Original:  Experiments 
Revised:  Experiments 5.1 Investigation of training trajectory distribution

Original: 5.1 Investigation of training trajectory distribution

Original: On the smaller 200k node graph, we investigate different choices for the trajectory policy distributions
Revised: On the smaller 200k node graph, we investigate different choices for the trajectory policy distributions

Original: E described in Section 3: (i) Random Forward Trajectories , where we randomly sample startnodes and do a random walk to get a target, (ii) Reverse Trajectories , where we randomly sample atarget node and and do a random walk on the reverse graph to get a start node, and (iii) Random
Revised: E described in Section 3: (i) Random Forward Trajectories , where we randomly sample start nodes and do a random walk to get a target, (ii) Reverse Trajectories , where we randomly sample a target node and and do a random walk on the reverse graph to get a start node, and (iii) Random

Original: Shortest Paths where we randomly sample a start node and take a random walk and then computethe shortest path between the start and target nodes.
Revised: Shortest Paths where we randomly sample a start node and take a random walk and then compute the shortest path between the start and target nodes.

Original: For each of these we trained policies on 5 ,10 , and 20 step navigation tasks.
Revised: For each of these we trained policies on 5 , 10 , and 20 step navigation tasks.

Original: We evaluate these in two different ways: (i) forward sampling ofsource and target in the disjoint evaluation graph, (ii) reverse sampling of source and target.
Revised: We evaluate these in two different ways: (i) forward sampling of source and target in the disjoint evaluation graph, (ii) reverse sampling of source and target.

Original: Theresults of this experiment are shown in Table 1.
Revised: The results of this experiment are shown in Table 1.

Original: Unsurprisingly, we can see that forward and reversetrained policies both do better when evaluated in the way that matches training.
Revised: Unsurprisingly, we can see that forward and reverse trained policies both do better when evaluated in the way that matches training.

Original: However, forwardtrajectory policies generalize much better to reverse navigation.
Revised: However, forward trajectory policies generalize much better to reverse navigation.

Original: When we analyze the traces of theseexperiments, we see that in the reverse sampling, many target nodes are in “dead-ends” where manytarget nodes do not have long random trajectories to sample.
Revised: When we analyze the traces of these experiments, we see that in the reverse sampling, many target nodes are in “dead-ends” where many target nodes do not have long random trajectories to sample.

Original: This results in easier navigation (as wecan see from higher accuracy in reverse trajectory trained policies on reverse navigation) and moreextreme overfitting, leading to poor performance of reverse trajectory on forward navigation.
Revised: This results in easier navigation (as we can see from higher accuracy in reverse trajectory trained policies on reverse navigation) and more extreme overfitting, leading to poor performance of reverse trajectory on forward navigation.

Original: Not surprisingly, we also see that training on a shortest path trajectories leads to better performanceboth in the forward and reverse sampling.
Revised: Not surprisingly, we also see that training on a shortest path trajectories leads to better performance both in the forward and reverse sampling.

Original: This advantage is especially seen in the 20 -step navigationtask.
Revised: This advantage is especially seen in the 20 -step navigation task.

Original: Ideally, we would always train on shortest paths.
Revised: Ideally, we would always train on shortest paths.

Original: However, the time complexity of precomputing all shortest paths in a graph using Dijkstra’s algorithm is O ( V ( E + V logV )) for thenumber of nodes V and number of edges E in the graph.
Revised: However, the time complexity of precomputing all shortest paths in a graph using Dijkstra’s algorithm is O ( V ( E + V logV )) for the number of nodes V and number of edges E in the graph.

Original: On the full 38 M Wikipedia graph, thiscomputation would be completely intractable.
Revised: On the full 38 M Wikipedia graph, this computation would be completely intractable.

Original: Luckily, the random forward trajectories performalmost as well and can be computed in constant time with respect to the size of the graph and onlylinear time with respect to the trajectory length.
Revised: Luckily, the random forward trajectories perform almost as well and can be computed in constant time with respect to the size of the graph and only linear time with respect to the trajectory length.

Original: For all subsequent experiments, we use the Random
Revised: For all subsequent experiments, we use the Random

Original: Forward Trajectories for our trajectory policy and evaluate on the forward version of the navigationtasks, where we call our approach Random Forward Behavioral Cloning (RFBC) . 
Revised: Forward Trajectories for our trajectory policy and evaluate on the forward version of the navigation tasks, where we call our approach Random Forward Behavioral Cloning (RFBC) . [Table]

Original: 5.2 Navigation
Revised: 5.2 Navigation

Original: We next do a more thorough set of experiments on the smaller 200 k Wikipedia graph.
Revised: We next do a more thorough set of experiments on the smaller 200 k Wikipedia graph.

Original: Because thegraph is much smaller than the full Wikipedia and thus prone to overfitting, for all of our methods and[Table] [Table] baselines we use the fixed RoBERTA embeddings ϕ 
Revised: Because the graph is much smaller than the full Wikipedia and thus prone to overfitting, for all of our methods and baselines we use the fixed RoBERTA embeddings ϕ ( s i ) , the simplest 1-layer feed-forward network, and MiniBERT Turc et al.

Original: ( s i ) , the simplest 1-layer feed-forward network,and MiniBERT Turc et al.

Original: [2019] for target sentence embedding ϕ
Revised: [2019] for target sentence embedding ϕ

Original: ( s g ) .
Revised: ( s g ) .

Original: Alongside our approach [Table] we employ the following baselines/ablations:
Revised: Alongside our approach, we employ the following baselines/ablations:

Original: • RFBC:
Revised: • RFBC:

Original: Random-forward behavioral cloning (our approach).
Revised: Random-forward behavioral cloning (our approach).

Original: • RL: Navigation using a policy gradients RL agent with same policy architecture as our model.
Revised: • RL: Navigation using a policy gradients RL agent with same policy architecture as our model.

Original: See Sec.
Revised: See Sec.

Original: A.5 for more details.• RFBC + RF: RFBC but with random features sampled from the unit sphere instead of ϕ• RL + RF: RL with random 
Revised: A.5 for more details. • RFBC + RF: RFBC but with random features sampled from the unit sphere instead of ϕ • RL + RF: RL with random features.

Original: features.• Random: policy that chooses random action (out-link) at each timestep.• Greedy: selects the action embedding which has the smallest cosine distance with the target.• Random DFS: DFS of depth equal to number of steps, choosing actions at random.• Greedy DFS: DFS but select action with smallest cosine distance to the target.
Revised: • Random: policy that chooses random action (out-link) at each timestep. • Greedy: selects the action embedding which has the smallest cosine distance with the target. • Random DFS: DFS of depth equal to number of steps, choosing actions at random. • Greedy DFS: DFS but select action with smallest cosine distance to the target.

Original: The results evaluated on the held-out graph are shown in columns 1-4 of Table 2.
Revised: The results evaluated on the held-out graph are shown in columns 1-4 of Table 2.

Original: Our approach(RFBC) performs well, achieving 77% in the multistep case.
Revised: Our approach (RFBC) performs well, achieving 77% in the multistep case.

Original: Performance also drops as the distanceto the target increases.
Revised: Performance also drops as the distance to the target increases.

Original: In contrast, the RL agent performs poorly (around 40 % on navigation tasks).
Revised: In contrast, the RL agent performs poorly (around 40 % on navigation tasks).

Original: Because we generate virtually infinite expert trajectories with RFBC, RL performs worse as it istrained with essentially the same amount of exploration, but with a more sparse signal than behavioralcloning.
Revised: Because we generate virtually infinite expert trajectories with RFBC, RL performs worse as it is trained with essentially the same amount of exploration, but with a more sparse signal than behavioral cloning.

Original: Using random features in place of ϕ ( . ) caused the performance to drop to near chance,showing that the models are utilizing the semantics at each node and are not relying on a generalsearch strategy of some kind.
Revised: Using random features in place of ϕ ( . ) caused the performance to drop to near chance, showing that the models are utilizing the semantics at each node and are not relying on a general search strategy of some kind.

Original: Random methods also do quite poorly, even when adding a DFS.
Revised: Random methods also do quite poorly, even when adding a DFS.

Original: Greedy methods do a bit better than random, especially with DFS, but still well below our method.
Revised: Greedy methods do a bit better than random, especially with DFS, but still well below our method.

Original: We also consider the more challenging sentence search task described in Section 4.1.
Revised: We also consider the more challenging sentence search task described in Section 4.1.

Original: This taskrequires not only finding a known target node, but learning to find a target node given only a singlerandom sentence from that node, more closely matching our downstream task and the way humanstend to approach finding information on the web.
Revised: This task requires not only finding a known target node, but learning to find a target node given only a single random sentence from that node, more closely matching our downstream task and the way humans tend to approach finding information on the web.

Original: Columns 5-7 of Table 2 show performance on thistask for our approach and RL.
Revised: Columns 5-7 of Table 2 show performance on this task for our approach and RL.

Original: Our method has reduced performance on the task due to the extradifficulty of learning the target embedding, but still performs reasonably.
Revised: Our method has reduced performance on the task due to the extra difficulty of learning the target embedding, but still performs reasonably.

Original: RL, however, degrades tonear chance performance.
Revised: RL, however, degrades to near chance performance.

Original: 5.3 Navigation on full 38M node Wikipedia graph:
Revised: 5.3 Navigation on full 38M node Wikipedia graph:

Original: Our method scales naturally to much larger graphs, 
Revised: Our method scales naturally to much larger graphs, which we demonstrate by training on the entire

Revised: Wikipedia 2017 graph.

Revised: We evaluate on the entire Wikipedia 2018 graph.

Revised: Each graph is ∼ 38 M nodes and ∼ 380 M edges (see Table 3 for details).

Original: [Table] [Table]
Revised: [Table] [Table]

Original: We explore both architectures described in Section 4.2 for the policy network: (i) the single feedforward layer (as used in the smaller graph) and (ii) the 4-layer Transformer model.
Revised: We explore both architectures described in Section 4.2 for the policy network: (i) the single feedforward layer (as used in the smaller graph) and (ii) the 4-layer Transformer model.

Original: We also compareusing a fixed RoBERTA model for ϕ versus pre-training a text transformer directly on the navigationtask (see Appendix B.1).
Revised: We also compare using a fixed RoBERTA model for ϕ versus pre-training a text transformer directly on the navigation task (see Appendix B.1).

Original: DistillBERT Sanh et al.
Revised: DistillBERT Sanh et al.

Original: [2019] is used for ϕ ( n g ).
Revised: [2019] is used for ϕ ( n g )

Original:  Table 2 shows the results on the full graph, evaluating on { 5 , 10 , 20 } step tasks, for both navigationand sentence search.
Revised: [Table] Table 4 shows the results on the full graph, evaluating on { 5 , 10 , 20 } step tasks, for both navigation and sentence search.

Original: Figure 2 shows example trajectories of the trained agent.
Revised: Figure 2 shows example trajectories of the trained agent.

Original: The first thing to noteis that the overall navigation and sentence search performance of our methods is high.
Revised: The first thing to note is that the overall navigation and sentence search performance of our methods is high.

Original: The[-
-]
Revised: The[-
-]

Original: Wikipedia graph we use for evaluation contains over 5 million articles, over 38 million nodes and387 million edges.
Revised: Wikipedia graph we use for evaluation contains over 5 million articles, over 38 million nodes and 387 million edges.

Original: On this graph (which we generalize to from the previous year’s graph) our bestmethod can find an article 20 steps away 92 .
Revised: On this graph (which we generalize to from the previous year’s graph) our best method can find an article 20 steps away 92 .

Original: 2% percent of the time and can locate it given just asingle sentence of that article 90 .
Revised: 2% percent of the time and can locate it given just a single sentence of that article 90 .

Original: 2% of the time.
Revised: 2% of the time.

Original: This compares favorably with a similar evaluationperformed in Nogueira and Cho [2016], where their WebNav approach achieved a 12 . 5% success ratefor 16 step tasks on a 12M node graph.
Revised: This compares favorably with a similar evaluation performed in Nogueira and Cho [2016], where their WebNav approach achieved a 12 . 5% success rate for 16 step tasks on a 12M node graph.

Original: Compared to the performance of our method on the smaller200 k graph, we see that performance has greatly improved, most likely due to the addition of ordersof magnitude more training data.
Revised: Compared to the performance of our method on the smaller 200 k graph, we see that performance has greatly improved, most likely due to the addition of orders of magnitude more training data.

Original: In particular we see that sentence search performance on 20 -stepimproves greatly from 34 .
Revised: In particular we see that sentence search performance on 20 -step improves greatly from 34 .

Original: 6% to a best performance of 90 . 2% .
Revised: 6% to a best performance of 90 . 2% .

Original: Additionally, in all cases, our learned embeddings perform better than using the fixed features,derived from large language models.
Revised: Additionally, in all cases, our learned embeddings perform better than using the fixed features, derived from large language models.

Original: In general, feed-forward policies perform best, except for 20step navigation tasks.
Revised: In general, feed-forward policies perform best, except forstep navigation tasks.

Original: For longer trajectories, the Transformer policy network achieves slightly betterperformance, while on sentence search, the feed-forward model is consistently superior.
Revised: For longer trajectories, the Transformer policy network achieves slightly better performance, while on sentence search, the feed-forward model is consistently superior.

Original: 5.4 Application to fact verification
Revised: 5.4 Application to fact verification

Original: We now evaluate the ability of our navigation approach to select evidence on the FEVER developmentset Thorne et al.
Revised: We now evaluate the ability of our navigation approach to select evidence on the FEVER development set Thorne et al.

Original: [2018] with results shown in Table 5.
Revised: [2018] with results shown in Table 5.

Original: The primary metric used is F1@top5 butwe also give precision and recall.
Revised: The primary metric used is F1@top5 but we also give precision and recall.

Original: Paired with a basic retrieval (BM25) and re-ranker (TF-IDF), theapproach obtains a respectable F1 of 0.46.
Revised: Paired with a basic retrieval (BM25) and re-ranker (TF-IDF), the approach obtains a respectable F1 of 0.46.

Original: If we remove our navigation component then this dropssubstantially to 0.36, demonstrating its utility.
Revised: If we remove our navigation component then this drops substantially to 0.36, demonstrating its utility.

Original: Swapping to a more powerful re-ranker (BigBird)boosts the F1 to 0.731, which is competitive with the state-of-the-art (FEVER leaderboard [2022]
Revised: Swapping to a more powerful re-ranker (BigBird)

Original: [Table]
Revised: [Table]

Original: et al.
Revised: Embed train (ours) + Feed-forward 96.

Original: [Table]
Revised: Embed train (ours) + Transformer 93.

Original: [Table]
Revised: [Table]

Original: BM25 + RFBC (Ours) + BigBird 0. Stammbach [2021] 0. rank #6; rank #1 F1 is 0.799).
Revised: boosts the F1 to 0.731, which is competitive with the state-of-the-art (FEVER leaderboard [2022] rank #6; rank #1 F1 is 0.799).

Original: We also compare to a leading approach Stammbach [2021] thatsimilarly relies on the BigBird re-ranker, with our approach having a significantly better F1 score.
Revised: We also compare to a leading approach Stammbach [2021] that similarly relies on the BigBird re-ranker, with our approach having a significantly better F1 score.

Original: A possible explanation is that their approach overwhelms the re-ranker with all possible hyperlinkswhereas, navigation being inherently sparse, our model presents a more refined set for re-ranking.
Revised: A possible explanation is that their approach overwhelms the re-ranker with all possible hyperlinks whereas, navigation being inherently sparse, our model presents a more refined set for re-ranking.

Original: Wenote that our approach: (a) is significantly simpler than many top ranked approaches and (b) selectsevidence over a much larger set than the curated version of Wikipedia used in FEVER (38M vs 5M).
Revised: We note that our approach: (a) is significantly simpler than many top ranked approaches and (b) selects evidence over a much larger set than the curated version of Wikipedia used in FEVER (38M vs 5M).

Original: Examples of successful navigation traces are shown in Figure 3. 
Revised: Examples of successful navigation traces are shown in Figure 3. [Table] [Figure] [Figure]

Original: Discussion
Revised: [Table] (cid:19)(cid:29)(cid:3)(cid:55)(cid:75)(cid:72)(cid:3)(cid:59)(cid:16)(cid:41)(cid:76)(cid:79)(cid:72)(cid:86)(cid:3)(cid:11)(cid:27)(cid:23)(cid:12)(cid:20)(cid:29)(cid:3)(cid:43)(cid:88)(cid:80)(cid:69)(cid:88)(cid:74)(cid:3)(cid:11)(cid:55)(cid:75)(cid:72)(cid:3)(cid:59)(cid:16)(cid:41)(cid:76)(cid:79)(cid:72)(cid:86)(cid:12)(cid:21)(cid:29)(cid:3)(cid:43)(cid:88)(cid:80)(cid:69)(cid:88)(cid:74)(cid:3)(cid:11)(cid:55)(cid:75)(cid:72)(cid:3)(cid:59)(cid:16)(cid:41)(cid:76)(cid:79)(cid:72)(cid:86)(cid:12) [Table] 5.5 Application to Question Answering

Revised: Finally, we present results on another downstream task where navigation helps.

Revised: In particular, we consider the task of finding correct evidence passage for open domain question answering.

Revised: We use the Natural Question (NQ) open-domain dataset presented in Lee et al.

Revised: [2019] which has been aligned to Wikipedia dump of December 20, 2018 by [Karpukhin et al., 2020].

Revised: Most prior works in literature utilize the processed collection of 21M passages provided by by [Karpukhin et al., 2020], but unfortunately we cannot use it as it has no hyperlink information which is crucial for our graph building.

Revised: So we re-align the questions in NQ to our 2018 graph with 38M passages (i.e. nodes).

Revised: We compute the shortest-path distance between top-1 retrieval of BM25 and the target node in this graph in Figure 10.

Revised: As can be seen, most path lengths are relatively short, which implies that BM25 lands us in the right vicinity and by a small amount of navigation around we will be able to find the right evidence passage.

Revised: Since we target navigating to the exact evidence passage required to answer the question, we use recall@{1,2,3,4,5} for finding the gold evidence passage as our metric.

Revised: More commonly the metric marks a retrieved passage to be correct if it contains the answer string, but this causes a lot of false positives, e.g. the answer string appears in a totally irrelevant context.

Revised: Our method is similar as to our approach to FEVER.

Revised: For each question, we first run BM25 over all nodes in the graph and take the top-5 matches as starting nodes.

Revised: From each starting node, the agent then navigates for 20 steps.

Revised: All the 100 nodes visited by the agent is then ranked by a simple cross-attention model.

Revised: For the cross-attention model, we follow Hofstätter et al.

Revised: [2020] to train 6-layer BigBird [Zaheer et al., 2020].

Revised: [Table]

Revised: [Table]

Revised: The results are tabulated in Table 7.

Revised: Paired with a basic retrieval (BM25) and re-ranker (BigBird), the approach obtains a respectable Recall@1 of 31.6.

Revised: If we remove our navigation component then this drops substantially to 22.2, demonstrating its utility.

Revised: As a reference, we also ran a state-of-the-art system RocketQA

Revised: [Qu et al., 2021] on our setup and evaluated it using our harder metric.

Revised: It is worthwhile to note that our approach (a) is significantly simpler than state-of-the-art approaches like RocketQA which employ 4 stages of dual-encoder and cross-attention models

Revised: [Table]

Revised: and (b) selects evidence over a larger set than the commonly used preprocess Wikipedia passages (38M vs 21M).

Revised:  Discussion

Original: We have presented a simple and effective scheme for navigating a large Wikipedia graph that isapplicable to more general web navigation problems.
Revised: We have presented a simple and effective scheme for navigating a large Wikipedia graph that is applicable to more general web navigation problems.

Original: We show that behavioral cloning of randomtrajectories is a viable approach to learning both entity embeddings and a navigation policy.
Revised: We show that behavioral cloning of random trajectories is a viable approach to learning both entity embeddings and a navigation policy.

Original: Whenapplied to the FEVER verification task, they offer highly competitive performance, whilst beingcomplementary to existing approaches.
Revised: When applied to the FEVER verification task and the Open-Ended Natural Questions dataset, they offer highly competitive performance, whilst being complementary to existing approaches.

Original: One limitation of our approach when we move from Wikipediato the wider Internet is that our scheme relies on a good target encoder.
Revised: One limitation of our approach when we move from Wikipedia to the wider Internet is that our scheme relies on a good target encoder.

Original: For navigation and ourdownstream task, there was a clear ground-truth target node available training, but in other settingsthis might not be the case.
Revised: For navigation and our downstream tasks, there was a clear ground-truth target node available training, but in other settings this might not be the case.

Original: Another limitation is that we require a re-ranker to decide on the finalretrieved sentence, but ideally the agent would decide for itself when it has reached the correct node.[Table] [Figure] [Figure]
Revised: Another limitation is that we require a re-ranker to decide on the final retrieved sentence, but ideally the agent would decide for itself when it has reached the correct node.

Original: [Table]

Original: Checklist
Revised: Checklist

Original: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
Revised: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?

Original: [Yes] (b) Did you describe the limitations of your work?
Revised: [Yes] (b) Did you describe the limitations of your work?

Original: [Yes] (c) Did you discuss any potential negative societal impacts of your work?
Revised: [Yes] (c) Did you discuss any potential negative societal impacts of your work?

Original: [N/A] (d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
Revised: [N/A] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?

Original: [Yes]2.
Revised: [Yes] 2.

Original: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
Revised: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?

Original: [N/A] (b) Did you include complete proofs of all theoretical results?
Revised: [N/A] (b) Did you include complete proofs of all theoretical results?

Original: [N/A] 
Revised: [N/A] 3.

Original: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main exper- imental results (either in the supplemental material or as a URL)?
Revised: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Original: [Yes] The paper and supplemental provide full details on how to reproduce the simple approach.
Revised: [Yes] The paper and supplemental provide full details on how to reproduce the simple approach.

Original: The datasets used are all public.
Revised: The datasets used are all public.

Original: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
Revised: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?

Original: [Yes] In appendix.
Revised: [Yes] In appendix.

Original: (c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)?
Revised: (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Original: [N/A] Our experiments are on large datasets and thus did not require multiple runs.
Revised: [N/A] Our experiments are on large datasets and thus did not require multiple runs.

Original: (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
Revised: (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?

Original: [No] The experiments were run on a proprietary infrastructure on which it is hard to estimate the total amount of compute.
Revised: [No] The experiments were run on a proprietary infrastructure on which it is hard to estimate the total amount of compute.

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?

Original: [Yes] (b) Did you mention the license of the assets?
Revised: [Yes] (b) Did you mention the license of the assets?

Original: [Yes] See Thorne et al.
Revised: [Yes] See Thorne et al.

Original: reference.
Revised: reference.

Original: (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [No] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
Revised: [No] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?

Original: [N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
Revised: [N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?

Original: [N/A] 
Revised: [N/A] 5.

Original: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
Revised: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?

Original: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
Revised: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?

Original: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
Revised: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?

Original: [N/A]
Revised: [N/A]

Original: A Dataset / Environment DetailsA.1 Wikipedia data processing
Revised: A Dataset / Environment Details A.1 Wikipedia data processing

Original: We start by downloading English Wikipedia snapshot (the pages-articles.xml.bz2 file) from Wikimedia 2 or Internet Archive.
Revised: We start by downloading English Wikipedia snapshot (the pages-articles.xml.bz2 file) from Wikimedia 2 or Internet Archive.

Original: 3 We extract text from English Wikipedia for a given snapshot using
Revised: 3 We extract text from English Wikipedia for a given snapshot using

Original: Gensim’s WikiCorpus class.
Revised: Gensim’s WikiCorpus class.

Original: For each page, this tool extracts the plain text and hyperlinks, and stripsout all structured data such as lists and figures.
Revised: For each page, this tool extracts the plain text and hyperlinks, and strips out all structured data such as lists and figures.

Original: To avoid pages without sufficient textual content, wefilter out categories, listical, disambiguation pages and any other page with less than 200 characters.
Revised: To avoid pages without sufficient textual content, we filter out categories, listical, disambiguation pages and any other page with less than 200 characters.

Original: Each article ( µ = 1000 words) is split into paragraph-sized blocks ( µ = 100 words), which formthe nodes in our Wikipedia Graph.
Revised: Each article ( µ = 1000 words) is split into paragraph-sized blocks ( µ = 100 words), which form the nodes in our Wikipedia Graph.

Original: This was done both for conceptual reasons (humans don’t readan entire article at once but look at bits and pieces of it at one go) and for modeling reasons (mostlanguage models have an upper input token length that is smaller than the average Wikipedia page).
Revised: This was done both for conceptual reasons (humans don’t read an entire article at once but look at bits and pieces of it at one go) and for modeling reasons (most language models have an upper input token length that is smaller than the average Wikipedia page).

Original: The edges in the graph are formed in three ways:
Revised: The edges in the graph are formed in three ways:

Original: 1. for nodes in the same article, we add “previous node” and “next node” links to connect them in a chain; 
Revised: 1. for nodes in the same article, we add “previous node” and “next node” links to connect them in a chain; 2.

Original: organic hyperlinks – the internal links connecting Wikipedia pages to each other; and 
Revised: organic hyperlinks – the internal links connecting Wikipedia pages to each other; and 3.

Original: we additionally run entity linking 5 because repeat mentions of an entity in Wikipedia lackhyperlinks, which is problematic when articles are split up.
Revised: we additionally run entity linking 5 because repeat mentions of an entity in Wikipedia lack hyperlinks, which is problematic when articles are split up.

Original: This constitutes our Wikipedia graph construction, where the text blocks act as vertices and theorganic hyperlinks along with entity links are the edges.
Revised: This constitutes our Wikipedia graph construction, where the text blocks act as vertices and the organic hyperlinks along with entity links are the edges.

Original: We store the graph as an adjacency listwith metadata, using a memory mapped key-value data-structure to enable fast random access duringnavigation.
Revised: We store the graph as an adjacency list with metadata, using a memory mapped key-value data-structure to enable fast random access during navigation.

Original: We run the above graph generation pipeline for two different snapshots of Wikipedia:
Revised: We run the above graph generation pipeline for two different snapshots of Wikipedia:

Original: June 01 2017 6 : This corresponds to FEVER fact verification dataset.2.
Revised: June 01 2017 6 : This corresponds to FEVER fact verification dataset. 2.

Original: December 2018 7 : This corresponds to Natural Questions open dataset.
Revised: December 2018 7 : This corresponds to Natural Questions open dataset.

Original: A.2 Smaller 200k node graph construction
Revised: A.2 Smaller 200k node graph construction

Original: We construct smaller train and eval graphs with 200k nodes each for the following reasons:
Revised: We construct smaller train and eval graphs with 200k nodes each for the following reasons:

Original: Checking strict generalization by making train and eval graphs to be totally disjoint, i.e. nocommon nodes or edges.
Revised: Checking strict generalization by making train and eval graphs to be totally disjoint, i e. no common nodes or edges.

Original: The full 2017 and 2018 are sufficiently different but not entirelydisjoint.2.
Revised: The full 2017 and 2018 are sufficiently different but not entirely disjoint.

Original: Running experiments on full graphs are expensive, so for more thorough evaluation acrossmultiple baselines we constructed the smaller graphs.
Revised: Running experiments on full graphs are expensive, so for more thorough evaluation across multiple baselines we constructed the smaller graphs.

Original: To construct these two smaller graphs, we start with the full 2018 graph.
Revised: To construct these two smaller graphs, we start with the full 2018 graph.

Original: We sort the nodes by theirin-degrees.
Revised: We sort the nodes by their in-degrees.

Original: We mark nodes with odd ranks to be in train and even ranks to be in eval, which ensurescomplete separation between the two.
Revised: We mark nodes with odd ranks to be in train and even ranks to be in eval, which ensures complete separation between the two.

Original: For the train graph we start with rank 1 node (and for eval withrank 2 node) and then select all its neighbors which have odd rank (even rank).
Revised: For the train graph we start with rank 1 node (and for eval with rank 2 node) and then select all its neighbors which have odd rank (even rank).

Original: Then all the odd(even) nodes connected to the selected ones are picked.
Revised: Then all the odd (even) nodes connected to the selected ones are picked.

Original: This process is continued until the desirednumber of nodes are selected.
Revised: This process is continued until the desired number of nodes are selected.

Original: A.3 Dataset statistics
Revised: A.3 Dataset statistics

Original: We construct two versions of the full Wikipedia graph following the method outlined in Sec.
Revised: We construct two versions of the full Wikipedia graph following the method outlined in Sec.

Original: A.1.
Revised: A.1.

Original: The main summary statistics (number of articles, nodes, edges, words per node, median path length)of the resulting graphs are listed in Table 3.
Revised: The main summary statistics (number of articles, nodes, edges, words per node, median path length) of the resulting graphs are listed in Table 3.

Original: A more detailed picture of the graph can be obtained by looking at its degree distribution, whichis presented in Figure 4.
Revised: A more detailed picture of the graph can be obtained by looking at its degree distribution, which is presented in Figure 4.

Original: It shows the graph has tell-tale sign of web-graph: it has a power-lawdistribution with a few hub nodes.
Revised: It shows the graph has tell-tale sign of web-graph: it has a power-law distribution with a few hub nodes

Original: [Figure]
Revised: [Figure]

Original: We try to estimate how far are two nodes in the graph.
Revised: We try to estimate how far are two nodes in the graph.

Original: Calculating the entire distribution of shortestpath length is prohibitively expensive for such large graph.
Revised: Calculating the entire distribution of shortest path length is prohibitively expensive for such large graph.

Original: We follow Ye et al.
Revised: We follow Ye et al.

Original: [2010] method toget an approximation.
Revised: [2010] method to get an approximation.

Original: We start by selecting 250k nodes having highest in-degree, which is a goodapproximation of the giant component in a web graph as required by Ye et al.
Revised: We start by selecting 250k nodes having highest in-degree, which is a good approximation of the giant component in a web graph as required by Ye et al.

Original: We thenperform a single source to all shortest paths on each of these 250k nodes using Dijkstra’s algorithmfor sparse graphs.
Revised: We then perform a single source to all shortest paths on each of these 250k nodes using Dijkstra’s algorithm for sparse graphs.

Original: This resulted in computing approximately 1 trillion shortest paths based on whichthe estimated distribution of shortest path length is shown in Figure5.
Revised: This resulted in computing approximately 1 trillion shortest paths based on which the estimated distribution of shortest path length is shown in Figure

Original: [Figure]
Revised: [Figure]

Original:  Finally, we would like to point out that there is significantevolution (difference) between the 2017 and 2018 graph.
Revised: c t i o n o f N o d e s Finally, we would like to point out that there is significant evolution (difference) between the 2017 and 2018 graph.

Original: Some important statistics reflecting the changes across thetwo graphs are:
Revised: Some important statistics reflecting the changes across the two graphs are:

Original: • New articles: 404,773 (7.5%) • New nodes: 3,871,578 (10%) 
Revised: • New articles: 404,773 (7.5%) • New nodes: 3,871,578 (10%) •

Original: • Deleted articles: 58,476 (1.1%)[Figure] • Deleted nodes: 874,372 (2.3%)
Revised: Deleted articles: 58,476 (1.1%) • Deleted nodes: 874,372 (2.3%)

Original: Further in Figure 6, we show histogram of changes inedges among the common nodes between the two graphs.
Revised: Further in Figure 6, we show histogram of changes in edges among the common nodes between the two graphs.

Original: Apart from addition/deletion of organic hyperlinks, justmodifications of text would yield to differences as the chunking will be different.
Revised: Apart from addition/deletion of organic hyperlinks, just modifications of text would yield to differences as the chunking will be different.

Original: This shows significant generalization is needed to successfully navigate across the two graphs, simply memorizationwill not work.
Revised: This shows significant generalization is needed to successfully navigate across the two graphs, simply memorization will not work.

Original: [Figure]
Revised: [Figure]

Original: A.4 FEVER data processing
Revised: A.4 FEVER data processing

Original: [Figure]

Original: FEVER Thorne et al.
Revised: FEVER Thorne et al.

Original: [2018] is a dataset containing 185 , 445 claims labeled as S UPPORTED , R EFUTED , or N OT
Revised: [2018] is a dataset containing 185 , 445 claims labeled as S UPPORTED , R EFUTED , or N OT

Original: E NOUGH I NFO .
Revised: E NOUGH I NFO .

Original: Verifiable claims (i.e. S UPPORTED or R EFUTED ) are annotated withevidence sentences supporting this classification, which are drawn from a preprocessed version of the
Revised: Verifiable claims (i.e. S UPPORTED or R EFUTED ) are annotated with evidence sentences supporting this classification, which are drawn from a preprocessed version of the

Original: June 2017 Wikipedia snapshot.
Revised: June 2017 Wikipedia snapshot.

Original: Notably, only the article introductions are retained, and claims aregenerated from a curated set of approximately 50 , 000 popular pages.
Revised: Notably, only the article introductions are retained, and claims are generated from a curated set of approximately 50 , 000 popular pages.

Original: To make an aligned navigation graph for FEVER, we start with the June 01 2017 snapshot of
Revised: To make an aligned navigation graph for FEVER, we start with the June 01 2017 snapshot of

Original: Wikipedia and build the graph using the procedure described in the previous section.
Revised: Wikipedia and build the graph using the procedure described in the previous section.

Original: Because ofdifferences in text preprocessing, FEVER’s version of Wikipedia does not precisely align to ourown.
Revised: Because of differences in text preprocessing, FEVER’s version of Wikipedia does not precisely align to our own.

Original: We then try to match FEVER’s Wikipedia articles to our graph, relying on Wikipedia URLredirections for handling disambiguations.
Revised: We then try to match FEVER’s Wikipedia articles to our graph, relying on Wikipedia URL redirections for handling disambiguations.

Original: This resulted in 99 .
Revised: This resulted in 99 .

Original: 5% matching of articles between ournavigation graph and the FEVER version of Wikipedia.
Revised: 5% matching of articles between our navigation graph and the FEVER version of Wikipedia.

Original: Second, we align the evidence sentences in
Revised: Second, we align the evidence sentences in

Original: FEVER to sentences from our text blocks using a fuzzy string match.
Revised: FEVER to sentences from our text blocks using a fuzzy string match.

Original: Specifically, we use the tokenset ratio to score the similarity between an evidence sentence and the text in a graph node.
Revised: Specifically, we use the token set ratio to score the similarity between an evidence sentence and the text in a graph node.

Original: If the scorebetween a sentence and a node is > = 80 , the node is added as an evidence node for the given claim.
Revised: If the score between a sentence and a node is > = 80 , the node is added as an evidence node for the given claim.

Original: This threshold is sufficiently high to minimize the chances of a false positive, while also matching ahigh percentage of evidence sentences.
Revised: This threshold is sufficiently high to minimize the chances of a false positive, while also matching a high percentage of evidence sentences.

Original: Some sentences are difficult to match, particularly those thatare split between two text blocks.
Revised: Some sentences are difficult to match, particularly those that are split between two text blocks.

Original: Ultimately, 93 .
Revised: Ultimately, 93 .

Original: 1% of all evidence sentences were matched to anode.
Revised: 1% of all evidence sentences were matched to a node.

Original: This gives us an augmented FEVER dataset, where for each claim we have a corresponding setof evidence nodes in the graph.
Revised: This gives us an augmented FEVER dataset, where for each claim we have a corresponding set of evidence nodes in the graph.

Original: These nodes are then used as navigation targets for fine-tuning duringthe training phase.
Revised: These nodes are then used as navigation targets for fine-tuning during the training phase.

Original: We generate trajectories for BC by running BM25 over all nodes in the graph,taking the top 10 matches as starting nodes, and finding shortest paths to evidence nodes.
Revised: We generate trajectories for BC by running BM25 over all nodes in the graph, taking the top 10 matches as starting nodes, and finding shortest paths to evidence nodes.

Original: We compute the shortest-path distance between top-1 retrieval of BM25 and target node in this graphin Figure 7.
Revised: We compute the shortest-path distance between top-1 retrieval of BM25 and target node in this graph in Figure 7.

Original: As can be seen that most path lengths are relatively short, which implies that BM25lands us in the right vicinity and by a small amount of navigation around we will be able to find theright evidence passage.
Revised: As can be seen that most path lengths are relatively short, which implies that BM lands us in the right vicinity and by a small amount of navigation around we will be able to find the right evidence passage.

Original: [Figure]
Revised: [Figure]

Original: A.5 RL Baseline
Revised: A.5 RL Baseline

Original: For the purposes of the RL baseline, we can view graph navigation as a goal-conditioned MDP.
Revised: For the purposes of the RL baseline, we can view graph navigation as a goal-conditioned MDP.

Original: 8 Ourobservation is the same as the inputs to the RFBC models.
Revised: Our observation is the same as the inputs to the RFBC models.

Original: The reward function R for current node nand goal node n g we can write simply as: 
Revised: The reward function R for current node n and goal node n g we can write simply as: [Equation]

Original: R = (cid:26) 1 n = n g 0 otherwise (5)
Revised: (cid:26) [Equation] g 0 otherwise (5)

Original: See Appendix B.2 for model and training details of RL baselines.
Revised: See Appendix B.2 for model and training details of RL baselines.

Original: B Experimental details 
Revised: B Experimental details B.1 Embedding pretraining

Original: B.1 Embedding pretraining

Original: This section details our embedding pretraining procedure, as introduced in Section 3.
Revised: This section details our embedding pretraining procedure, as introduced in Section 3.

Original: Model details
Revised: Model details

Original: The transformer is fine-tuned from a pre-trained RoBERTa
Revised: The transformer is fine-tuned from a pre-trained RoBERTa

Original: [Liu et al., 2019] model.
Revised: [Liu et al., 2019] model.

Original: As in Equation 1 our model is trained to optimize
Revised: As in Equation 1 our model is trained to optimize

Original: L ( θ ) =

Original: E E ( n

Original: [Equation]

Original: ...,n T ) (cid:34) T − 1 (cid:88) t = log p θ ( n t | n t , n g

Original: [Equation]
Revised: [Equation]

Original:  where we set n g = n T .
Revised: 6) where we set n g = n T .

Original: We parameterize this distribution as
Revised: We parameterize this distribution as

Original: [Equation]
Revised: [Equation]

Original: where s ϕ ( · ) and a ϕ ( · , · ) are functions which extract embeddings for an entire node and a node-actioncombination, respectively and f θ ( · , · , · ) is a function which combines these embeddings to produceaction probabilities.
Revised: where s ϕ ( · ) and a ϕ ( · , · ) are functions which extract embeddings for an entire node and a node-action combination, respectively and f θ ( · , · , · ) is a function which combines these embeddings to produce action probabilities.

Original: At their core, s ϕ and a ϕ are based on the same transformer model.
Revised: At their core, s ϕ and a ϕ are based on the same transformer model.

Original: We tokenize the text at node n to produce L tokens and pass these tokens through the transformer to produce Lembeddings.
Revised: We tokenize the text at node n to produce L tokens and pass these tokens through the transformer to produce L embeddings.

Original: The state representation s ϕ ( n ) is produced by taking the mean of these embeddings and passingthis vector through a tanh nonlinearity.
Revised: The state representation s ϕ ( n ) is produced by taking the mean of these embeddings and passing this vector through a tanh nonlinearity.

Original: The node-action embedding a ϕ ( n, n ′ ) embeds the actionof moving from node n to its neighbor n ′ .
Revised: The node-action embedding a ϕ ( n, n ′ ) embeds the action of moving from node n to its neighbor n ′ .

Original: Let us assume the text at node n is “Barack Obama wasthe President of the United States during...” where the substrings “Barack Obama” and “Presidentof the United States” correspond to hyperlinks to other neighboring articles n ′ and n ′′ .
Revised: Let us assume the text at node n is “Barack Obama was the President of the United States during...” where the substrings “Barack Obama” and “President of the United States” correspond to hyperlinks to other neighboring articles n ′ and n ′′ .

Original: Then weconstruct the node-action embedding a ϕ ( n, n ′ ) by passing the tokenized text of node n throughour transformer, again producing L transformed token embeddings.
Revised: Then we construct the node-action embedding a ϕ ( n, n ′ ) by passing the tokenized text of node n through our transformer, again producing L transformed token embeddings.

Original: Then we take the transformedtokens which correspond to the text “Barack Obama” and take their mean and pass this through thetanh nonlinearity.
Revised: Then we take the transformed tokens which correspond to the text “Barack Obama” and take their mean and pass this through the tanh nonlinearity.

Original: If we instead wanted to produce a ϕ ( n, n ′′ ) we would take the transformed tokenswhich correspond to the text “President of the United States” instead, take the mean and apply thenonlinearity.
Revised: If we instead wanted to produce a ϕ ( n, n ′′ ) we would take the transformed tokens which correspond to the text “President of the United States” instead, take the mean and apply the nonlinearity.

Original: This is visualized in Figure 8.
Revised: This is visualized in Figure 8.

Original: The function f θ which combines these embeddings first concatenates them and passes the combinedinput through a simple neural network with a single hidden layer and 1 output neuron.
Revised: The function f θ which combines these embeddings first concatenates them and passes the combined input through a simple neural network with a single hidden layer and 1 output neuron.

Original: Training details We train these models to optimize Equation 6.
Revised: Training details We train these models to optimize Equation 6.

Original: The objective is optimized withthe Adam [Kingma and Ba, 2014] optimizer using an initial learning rate of 1 e − 5 with a one-cyclecosine decay schedule and linear warm up of 10 K steps.
Revised: The objective is optimized with the Adam [Kingma and Ba, 2014] optimizer using an initial learning rate of 1 e − 5 with a one-cycle cosine decay schedule and linear warm up of 10 K steps.

Original: The training batch size is 512 and the modelis trained over 3 M iterations.
Revised: The training batch size is 512 and the model is trained over 3 M iterations.

Original: Node text is clipped (or padded) to 200 tokens.
Revised: Node text is clipped (or padded) to 200 tokens.

Original: The weights of thetransformer are initialized from a pretrained RoBERTa
Revised: The weights of the transformer are initialized from a pretrained RoBERTa

Original: [Liu et al., 2019] masked language model.
Revised: [Liu et al., 2019] masked language model.

Original: B.2 Small graph experiments
Revised: B.2 Small graph experiments

Original: Model details In the small graph experiments, we use the simple policy network FF as describedin Section 4.2.
Revised: Model details In the small graph experiments, we use the simple policy network FF as described in Section 4.2.

Original: As discussed, we use a 1-layer MLP (meaning a single learned linear layer with noactivation function).
Revised: As discussed, we use a 1-layer MLP (meaning a single learned linear layer with no activation function).

Original: We use a hidden size of 768 to match the size of ϕ .
Revised: We use a hidden size of 768 to match the size of ϕ .

Original: Because the final logits arethe result of an inner product between each state and action embedding of size 768 , to aid networktraining, we add a normalization layer (jax.nn.normalize) before the inner product.
Revised: Because the final logits are the result of an inner product between each state and action embedding of size 768 , to aid network training, we add a normalization layer (jax.nn.normalize) before the inner product.

Original: For the action embedding a i = ϕ
Revised: For the action embedding a i = ϕ

Original: ( s i ) we also concatenate a 1-hot vector that represents the actiontype (e.g. whether it was a link action, or a next action, or a prev action).
Revised: ( s i ) we also concatenate a 1-hot vector that represents the action type (e.g. whether it was a link action, or a next action, or a prev action).

Original: We also concatenate anotherbit that indicates whether the state s i of the particular action has been visited before.
Revised: We also concatenate another bit that indicates whether the state s i of the particular action has been visited before.

Original: For the purposes of the RL baseline, we can view graph navigation as a goal-conditioned MDP.
Revised: For the purposes of the RL baseline, we can view graph navigation as a goal-conditioned MDP.

Original: 9 Ourobservation is the same as the inputs to the RFBC models.
Revised: 9 Our observation is the same as the inputs to the RFBC models.

Original: The reward function R for current node nand goal node n g we can write simply as:
Revised: The reward function R for current node n and goal node n g we can write simply as:

Original: [Equation]
Revised: [Equation]

Original: For the RL baseline, we used IMPALA [Espeholt et al., 2018], which is a kind of policy gradientapproach, and thus has a policy network π and a “baseline” or value function network.
Revised: For the RL baseline, we used IMPALA [Espeholt et al., 2018], which is a kind of policy gradient approach, and thus has a policy network π and a “baseline” or value function network.

Original: Since thepolicy network has the same output space as our RFBC models, we use an identical architecture forthis.
Revised: Since the policy network has the same output space as our RFBC models, we use an identical architecture for this.

Original: For the value network, we use a very similar architecture.
Revised: For the value network, we use a very similar architecture.

Original: We again use ϕ as our encoding ofthe current node and the target node, again concatenate them together.
Revised: We again use ϕ as our encoding of the current node and the target node, again concatenate them together.

Original: We then feed this through a1-layer MLP to compute the value.
Revised: We then feed this through a 1-layer MLP to compute the value.

Original: Training details The procedure for generating the trajectories we use for the BC loss is described in
Revised: Training details The procedure for generating the trajectories we use for the BC loss is described in

Original: Section 3.1.
Revised: Section 3.1.

Original: Except for our experiment in Section 5.1, we always use the random forward trajectories.
Revised: Except for our experiment in Section 5.1, we always use the random forward trajectories.

Original: Following Lin et al.
Revised: Following Lin et al.

Original: [2018], in the navigation experiments, we randomly drop out edges in the traininggraph with probability 0 .
Revised: [2018], in the navigation experiments, we randomly drop out edges in the training graph with probability 0 .

Original: 5 to reduce over-fitting.
Revised: 5 to reduce over-fitting.

Original: As stated in Section 4.1, we set max steps to
Revised: As stated in Section 4.1, we set max steps to

Original: B = 100 .
Revised: B = 100 .

Original: For RFBC training on the small graph, we use RMSProp with a learning rate of 0 .
Revised: For RFBC training on the small graph, we use RMSProp with a learning rate of 0 .

Original: 01 , adecay of 0 .
Revised: 01 , a decay of 0 .

Original: 9 an epsilon of 1 e − 10 .
Revised: 9 an epsilon of 1 e − 10 .

Original: For the sentence search tasks, our models train a separate ϕ for the target embedding.
Revised: For the sentence search tasks, our models train a separate ϕ for the target embedding.

Original: As we stated,we use MiniBERT for this embedding ϕ target and train it with the same optimization settings, exceptthat we reduce the learning rate for these weights to 1 e − 4 .
Revised: As we stated, we use MiniBERT for this embedding ϕ target and train it with the same optimization settings, except that we reduce the learning rate for these weights to 1 e − 4 .

Original: We use a batch size of 512 and we trainfor 50 , 000 network update steps.
Revised: We use a batch size of 512 and we train for 50 , 000 network update steps.

Original: For RL training, we use VTRACE [Espeholt et al., 2018] loss, using the reward above.
Revised: For RL training, we use VTRACE [Espeholt et al., 2018] loss, using the reward above.

Original: We againuse RMSProp with a learning rate of 0 .
Revised: We again use RMSProp with a learning rate of 0 .

Original: 01 , a decay of 0 .
Revised: 01 , a decay of 0 .

Original: 9 an epsilon of 1 e − 10 .
Revised: 9 an epsilon of 1 e − 10 .

Original: We set the baselinecost to 0 .
Revised: We set the baseline cost to 0 .

Original: 5 , trajectory length (the number of timesteps the RL loss backprops through) to 100 , batchsize again to 512 and max update steps to 50 , 000 .
Revised: 5 , trajectory length (the number of timesteps the RL loss backprops through) to 100 , batch size again to 512 and max update steps to 50 , 000 .

Original: We did a parameter sweep for entropy costs of0 . 1 , 0 . 01 , 0 . 001 and gamma of 0 . 8 , 0 . 9 .
Revised: We did a parameter sweep for entropy costs of 0 . 1 , 0 . 01 , 0 . 001 and gamma of 0 . 8 , 0 . 9 .

Original: To give RL the best chance possible, we choose the maximumover this sweep for each experiment (but still none of these match RFBC performance).
Revised: To give RL the best chance possible, we choose the maximum over this sweep for each experiment (but still none of these match RFBC performance).

Original: B.3 Full Wikipedia navigation experiments
Revised: B.3 Full Wikipedia navigation experiments

Original: On the full Wikipedia training experiments in Section 5.3, we evaluate on navigation and sentence search using either our feed-forward or transformer model and using either RoBERTA fixedembeddings, or our trained embeddings described in Appendix B.1.
Revised: On the full Wikipedia training experiments in Section 5.3, we evaluate on navigation and sentence search using either our feed-forward or transformer model and using either RoBERTA fixed embeddings, or our trained embeddings described in Appendix B.1.

Original: The feed-forward model is identical to the one described in Appendix B.2.
Revised: The feed-forward model is identical to the one described in Appendix B.2.

Original: The transformer model isthe standard transformer model from Vaswani et al.
Revised: The transformer model is the standard transformer model from Vaswani et al.

Original: [2017] with 4 layers, an attention size of 64 , 12heads, and mlp hidden size of 3072 and dropout rate of 0 .
Revised: [2017] with 4 layers, an attention size of 64 ,heads, and mlp hidden size of 3072 and dropout rate of 0 .

Original: DistillBERT Sanh et al.
Revised: DistillBERT Sanh et al.

Original: [2019] is used forϕ ( n g ) for the sentence search experiments.
Revised: [2019] is used for ϕ ( n g ) for the sentence search experiments.

Original: The learning parameters and all other relevant training parameters for all models are identical tothose in Appendix B.2 except that the learning rate for the transformer model is lower ( 1 e − 4 ) as theearlier learning rate was too high for transformer models.
Revised: The learning parameters and all other relevant training parameters for all models are identical to those in Appendix B.2 except that the learning rate for the transformer model is lower ( 1 e − 4 ) as the earlier learning rate was too high for transformer models.

Original: B.4 Downstream experiments
Revised: B.4 Downstream experiments

Original: Training details For the FEVER benchmark, our model uses learned embeddings for ϕ with asingle feed-forward layer for the policy network.
Revised: Training details For the FEVER benchmark, our model uses learned embeddings for ϕ with a single feed-forward layer for the policy network.

Original: The model is pretrained on the 5 -step sentencesearch task, and finetuned on trajectories generated from the augmented FEVER dataset described in
Revised: The model is pretrained on the 5 -step sentence search task, and finetuned on trajectories generated from the augmented FEVER dataset described in

Original: A.4 using the the following loss function:
Revised: A.4 using the the following loss function:

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where L BC is the normal BC loss we use in our other experiments
Revised: where L BC is the normal BC loss we use in our other experiments.

Original: Only the ϕ target weights are finetuned for this task.
Revised: Only the ϕ target weights are finetuned for this task.

Original: We use AdamW with a constant learning rate of1e-6, b1=0.9, b2=0.999 , and eps=1e-6 .
Revised: We use AdamW with a constant learning rate of 1e-6, b1=0.9, b2=0.999 , and eps=1e-6 .

Original: The model is finetuned for 100 , 000 update steps witha batch size of 512 .
Revised: The model is finetuned for 100 , 000 update steps with a batch size of 512 .

Original: No edge dropout is used during finetuning.
Revised: No edge dropout is used during finetuning.

Original: Evaluation details For each claim in the FEVER development dataset, we first run BM25 overall nodes in the graph and take the top 5 matches as starting nodes.
Revised: Evaluation details For each claim in the FEVER development dataset, we first run BM25 over all nodes in the graph and take the top 5 matches as starting nodes.

Original: From each node, the finetunedmodel then navigates for 20 steps.
Revised: From each node, the finetuned model then navigates for 20 steps.

Original: We collect all sentences in all nodes visited by the agent overthe 100 total navigation steps, and match them to FEVER evidence sentences using the WRatiofunction in the fuzzywuzzy package.
Revised: We collect all sentences in all nodes visited by the agent over the 100 total navigation steps, and match them to FEVER evidence sentences using the WRatio function in the fuzzywuzzy package.

Original: 10 Two sentence ranking methods are explored: Gensim’s
Revised: 10 Two sentence ranking methods are explored: Gensim’s

Original: TF-IDF implementation, 11 and the BigBird re-ranker of Stammbach [2021], using their open sourcemodel and weights.
Revised: TF-IDF implementation, 11 and the BigBird re-ranker of Stammbach [2021], using their open source model and weights.

Original: Finally, the top-5 evidence sentences are then submitted to the official FEVERscorer 13 to compute the accuracy, recall, and F1 scores.
Revised: Finally, the top-5 evidence sentences are then submitted to the official FEVER scorer 13 to compute the accuracy, recall, and F1 scores.

Original: C Further experiments 
Revised: C Further experiments C.1 RFBC + RL

Original: C.1 RFBC + RL

Original: We additionally try finetuning an RL policy starting from our RFBC-trained navigation policies.
Revised: We additionally try finetuning an RL policy starting from our RFBC-trained navigation policies.

Original: Weuse the same network and learning settings as we did in the other RL experiments.
Revised: We use the same network and learning settings as we did in the other RL experiments.

Original: We train for anadditional 10,000 network updates (512M environment steps).
Revised: We train for an additional 10,000 network updates (512M environment steps).

Original: We add the final numbers in Table 2and show the training curve in Figure 9.
Revised: We add the final numbers in Tableand show the training curve in Figure 9.

Original: We can see from the training curve that RL training accuracyfluctuates around the point where RFBC training left off.
Revised: We can see from the training curve that RL training accuracy fluctuates around the point where RFBC training left off.

Original: From the results in Table 2 we see littlestatistically significant difference (about 1 point in either direction).
Revised: From the results in Table 2 we see little statistically significant difference (about 1 point in either direction).

Original: C.2 Question Answer Downstream Task
Revised: C.2 Question Answer Downstream Task

Original: Finally, we present results on another downstream task where navigation helps.
Revised: Finally, we present results on another downstream task where navigation helps.

Original: In particular, weconsider the task of finding correct evidence passage for open domain question answering.
Revised: In particular, we consider the task of finding correct evidence passage for open domain question answering.

Original: Weuse the Natural Question (NQ) open-domain dataset presented in Lee et al.
Revised: We use the Natural Question (NQ) open-domain dataset presented in Lee et al.

Original: [2019] which has beenaligned to Wikipedia dump of December 20, 2018 by [Karpukhin et al., 2020].
Revised: [2019] which has been aligned to Wikipedia dump of December 20, 2018 by [Karpukhin et al., 2020].

Original: Most prior works inliterature utilize the processed collection of 21M passages provided by by [Karpukhin et al., 2020],but unfortunately we cannot use it as it has no hyperlink information which is crucial for our graphbuilding.
Revised: Most prior works in literature utilize the processed collection of 21M passages provided by by [Karpukhin et al., 2020], but unfortunately we cannot use it as it has no hyperlink information which is crucial for our graph building.

Original: So we re-align the questions in NQ to our 2018 graph with 38M passages (i.e. nodes).
Revised: So we re-align the questions in NQ to our 2018 graph with 38M passages (i.e. nodes).

Original: Wecompute the shortest-path distance between top-1 retrieval of BM25 and target node in this graph in
Revised: We compute the shortest-path distance between top-1 retrieval of BM25 and target node in this graph in

Original: Figure 10.
Revised: Figure 10.

Original: As can be seen that most path lengths are relatively short, which implies that BM25 landsus in the right vicinity and by a small amount of navigation around we will be able to find the rightevidence passage.
Revised: As can be seen that most path lengths are relatively short, which implies that BM25 lands us in the right vicinity and by a small amount of navigation around we will be able to find the right evidence passage.

Original: Since we target navigating to the exact evidence passage required to answer the question, we userecall@{1,2,3,4,5} for finding the gold evidence passage as our metric.
Revised: Since we target navigating to the exact evidence passage required to answer the question, we use recall@{1,2,3,4,5} for finding the gold evidence passage as our metric.

Original: More commonly the metricmarks a retrieved passage to be correct if it contains the answer string, but this causes a lot of falsepositives, e.g. the answer string appears in a totally irrelevant context.
Revised: More commonly the metric marks a retrieved passage to be correct if it contains the answer string, but this causes a lot of false positives, e.g. the answer string appears in a totally irrelevant context.

Original: Our method: For each question, we first run BM25 over all nodes in the graph and take the top-5matches as starting nodes.
Revised: Our method: For each question, we first run BM25 over all nodes in the graph and take the topmatches as starting nodes.

Original: From each starting node, the agent then navigates for 20 steps.
Revised: From each starting node, the agent then navigates for 20 steps.

Original: All the 100nodes visited by the agent is then ranked by a simple cross-attention model.
Revised: All thenodes visited by the agent is then ranked by a simple cross-attention model.

Original: For the cross-attentionmodel, we follow Hofstätter et al.
Revised: For the cross-attention model, we follow Hofstätter et al.

Original: [2020] to train 6-layer BigBird [Zaheer et al., 2020].
Revised: [2020] to train 6-layer BigBird [Zaheer et al., 2020].

Original: [Table]
Revised: [Table]

Original: The results are tabulated in Table 6.
Revised: The results are tabulated in Table 7.

Original: Paired with a basic retrieval (BM25) and re-ranker (BigBird), theapproach obtains a respectable Recall@1 of 31.6.
Revised: Paired with a basic retrieval (BM25) and re-ranker (BigBird), the approach obtains a respectable Recall@1 of 31.6.

Original: If we remove our navigation component then thisdrops substantially to 22.2, demonstrating its utility.
Revised: If we remove our navigation component then this drops substantially to 22.2, demonstrating its utility.

Original: As a reference, we also ran a state-of-the-artsystem RocketQA
Revised: As a reference, we also ran a state-of-the-art system RocketQA

Original: [Qu et al., 2021] on our setup and evaluated it using our harder metric.
Revised: [Qu et al., 2021] on our setup and evaluated it using our harder metric.

Original: It isworthwhile to note that our approach (a) is significantly simpler than state-of-the-art approacheslike RocketQA which employ 4 stages of dual-encoder and cross-attention models, and (b) selectsevidence over a larger set than the commonly used preprocess Wikipedia passages (38M vs 21M).
Revised: It is worthwhile to note that our approach (a) is significantly simpler than state-of-the-art approaches like RocketQA which employ 4 stages of dual-encoder and cross-attention models, and (b) selects evidence over a larger set than the commonly used preprocess Wikipedia passages (38M vs 21M).

Original: [Figure]
Revised: [Figure]

Original: D Variational Interpretation of our Approach
Revised: D Variational Interpretation of our Approach

Original: We present an alternative motivation of our method based on variational inference in a latent variablemodel of states.
Revised: We present an alternative motivation of our method based on variational inference in a latent variable model of states.

Original: Assume we are given some goal node n g .
Revised: Assume we are given some goal node n g .

Original: We would like to parameterize a modelp θ ( n T | n g ) which will generate a trajectory of states n 0 , n 1 , . . .
Revised: We would like to parameterize a model p θ ( n T | n g ) which will generate a trajectory of states n 0 , n 1 , . . .

Original: , n T such that n T = n g .
Revised: , n T such that n T = n g .

Original: We defineour model as a latent variable model
Revised: We define our model as a latent variable model

Original: [Equation]
Revised: [Equation]

Original: i.e an autoregressive model which samples an initial state from n 0 ∼ p ( n 0 ) and then samplessubsequent states n t ∼ p θ ( n t | n t − 1 , n g ) .
Revised: i.e an autoregressive model which samples an initial state from n 0 ∼ p ( n 0 ) and then samples subsequent states n t ∼ p θ ( n t | n t − 1 , n g ) .

Original: The transition distribution samples the next node n t fromthe neighbors of n t − 1 in our graph.
Revised: The transition distribution samples the next node n t from the neighbors of n t − 1 in our graph.

Original: We would like the probability that n T = n g to be large, thus wewill train our model to maximize log p θ ( n T = n g | n g ) .
Revised: We would like the probability that n T = n g to be large, thus we will train our model to maximize log p θ ( n T = n g | n g ) .

Original: For latent-variable models such as this, we can rewrite the marginal likelihood as
Revised: For latent-variable models such as this, we can rewrite the marginal likelihood as

Original: [Equation]
Revised: [Equation]

Original: and can obtain a lower-bound on this quantity by replacing the intractable posteriorp θ ( n 0 , . .
Revised: and can obtain a lower-bound on this quantity by replacing the intractable posterior p θ ( n 0 , . .

Original: . , n T − 1 | n T , n g ) with a variational approximation q ( n 0 , . .
Revised: . , n T − 1 | n T , n g ) with a variational approximation q ( n 0 , . .

Original: . , n T − 1 | n T , n g ) , i.e
Revised: . , n T − 1 | n T , n g ) , i.e

Original: L θ ( n T , n g ; q ) := log p θ ( n T | n g ) ≥ E q ( n [Equation] n T
Revised: L θ ( n T , n g ; q ) :

Original: [Equation]

Original:  T [Equation] g )(cid:34) p [Equation]
Revised: = log p θ ( n T | n g )

Original: [Equation]

Original: (cid:35)

Original: [Equation]
Revised: [Equation]

Original: The above bound becomes tight when q ( n 0 , . .
Revised: The above bound becomes tight when q ( n 0 , . .

Original: . , n T − 1 | n T , n g ) =
Revised: . , n T − 1 | n T , n g ) =

Original: p θ ( n 0 , . . .
Revised: p θ ( n 0 , . . .

Original: , n T − 1 | n T , n g ) .
Revised: , n T − 1 | n T , n g ) .

Original: Thus,we can optimize our model parameters θ to maximize L θ ( n T , n g ; q ) .
Revised: Thus, we can optimize our model parameters θ to maximize L θ ( n T , n g ; q ) .

Original: This approach has had a longhistory of successfully training latent-variable generative models [Kingma and Welling, 2013, Hoet al., 2020].
Revised: This approach has had a long history of successfully training latent-variable generative models [Kingma and Welling, 2013, Ho et al., 2020].

Original: In the context of this work, we can view q ( n 0 , . . .
Revised: In the context of this work, we can view q ( n 0 , . . .

Original: , n T − 1 | n T , n g ) as a distribution over trajectoriesof nodes which end at our goal node.
Revised: , n T − 1 | n T , n g ) as a distribution over trajectories of nodes which end at our goal node.

Original: We can interpret the various trajectory generation methodsintroduced in Section 3.1 as different variational approximations q .
Revised: We can interpret the various trajectory generation methods introduced in Section 3.1 as different variational approximations q .

Original: Ideally, the option which mostclosely approximates the true posterior p θ ( n 0 , . .
Revised: Ideally, the option which most closely approximates the true posterior p θ ( n 0 , . .

Original: . , n T − 1 | n T , n g ) would be the most desirable but ingeneral this distribution is intractable.
Revised: . , n T − 1 | n T , n g ) would be the most desirable but in general this distribution is intractable.

Original: We find that using simple random trajectories provides a good-enough approximation to the posteriorto enable us to train a model which reliably finds the goal state.
Revised: We find that using simple random trajectories provides a good-enough approximation to the posterior to enable us to train a model which reliably finds the goal state.

Original: This result is not completely surprisingin context of prior work [Dai et al., 2020] which successfully trains latent-variable models usingrandom trajectories as an inference model.
Revised: This result is not completely surprising in context of prior work [Dai et al., 2020] which successfully trains latent-variable models using random trajectories as an inference model.

Original: E Efficiency Analysis
Revised: E Efficiency Analysis

Original: In this section we analyze the asmpytotic runtime of our method and baselines.
Revised: In this section we analyze the asmpytotic runtime of our method and baselines.

Original: In Table 7 we showthe asymptotic runtime and whether or not it is scalable for each method.
Revised: In Table 8 we show the asymptotic runtime and whether or not it is scalable for each method.

Original: We also show the 5-stepaccuracy from Table 2 for quick reference.
Revised: We also show the 5-step accuracy from Table 2 for quick reference.

Original: T is the trajectory length (either the length of the trajectory for BC methods or the maximum allowedtrajectory for other methods.
Revised: T is the trajectory length (either the length of the trajectory for BC methods or the maximum allowed trajectory for other methods.

Original: For BC, T is guaranteed to be shorter than the max, but this is a constantfactor difference.
Revised: For BC, T is guaranteed to be shorter than the max, but this is a constant factor difference.

Original: E out is the average number of outgoing edges (i.e. actions) at a node.
Revised: E out is the average number of outgoing edges (i e ) at a node.

Original: (small - 10 1 )

Original: E is the total edges in the graph (very large - 10 8 ).
Revised: E is the total edges in the graph (very large - 10 8 ).

Original: Vis the number of nodes in the graph (large10 7 ).
Revised: V number of nodes in the graph (large10 7 ).

Original: [Table]
Revised: [Table]

Original: For most methods, the runtime is O ( E out T ) because the method does some constant amount of workfor each possible actions E o ut for O ( T ) steps.
Revised: For most methods, the runtime is O ( E out T ) because the method does some constant amount of work for each possible actions E o ut for O ( T ) steps.

Original: For the random methods it is O ( T ) because the workis constant per step (choose randomly).
Revised: For the random methods it is O ( T ) because the work is constant per step (choose randomly).

Original: Shortest path distance has the worst runtime.
Revised: Shortest path distance has the worst runtime.

Original: The O ( ElogV )term is the runtime of finding a shortest path (Dijkstra’s algorithm with Fibonacci heap).
Revised: The O ( ElogV ) term is the runtime of finding a shortest path (Dijkstra’s algorithm with Fibonacci heap).

Original: As the graphgets larger, this term because infeasible (we could not even run this on the full Wikipedia graph).
Revised: As the graph gets larger, this term because infeasible (we could not even run this on the full Wikipedia graph).

Original: DFSmight seem like it should be larger than O ( E out T ) or O ( T ) , but because we restrict the maximumallowable steps, it is only proportional to T and (for greedy) E out .
Revised: DFS might seem like it should be larger than O ( E out T ) or O ( T ) , but because we restrict the maximum allowable steps, it is only proportional to T and (for greedy) E out .

Original: From this Table, we see that all methods (except shortest path) have scalable runtimes.
Revised: From this Table, we see that all methods (except shortest path) have scalable runtimes.

Original: The dominantfactor is E out , which theoretically grows slowly with graph size, but in practice, there is a practicalmaximum number of links you have in a paragraph of text, and the runtime difference in ourexperiments going from 200k nodes to 38M nodes is only 1.5x despite the 200-fold increase in graphsize.
Revised: The dominant factor is E out , which theoretically grows slowly with graph size, but in practice, there is a practical maximum number of links you have in a paragraph of text, and the runtime difference in our experiments going from 200k nodes to 38M nodes is only 1.5x despite the 200-fold increase in graph size.