Original: Learning to Navigate Wikipedia by Taking Random Walks
Revised: Learning to Navigate Wikipedia by Taking Random Walks

Original: Anonymous Author(s)
Revised: Anonymous Author(s)

Original: AffiliationAddress email
Revised: AffiliationAddress email

Original: Abstract
Revised: Abstract

Original: A fundamental ability of an intelligent web-based agent is seeking out and acquiringnew information.
Revised: A fundamental ability of an intelligent web-based agent is seeking out and acquiringnew information.

Original: Internet search engines reliably find the correct vicinity but thetop results may be a few links away from the desired target.
Revised: Internet search engines reliably find the correct vicinity but thetop results may be a few links away from the desired target.

Original: A complementaryapproach is navigation via hyperlinks, employing a policy that comprehends localcontent and selects a link that moves it closer to the target.
Revised: A complementaryapproach is navigation via hyperlinks, employing a policy that comprehends localcontent and selects a link that moves it closer to the target.

Original: In this paper we showthat using behavioral cloning of randomly sampled trajectories is sufficient to learnan effective link selection policy.
Revised: In this paper we showthat using behavioral cloning of randomly sampled trajectories is sufficient to learnan effective link selection policy.

Original: We demonstrate the approach on a graph versionof Wikipedia with 38 M nodes and 387 M edges.
Revised: We demonstrate the approach on a graph versionof Wikipedia with 38 M nodes and 387 M edges.

Original: The model is able to efficientlynavigate between nodes 5 and 20 steps apart 96% and 92% of the time, respectively.
Revised: The model is able to efficientlynavigate between nodes 5 and 20 steps apart 96% and 92% of the time, respectively.

Original: We then use the resulting embeddings and policy in a downstream fact verificationtask where, in combination with basic TF-IDF search and ranking methods, theyare able to obtain competitive results to state-of-the-art methods
Revised: We then use the resulting embeddings and policy in a downstream fact verificationtask where, in combination with basic TF-IDF search and ranking methods, theyare able to obtain competitive results to state-of-the-art methods

Original:  Introduction
Revised:  Introduction

Original: The ability to gather new knowledge about the world is a fundamental aspect of intelligence.
Revised: The ability to gather new knowledge about the world is a fundamental aspect of intelligence.

Original: Basedonly on a few words, a fact, question, or even vague idea, humans have the ability to use the internetto find extremely specific information about the world.
Revised: Basedonly on a few words, a fact, question, or even vague idea, humans have the ability to use the internetto find extremely specific information about the world.

Original: From the important (how to administer CPR)to the trivial (the LA Raiders won the 1983 SuperBowl), virtually any knowledge is a click away.
Revised: From the important (how to administer CPR)to the trivial (the LA Raiders won the 1983 SuperBowl), virtually any knowledge is a click away.

Original: In this work, we focus in one particular aspect of this ability: web navigation.
Revised: In this work, we focus in one particular aspect of this ability: web navigation.

Original: In general, navigationis a key component of an embodied agent: the ability to move efficiently toward a target.
Revised: In general, navigationis a key component of an embodied agent: the ability to move efficiently toward a target.

Original: In knownenvironments, where map information is available, shortest-path algorithms provide a viable solution.
Revised: In knownenvironments, where map information is available, shortest-path algorithms provide a viable solution.

Original: However, in novel settings, the agent lacks such global information and must instead navigate to thetarget, using its understanding of the local environment to select actions.
Revised: However, in novel settings, the agent lacks such global information and must instead navigate to thetarget, using its understanding of the local environment to select actions.

Original: Other approaches to web agents have focused mostly on retrieval or search engines.
Revised: Other approaches to web agents have focused mostly on retrieval or search engines.

Original: These, however,only provide a partial solution, typically getting close to a desired target but often not exactly to theright page.
Revised: These, however,only provide a partial solution, typically getting close to a desired target but often not exactly to theright page.

Original: In this work, we present an approach for navigation on graph-structured web data thatuses hyperlinks within articles to navigate toward a target.
Revised: In this work, we present an approach for navigation on graph-structured web data thatuses hyperlinks within articles to navigate toward a target.

Original: It complements search engines, usingthem to provide a sensible starting point for a local search for specific information.
Revised: It complements search engines, using them

Original: To be able to navigate on the web effectively, we must read the text of the page, see how concepts inthe current page are related to a possible next page, and use our understanding to learn a policy tomake the right navigation decision.
Revised: To be able to navigate on the web effectively, we must read the text of the page, see how concepts inthe current page are related to a possible next page, and use our understanding to learn a policy tomake the right navigation decision.

Original: Consider the scenario shown in Figure 1.
Revised: Consider the scenario shown in Figure 1.

Original: The agent is tryingto navigate to the target node N ORTH A MERICA while currently at the node P RESIDENTS OF THE
Revised: The agent is tryingto navigate to the target node N ORTH A MERICA while currently at the node P RESIDENTS OF THE

Original: U NITED S TATES containing two hyperlinks: B ARACK O BAMA and USA.
Revised: U NITED S TATES containing two hyperlinks: B ARACK O BAMA and USA.

Original: Our approach learns anembedding from the text of the current page and the text for each hyperlink node and passes theseto a goal-conditioned policy that selects the best possible action.
Revised: Our approach learns anembedding from the text of the current page and the text for each hyperlink node and passes theseto a goal-conditioned policy that selects the best possible action.

Original: To make the correct choice in thisexample, the agent should associate USA with N ORTH A MERICA
Revised: To make the correct choice in thisexample, the agent should associate USA with N ORTH A MERICA

Original: Barack Obama
Revised: Barack Obama

Original: USA
Revised: USA

Original: Current Node
Revised: Current Node

Original: Presidents of the United States
Revised: Presidents of the United States

Original: Node
Revised: Node

Original: Node Text “Barack Obama was the President of the United States…”
Revised: Node Text “Barack Obama was the President of the United States…”

Original: “Barack Obama is an American Politician…” “The USA is a nation in North America…” ϕ
Revised: “Barack Obama is an American Politician…” “The USA is a nation in North America…” ϕ

Original: Transformer s t a“The USA is a nation in North America…”
Revised: Transformer s t a“The USA is a nation in North America…”

Original: a
Revised: a

Original: North America “North America is a continent on the…” s g
Revised: North America “North America is a continent on the…” s g

Original: [Figure]
Revised: [Figure]

Original: MLP
Revised: MLP

Original: Action probability pp
Revised: Action probability pp

Original: Node
Revised: Node

Original: Nodeϕ ϕ ϕ θ θ
Revised: Nodeϕ ϕ ϕ θ θ

Original: The particular web environment we consider here is Wikipedia, converted into a graph form.
Revised: The particular web environment we consider here is Wikipedia, converted into a graph form.

Original: Eachof the 38M paragraphs is represented by a node; edges are links within and between articles.
Revised: Eachof the 38M paragraphs is represented by a node; edges are links within and between articles.

Original: Weintroduce an approach for navigation on this graph based on behavioral cloning of automaticallygenerated trajectories.
Revised: Weintroduce an approach for navigation on this graph based on behavioral cloning of automaticallygenerated trajectories.

Original: This allows for unsupervised pre-training of both web page embeddingsand navigation policy.
Revised: This allows for unsupervised pre-training of both web page embeddingsand navigation policy.

Original: We explore different trajectory distributions for training the model, the bestpractical choice being equivalent to random walks on the graph.
Revised: We explore different trajectory distributions for training the model, the bestpractical choice being equivalent to random walks on the graph.

Original:  Our contributions are: (i) we demonstrate that our pre-training scheme enables the model to reliably( > 90% success) navigate between pairs of nodes in a 387 M-edge Wikipedia graph; (ii) we investigateseveral strategies for sampling random trajectories from the graph, concluding that a simple forwardrandom walk is  the best choice in practical settings; (iii) we show that paragraph embeddings trainedfor navigation outperform those from language models (trained for masked language modelling) and(iv)
Revised: Contributions: Our work presents the first viable solution to the problem of web navigation, asdemonstrated by reliable (>90% success) navigation on a graph of size 10 7 . Previous attemptssuch as Nogueira and Cho [2016] show a far inferior performance, well below that needed forpractical utility. Furthermore, as there is nothing Wikipedia-specific about our approach and it scalesgracefully, the method is applicable to general web settings. We also demonstrate the relevance ofthe

Revised: pre-train + fine-tune paradigm to the web domain, showing that pre-training for navigation is asuitable objective to aid downstream tasks with limited data such as Q&A.

Original: we show the utility of our navigation scheme for the FEVER Thorne et al.
Revised: We demonstrate this byapplying our navigation approach on two challenging Q&A tasks (FEVER Thorne et al.

Original: [2018] fact verificationtask, demonstrating a significant boost when integrated into a simple TF-IDF scheme, as well asobtaining a F1 score competitive with SOTA (73.1% vs 79.9% SOTA) with a more complex re-ranker.
Revised: [2018] and Natural Questions Lee et al. In both settings, we show a jump in performance over existingsearch-based approaches that rely on retrieval followed by re-ranking, matching state-of-the-artmethods despite using far simpler retrieval and re-ranking mechanisms.

Original:  Background
Revised:  Background

Original: Wikipedia as a navigation environment: Human behavior on Wikipedia [West and Leskovec,2012a] was collected via the Wikispeedia game [West et al., 2009], which asked the players to movebetween a pair of arbitrarily chosen articles in the fewest steps.
Revised: Wikipedia as a navigation environment: Human behavior on Wikipedia [West and Leskovec,2012a] was collected via the Wikispeedia game [West et al., 2009], which asked the players to movebetween a pair of arbitrarily chosen articles in the fewest steps.

Original: This rule allows the emergence of asemantic distance between concepts that facilitates retrieval.
Revised: This rule allows the emergence of asemantic distance between concepts that facilitates retrieval.

Original: West and Leskovec [2012b] conductedinitial studies on this dataset for navigation, using supervised learning and reinforcement learningmethods on a small subset of articles ( 4 . 6 K) with simple TF-IDF features.
Revised: West and Leskovec [2012b] conductedinitial studies on this dataset for navigation, using supervised learning and reinforcement learningmethods on a small subset of articles ( 4 . 6 K) with simple TF-IDF features.

Original: Most relevant to our work, Nogueira and Cho [2016] explore the same problem of navigation on alarge Wikipedia graph ( 12 M nodes).
Revised: Most relevantly, Nogueira and Cho [2016] explore the same problem of navigation on a large Wikipedia graph ( 12 M nodes).

Original: They train LSTM agents to perform navigation on the graph,using Bag-of-Word features for each article.
Revised: They train LSTM agents to perform navigation on the graph, using Bag-of-Word features for each article.

Original: The trained agent is then applied to a Jeopardy Q&A task.
Revised: The trained agent is then applied to a Jeopardy Q&A task.

Original: Our approach revisits this framing but with a number of important differences: (i)  on a significantlylarger graph ( 38 M vs 12 M); (ii) using modern text features, e.g. BERT features fine-tuned with a
Revised: Our approach revisits this framing but with a number of important differences: (i) training on asignificantly larger graph ( 38 M vs 12 M nodes, 51 .

Original: Transformer-based navigation policy; (iii) they use a single fixed start node, whereas ours is ableto navigate from any start node; (iv) removal of beam search, or other complex search mechanismsat evaluation time.
Revised: 5 M vs 380 M edges); (ii) using Transformer-basedarchitectures for paragraph encodings and for the navigation policy; (iii) they use a single fixed startnode, whereas ours is able to navigate from any start node; (iv) removal of beam search, or othercomplex search mechanisms at evaluation time.

Original: Collectively, our approach boosts the navigation success rate from 12 .
Revised: Collectively, our approach boosts the navigationsuccess rate from 12 .

Original: 5% in[Nogueira and Cho, 2016] to over 90% , 1 despite a much larger graph.
Revised: 5% in [Nogueira and Cho, 2016] to over 90% , 1 despite a much larger graph.

Original: Fact verification and Q&A using Wikipedia:
Revised: Fact verification and Q&A using Wikipedia:

Original: Wikipedia is a rich resource for evaluating naturallanguage understanding through question and answering tasks.
Revised: Wikipedia is a rich resource for evaluating languageunderstanding through Q&A tasks.

Original: Chen et al.
Revised: Chen et al.

Original: [2017] combine TF-IDFretrieval with an LSTM document reader to find answers to free-form questions within Wikipediadocuments.
Revised: [2017] combine TF-IDF retrieval with an LSTM document reader to find answers to free-form questions within Wikipedia documents.

Original: Contemporary Q&A systems [Semnani and Pandey, 2020, Qu et al., 2020] extendthis general approach to more complex pipelines, combining traditional and neural methods withinverted index search/retrieval schemes, followed by document readers that perform re-ranking.
Revised: Contemporary Q&Asystems [Semnani and Pandey, 2020, Qu et al., 2020] extend this general approach to more complexpipelines, combining traditional and neural methods with inverted index search/retrieval schemes,followed by document readers that perform re-ranking.

Original: We demonstrate our navigation approach in conjunction with very basic retrieval and re-rankingapproaches, so as to clearly demonstrate the value of navigation.
Revised: We demonstrate our navigation approach in conjunction with very basic retrieval and re-ranking approaches, so as to clearly demonstrate thevalue of navigation.

Original: In common with our approach, Wanget al.
Revised: In common with our approach, Wang et al.

Original: [2021] also use a graph version of Wikipedia, but with the goal of aligning it to a knowledgebase, rather than navigation.
Revised: [2021] also use a graph version of Wikipedia, but with the goal of aligning it to a knowledge base, rather than navigation.

Original: Stammbach and Neumann [2019] propose a scheme for finding evidencefor the FEVER Thorne et al.
Revised: Stammbachand Neumann [2019] propose a scheme for finding evidence for the FEVER Thorne et al.

Original: [2018] benchmark that explores all links one step away from an initialretrieval set.
Revised: [2018]benchmark that explores all links one step away from an initial retrieval set.

Original: This can be viewed as a 1-step navigation operation, in contrast to our scheme thatpermits an arbitrary number of steps.
Revised: This can be viewed as a1-step navigation operation, in contrast to our scheme that permits an arbitrary number of steps.

Original: Navigation in knowledge bases: Graph navigation is a key element in utilizing knowledge bases(KB), where facts are represented as tuples of entity nodes connected by a labelled edge correspondingto relation between the two nodes.
Revised: Navigation in knowledge bases: Graph navigation is a key element in utilizing knowledge bases(KB), where facts are represented as tuples of entity nodes connected by a labelled edge correspondingto relation between the two nodes.

Original: For example, Das et al.
Revised: For example, Das et al.

Original: [2017] answer complex queries bynavigating the graph, inferring missing edges along the way, and finding a valid path to the answernode.
Revised: [2017] answer complex queries bynavigating the graph, inferring missing edges along the way, and finding a valid path to the answernode.

Original: In our case, the graph is fixed but consists of natural language text and is partially observed.
Revised: In our case, the graph is fixed but consists of natural language text and is partially observed.

Original: This makes the task much harder on two fronts.
Revised: This makes the task much harder on two fronts.

Original: First, the policy must comprehend the naturallanguage actions, without access to crisply labelled relation actions.
Revised: First, the policy must comprehend the naturallanguage actions, without access to crisply labelled relation actions.

Original: Second, the action space isopen and varying in size, unlike in a KB which typically has a fixed schema, i.e. relations/actionscome from a fixed size set.
Revised: Second, the action space isopen and varying in size, unlike in a KB which typically has a fixed schema, i.e. relations/actionscome from a fixed size set.

Original: Perhaps the closest work involving natural language actions, Fu et al.[2019] augment a KB with textual nodes.
Revised: Perhaps the closest work involving natural language actions, Fu et al.[2019] augment a KB with textual nodes.

Original: However, it still operates under the fixed KB schema.
Revised: However, it still operates under the fixed KB schema.

Original: Further improvements on navigation in KB have been shown by incorporating tricks like rewardshaping [Xiong et al., 2017] and action dropout [Lin et al., 2018].
Revised: Further improvements on navigation in KB have been shown by incorporating tricks like rewardshaping [Xiong et al., 2017] and action dropout [Lin et al., 2018].

Original: We adopted the action dropoutwhere some outgoing edges are masked during training to enable more effective path exploration.
Revised: We adopted the action dropoutwhere some outgoing edges are masked during training to enable more effective path exploration.

Original: Pushing more on exploration, inspired by AlphaGo, Shen et al.
Revised: Pushing more on exploration, inspired by AlphaGo, Shen et al.

Original: He et al.
Revised: He et al.

Original: [2022] combine thepolicy with Monte-Carlo tree search to navigate KB.
Revised: [2022] combine thepolicy with Monte-Carlo tree search to navigate KB.

Original: We find that no search method is necessary forour approach at evaluation time, with good performance obtained from the policy alone.
Revised: We find that no search method is necessary forour approach at evaluation time, with good performance obtained from the policy alone.

Original: Internet powered Q&A:
Revised: Internet powered Q&A:

Original: Several recent works have attempted complex question answering usingthe internet.
Revised: Several recent works have attempted complex question answering usingthe internet.

Original: Talmor and Berant [2018] retrieve snippets of information from the web with a searchengine to answer a structured decomposition of the question with a simpler Q&A model.
Revised: Talmor and Berant [2018] retrieve snippets of information from the web with a searchengine to answer a structured decomposition of the question with a simpler Q&A model.

Original: Therecent WebGPT Nakano et al.
Revised: Therecent WebGPT Nakano et al.

Original: [2021] system takes a broadly similar approach to long-form questionanswering.
Revised: [2021] system takes a broadly similar approach to long-form questionanswering.

Original: WebGPT accures information from the Bing search API as well as some navigation,and passes it to an LM to generate the answer text.
Revised: WebGPT accures information from the Bing search API as well as some navigation, andpasses it to an LM to generate the answer text.

Original: The complexity of the system necessitatesthe use of human demonstrations for training.
Revised: The complexity of the system necessitates the use ofhuman demonstrations for training.

Original: Related work from Lazaridou et al.
Revised: Related work from Lazaridou et al.

Original: [2022] utilizesa few-shot supervision of an LM to perform Google queries, which are then used by the LM togenerate an answer.
Revised: [2022] utilizes a few-shotsupervision of an LM to perform Google queries, which are then used by the LM to generate ananswer.

Original: They show that by incorporating information from Google queries, the answersare more factual than those generated directly from LMs. 
Revised: They show that by incorporating information from Google queries, the answers are morefactual than those generated directly from LMs. Another line of work, has leveraged link structure ofwebpages found on the internet to improve Q&A.

Revised: In LinkBERT [Yasunaga et al., 2022] is pretrainedto capture dependencies between documents and results in improvement for for multi-hop reasoningand few-shot reading comprehension.

Revised: Asai et al.

Original:  In contrast to these works, we focus onmastering navigation rather than any combination with search/retrieval, albeit in more limited domain(Wikipedia vs the internet).
Revised: [2020] use the link structure to enhance open domain Q&A by learning to retrieve according to reasoning paths. In contrast to these works, we focus onmastering navigation rather than any combination with search/retrieval, albeit in more limited domain(Wikipedia vs the internet).

Original: However, our approach could potentially act as a core component in ageneral web navigation, being more versatile on account of not requiring supervision.
Revised: However, our approach could potentially act as a core component in ageneral web navigation, being more versatile on account of not requiring supervision.

Original:  Approach
Revised:  Approach

Original: We consider navigation in a web environment W = { N , L } , where N := { n i } N i =1 is the set ofnodes, each n i representing a web page with content s i .
Revised: We consider navigation in a web environment W = { N , L } , where N := { n i } Ni =1 is the set ofnodes, each n i representing a web page with content s i .

Original: The links between pages are representedby L := { l i } Ni =1 , where l i = { l 1 i , . . .
Revised: The links between pages are representedby L := { l i } Ni =1 , where l i = { l 1 i , . . .

Original: , l ji , . . .
Revised: , l ji , . . .

Original: , l M i i } are the set of M i (directed) out-links from node i .
Revised: , l M i i } are the set of M i (directed) out-links from node i .

Original: We define NH ( n i ) as the set of nodes reached by following all links l i from node n i , where the j thneighbor is given by l ji ( n i ) .
Revised: We define NH ( n i ) as the set of nodes reached by following all links l i from node n i , where the j thneighbor is given by l ji ( n i ) .

Original: We assume some target node n g and some starting node n 0 are given, and wish to train an agent thatcan navigate from n 0 to n g by walking along the edges of the graph within T steps.
Revised: We assume some target node n g and some starting node n 0 are given, and wish to train an agent thatcan navigate from n 0 to n g by walking along the edges of the graph within T steps.

Original: To accomplish this,we construct a parametric policy p θ ( n t +1 | n t , n g ) which, at time step t , parameterizes a distributionover n t +1 ∈ NH ( n t ) .
Revised: To accomplish this,we construct a parametric policy p θ ( n t +1 | n t , n g ) which, at time step t , parameterizes a distributionover n t +1 ∈ NH ( n t ) .

Original: We train this model via behavioral cloning to maximize
Revised: We train this model via behavioral cloning to maximize

Original: [Equation]
Revised: [Equation]

Original: where E ( n 0 , . . . , n T ) is our trajectory distribution which generates trajectories ( n 0 , . .
Revised: where E ( n 0 , . . . , n T ) is our trajectory distribution which generates trajectories ( n 0 , . .

Original: . , n T )
Revised: . , n T )

Original: that wewould like our model to replicate (i.e. setting n g = n T ).
Revised: that wewould like our model to replicate (i.e. setting n g = n T ).

Original: 3.1 Trajectory distribution
Revised: 3.1 Trajectory distribution

Original: This framework opens up many choices for E .
Revised: This framework opens up many choices for E .

Original: Clearly some will result in ineffective navigationstrategies while others may enable us to train a policy which is capable of generalizing to new graphsand downstream applications.
Revised: Clearly some will result in ineffective navigationstrategies while others may enable us to train a policy which is capable of generalizing to new graphsand downstream applications.

Original: We elaborate on some choices below.
Revised: We elaborate on some choices below.

Original: Reverse trajectories We define E ( n T ) =
Revised: Reverse trajectories We define E ( n T ) =

Original: Uniform ( N ) (the uniform distribution over nodes in thegraph).
Revised: Uniform ( N ) (the uniform distribution over nodes in thegraph).

Original: Now we can define
Revised: Now we can define

Original:  [Equation]
Revised: E ( [Equation] = [Equation]

Revised: { n

Original: or, a distribution which uniformly walks the reverse graph (with transposed adjacency matrix).
Revised: [Equation] n i ∈ NH [Equation]

Original:  Webegin with some uniformly chosen target node and walk backward randomly to n 0 in such a way thatwe can reach the target by walking forward from n 0 .
Revised: n t ) } [Equation] if n t +1 ∈ NH ( n t ) , else(2) or, a distribution which uniformly walks the reverse graph (with transposed adjacency matrix). Webegin with some uniformly chosen target node and walk backward randomly to n 0 in such a way thatwe can reach the target by walking forward from n 0 .

Original: Intuitively, training with the reverse trajectories defined above should allow our model to be able toreach any node in the graph from any node it is connected to.
Revised: Intuitively, training with the reverse trajectories defined above should allow our model to be able toreach any node in the graph from any node it is connected to.

Original: Unfortunately, in realistic web graphdata, there exist many “dead-ends” or nodes with no in-going edges.
Revised: Unfortunately, in realistic web graphdata, there exist many “dead-ends” or nodes with no in-going edges.

Original: The trajectories get stuck atthese nodes leading to low-diversity training data.
Revised: The trajectories get stuck atthese nodes leading to low-diversity training data.

Original: Random forward trajectories In our real-world knowledge graph data, there are far fewer nodeswhich have no out-going edges.
Revised: Random forward trajectories In our real-world knowledge graph data, there are far fewer nodeswhich have no out-going edges.

Original: Thus we propose an alternative method to generate target nodesbased on randomly walking forward.
Revised: Thus we propose an alternative method to generate target nodesbased on randomly walking forward.

Original: We define E ( n 0 ) to be the Uniform ( N ) and
Revised: We define E ( n 0 ) to be the Uniform ( N ) and

Original: E ( n t +1 | n t ) =
Revised: E ( n t +1 | n t ) =

Original: | NH ( n t ) | if n t +1 ∈ NH ( n t ) , else 0 .
Revised: | NH ( n t ) | if n t +1 ∈ NH ( n t ) , else 0 .

Original: (3) or, a distribution which uniformly picks a starting node and randomly walks the forward graph for
Revised: (3) or, a distribution which uniformly picks a starting node and randomly walks the forward graph for

Original: T steps.
Revised: T steps.

Original: Training in this way should encourage our model to be able to reach any node which isreachable from some uniformly chosen start.
Revised: Training in this way should encourage our model to be able to reach any node which isreachable from some uniformly chosen start.

Original: We note that the target node distribution E ( n T ) will notbe uniform and instead be a function of the graph structure.
Revised: We note that the target node distribution E ( n T ) will notbe uniform and instead be a function of the graph structure.

Original: We also note that while this generates a
Revised: We also note that while this generates a

Original: T -step trajectory, because it is generated by random walk, the shortest path distance between startand target nodes may be smaller.
Revised: T -step trajectory, because it is generated by random walk, the shortest path distance between startand target nodes may be smaller.

Original: Random shortest paths We define E ( n T ) =
Revised: Random shortest paths We define E ( n T ) =

Original: Uniform ( N ) and let E ( n 0 | n T ) be the uniformdistribution of nodes whose shortest path length to n T is T steps.
Revised: Uniform ( N ) and let E ( n 0 | n T ) be the uniformdistribution of nodes whose shortest path length to n T is T steps.

Original: We then define E ( n t | n 0 , n T ) tobe the delta distribution that n t is the t -th node of this shortest path.
Revised: We then define E ( n t | n 0 , n T ) tobe the delta distribution that n t is the t -th node of this shortest path.

Original: While it can be challenging tocompute the probabilities of this distribution efficiently we can draw samples from it using a shortestpath finding algorithm such as Dijkstra’s algorithm.
Revised: While it can be challenging tocompute the probabilities of this distribution efficiently we can draw samples from it using a shortestpath finding algorithm such as Dijkstra’s algorithm.

Original: 3.2 Model parameterization
Revised: 3.2 Model parameterization

Original: From above, we can see the core of our model is the p θ ( n t +1 | n t , n T ) distribution.
Revised: From above, we can see the core of our model is the p θ ( n t +1 | n t , n T ) distribution.

Original: Given a sampledtrajectory that begins at n 0 , this is the distribution which attempts to choose the correct action totake at timestep t + 1 starting at node n t in order to get to target node n T .
Revised: Given a sampledtrajectory that begins at n 0 , this is the distribution which attempts to choose the correct action totake at timestep t + 1 starting at node n t in order to get to target node n T .

Original: We parameterize thisdistribution as a function of the text available at node n t .
Revised: We parameterize thisdistribution as a function of the text available at node n t .

Original: [Equation]
Revised: [Equation]

Original: where f θ embeds the text s g , s t and s t +1 of the target, current and destination nodes, respectively.
Revised: where f θ embeds the text s g , s t and s t +1 of the target, current and destination nodes, respectively.

Original: Further details on how we apply this general model to Wikipedia are given in Section 4.2.
Revised: Further details on how we apply this general model to Wikipedia are given in Section 4.2 and analternative interpretation of this approach based on variational inference in a latent-variable model ispresented in Appendix D.

Original:  Application to Wikipedia
Revised:  Application to Wikipedia

Original: 4.1 Navigation and sentence search tasks
Revised: 4.1 Navigation and sentence search tasks

Original: Data processing: We convert a snapshot of English Wikipedia into a graph but split each article( ¯ µ = 1000 words) into paragraph-sized blocks ( ¯ µ = 100 words), which form the nodes .
Revised: Data processing: We convert a snapshot of English Wikipedia into a graph but split each article( ¯ µ = 1000 words) into paragraph-sized blocks ( ¯ µ = 100 words), which form the nodes .

Original: Edges inthe graph are the union of (i) organic hyperlinks, (ii) additional entity linking and (iii) next/previousparagraph links.
Revised: Edges inthe graph are the union of (i) organic hyperlinks, (ii) additional entity linking and (iii) next/previousparagraph links.

Original: The full graph has ∼ 38 M nodes and 387 M edges.
Revised: The full graph has ∼ 38 M nodes and 387 M edges.

Original: For initial experiments, we also use a smaller subsampled version of 200 k nodes.
Revised: For initial experiments, we also use a smaller subsampled version of 200 k nodes.

Original: For this we sub-sample disjoint sets of 200k train /200k evaluation Wikipedia nodes.
Revised: For this we sub-sample disjoint sets of 200k train /200k evaluation Wikipedia nodes.

Original: See Section A.2 for full details.
Revised: See Section A.1 for full details.

Original: T-step navigation: The most basic task is T -step navigation.
Revised: T-step navigation: The most basic task is T -step navigation.

Original: At the start of the episode, we randomlysample a start node and generate a T -step random walk trajectory.
Revised: At the start of the episode, we randomlysample a start node and generate a T -step random walk trajectory.

Original: We then take the last node of thistrajectory as the target n g .
Revised: We then take the last node of thistrajectory as the target n g .

Original: The agent is then given a time budget B (in most experiments, we fix maxsteps to B = 100 ), and succeeds if it reaches the target node within B steps.
Revised: The agent is then given a time budget B (in most experiments, we fix maxsteps to B = 100 ), and succeeds if it reaches the target node within B steps.

Original: In our experiments, weshow results on various graphs and splits with T = { 5 , 10 , 20 } -step navigation.1-T Multistep navigation: The Multistep task is much like the previous setting, except that insteadof generating fixed T -step trajectories, we sample from Uniform (1 , T ) , T = 20 .
Revised: In our experiments, weshow results on various graphs and splits with T = { 5 , 10 , 20 } -step navigation.1-T Multistep navigation: The Multistep task is much like the previous setting, except that insteadof generating fixed T -step trajectories, we sample from Uniform (1 , T ) , T = 20 .

Original: T-step sentence search: This is the same setup as T -step navigation, except that rather than providethe full text for the target node n g , only one sentence within that text is selected at random to computethe target embeddings from.
Revised: T-step sentence search: This is the same setup as T -step navigation, except that rather than providethe full text for the target node n g , only one sentence within that text is selected at random to computethe target embeddings from.

Original: Thus agents have the more challenging task of locating the correct node,given only a small snippet of its text (see Sec.
Revised: Thus agents have the more challenging task of locating the correct node,given only a small snippet of its text (see Sec.

Original: 4.2 for details on target representation).
Revised: 4.2 for details on target representation).

Original: We includethis task because one of the difficulties of web navigation is establishing which page should be thetarget in the first place.
Revised: We includethis task because one of the difficulties of web navigation is establishing which page should be thetarget in the first place.

Original: It also has direct relevance to our downstream fact verification task.
Revised: It also has direct relevance to our downstream fact verification task.

Original: 4.2 Model architecture
Revised: 4.2 Model architecture

Original: State representation:
Revised: State representation:

Original: For each node n i we embed its text
Revised: For each node n i we embed its text

Original: s i using a Transformer ϕ ( . ) to produceϕ ( s i ) .
Revised: s i using a Transformer ϕ ( . ) to produceϕ ( s i ) .

Original: By basing the state representation on semantic content, rather than i.e. node index, the agentis able to learn about the relationships between entities in a manner that allows for generalizationto new graphs at evaluation time, instead of memorizing the structure of the training graph.
Revised: By basing the state representation on semantic content, rather than i.e. node index, the agentis able to learn about the relationships between entities in a manner that allows for generalizationto new graphs at evaluation time, instead of memorizing the structure of the training graph.

Original: Following Karpukhin et al.
Revised: Following Karpukhin et al.

Original: [2020], we append to the node text the title of the Wikipedia article to which itbelongs.
Revised: [2020], we append to the node text the title of the Wikipedia article to which itbelongs.

Original: This often provides context to the node that might otherwise be missing – in many articles,the subject of the article will be referred to indirectly (e.g. it might state that “she” was born in 1945instead of repeating the subject’s name).
Revised: This often provides context to the node that might otherwise be missing – in many articles,the subject of the article will be referred to indirectly (e.g. it might state that “she” was born in 1945instead of repeating the subject’s name).

Original: For those experiments where the node embeddings are kept fixed, ϕ ( . ) is a pre-trained RoBERTA[Liu et al., 2019] model that encodes the node text & title.
Revised: For those experiments where the node embeddings are kept fixed, ϕ ( . ) is a pre-trained RoBERTA[Liu et al., 2019] model that encodes the node text & title.

Original: However, in our large-scale experiments,we pre-train the Transformer ϕ ( . ) directly using the text & title (see Appendix B.1).
Revised: However, in our large-scale experiments,we pre-train the Transformer ϕ ( . ) directly using the text & title (see Appendix B.1).

Original: In both cases, weapply the Transformer to the tokenized text, take the mean over the input tokens, and apply a tanhnonlinearity to produce ϕ ( s i ) .
Revised: In both cases, weapply the Transformer to the tokenized text, take the mean over the input tokens, and apply a tanhnonlinearity to produce ϕ ( s i ) .

Original: Target representation: Similarly, we use the article text s g of the target node as input to the
Revised: Target representation: Similarly, we use the article text s g of the target node as input to the

Original: Transformer model ϕ to compute its representation ϕ ( s g ) .
Revised: Transformer model ϕ to compute its representation ϕ ( s g ) .

Original: For some tasks, such as sentence search orfact verification, the target representation instead relies on a single sentence from each node or othertext specific to the task and a separate transformer model ϕ target is trained to embed the target.
Revised: For some tasks, such as sentence search orfact verification, the target representation instead relies on a single sentence from each node or othertext specific to the task and a separate transformer model ϕ target is trained to embed the target.

Original: Navigation policy network: The network operates on the target node n g and the current node n t ,and outputs a distribution over the possible actions that can be taken at n t .
Revised: Navigation policy network: The network operates on the target node n g and the current node n t ,and outputs a distribution over the possible actions that can be taken at n t .

Original: As the nodes in the graphhave a variable number of outgoing edges, the policy must produce a distribution with a variablenumber of outcomes.
Revised: As the nodes in the graphhave a variable number of outgoing edges, the policy must produce a distribution with a variablenumber of outcomes.

Original: To do this, we first concatenate the embeddings of s g and s t , and pass themthrough a 1-layer feed-forward network to produce a combined embedding e tg =
Revised: To do this, we first concatenate the embeddings of s g and s t , and pass themthrough a 1-layer feed-forward network to produce a combined embedding e tg =

Original: FF [ ϕ ( s t ) , ϕ ( s g )] .
Revised: FF [ ϕ ( s t ) , ϕ ( s g )] .

Original: Next we embed each possible action a i .
Revised: Next we embed each possible action a i .

Original: When we use a fixed ϕ , we directly use a i = ϕ
Revised: When we use a fixed ϕ , we directly use a i = ϕ

Original: ( s i ) forn
Revised: ( s i ) forn

Original: i ∈ NH ( n t ) .
Revised: i ∈ NH ( n t ) .

Original: However this is not feasible when pre-training our embedding model due to memoryconcerns.
Revised: However this is not feasible when pre-training our embedding model due to memoryconcerns.

Original: Instead, we compute a i = ϕ
Revised: Instead, we compute a i = ϕ

Original: ( s t [ l it ]) where s [ l i ] selects the words in s which belong tohyperlink i – see Appendix B.1 for more details.
Revised: ( s t [ l it ]) where s [ l i ] selects the words in s which belong tohyperlink i – see Appendix B.1 for more details.

Original: We then define the probability of moving to n i fromn t with target n g as p ( n i | n t , n g )
Revised: We then define the probability of moving to n i fromn t with target n g as p ( n i | n t , n g )

Original: ∝ exp ( e tg · a i ) which is normalized to produce a distribution overthe | NH ( n t ) | actions.
Revised: ∝ exp ( e tg · a i ) which is normalized to produce a distribution overthe | NH ( n t ) | actions.

Original: In our larger-scale model, we also include the trajectory leading up to the current node ( n 0 , . . . , n t ) .
Revised: In our larger-scale model, we also include the trajectory leading up to the current node ( n 0 , . . . , n t ) .

Original: We pass their embeddings, combined with the target embedding, [ ϕ ( s 0 ) , . . .
Revised: We pass their embeddings, combined with the target embedding, [ ϕ ( s 0 ) , . . .

Original: , ϕ ( s t ) , ϕ ( s g )] , through a4-layer transformer and use the output as e tg when computing the transition probabilities above.
Revised: , ϕ ( s t ) , ϕ ( s g )] , through a4-layer transformer and use the output as e tg when computing the transition probabilities above.

Original: Weexplore these two policy architecture choices, feed-forward and Transformer in Table 4.
Revised: Weexplore these two policy architecture choices, feed-forward and Transformer in Table 4.

Original: 4.3 Application to fact verification
Revised: 4.3 Application to fact verification

Original: Finally, we use our navigation approach to gather information for the task of fact
Revised: Finally, we use our navigation approach to gather information for the task of fact

Original: verification onthe FEVER benchmark Thorne et al.
Revised: verification onthe FEVER benchmark Thorne et al.

Original: [2018] which also uses Wikipedia as a knowledge base.
Revised: [2018] which also uses Wikipedia as a knowledge base.

Original: One important difference from our earlier navigation task is that the target node n g is not specifiedat evaluation time.
Revised: Oneimportant difference from our earlier navigation task is that the target node n g is not specifiedat evaluation time.

Original: Instead, we are given a claim whose veracity must be established by findinginformation in a particular node in the graph.
Revised: Instead, we are given a claim whose veracity must be established by finding information in a particular node in the graph.

Original: To determine which node this is, we use a target encoderwhich maps the claim to an embedding vector that can be used in place of ϕ ( s g ) in Section 4.2.
Revised: To determine which node this is, we use a target encoderwhich maps the claim to an embedding vector that can be used in place of ϕ ( s g ) in Section 4.2.

Original: Due to the limited size of the FEVER training set, we pre-train our graph embeddings, navigationpolicy and target encoder on the similar sentence search task.
Revised: Due to the limited size of the FEVER training set, we pre-train our graph embeddings, navigationpolicy and target encoder on the similar sentence search task.

Original: We then freeze the embeddings andpolicy and fine-tune the target encoder on ( n g , s g ) pairs from the FEVER training set.
Revised: We then freeze the embeddings andpolicy and fine-tune the target encoder on ( n g , s g ) pairs from the FEVER training set.

Original: To avoidover-fitting, we add an auxiliary loss which enforces the similarity between our target embedding andthe embedding of the ground truth node containing the sentence needed to resolve the claim.
Revised: To avoidover-fitting, we add an auxiliary loss which enforces the similarity between our target embedding andthe embedding of the ground truth node containing the sentence needed to resolve the claim.

Original: Trainingand evaluation on FEVER required alignment between our version of Wikipedia and the one used tocompile the benchmark and details of this can be found in Appendix A.4.
Revised: Trainingand evaluation on FEVER required alignment between our version of Wikipedia and the one used tocompile the benchmark and details of this can be found in Appendix A.4.

Original: Next, our navigation method requires a starting node.
Revised: Next, our navigation method requires a starting node.

Original: To put our agent close to the articles we needin the downstream task, we run a popular variant of TF-IDF, called BM25 [Robertson and Zaragoza,2009].
Revised: To put our agent close to the articles we needin the downstream task, we run a popular variant of TF-IDF, called BM25 [Robertson and Zaragoza,2009].

Original: We first create an index over all of the nodes in the graph, then take the top-5 BM25 matchesas the starting nodes, and run navigation for 20 steps on each.
Revised: We first create an index over all of the nodes in the graph, then take the top-5 BM25 matchesas the starting nodes, and run navigation for 20 steps on each.

Original: Note that BM25 does not find the exacttext (i.e. the hits@1 is not very high) but matches are in the right vicinity, usually in the same orsimilar article.
Revised: Note that BM25 does not find the exacttext (i.e. the hits@1 is not very high) but matches are in the right vicinity, usually in the same orsimilar article.

Original: Thus the agent only has to navigate a few steps to reach the target: for the FEVERdataset, the mean length of shortest path between start and target node is 4 .
Revised: Thus the agent only has to navigate a few steps to reach the target: for the FEVERdataset, the mean length of shortest path between start and target node is 4 .

Original: 3 steps.
Revised: 3 steps.

Original: The final component is a re-ranker that takes all the sentences from all nodes visited by the agent andassigns a match score between each one and the claim.
Revised: The final component is a re-ranker that takes all the sentences from all nodes visited by the agent andassigns a match score between each one and the claim.

Original: The ranked list of sentences can then be usedfor benchmark evaluation.
Revised: The ranked list of sentences can then be usedfor benchmark evaluation.

Original: We explore two types of ranker: (a) basic TF-IDF and (b) a BERT model,fine tuned using hard negatives mined from the TF-IDF ranker.
Revised: We explore two types of ranker: (a) basic TF-IDF and (b) a BERT model,fine tuned using hard negatives mined from the TF-IDF ranker.

Original: See Appendix A.4 for more details.
Revised: See Appendix A.4 for more details.

Original: Evaluation of our approach thus consists of: (i) given a claim, use BM-25 to return a list of promisingstart locations; (ii) from the top N = 10 of these, run our navigation model for 20 steps, using thetarget encoder to embed the claim, (iii) use the re-ranker to score sentences visited against the claimand (iv) select the top 5 for computing the F1 score, for comparison on the FEVER benchmark.
Revised: Evaluation of our approach thus consists of: (i) given a claim, use BM-25 to return a list of promisingstart locations; (ii) from the top N = 10 of these, run our navigation model for 20 steps, using thetarget encoder to embed the claim, (iii) use the re-ranker to score sentences visited against the claimand (iv) select the top 5 for computing the F1 score, for comparison on the FEVER benchmark.

Original:  Experiments
Revised:  Experiments

Original: 5.1 Investigation of training trajectory distribution
Revised: 5.1 Investigation of training trajectory distribution

Original: On the smaller 200k node graph, we investigate different choices for the trajectory policy distributions
Revised: On the smaller 200k node graph, we investigate different choices for the trajectory policy distributions

Original: E described in Section 3: (i) Random Forward Trajectories , where we randomly sample startnodes and do a random walk to get a target, (ii) Reverse Trajectories , where we randomly sample atarget node and and do a random walk on the reverse graph to get a start node, and (iii) Random
Revised: E described in Section 3: (i) Random Forward Trajectories , where we randomly sample startnodes and do a random walk to get a target, (ii) Reverse Trajectories , where we randomly sample atarget node and and do a random walk on the reverse graph to get a start node, and (iii) Random

Original: Shortest Paths where we randomly sample a start node and take a random walk and then computethe shortest path between the start and target nodes.
Revised: Shortest Paths where we randomly sample a start node and take a random walk and then computethe shortest path between the start and target nodes.

Original: For each of these we trained policies on 5 ,10 , and 20 step navigation tasks.
Revised: For each of these we trained policies on 5 ,10 , and 20 step navigation tasks.

Original: We evaluate these in two different ways: (i) forward sampling ofsource and target in the disjoint evaluation graph, (ii) reverse sampling of source and target.
Revised: We evaluate these in two different ways: (i) forward sampling ofsource and target in the disjoint evaluation graph, (ii) reverse sampling of source and target.

Original: Theresults of this experiment are shown in Table 1.
Revised: Theresults of this experiment are shown in Table 1.

Original: Unsurprisingly, we can see that forward and reversetrained policies both do better when evaluated in the way that matches training.
Revised: Unsurprisingly, we can see that forward and reversetrained policies both do better when evaluated in the way that matches training.

Original: However, forwardtrajectory policies generalize much better to reverse navigation.
Revised: However, forwardtrajectory policies generalize much better to reverse navigation.

Original: When we analyze the traces of theseexperiments, we see that in the reverse sampling, many target nodes are in “dead-ends” where manytarget nodes do not have long random trajectories to sample.
Revised: When we analyze the traces of theseexperiments, we see that in the reverse sampling, many target nodes are in “dead-ends” where manytarget nodes do not have long random trajectories to sample.

Original: This results in easier navigation (as wecan see from higher accuracy in reverse trajectory trained policies on reverse navigation) and moreextreme overfitting, leading to poor performance of reverse trajectory on forward navigation.
Revised: This results in easier navigation (as wecan see from higher accuracy in reverse trajectory trained policies on reverse navigation) and moreextreme overfitting, leading to poor performance of reverse trajectory on forward navigation.

Original: Not surprisingly, we also see that training on a shortest path trajectories leads to better performanceboth in the forward and reverse sampling.
Revised: Not surprisingly, we also see that training on a shortest path trajectories leads to better performanceboth in the forward and reverse sampling.

Original: This advantage is especially seen in the 20 -step navigationtask.
Revised: This advantage is especially seen in the 20 -step navigationtask.

Original: Ideally, we would always train on shortest paths.
Revised: Ideally, we would always train on shortest paths.

Original: However, the time complexity of precomputing all shortest paths in a graph using Dijkstra’s algorithm is O ( V ( E + V logV )) for thenumber of nodes V and number of edges E in the graph.
Revised: However, the time complexity of precomputing all shortest paths in a graph using Dijkstra’s algorithm is O ( V ( E + V logV )) for thenumber of nodes V and number of edges E in the graph.

Original: On the full 38 M Wikipedia graph, thiscomputation would be completely intractable.
Revised: On the full 38 M Wikipedia graph, thiscomputation would be completely intractable.

Original: Luckily, the random forward trajectories performalmost as well and can be computed in constant time with respect to the size of the graph and onlylinear time with respect to the trajectory length.
Revised: Luckily, the random forward trajectories performalmost as well and can be computed in constant time with respect to the size of the graph and onlylinear time with respect to the trajectory length.

Original: For all subsequent experiments, we use the Random
Revised: For all subsequent experiments, we use the Random

Original: Forward Trajectories for our trajectory policy and evaluate on the forward version of the navigationtasks, where we call our approach Random Forward Behavioral Cloning (RFBC) .[Table] [Table]
Revised: Forward Trajectories for our trajectory policy and evaluate on the forward version of the navigationtasks, where we call our approach Random Forward Behavioral Cloning (RFBC) .

Original: 5.2 Navigation
Revised: 5.2 Navigation

Original: We next do a more thorough set of experiments on the smaller 200 k Wikipedia graph.
Revised: We next do a more thorough set of experiments on the smaller 200 k Wikipedia graph.

Original: Because thegraph is much smaller than the full Wikipedia and thus prone to overfitting, for all of our methods andbaselines we use the fixed RoBERTA embeddings ϕ
Revised: Because thegraph is much smaller than the full Wikipedia and thus prone to overfitting, for all of our methods and [Table] [Table] baselines we use the fixed RoBERTA embeddings ϕ

Original: ( s i ) , the simplest 1-layer feed-forward network,and MiniBERT Turc et al.
Revised: ( s i ) , the simplest 1-layer feed-forward network,and MiniBERT Turc et al.

Original: [2019] for target sentence embedding ϕ
Revised: [2019] for target sentence embedding ϕ

Original: ( s g ) .
Revised: ( s g ) .

Original: Alongside our approach, we employ the following baselines/ablations:
Revised: Alongside our approach [Table] we employ the following baselines/ablations:

Original: • RFBC:
Revised: • RFBC:

Original: Random-forward behavioral cloning (our approach).
Revised: Random-forward behavioral cloning (our approach).

Original: • RL: Navigation using a policy gradients RL agent with same policy architecture as our model.
Revised: • RL: Navigation using a policy gradients RL agent with same policy architecture as our model.

Original: See Sec.
Revised: See Sec.

Original: A.5 for more details.• RFBC + RF: RFBC but with random features sampled from the unit sphere instead of ϕ• RL + RF: RL with random
Revised: A.5 for more details.• RFBC + RF: RFBC but with random features sampled from the unit sphere instead of ϕ• RL + RF: RL with random

Original: features.• Random: policy that chooses random action (out-link) at each timestep. • Greedy: selects the action embedding which has the smallest cosine distance with the target.• Random DFS: DFS of depth equal to number of steps, choosing actions at random.• Greedy DFS: DFS but select action with smallest cosine distance to the target.
Revised: features.• Random: policy that chooses random action (out-link) at each timestep.• Greedy: selects the action embedding which has the smallest cosine distance with the target.• Random DFS: DFS of depth equal to number of steps, choosing actions at random.• Greedy DFS: DFS but select action with smallest cosine distance to the target.

Original: The results evaluated on the held-out graph are shown in columns 1-4 of Table 2.
Revised: The results evaluated on the held-out graph are shown in columns 1-4 of Table 2.

Original: Our approach(RFBC) performs well, achieving 77% in the multistep case.
Revised: Our approach(RFBC) performs well, achieving 77% in the multistep case.

Original: Performance also drops as the distance tothe target increases.
Revised: Performance also drops as the distanceto the target increases.

Original: In contrast, the RL agent did not perform well (around 40 % on navigation tasks).
Revised: In contrast, the RL agent performs poorly (around 40 % on navigation tasks).

Original:  Using random features in place of ϕ ( . ) caused the performance to drop to near chance, showing thatthe models are utilizing the semantics at each node and are not relying on a general search strategy ofsome kind.
Revised: Because we generate virtually infinite expert trajectories with RFBC, RL performs worse as it istrained with essentially the same amount of exploration, but with a more sparse signal than behavioralcloning. Using random features in place of ϕ ( . ) caused the performance to drop to near chance,showing that the models are utilizing the semantics at each node and are not relying on a generalsearch strategy of some kind.

Original: Random methods also do quite poorly, even when adding a DFS.
Revised: Random methods also do quite poorly, even when adding a DFS.

Original: Greedy methods do abit better than random, especially with DFS, butare still well below our method.
Revised: Greedy methods do a bit better than random, especially with DFS, but still well below our method.

Original: We also consider the more challenging sentence search task described in Section 4.1.
Revised: We also consider the more challenging sentence search task described in Section 4.1.

Original: This taskrequires not only finding a known target node, but learning to find a target node given only a singlerandom sentence from that node, more closely matching our downstream task and the way humanstend to approach finding information on the web.
Revised: This taskrequires not only finding a known target node, but learning to find a target node given only a singlerandom sentence from that node, more closely matching our downstream task and the way humanstend to approach finding information on the web.

Original: Columns 5-7 of Table 2 show performance on thistask for our approach and RL.
Revised: Columns 5-7 of Table 2 show performance on thistask for our approach and RL.

Original: Our method has reduced performance on the task due to the extradifficulty of learning the target embedding, but still performs reasonably.
Revised: Our method has reduced performance on the task due to the extradifficulty of learning the target embedding, but still performs reasonably.

Original: RL, however, degrades tonear chance performance.[Table] [Table]
Revised: RL, however, degrades tonear chance performance.

Original: 5.3 Navigation on full 38M node Wikipedia graph:
Revised: 5.3 Navigation on full 38M node Wikipedia graph:

Original: Our method scales naturally to much larger graphs,which we demonstrate by training on the entire
Revised: Our method scales naturally to much larger graphs,

Original: Wikipedia 2017 graph.
Revised: [Table]

Original: We evaluate on the entire Wikipedia 2018 graph. Each graph is ∼ 38 M nodesand ∼ 380 M edges (see Table 3 for details).
Revised: [Table]

Original: We explore both architectures described in Section 4.2 for the policy network: (i) the single feedforward layer (as used in the smaller graph) and (ii) the 4-layer Transformer model.
Revised: We explore both architectures described in Section 4.2 for the policy network: (i) the single feedforward layer (as used in the smaller graph) and (ii) the 4-layer Transformer model.

Original: We also compareusing a fixed RoBERTA model for ϕ versus pre-training a text transformer directly on the navigationtask (see Appendix B.1).
Revised: We also compareusing a fixed RoBERTA model for ϕ versus pre-training a text transformer directly on the navigationtask (see Appendix B.1).

Original: DistillBERT Sanh et al.
Revised: DistillBERT Sanh et al.

Original: [2019] is used for ϕ ( n g ) .
Revised: [2019] is used for ϕ ( n g ) .

Original: Table 2 shows the results on the full graph, evaluating on { 5 , 10 , 20 } step tasks, for both navigationand sentence search.
Revised: Table 2 shows the results on the full graph, evaluating on { 5 , 10 , 20 } step tasks, for both navigationand sentence search.

Original: Figure 2 shows example trajectories of the trained agent.
Revised: Figure 2 shows example trajectories of the trained agent.

Original: The first thing to noteis that the overall navigation and sentence search performance of our methods is high.
Revised: The first thing to noteis that the overall navigation and sentence search performance of our methods is high.

Original: The 

Revised: The

Original: Wikipedia graph we use for evaluation contains over 5 million articles, over 38 million nodes and387 million edges.
Revised: Wikipedia graph we use for evaluation contains over 5 million articles, over 38 million nodes and387 million edges.

Original: On this graph (which we generalize to from the previous year’s graph) our bestmethod can find an article 20 steps away 92 .
Revised: On this graph (which we generalize to from the previous year’s graph) our bestmethod can find an article 20 steps away 92 .

Original: 2% percent of the time and can locate it given just asingle sentence of that article 90 .
Revised: 2% percent of the time and can locate it given just asingle sentence of that article 90 .

Original: 2% of the time.
Revised: 2% of the time.

Original: This compares favorably with a similar evaluationperformed in Nogueira and Cho [2016], where their WebNav approach achieved a 12 . 5% success ratefor 16 step tasks on a 12M node graph.
Revised: This compares favorably with a similar evaluationperformed in Nogueira and Cho [2016], where their WebNav approach achieved a 12 . 5% success ratefor 16 step tasks on a 12M node graph.

Original: Compared to the performance of our method on the smaller200 k graph, we see that performance has greatly improved, most likely due to the addition of ordersof magnitude more training data.
Revised: Compared to the performance of our method on the smaller200 k graph, we see that performance has greatly improved, most likely due to the addition of ordersof magnitude more training data.

Original: In particular we see that sentence search performance on 20 -stepimproves greatly from 34 .
Revised: In particular we see that sentence search performance on 20 -stepimproves greatly from 34 .

Original: 6% to a best performance of 90 . 2% .
Revised: 6% to a best performance of 90 . 2% .

Original: Additionally, in all cases, our learned embeddings perform better than using the fixed features,derived from large language models.
Revised: Additionally, in all cases, our learned embeddings perform better than using the fixed features,derived from large language models.

Original: In general, feed-forward policies perform best, except for 20step navigation tasks.
Revised: In general, feed-forward policies perform best, except for 20step navigation tasks.

Original: For longer trajectories, the Transformer policy network achieves slightly betterperformance, while on sentence search, the feed-forward model is consistently superior.
Revised: For longer trajectories, the Transformer policy network achieves slightly betterperformance, while on sentence search, the feed-forward model is consistently superior.

Original: 5.4 Application to fact verification
Revised: 5.4 Application to fact verification

Original: We now evaluate the ability of our navigation approach to select evidence on the FEVER developmentset Thorne et al.
Revised: We now evaluate the ability of our navigation approach to select evidence on the FEVER developmentset Thorne et al.

Original: [2018] with results shown in Table 5.
Revised: [2018] with results shown in Table 5.

Original: The primary metric used is F1@top5 butwe also give precision and recall.
Revised: The primary metric used is F1@top5 butwe also give precision and recall.

Original: Paired with a basic retrieval (BM25) and re-ranker (TF-IDF), theapproach obtains a respectable F1 of 0.46.
Revised: Paired with a basic retrieval (BM25) and re-ranker (TF-IDF), theapproach obtains a respectable F1 of 0.46.

Original: If we remove our navigation component then this drops Method Precision@
Revised: If we remove our navigation component then this dropssubstantially to 0.36, demonstrating its utility.

Original: Recall@ F1@ BM25 + TF-IDF
Revised: Swapping to a more powerful re-ranker (BigBird)boosts the F1 to 0.731, which is competitive with the state-of-the-art (FEVER leaderboard [2022]

Original: [Thorne et al., 2018] 0.
Revised: [Table] et al.

Original: BM25 + RFBC
Revised: [Table]

Original: (Ours) + TF-IDF 0.
Revised: [Table]

Original: BM25 + RFBC
Revised: BM25 + RFBC

Original: (Ours) + BigBird 0.
Revised: (Ours) + BigBird 0.

Original: Stammbach [2021] 0.
Revised: Stammbach [2021] 0.

Original: substantially to 0.36, demonstrating its utility. Swapping to a more powerful re-ranker (BigBird)boosts the F1 to 0.731, which is competitive with the state-of-the-art (FEVER leaderboard [2022]rank #6; rank #1 F1 is 0.799).
Revised: rank #6; rank #1 F1 is 0.799).

Original: We also compare to a leading approach Stammbach [2021] thatsimilarly relies on the BigBird re-ranker, with our approach having a significantly better F1 score.
Revised: We also compare to a leading approach Stammbach [2021] thatsimilarly relies on the BigBird re-ranker, with our approach having a significantly better F1 score.

Original: A possible explanation is that their approach overwhelms the re-ranker with all possible hyperlinkswhereas, navigation being inherently sparse, our model presents a more refined set for re-ranking.
Revised: A possible explanation is that their approach overwhelms the re-ranker with all possible hyperlinkswhereas, navigation being inherently sparse, our model presents a more refined set for re-ranking.

Original: Wenote that our approach: (a) is significantly simpler than many top ranked approaches and (b) selectsevidence over a much larger set than the curated version of Wikipedia used in FEVER (38M vs 5M).
Revised: Wenote that our approach: (a) is significantly simpler than many top ranked approaches and (b) selectsevidence over a much larger set than the curated version of Wikipedia used in FEVER (38M vs 5M).

Original: Examples of successful navigation traces are shown in Figure 3.
Revised: Examples of successful navigation traces are shown in Figure 3.

Original:  Discussion
Revised:  Discussion

Original: We have presented a simple and effective scheme for navigating a large Wikipedia graph that isapplicable to more general web navigation problems.
Revised: We have presented a simple and effective scheme for navigating a large Wikipedia graph that isapplicable to more general web navigation problems.

Original: We show that behavioral cloning of randomtrajectories is a viable approach to learning both entity embeddings and a navigation policy.
Revised: We show that behavioral cloning of randomtrajectories is a viable approach to learning both entity embeddings and a navigation policy.

Original: Whenapplied to the FEVER verification task, they offer highly competitive performance, whilst beingcomplementary to existing approaches.
Revised: Whenapplied to the FEVER verification task, they offer highly competitive performance, whilst beingcomplementary to existing approaches.

Original: One limitation of our approach when we move from Wikipediato the wider Internet is that our scheme relies on a good target encoder.
Revised: One limitation of our approach when we move from Wikipediato the wider Internet is that our scheme relies on a good target encoder.

Original: For navigation and ourdownstream task, there was a clear ground-truth target node available training, but in other settingsthis might not be the case.
Revised: For navigation and ourdownstream task, there was a clear ground-truth target node available training, but in other settingsthis might not be the case.

Original: Another limitation is that we require a re-ranker to decide on the finalretrieved sentence, but ideally the agent would decide for itself when it has reached the correct node.
Revised: Another limitation is that we require a re-ranker to decide on the finalretrieved sentence, but ideally the agent would decide for itself when it has reached the correct node.

Original: [Table]
Revised: [Table]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Table]
Revised: [Table]

Original: Checklist
Revised: Checklist

Original: For all authors...
Revised: For all authors...

Original: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?
Revised: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you describe the limitations of your work?
Revised: (b) Did you describe the limitations of your work?

Original: [Yes]
Revised: [Yes]

Original: (c) Did you discuss any potential negative societal impacts of your work?
Revised: (c) Did you discuss any potential negative societal impacts of your work?

Original: [N/A](d)
Revised: [N/A]

Original:  Have you read the ethics review guidelines and ensured that your paper conforms tothem?
Revised: (d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?

Original: [Yes]2.
Revised: [Yes]2.

Original: If you are including theoretical results...
Revised: If you are including theoretical results...

Original: (a) Did you state the full set of assumptions of all theoretical results? 
Revised: (a) Did you state the full set of assumptions of all theoretical results? [N/A] (b) Did you include complete proofs of all theoretical results?

Original: [N/A]
Revised: [N/A]

Original: (b) Did you include complete proofs of all theoretical results?
Revised: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main exper- imental results (either in the supplemental material or as a URL)? [Yes] The paper

Original: [N/A]
Revised: and supplemental provide full details on how to reproduce the simple approach. The datasets used are all public. (b) Did you specify all the training details (e.g., data splits, hyperparameters,

Revised: how they

Revised: were chosen)?

Revised: [Yes] In appendix.

Revised: (c) Did you report error bars (e.g., with respect to the random seed after running experi-

Revised: ments multiple times)?

Revised: [N/A] Our experiments are on large datasets and thus did not

Revised: require multiple runs.

Revised: (d) Did you include the total amount of compute and the type of resources used (e.g., type

Revised: of GPUs, internal cluster, or cloud provider)?

Revised: [No] The experiments were run on a

Revised: proprietary infrastructure on which it is hard to estimate the total amount of compute.

Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...

Revised: (a) If your work uses existing assets, did you cite the creators?

Revised: [Yes]

Revised: (b) Did you mention the license of the assets?

Revised: [Yes] See Thorne et al.

Original: If you ran experiments...
Revised: reference. (c) Did you include any new assets either in the supplemental material or as a URL? [No] (d) Did you discuss whether and how consent was obtained from people whose data you’re

Original: (a) Did you include the code, data, and instructions needed to reproduce the main exper-
Revised: using/curating? [N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable

Original: imental results (either in the supplemental material or as a URL)? [Yes] The paper
Revised: information or offensive content? [N/A] If you used crowdsourcing or conducted research with human subjects...

Original: and supplemental provide full details on how to reproduce the simple approach.
Revised: (a) Did you include the full text of instructions given to participants and screenshots, if applicable?

Original: The
Revised: [N/A]

Original: datasets used are all public. (b) Did you specify all the training details (e
Revised: (b) Did you describe any potential participant risks, with links to Institutional Review

Original: g., data splits, hyperparameters, how theywere chosen)?
Revised: Board (IRB) approvals, if applicable?

Original: [Yes] In appendix.(c) Did you report error bars (e
Revised: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount

Original: g., with respect to the random seed after running experiments multiple times)?
Revised: spent on participant compensation?

Original: [N/A] Our experiments are on large datasets and thus did not require multiple runs.
Revised: [N/A] A Dataset / Environment DetailsA.1 Wikipedia data processing We start by downloading English Wikipedia snapshot (the pages-articles.xml.bz2 file) from Wikimedia 2 or Internet Archive.

Original: (d) Did you include the total amount of compute and the type of resources used (e
Revised: 3 We extract text from English Wikipedia for a given snapshot using

Original: g., typeof GPUs, internal cluster, or cloud provider)?
Revised: Gensim’s WikiCorpus class. For each page, this tool extracts the plain text and hyperlinks, and stripsout all structured data such as lists and figures. To avoid pages without sufficient textual content, wefilter out categories, listical, disambiguation pages and any other page with less than 200 characters. Each article ( µ = 1000 words) is split into paragraph-sized blocks ( µ = 100 words), which formthe nodes in our Wikipedia Graph.

Revised: This was done both for conceptual reasons (humans don’t readan entire article at once but look at bits and pieces of it at one go) and for modeling reasons (mostlanguage models have an upper input token length that is smaller than the average Wikipedia page).

Revised: The edges in the graph are formed in three ways:

Revised: 1. for nodes in the same article, we add “previous node” and “next node” links to connect them

Revised: in a chain;

Revised: organic hyperlinks – the internal links connecting Wikipedia pages to each other; and

Revised: we additionally run entity linking 5 because repeat mentions of an entity in Wikipedia lackhyperlinks, which is problematic when articles are split up.

Revised: This constitutes our Wikipedia graph construction, where the text blocks act as vertices and theorganic hyperlinks along with entity links are the edges.

Revised: We store the graph as an adjacency listwith metadata, using a memory mapped key-value data-structure to enable fast random access duringnavigation.

Revised: We run the above graph generation pipeline for two different snapshots of Wikipedia:

Revised: June 01 2017 6 : This corresponds to FEVER fact verification dataset.2.

Revised: December 2018 7 : This corresponds to Natural Questions open dataset.

Revised: A.2 Smaller 200k node graph construction

Revised: We construct smaller train and eval graphs with 200k nodes each for the following reasons:

Revised: Checking

Revised: strict generalization by making train and eval graphs to be totally disjoint, i.e. nocommon nodes or edges.

Revised: The full 2017 and 2018 are sufficiently different but not entirelydisjoint.2.

Original: [No] The experimentswere run on aproprietary infrastructure on which it is hard to estimate the total amount of compute.
Revised: Running experiments on full graphs are expensive, so for more thorough evaluation acrossmultiple baselines we constructed the smaller graphs. To construct these two smaller graphs, we start with the full 2018 graph. We sort the nodes by theirin-degrees. We mark nodes with odd ranks to be in train and even ranks to be in eval, which ensurescomplete separation between the two.

Revised: For the train graph we start with rank 1 node (and for eval withrank 2 node) and then select all its neighbors which have odd rank (even rank).

Revised: Then all the odd(even) nodes connected to the selected ones are picked.

Revised: This process is continued until the desirednumber of nodes are selected.

Revised: A.3 Dataset statistics

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
Revised: We construct two versions of the full Wikipedia graph following the method outlined in Sec. A.1. The main summary statistics (number of articles, nodes, edges, words per node, median path length)of the resulting graphs are listed in Table 3.

Revised: A more detailed picture of the graph can be obtained by looking at its degree distribution, whichis presented in Figure 4.

Original: (a) If your work uses existing assets, did you cite the creators?
Revised: It shows the graph has tell-tale sign of web-graph: it has a power-lawdistribution with a few hub nodes. [Figure] We try to estimate how far are two nodes in the graph. Calculating the entire distribution of shortestpath length is prohibitively expensive for such large graph.

Revised: We follow Ye et al.

Revised: [2010] method toget an approximation.

Revised: We start by selecting 250k nodes having highest in-degree, which is a goodapproximation of the giant component in a web graph as required by Ye et al.

Revised: We thenperform a single source to all shortest paths on each of these 250k nodes using Dijkstra’s algorithmfor sparse graphs.

Revised: This resulted in computing approximately 1 trillion shortest paths based on whichthe estimated distribution of shortest path length is shown in Figure 5.

Original: [Yes]
Revised: [Figure]

Revised: Finally, we would like to point out that there is significantevolution (difference) between the 2017 and 2018 graph.

Revised: Some important statistics reflecting the changes across thetwo graphs are:

Revised: • New articles: 404,773 (7.5%)

Original: (b) Did you mention the license of the assets?
Revised: • New nodes: 3,871,578 (10%) • Deleted articles: 58,476 (1.1%) [Figure] • Deleted nodes: 874,372 (2.3%)

Revised: Further in Figure 6, we show histogram of changes inedges among the common nodes between the two graphs.

Revised: Apart from addition/deletion of organic hyperlinks, justmodifications of text would yield to differences as the chunking will be different.

Revised: This shows significant generalization is needed to successfully navigate across the two graphs, simply memorizationwill not work.

Revised: [Figure]

Revised: A.4 FEVER data processing

Original: [Yes] See Thorne et al.
Revised: [Figure] FEVER Thorne et al.

Revised: [2018] is a dataset containing 185 , 445 claims labeled as S UPPORTED , R EFUTED , or N OT

Revised: E NOUGH I NFO .

Revised: Verifiable claims (i.e. S UPPORTED or R EFUTED ) are annotated withevidence sentences supporting this classification, which are drawn from a preprocessed version of the

Original: reference.
Revised: June 2017 Wikipedia snapshot. Notably, only the article introductions are retained, and claims aregenerated from a curated set of approximately 50 , 000 popular pages. To make an aligned navigation graph for FEVER, we start with the June 01 2017 snapshot of Wikipedia and build the graph using the procedure described in the previous section.

Revised: Because ofdifferences in text preprocessing, FEVER’s version of Wikipedia does not precisely align to ourown.

Revised: We then try to match FEVER’s Wikipedia articles to our graph, relying on Wikipedia URLredirections for handling disambiguations.

Revised: This resulted in 99 .

Revised: 5% matching of articles between ournavigation graph and the FEVER version of Wikipedia.

Revised: Second, we align the evidence sentences in

Revised: FEVER to sentences from our text blocks using a fuzzy string match.

Revised: Specifically, we use the tokenset ratio to score the similarity between an evidence sentence and the text in a graph node.

Revised: If the scorebetween a sentence and a node is > = 80 , the node is added as an evidence node for the given claim.

Revised: This threshold is sufficiently high to minimize the chances of a false positive, while also matching ahigh percentage of evidence sentences.

Revised: Some sentences are difficult to match, particularly those thatare split between two text blocks.

Original: (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: Ultimately, 93 . 1% of all evidence sentences were matched to anode. This gives us an augmented FEVER dataset, where for each claim we have a corresponding setof evidence nodes in the graph. These nodes are then used as navigation targets for fine-tuning duringthe training phase.

Revised: We generate trajectories for BC by running BM25 over all nodes in the graph,taking the top 10 matches as starting nodes, and finding shortest paths to evidence nodes.

Revised: We compute the shortest-path distance between top-1 retrieval of BM25 and target node in this graphin Figure 7.

Revised: As can be seen that most path lengths are relatively short, which implies that BM25lands us in the right vicinity and by a small amount of navigation around we will be able to find theright evidence passage.

Original: [No]
Revised: [Figure]

Revised: A.5 RL Baseline

Revised: For the purposes of the RL baseline, we can view graph navigation as a goal-conditioned MDP.

Revised: 8 Ourobservation is the same as the inputs to the RFBC models.

Revised: The reward function R for current node nand goal node n g we can write simply as:

Revised: R = (cid:26) 1 n = n g 0 otherwise (5)

Revised: See Appendix B.2 for model and training details of RL baselines.

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’re
Revised: B.1 Embedding pretraining This section details our embedding pretraining procedure, as introduced in Section 3. Model details

Revised: The transformer is fine-tuned from a pre-trained RoBERTa

Revised: [Liu et al., 2019] model.

Revised: As in Equation 1 our model is trained to optimize

Original: using/curating?
Revised: L ( θ ) = E E ( n [Equation] ...,n T ) (cid:34) T − 1 (cid:88) t = log p θ ( n t | n t , n g

Revised: [Equation]

Revised: where we set n g = n T .

Revised: We parameterize this distribution as

Original: [N/A]
Revised: [Equation]

Revised: where s ϕ ( · ) and a ϕ ( · , · ) are functions which extract embeddings for an entire node and a node-actioncombination, respectively and f θ ( · , · , · ) is a function which combines these embeddings to produceaction probabilities.

Revised: At their core, s ϕ and a ϕ are based on the same transformer model.

Revised: We tokenize the text at node n to produce L tokens and pass these tokens through the transformer to produce Lembeddings.

Revised: The state representation s ϕ ( n ) is produced by taking the mean of these embeddings and passingthis vector through a tanh nonlinearity.

Revised: The node-action embedding a ϕ ( n, n ′ ) embeds the actionof moving from node n to its neighbor n ′ .

Revised: Let us assume the text at node n is “Barack Obama wasthe President of the United States during...” where the substrings “Barack Obama” and “Presidentof the United States” correspond to hyperlinks to other neighboring articles n ′ and n ′′ .

Revised: Then weconstruct the node-action embedding a ϕ ( n, n ′ ) by passing the tokenized text of node n throughour transformer, again producing L transformed token embeddings.

Revised: Then we take the transformedtokens which correspond to the text “Barack Obama” and take their mean and pass this through thetanh nonlinearity.

Revised: If we instead wanted to produce a ϕ ( n, n ′′ ) we would take the transformed tokenswhich correspond to the text “President of the United States” instead, take the mean and apply thenonlinearity.

Revised: This is visualized in Figure 8.

Revised: The function f θ which combines these embeddings first concatenates them and passes the combinedinput through a simple neural network with a single hidden layer and 1 output neuron.

Revised: Training details We train these models to optimize Equation 6.

Revised: The objective is optimized withthe Adam [Kingma and Ba, 2014] optimizer using an initial learning rate of 1 e − 5 with a one-cyclecosine decay schedule and linear warm up of 10 K steps.

Original: (e) Did you discuss whether the data you are using/curating contains personally identifiable
Revised: The training batch size is 512 and the modelis trained over 3 M iterations. Node text is clipped (or padded) to 200 tokens. The weights of thetransformer are initialized from a pretrained RoBERTa [Liu et al., 2019] masked language model.

Revised: B.2 Small graph experiments

Revised: Model details In the small graph experiments, we use the simple policy network FF as describedin Section 4.2.

Revised: As discussed, we use a 1-layer MLP (meaning a single learned linear layer with noactivation function).

Revised: We use a hidden size of 768 to match the size of ϕ .

Original: information or offensive content?
Revised: Because the final logits arethe result of an inner product between each state and action embedding of size 768 , to aid networktraining, we add a normalization layer (jax.nn.normalize) before the inner product. For the action embedding a i = ϕ ( s i ) we also concatenate a 1-hot vector that represents the actiontype (e.g. whether it was a link action, or a next action, or a prev action). We also concatenate anotherbit that indicates whether the state s i of the particular action has been visited before.

Revised: For the purposes of the RL baseline, we can view graph navigation as a goal-conditioned MDP.

Revised: 9 Ourobservation is the same as the inputs to the RFBC models.

Revised: The reward function R for current node nand goal node n g we can write simply as:

Original: [N/A]
Revised: [Equation]

Revised: For the RL baseline, we used IMPALA [Espeholt et al., 2018], which is a kind of policy gradientapproach, and thus has a policy network π and a “baseline” or value function network.

Revised: Since thepolicy network has the same output space as our RFBC models, we use an identical architecture forthis.

Revised: For the value network, we use a very similar architecture.

Revised: We again use ϕ as our encoding ofthe current node and the target node, again concatenate them together.

Revised: We then feed this through a1-layer MLP to compute the value.

Revised: Training details The procedure for generating the trajectories we use for the BC loss is described in

Revised: Section 3.1.

Revised: Except for our experiment in Section 5.1, we always use the random forward trajectories.

Revised: Following Lin et al.

Revised: [2018], in the navigation experiments, we randomly drop out edges in the traininggraph with probability 0 .

Revised: 5 to reduce over-fitting.

Revised: As stated in Section 4.1, we set max steps to

Original: If you used crowdsourcing or conducted research with human subjects...
Revised: B = 100 . For RFBC training on the small graph, we use RMSProp with a learning rate of 0 . 01 , adecay of 0 . 9 an epsilon of 1 e − 10 .

Revised: For the sentence search tasks, our models train a separate ϕ for the target embedding.

Revised: As we stated,we use MiniBERT for this embedding ϕ target and train it with the same optimization settings, exceptthat we reduce the learning rate for these weights to 1 e − 4 .

Revised: We use a batch size of 512 and we trainfor 50 , 000 network update steps.

Revised: For RL training, we use VTRACE [Espeholt et al., 2018] loss, using the reward above.

Revised: We againuse RMSProp with a learning rate of 0 .

Original: (a) Did you include the full text of instructions given to participants and screenshots, if
Revised: 01 , a decay of 0 . 9 an epsilon of 1 e − 10 . We set the baselinecost to 0 . 5 , trajectory length (the number of timesteps the RL loss backprops through) to 100 , batchsize again to 512 and max update steps to 50 , 000 .

Revised: We did a parameter sweep for entropy costs of0 .

Revised: 1 , 0 . 01 , 0 . 001 and gamma of 0 . 8 , 0 . 9 .

Revised: To give RL the best chance possible, we choose the maximumover this sweep for each experiment (but still none of these match RFBC performance).

Revised: B.3 Full Wikipedia navigation experiments

Revised: On the full Wikipedia training experiments in Section 5.3, we evaluate on navigation and sentence search using either our feed-forward or transformer model and using either RoBERTA fixedembeddings, or our trained embeddings described in Appendix B.1.

Revised: The feed-forward model is identical to the one described in Appendix B.2.

Original: applicable?
Revised: The transformer model isthe standard transformer model from Vaswani et al. [2017] with 4 layers, an attention size of 64 , 12heads, and mlp hidden size of 3072 and dropout rate of 0 . DistillBERT Sanh et al. [2019] is used forϕ ( n g ) for the sentence search experiments.

Revised: The learning parameters and all other relevant training parameters for all models are identical tothose in Appendix B.2 except that the learning rate for the transformer model is lower ( 1 e − 4 ) as theearlier learning rate was too high for transformer models.

Revised: B.4 Downstream experiments

Revised: Training details For the FEVER benchmark, our model uses learned embeddings for ϕ with asingle feed-forward layer for the policy network.

Revised: The model is pretrained on the 5 -step sentencesearch task, and finetuned on trajectories generated from the augmented FEVER dataset described in

Revised: A.4 using the the following loss function:

Original: [N/A]
Revised: [Equation]

Original: (b) Did you describe any potential participant risks, with links to Institutional Review
Revised: where L BC is the normal BC loss we use in our other experiments Only the ϕ target weights are finetuned for this task. We use AdamW with a constant learning rate of1e-6, b1=0.9, b2=0.999 , and eps=1e-6 . The model is finetuned for 100 , 000 update steps witha batch size of 512 .

Revised: No edge dropout is used during finetuning.

Revised: Evaluation details For each claim in the FEVER development dataset, we first run BM25 overall nodes in the graph and take the top 5 matches as starting nodes.

Revised: From each node, the finetunedmodel then navigates for 20 steps.

Revised: We collect all sentences in all nodes visited by the agent overthe 100 total navigation steps, and match them to FEVER evidence sentences using the WRatiofunction in the fuzzywuzzy package.

Revised: 10 Two sentence ranking methods are explored: Gensim’s

Revised: TF-IDF implementation, 11 and the BigBird re-ranker of Stammbach [2021], using their open sourcemodel and weights.

Revised: Finally, the top-5 evidence sentences are then submitted to the official FEVERscorer 13 to compute the accuracy, recall, and F1 scores.

Revised: C Further experiments

Revised: C.1 RFBC + RL

Revised: We additionally try finetuning an RL policy starting from our RFBC-trained navigation policies.

Revised: Weuse the same network and learning settings as we did in the other RL experiments.

Revised: We train for anadditional 10,000 network updates (512M environment steps).

Revised: We add the final numbers in Table 2and show the training curve in Figure 9.

Revised: We can see from the training curve that RL training accuracyfluctuates around the point where RFBC training left off.

Revised: From the results in Table 2 we see littlestatistically significant difference (about 1 point in either direction).

Revised: C.2 Question Answer Downstream Task

Revised: Finally, we present results on another downstream task where navigation helps.

Revised: In particular, weconsider the task of finding correct evidence passage for open domain question answering.

Revised: Weuse the Natural Question (NQ) open-domain dataset presented in Lee et al.

Revised: [2019] which has beenaligned to Wikipedia dump of December 20, 2018 by [Karpukhin et al., 2020].

Revised: Most prior works inliterature utilize the processed collection of 21M passages provided by by [Karpukhin et al.,

Revised: 2020],but unfortunately we cannot use it as it has no hyperlink information which is crucial for our graphbuilding.

Revised: So we re-align the questions in NQ to our 2018 graph with 38M passages (i.e. nodes).

Revised: Wecompute the shortest-path distance between top-1 retrieval of BM25 and target node in this graph in

Revised: Figure 10.

Revised: As can be seen that most path lengths are relatively short, which implies that BM25 landsus in the right vicinity and by a small amount of navigation around we will be able to find the rightevidence passage.

Revised: Since we target navigating to the exact evidence passage required to answer the question, we userecall@{1,2,3,4,5} for finding the gold evidence passage as our metric.

Revised: More commonly the metricmarks a retrieved passage to be correct if it contains the answer string, but this causes a lot of falsepositives, e.g. the answer string appears in a totally irrelevant context.

Revised: Our method: For each question, we first run BM25 over all nodes in the graph and take the top-5matches as starting nodes.

Revised: From each starting node, the agent then navigates for 20 steps.

Revised: All the 100nodes visited by the agent is then ranked by a simple cross-attention model.

Original: Board (IRB) approvals, if applicable?
Revised: For the cross-attentionmodel, we follow Hofstätter et al. [2020] to train 6-layer BigBird [Zaheer et al., 2020]. [Table] The results are tabulated in Table 6.

Revised: Paired with a basic retrieval (BM25) and re-ranker (BigBird), theapproach obtains a respectable Recall@1 of 31.6.

Revised: If we remove our navigation component then thisdrops substantially to 22.2, demonstrating its utility.

Revised: As a reference, we also ran a state-of-the-artsystem RocketQA

Revised: [Qu et al., 2021] on our setup and evaluated it using our harder metric.

Revised: It isworthwhile to note that our approach (a) is significantly simpler than state-of-the-art approacheslike RocketQA which employ 4 stages of dual-encoder and cross-attention models, and (b) selectsevidence over a larger set than the commonly used preprocess Wikipedia passages (38M vs 21M).

Original: [N/A]
Revised: [Figure]

Revised: D Variational Interpretation of our Approach

Revised: We present an alternative motivation of our method based on variational inference in a latent variablemodel of states.

Original: (c) Did you include the estimated hourly wage paid to participants and the total amount
Revised: Assume we are given some goal node n g . We would like to parameterize a modelp θ ( n T | n g ) which will generate a trajectory of states n 0 , n 1 , . . . , n T such that n T = n g . We defineour model as a latent variable model

Revised: [Equation]

Revised: i.e an autoregressive model which samples an initial state from n 0 ∼ p ( n 0 ) and then samplessubsequent states n t ∼ p θ ( n t | n t − 1 , n g ) .

Revised: The transition distribution samples the next node n t fromthe neighbors of n t − 1 in our graph.

Revised: We would like the probability that n T = n g to be large, thus wewill train our model to maximize log p θ ( n T = n g | n g ) .

Revised: For latent-variable models such as this, we can rewrite the marginal likelihood as

Revised: [Equation]

Revised: and can obtain a lower-bound on this quantity by replacing the intractable posteriorp θ ( n 0 , . .

Revised: . , n T − 1 | n T , n g ) with a variational approximation q ( n 0 , . .

Revised: . , n T − 1 | n T , n g ) , i.e

Revised: L θ ( n T , n g ; q ) :

Revised: = log p θ ( n T | n g ) ≥ E q ( n

Revised: [Equation]

Revised: n T

Revised: [Equation]

Revised: T

Original: spent on participant compensation?
Revised: [Equation] g ) (cid:34) p [Equation] [Equation]

Revised: (cid:35)

Revised: [Equation]

Revised: The above bound becomes tight when q ( n 0 , . .

Revised: . , n T − 1 | n T , n g ) =

Revised: p θ ( n 0 , . . .

Revised: , n T − 1 | n T , n g ) .

Revised: Thus,we can optimize our model parameters θ to maximize L θ ( n T , n g ; q ) .

Revised: This approach has had a longhistory of successfully training latent-variable generative models [Kingma and Welling, 2013, Hoet al., 2020].

Revised: In the context of this work, we can view q ( n 0 , . . .

Revised: , n T − 1 | n T , n g ) as a distribution over trajectoriesof nodes which end at our goal node.

Revised: We can interpret the various trajectory generation methodsintroduced in Section 3.1 as different variational approximations q .

Revised: Ideally, the option which mostclosely approximates the true posterior p θ ( n 0 , . .

Revised: . , n T − 1 | n T , n g ) would be the most desirable but ingeneral this distribution is intractable.

Revised: We find that using simple random trajectories provides a good-enough approximation to the posteriorto enable us to train a model which reliably finds the goal state.

Revised: This result is not completely surprisingin context of prior work [Dai et al., 2020] which successfully trains latent-variable models usingrandom trajectories as an inference model.

Revised: E Efficiency Analysis

Revised: In this section we analyze the asmpytotic runtime of our method and baselines.

Revised: In Table 7 we showthe asymptotic runtime and whether or not it is scalable for each method.

Revised: We also show the 5-stepaccuracy from Table 2 for quick reference.

Revised: T is the trajectory length (either the length of the trajectory for BC methods or the maximum allowedtrajectory for other methods.

Revised: For BC, T is guaranteed to be shorter than the max, but this is a constantfactor difference.

Revised: E out is the average number of outgoing edges (i.e. actions) at a node.

Revised: (small - 10 1 )

Revised: E is the total edges in the graph (very large - 10 8 ).

Revised: V is the number of nodes in the graph (large10 7 ).

Original: [N/A]
Revised: [Table]

Revised: For most methods, the runtime is O ( E out T ) because the method does some constant amount of workfor each possible actions E o ut for O ( T ) steps.

Revised: For the random methods it is O ( T ) because the workis constant per step (choose randomly).

Revised: Shortest path distance has the worst runtime.

Revised: The O ( ElogV )term is the runtime of finding a shortest path (Dijkstra’s algorithm with Fibonacci heap).

Revised: As the graphgets larger, this term because infeasible (we could not even run this on the full Wikipedia graph).

Revised: DFSmight seem like it should be larger than O ( E out T ) or O ( T ) , but because we restrict the maximumallowable steps, it is only proportional to T and (for greedy) E out .

Revised: From this Table, we see that all methods (except shortest path) have scalable runtimes.

Revised: The dominantfactor is E out , which theoretically grows slowly with graph size, but in practice, there is a practicalmaximum number of links you have in a paragraph of text, and the runtime difference in ourexperiments going from 200k nodes to 38M nodes is only 1.5x despite the 200-fold increase in graphsize.