Paper:
End-to-End Goal-Driven Web Navigation
We propose a goal-driven web navigation as a benchmark task for evaluating an agent with abilities to understand natural language and plan on partially observed environments. In this challenging task, an agent navigates through a website, which is represented as a graph consisting of web pages as nodes and hyperlinks as directed edges, to find a web page in which a query appears. The agent is required to have sophisticated high-level reasoning based on natural languages and efficient sequential decision-making capability to succeed. We release a software tool, called WebNav, that automatically transforms a website into this goal-driven web navigation task, and as an example, we make WikiNav, a dataset constructed from the English Wikipedia. We extensively evaluate different variants of neural net based artificial agents on WikiNav and observe that the proposed goal-driven web navigation well reflects the advances in models, making it a suitable benchmark for evaluating future progress. Furthermore, we extend the WikiNav with questionanswer pairs from Jeopardy! and test the proposed agent based on recurrent neural networks against strong inverted index based search engines. The artificial agents trained on WikiNav outperforms the engined based approaches, demonstrating the capability of the proposed goal-driven navigation as a good proxy for measuring the progress in real-world tasks such as focused crawling and question-answering.

Review:
This paper introduces a task used as a benchmark in evaluating language understanding agents. The task takes a website and converts it into a graph of individual pages as nodes with hyperlinks represented by edges. Final performance is measured by starting from a page node and reaching a desired target node as represented by a sentence contained within the page for that node. The authors use Wikipedia to instantiate a dataset for this task and evaluate several existing neural net approaches. they also extend this test to include an evaluation of the existing Jeopardy question answer benchmark.
Overall this is an interesting paper but its contribution is very broad rather than having a specific  deeper contribution. the primary contribution seems to be framing Benchmark tests in this web navigation framework however their needs to be more discussion of related work and related benchmark test for comparison when introducing a new benchmark. If this is intended to be the main contribution of the paper the authors could save space by moving discussion of the specific neural network approaches to an appendix. also when introducing this new benchmark it would help to have more in-depth analysis of the dataset and we're evaluating approaches succeed and fail by way of examples in the paper. it's nice that the tool to create this type of data set is being released but I'm skeptical that this framework will create meaningful data sets for anything outside of Wikipedia. again given Wikipedia is the focus more comparison on other evaluations using Wikipedia data are needed. In particular there is some recent work on question answering from Wikipedia tables that seems relevant from the deep learning community. the writing is fairly clear overall but some details are omitted due to packing in a wide variety of information into what could easily be a journal length paper.   The wording of constraints 3 and 4 is a bit confusing. It seems like 3 is imposing a maximum number of allowed hops but please clarify.   It seems a bit strange to use the number of sentences in a query as the way of defining its difficulty. Sentence lengths can vary widely especially when selected for highly representative sentences of pages. it would be nice to see metrics of query difficulty related to number of words.  While it isn't the main focus of this paper a feed forward network with single hidden layer seems obviously too simple for this task.  The task as framed isn't a very natural way to explore a resource like Wikipedia so it's not surprising that humans perform poorly. I'm not sure that comparing to human performance is even meaningful  other than to say humans aren't good at exploring Wikipedia in this way.  It would be nice to see more analysis on the Jeopardy test to understand what about the queries makes the proposed approach outperform standard search engines.  intuitively it seems that Jeopardy questions are intentionally difficult when  posed purely as search queries.