DOM-Q-NET: G ROUNDED RL ON S TRUCTURED L ANGUAGE
Anonymous authors
Paper under double-blind review
A BSTRACT
The ability for agents to interact with the web would allow for signiﬁcant improvements in knowledge understanding and representation learning.
However, web navigation tasks are difﬁcult for current deep reinforcement learning (RL) models due to large discrete action space and varying number of actions between the states.
In this work, we introduce DOM-Q-NET, a novel architecture for RLbased web navigation to address both of these problems.
It parametrizes Q functions with separate networks for different action categories, clicking DOM and typing a string input. DOM-Q-NET utilizes a graph neural network to represent the tree-structured HTML of a standard web page.
We demonstrate the capabilities of our model on the WorldOfBits(MiniWoB) environment where we can match or outperform existing work without the use of expert demonstrations. Furthermore, we show 2x improvements in sample efﬁciency when training in the multi-task setting, allowing our model to transfer learned behaviours across tasks.
I NTRODUCTION
Over the past years, deep reinforcement learning (RL) has shown huge success in solving tasks such as playing arcade games (Mnih et al., 2015) and manipulating robotic arms (Levine et al., 2016).
Recent advances in neural networks allow RL agents to learn the control policies from raw pixels without feature engineering by human experts.
However, most of the deep RL methods focus on solving problems in either simulated physics environments where the inputs to the agents are joint angels and velocities, or simulated video games where the inputs are rendered graphics.
Agents trained in such simulated environments have little knowledge about the rich semantics of the world.
The World Wide Web (WWW) is a rich repository of knowledge about the real world.
To navigate in this complex web environment, an agent needs to learn about the semantic meaning of text, images and the relationships between them.
Each action of navigation corresponds to interacting with DOMs from tree structured HTML.
Tasks like ﬁnding a friend on a social network, clicking into a web page, and rating a place on Google maps can be framed as accessing the DOM and potentially modifying its value with user input.
Unlike Atari games, the difﬁculty of web tasks comes from their diversity, large action space and sparse reward signals.
A common solution for the agent is to mimic the expert demonstration by imitation learning, and so efﬁciency of using demonstrations has been paid attention in previous works (Shi et al., 2017;
Liu et al., 2018).
However, the diverse set of tasks also include simple ones that agent can learn and generalize some actions to harder tasks.
This capability is essential because the action space at each time step varies when visiting different web pages.
Two previous works (Shi et al., 2017; Liu et al., 2018) have evaluated RL-based web navigation by attempting to solve MiniWoB (Shi et al., 2017) benchmark tasks.
Liu et al.
(2018) achieved state of the art performance with very few expert demonstrations, but their exploration policy requires constrained action sets, hand-crafted with expert knowledge in HTML.
In this work, our contribution is to propose a novel architecture, DOM-Q-NET, that parametrizes factorized Q functions for web navigation, which can be trained to match or outperform existing work on MiniWoB without using any expert demonstration. Graph neural network (Scarselli et al., 2009; Li et al., 2016) is used as the main backbone to provide three levels of state and action representations, which are selectively fed into three factorized Q networks.
Essentially, it uses neural message passing and readout of the node representations (Gilmer et al., 2017) from ‘local’ representations of the web pages to produce ‘neighbor’ and ‘global’ representations.
Three representational modules are used by factorized Q networks that use three separate multilayer perceptrons(MLP)
(Rumelhart et al., 1985) for parametrizing the factorized Q values of distinct action categories, “click”, “type” and “mode”.
The entire architecture is fully differentiable, and all of its components are jointly trained. Another contribution is that it is the ﬁrst architecture to perform multitask learning of web navigation tasks, and demonstrate the transferability of learned behaviors on the web interface.
We show that the multi-task agent achieves an average of 2x sample efﬁciency comparing to the single task agent.
B ACKGROUND 2. R EPRESENTING WEB PAGES USING DOM S
We are interested in the web navigation setting where the agent can click and type.
The Document Object Model (DOM) is a programming interface for HTML documents and it deﬁnes logical structure of such documents.
DOMs are connected in a tree structure, and we frame web navigation as accessing a DOM and potentially modifying it by user input.
As an element object, each DOM has a “tag” and attributes such as “class”, “is focused”, similar to objects in Object Oriented
Programming.
The values of attributes are used by browsers to render web page for users.
2. R EINFORCEMENT LEARNING
In the traditional reinforcement learning setting, an agent interacts with an inﬁnite-horizon, discounted Markov Decision Process (MDP) to maximize its total discounted future rewards.
An MDP is deﬁned as a tuple ( S , A , T, R, γ ) where S and A are the state space and the action space respectively, T ( s (cid:48) | s, a ) is the transition probability of reaching state s (cid:48) ∈ S by taking action a ∈ A from s ∈ S , R is the immediate reward by the transition, and γ is a discount factor.
Then we deﬁne the
Q -value function for a tuple of actions to be Q π ( s, a ) =
E [ (cid:80) Tt =
γ t r t | s
=
s, a 0 = a ] , where T is the number of timesteps till termination.
The formula represents the expected future discounted reward starting from state s , performing a and follow the policy until termination.
The optimal
Q-value function Q ∗ ( s, a ) = max π Q π ( s, a ) , ∀ s ∈ S , a ∈ A (Sutton & Barto, 1998) satisﬁes the
Bellman optimality equation Q ∗ ( s, a ) =
E s (cid:48)
[ r + γ max a (cid:48) ∈A Q ∗ ( s (cid:48) , a (cid:48) )] .
2. G RAPH NEURAL NETWORKS
For an undirected graph G =
( V, E ) , the Message Passing Neural Network (MPNN) framework (Gilmer et al., 2017) formulates two phases of forward pass to update the node-level feature representations h v , where v ∈ V , and graph-level feature vector ˆ y .
Message passing phase updates hidden states of each node by applying a vertex update function U t over current hidden state and message h t +1 v =
U t ( h tv , m t +1 v ) , where passed message m t +1 v is computed as m t +1 v = (cid:80) ω ∈ N ( v ) M
( h tv , h tw , e vw ) .
N(v) denotes the neighbors of v in G, e vw is an edge feature.
This process runs for T timesteps.
The readout phase uses readout function R, and computes graph level feature vector ˆ y =
R ( h Tv | v ∈ G ) .
Neural networks are used to parametrize each of the functions M t , U t and R t .
2. R EINFORCEMENT L EARNING WITH G RAPH N EURAL N ETWORKS
There have been works in robot locomotion that uses graph neural networks (GNNs) to model the physical body (Wang et al., 2018; Hamrick et al., 2018).
NerveNet demonstrates that policies learned with GNN transfers better to other learning tasks than policies learned with MLP (Wang et al., 2018).
It uses GNN to parametrize the entire policy whereas DOM-Q-NET uses a GNN to provide representational modules for factorized Q functions.
Note that the graph structure of a robot is static whereas the graph structure of a web page can change at each time step.
Locomotion control tasks provide dense reward whereas web navigation tasks are sparse reward problems with only 0/1 reward at the end of the episode.
For web navigation, our model also needs to account for the dependency of actions on goal instructions.
2. S EMANTIC P ARSING
The problem of mapping natural language to an executable logical form, or formal language, has been studied extensively since a decade ago.
(Branavan et al., 2009; Mooney, 2008).
One related line of work is to perform lookups on Wikipedia tables, and answer compositional questions on
W IKI T ABLE Q UESTIONS dataset (Pasupat & Liang, 2015).
Along with the question, a generated knowledge graph from the table is provided as an input.
To use semantic parsing, some interesting works like Krishnamurthy et al.
(2017) and Liang et al.
(2018) apply encoder-decoder architectures to output the logical form that can be executed.
Krishnamurthy et al.
(2017) proposed Entity Linking that improved its performance on this dataset.
It computes the weighted average of the embeddings of entities in the knowledge graph, and the weights are computed by the similarity score between a question token and each entity.
Liang et al.
(2018) uses policy optimization of mapping to the executable formal language to further improve the performance of applying semantic parsing to this dataset.
In those cases, the use of semantic parsing assumes there is an executor performing the sequence of actions based on the formal language.
However, it is not trivial to design a comprehensive formal language and its executor for web navigation.
The goal of applying RL for web navigation is to train an agent end-to-end to control clicking and typing in the standard browser without such restrictions.
2. P REVIOUS W ORKS ON RL ON WEB INTERFACES
Shi et al.
(2017) constructed benchmark tasks, Mini World of Bits, that consist of many toy tasks of web navigation .
This environment provides both the image and HTML of a web page.
Shi et al. (2017) showed a benchmark result that the agent with visual input does not solve most of the problems, even given the demonstrations.
Then Liu et al. (2018) proposed DOM-NET architecture that uses a series of attention between DOM elements and the goal, and workﬂow guided-exploration, which uses predeﬁned formal language to constrain the action space of an agent at each state.
They achieved new state of the art performance and sample efﬁciency of using demonstrations with the constrained action sets.
Unlike these previous works, we aim to tackle web navigation tasks without any expert demonstration or prior knowledge.
N EURAL DOM Q N ETWORK
tag: "span" text: "wZQMj24" tag: "span" text: "GvCqw" tag: "span" text: "sr" tag:"Input_checkbox"text: "" tag:"Input_checkbox"text: "" tag:"
[Figure] _checkbox"text: "" DOM i tag:"Button"
[Figure]
: "Submit" tag
[Figure]
div text: ""
[Figure]
click
Submit [1,0,0,0,0] [0,1,0,0,0] [0,0,1,0,0] [0,0,0,1,0] [0,0,0,0,1]
[Figure]
Consider the problem of navigating through multiple web pages or menus to locate a piece of information.
Let V be the set of DOMs in the current web page.
There are often multiple goals that can be achieved in the same web environment.
We consider goals that are presented to the agent in the form of a natural language sentence, e.g. “Select sr and click Submit” shown in ﬁgure 1 and “Use the textbox to enter Kanesha and press Search, then ﬁnd and click the 9th search result” in ﬁgure 2.
Let G represent the set of word tokens in the given goal sentence.
The RL agent will only receive a reward if it successfully accomplishes the goal, i.e. it is a sparse reward problem.
The primary means of navigation are through interaction with the buttons and the text ﬁelds on the web pages.
There are two major challenges in representing the state-action value function for web navigation: learning from the enormous action space; and the number of actions can vary drastically between the states.
We propose DOM-Q-NET to address both of the problems in the following.
3. A CTION SPACE FOR WEB NAVIGATION
Unlike typical RL tasks that require choosing only one action a from an action space, A , such as choosing one from all combinations of controller’s joint movements for Atari game (Mnih et al., 2015), we frame acting on the web with three distinct categories of actions:
• DOM selection a dom chooses a single DOM in the current web page, a dom ∈ V .
The DOM selection covers the typical interactive actions such as clicking buttons or check-boxes as well as choosing which text box to ﬁll in the string input.
• Word token selection a token ∈ G picks a work token from the given goal sentence to ﬁll in the selected text box.
The assumption that typed string comes from goal sentence aligns with previous work Liu et al.
In the domain of semantic parsing, similar restrictions also exist for parsing questions in WikiTableQuestions (Liang et al., 2018).
• Click or type mode a mode ∈ { click , type } action tells the environment whether the agent’s intention is to “click” or “type” when interacting with the web page.
We can represent a mode as a binary action.
At each time step, the environment receives a tuple of actions, namely a = ( a dom , a token , a mode ) though it will only use a token when a
[Figure] 3. F ACTORIZED Q FUNCTION
One way to represent the state-action value function is to consider all the permutations of a dom and a token .
For example, Mnih et al. (2015) used permutations to ﬂatten out the combinations of joystick direction and clicking for atari games.
However, for MiniWoB environment, this introduces an enormous action space in | V |×|G| number of states.
The number of DOMs | V | and |G| can reach up to 60 and 18 for most medium tasks and the total number of actions can be over 1 , 000 for some hard tasks.
To reduce the number of actions, we consider a factorized state-action value function where the action value of a dom and a token are independent to each other.
Formally, we deﬁne the optimal
Q -value function as the sum of the individual value functions of the three action categories:
[Equation]
Under the independence assumption, we can ﬁnd the optimal policy by selecting the greedy actions w.r.t.
each Q -value function individually.
Therefore, the computation cost for the optimal action of the factorized Q function is linear in the number of DOM elements and the number of word tokens rather than quadratic.
[Equation]
3. L EARNING STATE - ACTION EMBEDDINGS OF WEB PAGES
Many web actions such as clicking different checkboxes and ﬁlling unseen type of forms share similar tag or class attributes.
Our goal is to design a neural network architecture that can effectively capture such invariance for web pages and yet is ﬂexible to deal with the varying number of DOM elements and goal tokens at different time steps.
Furthermore, when locating a piece of information on the web, an agent needs to be aware of both the local information, e.g. the name of button and its surrounding texts, and the global information, e.g. the general theme, of the web page.
The cue for click a particular button from the menu is likely scattered.
To address the above problem, we propose a graph neural network (GNN) based RL agent that computes the factorized Q -value for each DOM present in the current web page, called “DOMQ-NET” shown in Figure 1.
DOM-Q-NET uses additional information of tree structured DOM elements in HTML to guide the learning of state-action representations or embeddings e , which is shared among all its factorized Q outputs.
Explicitly models the HTML structures provide relational information among the DOM elements to the RL agents.
Given a web page, DOM-Q-NET learns a concatenated embedding vector e i = (cid:104) e ilocal , e ineighbor , e global (cid:105) from low level and high level modules correspond to node-level, and graph-level output of the GNN and are jointly shared and trained to represent the DOM v i ∈ V :
Local Module e ilocal is simply the concatenation of each embedded attribute e Attr of the DOM v i , which includes the tag, class, focus, tampered and text information of the DOM element.
In particular, we use a maximum of cosine distance between text and goal tokens to measure the ‘soft’ alignment of the DOM v i with respect to the j th word token embedding e jgoal in the goal.
Liu et al.
(2018) uses a direct alignment to obtain tokens that appear in goal but our method can detect synonyms that are not exactly matched.
[Equation]
This module provides the action representation of clicking each DOM and is generally considered as the skip connection of GNN.
Neighbor Module e ineighbor computes the neighborhood context of the DOM v i using a graph neural network (GNN) with the weights w GNN on the tree structure of the HTML page.
Propagation model is a neural message passing system (Gilmer et al., 2017) between the nodes of the HTML of the web page.
The initialization of the propagation is set to the local embedding.
m t is an intermediate state between message passing.
We apply T number of message passing steps to obtain the ﬁnal neighbor embedding.
We adopt GRU gated version (Li et al., 2016) for state updates between aggregation time steps.
[Equation]
Note that this module contains both action and state representation of the DOM, so the Q -value function can still be approximated with only this module.
Global Module e global is the output model (Gilmer et al., 2017) of the HTML tree structure of the current web page.
It provides the high level state of the web page and is used by all Q -value functions.
We investigate two approaches to obtain global embedding with and without explicit goal information.
We use max-pooling to aggregate all of the DOM embeddings on the web page.
[Equation]
2) Alternatively, we use attention to compute the output model of the GNN where the attention query is the goal vector.
In contrast to Velickovic et al.
(2018), this attention mechanism is used at the readout stage, and is unique to goal-oriented RL settings.
Figure 1 shows each goal token e token is concatenated with one-hot positional encoding vector e pos .
Position-wise feed-forward networks with ReLU activation is applied to each concatenated representation before being maxpooled to obtain the goal vector h goal . Motivated by Vaswani et al.
(2017), we use scaled dot product attention with local embeddings e ilocal as keys, and neighbor embeddings e ineighbor as values. Note that E local and E neighbor are packed representations of ( e 1 local , ..., e Vlocal ) and ( e 1 neighbor , ..., e Vneighbor ) respectively, where E local ∈ R ( V,d k ) ,
E neighbor ∈ R
( V,d k ) and d k is the dimension of text token embedding.
We will call this “goal attention”, and show its simpliﬁed diagram in the appendix 6.2.
[Equation]
Note that the simplest method is to concatenate each node-level feature with goal vector though this will increase the size of the network, and still leads to the same performance.
Further details and comparisons are shown in the appendix 6.3, 6.6. Learning To obtain Q -value function Q idom for the DOM v i , the concatenation of the DOM embedding e i = (cid:104) e ilocal , e ineighbor , e global (cid:105) of all levels are fed into a two layer MLP with weights w dom , Q idom =
MLP ( e i ; w dom ) .
Similarly, the Q -value functions for the word token and the mode are computed using w token and w mode respectively, with a shared e global state module. Here, each e token is fed into a MLP with w token for computing the action value of its word token, see Figure 1.
All the model parameters including the embedding matrices are learned from scratch.
Let θ =
( E, w GNN , w dom , w token , w mode ) be the DOM-Q-NET model parameters including the embedding matrices, the weights of a graph neural network, and weights of the factorized Q -value function.
The model parameters are updated by minimizing the mean squared TD error(Sutton, 1988):
[Equation]
where the transition pairs ( s, a, r, s (cid:48) ) are sampled from the replay buffer and y DQN is the factorized target Q-value with the target network parameters θ − as in the standard DQN algorithm.
[Equation]
E XPERIMENTS
In this section, we ﬁrst evaluate the generalization capability of DOM-Q-NET for large action space by comparing it with previous works (Shi et al., 2017; Liu et al., 2018) that use expert demonstrations.
Tasks with various difﬁculties, as deﬁned in the appendix 6.4, are chosen from MiniWoB.
Then we investigate the gain in sample efﬁciency with our model from multitask learning.
We also perform an ablation study to justify the effectiveness of each representational module, followed by the comparison of gains in sample efﬁciency from goal attention in multitask and single task settings.
Hyperparameters of the experiments are explained in the appendix 6.1, and we will release the code for reproducibility upon acceptance of the paper.
4. DOM-Q-NET B ENCHMARK M INI W O B
We chooses an off-policy Q learning algorithm, using the four components of Rainbow DQN (Hessel et al., 2018), to train our agent because the web navigation tasks are sparse reward problems, and off-policy learning with a replay buffer is more efﬁcient.
Four components are DDQN (Van Hasselt et al., 2016), Prioritized replay (Schaul et al., 2016), Multi-step learning (Sutton, 1988) and NoisyNet (Fortunato et al., 2018).
Selected tasks require clicking DOM elements and typing strings, which is the same setting as Liu et al.
The agent receives +1 reward if the task is completed correctly, and 0 reward otherwise.
We perform N = 3 steps of neural message passing of GNN for all the tasks except “Social Media”, for which we use N = 7 steps to address the large DOM space.
Evaluation metric: We plot the moving average of the rewards for last 100 episodes during the training. We follow the previous works (Shi et al., 2017; Liu et al., 2018), and report success rate, which is the percentage of test episodes that ends up with reward +1.
Each reported success rate is based on the average of 4 different runs, and appendix 6.5 explains the experiment protocol.
Results: Figure 3 shows that DOM-Q-NET reaches 100% success rate for most of the tasks selected by Liu et al.
(2018), except for “Click widget”, “Social Media” and “Email inbox”.
Our model still reaches 86% success rate for “Social Media” task, and the use of goal attention enables the model to solve “Click widget” and “Social Media” with 100% success rate.
We did not use any prior knowledge such as providing constraints on the action set during exploration (Liu et al., 2018) and using pre-deﬁned ﬁelds of the goal or expert demonstrations.
In particular, our model solves a longhorizon task “choose-date” that none of the previous works with demonstrations is able to solve.
This task contains many similar actions but has a large action space.
Even with imitation learning or guided exploration, the neural network needs to learn a representation that generalizes for unseen diverse set of DOM states and actions, which our model proves to do.
4. M ULTITASK
To evaluate the transferability of learned behaviours, we perform multitask learning of 9 tasks.
In this setting, the agent makes one action for each of M tasks at each time step and receives the next state and a reward for that task.
All the transitions are stored in the same replay buffer.
At each timestep, the network also updates its parameters M times.
This means that the agent can make more updates for a particular task by sampling the corresponding transitions more frequently in earlier stages, and sample transitions for other tasks later.
Hence, the evaluation criteria of sample efﬁciency for multitask versus non-multitask learning is to compare the total number of frames used for solving all the tasks against the sum of the number of frames used for solving each task for a single task agent.
Multitask agent solves the 9 tasks with about twice better sample efﬁciency, using approximately 63000 frames in total, whereas the single task agents use approximately a total of 127000 frames.
Figure 4 shows the plots for 6 out of the 9 tasks.
Particularly, “login-user” and “Click-Checkboxes” are solved with 40000 less frames using multitasking, but this gain is not as obvious when the task is relatively simpler, as in the case of “Navigate-Tree”.
This can be attributed to the capability of doing implicit curriculum learning by biased sampling of transitions for particular tasks at different training period with prioritized replay buffer.
Appendix 6.6.1 shows the full comparisons.
Multitask learning is not applicable to the previous work using workﬂow-guided exploration(WGE) with DOMNET (Liu et al., 2018), because WGE needs structured key-value inputs as a result of us-
[Figure]
has better sample efﬁciency among the shown tasks.
g a=goal attention.
[Figure] r [Figure] a r
[Figure]
l
[Figure]
i s
[Figure]
ing the formal language for constraining actions.
Since the embeddings of DOMNET for key-value inputs are fed into the network without being aggregated, the dimension of the weight matrices is task-dependent.
Hence, single task DOMNET with WGE cannot be extended to multitask learning. 4. A BLATION S TUDY ON THE DOM R EPRESENTATION M ODULES We perform ablation experiments to justify the effectiveness of each module for Q dom network.
Three discounted versions that omit some modules for computing Q dom , (a) e dom = e local , (b) e dom = e neighbor , (c) e dom =
[ e Tlocal , e Tneighbor ] T , are compared against DOM-Q-NET.
Figure 5 shows the two tasks chosen, and the failure case for click-checkboxes task shows that DOM selection without neighborhood module will simply not work because many individual DOMs will have exactly same representation without neural message passing.
Liu et al.
(2018) addressed this issue by hand-crafting the propagation function.
The faster convergence of DOM-Q-NET to optimal behaviour indicates the limitation of neighbor module and how global and local module provide a shortcut to highest and lowest levels of representation of the web page.
[Figure]
E FFECTIVENESS OF G OAL A TTENTION
Most of the MiniWoB tasks have only one type of expected control policy, such as “put a query word in the search and ﬁnd the matched link”, where word token and link have alignments with the DOM elements.
Hence, our DOM-Q-NET solves most of the tasks without feeding the goal vector representation to each Q network, with exceptions like “click-widget”.
Appendix 6.6 shows comparisons of DOM-Q-NET with different goal encoding methods including goal attention.
The effect of goal attention is not obvious, as seen in some tasks.
However, Figure 6 shows that the gain in sample efﬁciency from using goal attention is considerable and much bigger in multitask than the gain in single task setting.
This means that the agent successfully learns to pay attention to different parts of the DOM tree given different goal instructions when multitasking.
D ISCUSSION
We propose a new architecture for parameterizing factorized Q functions using goal attention, local word embeddings and a graph neural network(GNN), and contributes to the formulation of web navigation with this model. Without any demonstration, it solves hard tasks with large action space, and transfers learned behaviours when multitasking, which are two important factors for web navigation.
For future work, we investigate exploration strategies for tasks like “email-inbox” where the environment does not have a simple version of the task that the agent can use to generalize learned behaviours.
Liu et al.
(2018) demonstrated an interesting way to guide the exploration.
Another problem is to reduce the computational cost of evaluating the Q values for each DOM element.
Finally, we intend on applying our methods to using search engines.
Tasks like question answering could beneﬁt from the ability of an agent to query search, navigate the results page and obtain relevant information for solving the desired goal.
The ability to query and navigate search could also be used to bootstrap agents in realistic environments to obtain task-oriented knowledge and improve sample efﬁciency
A PPENDIX 6. H YPERPARAMETERS
Following hyperparameters are used throught all the experiments presented in this paper. [Table] [Table]
[Table] 6. G OAL A TTENTION O UTPUT M ODEL Figure 7 shows vanilla goal-attention output model of the graph neural network. The graph-level representation, h global , is computed by the weighted average of node-level representations, { h 1 .....h V } .
The weights, { α 1 .....α V } , are computed with goal vector as query and node-level features as key and values. For DOM-Q-NET, we use a scaled dot product attention (Vaswani et al., 2017) with local embddings as keys and neighbor embeddings as values i. This is illustrated in 3.3. We frame our goal-attention on the line of works for GNNs(Scarselli et al., 2009), and enable GNNs to be used as a parametrized state for goal-oriented RL.
h=
f ( ,..., ) h goal ee N hhhhhαααααα=
h global ∑ i =
V α i h
i
[Figure]
[Figure] G
[Figure]
E
[Figure]
Three types of goal encoding module for global module are investigated.
Goal vector concatenation with node-level features 2.
Goal Attention, as illustrated in 6. 3.
Both Goal vector concatenation and attention, as shown in ﬁgure [Figure] e [Figure]
e jgoal
[Figure] e jpos e Ngoal [Figure] e Npos = f
) h
[Figure]
ee N ( ..
[Figure]
Vneighbor
[Figure] [Figure]
a l
C o n
[Figure]
t e n a
[Figure]
Benchmark results for multitask and 23 tasks in the appendix 6.6 also compare the performances of using different goal encoding modules.
[Figure]
• Easy Task : Any task solvable under 5000 timesteps by single task DOM-Q-NET { click-dialog, click-test, focus-text, focus-text-2, click-test-2, click-button, click-link, click-button- sequence, click-tab, click-tab-2, Navigate-tree } • Medium Task : Any task solvable under 50000 timesteps by single task DOM-Q-NET { enter-text, click-widget, click-option, click-checkboxes, enter-text-dynamic, enterpassword, login-user, email-inbox-delete } • Hard Task : Any task solvable under 200000 timesteps by single task DOM-Q-NET, or any task for which the agent does not reach 100% success rate. { choose-date, search-engine, social-media, email-inbox }
6. E XPERIMENT P ROTOCOL
We report the success rate of the 100 test episodes at the end of the training once the agent has converged to its highest performance. The ﬁnal success rate reported in Figure 3 is based on the average of success rate from 4 different random seeds/runs. In detail, we evaluate the RL agent after training for a ﬁxed number of frames depending on the difﬁculty of the task, as illustrated in the appendix 6.4.
As shown in table 4, the results presented in this paper is based on a total ofexperiments for the set of hyperparameters in table 1, 2,
[Table]
[Table]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
We present the learning curves of both single and multitask agents that provided the results reported in this paper. Each plot is accompanied with the learning curves of a DOM-Q-NET agent with different goal encoding modules 6.3. X-axis and Y-axis represent the timestep and moving average of last 100 rewards respectively.
For medium and hard tasks, we also show the fraction of transitions with positive/non-zero rewards in the replay buffer, and number of unique positive transitions sampled throughout the training.
This is to demonstrate the sparsity of the reward for each task, and investigate whether the actual problem is due to the lack of exploration. (Note that we are using multistep-bootstrap (Sutton, 1988) so some transitions that do not directly lead to the rewards are still analyzed as positive here) 6.6. M ULTITASK (9 T ASKS ) RESULTS
The following shows the results for 9 tasks used in Multitasking with different goal encoder modules.
6.6. S OME E ASY AND M EDIUM T ASKS
Benchmark DOM-Q-NET performance versus DOM-Q-NET + different goal encoding modules for global modules.
The plots on the left average moving reward last 100 episodes
6.6. H ARD T ASKS
[Figure] [Table]
D IFFERENCES BETWEEN RL AND S EMANTIC P ARSING IN USING LANGUAGE
The main difference between the interest of using RL for web navigation and using semantic parsing for Question Answering with a table:
For web navgiation, we are interested allowing our agent click the DOM and type inputs, which avoids the issue of designing rich primitives and formal language for web navigation.
For question answering using a table, the formal language and primitives are simpler to design because the data format is always a table, but with rich set of compositional questions.
[Equation] [Table] Environment MiniWoB
WikiTableQuestions
Are HTML attributes (tag, class etc) used? Yes No Is HTML structure used?
Yes somewhat (only table structure) Is preprocessing the data structure required? No (Use HTML) Yes
Need Knowledge graph of a table
Input type HTML Knowledge graph Output action type 0(click) or 1(type)
Formal(Machine) language
Executor of output Standard browser Executor for formal language Example of Web navigation with RL :
• Goal: “Use the textbox to enter Rex and Press search, then ﬁnd and click the 4th result” • Input: HTML page • Output: 0(click) or 1(type) along with the index of the DOM element.
Example of Question Answering from a table with Semantic parsing :
(Pasupat & Liang, 2015) • Question: “Greece held its last Summer Olympics in which year?” • Input: Knowledge graph generated from HTML table • Output: R [ λx [ Y ear.
Date.x ]] .argmax ( Country.
Greece, Index )