R EINFORCEMENT L EARNING ON W EB I NTERFACES USING W ORKFLOW -G UIDED E XPLORATION
A BSTRACT
Reinforcement learning (RL) agents improve through trial-and-error, but when reward is sparse and the agent cannot discover successful action sequences, learning stagnates.
This has been a notable problem in training deep RL agents to perform web-based tasks, such as booking ﬂights or replying to emails, where a single mistake can ruin the entire sequence of actions.
A common remedy is to “warmstart” the agent by pre-training it to mimic expert demonstrations, but this is prone to overﬁtting.
Instead, we propose to constrain exploration using demonstrations.
From each demonstration we induce high-level “workﬂows” which constrain the allowable actions at each time step to be similar to those in the demonstration (e g., “Step 1: click on a textbox; Step 2: enter some text”).
Our exploration policy then learns to identify successful workﬂows and samples actions that satisfy these workﬂows.
Workﬂows prune out bad exploration directions and accelerate the agent’s ability to discover rewards.
We use our approach to train a novel neural policy designed to handle the semi-structured nature of websites and evaluate on a suite of web tasks, including the recent World of Bits benchmark.
We achieve new state-of-the-art results and show that workﬂow-guided exploration improves sample efﬁciency over behavioral cloning by more than 100x
I NTRODUCTION
We are interested in training reinforcement learning (RL) agents to use the Internet (e.g., to book ﬂights or reply to emails) by directly controlling a web browser.
Such systems could expand the capabilities of AI personal assistants (Stone & Soper, 2014), which are currently limited to interacting with machine-readable APIs, rather than the much larger world of human-readable web interfaces.
Reinforcement learning agents could learn to accomplish tasks using these human-readable web interfaces through trial-and-error (Sutton & Barto, 1998).
But this learning process can be very slow in tasks with sparse reward, where the vast majority of naive action sequences lead to no reward signal (Vecerik et al., 2017; Nair et al., 2017).
This is the case for many web tasks, which involve a large action space (the agent can type or click anything) and require a well-coordinated sequence of actions to succeed.
A common countermeasure in RL is to pre-train the agent to mimic expert demonstrations via behavioral cloning (Pomerleau, 1991; Kim et al., 2013), encouraging it to take similar actions in similar states.
But in environments with diverse and complex states such as websites, demonstrations may cover only a small slice of the state space, and it is difﬁcult to generalize beyond these states (overﬁtting).
Indeed, previous work has found that warm-starting with behavioral cloning often fails to improve over pure RL (Shi et al., 2017).
At the same time, simple strategies to combat overﬁtting (e.g. using fewer parameters or regularization) cripple the policy’s ﬂexibility (Bitzer et al., 2010), which is required for complex spatial and structural reasoning in user interfaces.
In this work, we propose a different method for leveraging demonstrations.
Rather than training an agent to directly mimic them, we use demonstrations to constrain exploration .
By pruning away bad exploration directions, we can accelerate the agent’s ability to discover sparse rewards.
Furthermore, train explore demonstrations replay buffer workflow policy π neural policy π workflow lattices
DOM NET p ( a | s )
V ( s ) train episodes episodes store store w n s state action train explore
Preprocessing: for all demonstrations d do
Induce workﬂow lattice from d
Every iteration:
Observe an initial environment state π w samples a workﬂow from a lattice Roll out an episode e from the workﬂow
Use e to update π w if e gets reward +1 then
Add e to replay buffer
Periodically: if replay buffer size > threshold then Sample episodes from replay buffer
Update π n with sampled episodes
Observe an initial environment state π n rolls out episode e Update π n and critic V with e if e gets reward +1 then
Add e to replay buffer because the agent is not directly exposed to demonstrations, we are free to use a sophisticated neural policy with a reduced risk of overﬁtting.
To constrain exploration, we employ the notion of a “workﬂow” (Deka et al., 2016).
For instance, given an expert demonstration of how to forward an email, we might infer the following workﬂow:
Click an email title → Click a “Forward” button → Type an email address into a textbox → Click a “Send” button
This workﬂow is more high-level than an actual policy: it does not tell us exactly which email to click or which textbox to type into, but it helpfully constrains the set of actions at each time step.
Furthermore, unlike a policy, it does not depend on the environment state: it is just a sequence of steps that can be followed blindly.
In this sense, a workﬂow is environment-blind .
The actual policy certainly should not be environment-blind, but for exploration, we found environment-blindness to be a good inductive bias.
To leverage workﬂows, we propose the workﬂow-guided exploration (WGE) framework as illustrated in Figure 1:
For each demonstration, we extract a lattice of workﬂows that are consistent with the actions observed in the demonstration (Section 3).
We then deﬁne a workﬂow exploration policy π w (Section 4), which explores by ﬁrst selecting a workﬂow, and then sampling actions that ﬁt the workﬂow.
This policy gradually learns which workﬂow to select through reinforcement learning.
Reward-earning episodes discovered during exploration enter a replay buffer, which we use to train a more powerful and expressive neural network policy π n (Section 5).
A key difference between the web and traditional RL domains such as robotics (Atkeson & Schaal, 1997) or game-playing (Bellemare et al., 2013) is that the state space involves a mix of structured (e.g. HTML) and unstructured inputs (e.g. natural language and images).
This motivates us to propose a novel neural network policy (DOM NET ), speciﬁcally designed to perform ﬂexible relational reasoning over the tree-structured HTML representation of websites.
We evaluate workﬂow-guided exploration and DOM NET on a suite of web interaction tasks, including the MiniWoB benchmark of (Shi et al., 2017), the ﬂight booking interface for Alaska Airlines, and a new collection of tasks that we constructed to study additional challenges such as noisy environments, variation in natural language, and longer time horizons.
Compared to previous results on MiniWoB Shi et al.
(2017), which used 10 minutes of demonstrations per task (approximately 200 demonstrations on average), our system achieves much higher success rates and establishes new state-of-the-art results with only 3–10 demonstrations per task.
S ETUP
In the standard reinforcement learning setup, an agent learns a policy π ( a | s ) that maps a state s to a probability distribution over actions a .
At each time step t , the agent observes an environment state s t and chooses an action a t , which leads to a new state s t +1 and a reward r t
= r
( s t , a t ) .
The goal is to maximize the expected return E [ R ] , where R = (cid:80) t γ t r t +1 and γ is a discount factor.
Typical reinforcement learning agents learn through trial-and-error: rolling out episodes ( s 1 , a 1 , . . .
, s T , a T ) and adjusting their policy based on the results of those episodes.
We focus on settings where the reward is delayed and sparse.
Speciﬁcally, we assume that (1) the agent receives reward only at the end of the episode, and (2) the reward is high (e.g., +1 ) for only a small fraction of possible trajectories and is uniformly low (e.g., − 1 ) otherwise.
With large state and action spaces, it is difﬁcult for the exploration policy to ﬁnd episodes with positive rewards, which prevents the policy from learning effectively.
We further assume that the agent is given a goal g , which can either be a structured key-value mapping (e.g., {task: forward, from: Bob, to: Alice}) or a natural language utterance (e.g., “Forward
Bob’s message to Alice” ).
The agent’s state s consists of the goal g and the current state of the web page, represented as a tree of elements (henceforth DOM tree ).
We restrict the action space to click actions Click(e) and type actions Type(e,t) , where e is a leaf element of the DOM tree, and t is a string from the goal g (a value from a structured goal, or consecutive tokens from a natural language goal).
Figure 2 shows an example episode for an email processing task.
The agent receives +1 reward if the task is completed correctly, and − 1 reward otherwise.
I NDUCING WORKFLOWS FROM DEMONSTRATIONS
Given a collection of expert demonstrations d = (˜ s 1 , ˜ a 1 , . . .
, ˜ s T , ˜ a T ) , we would like explore actions a t that are “similar” to the demonstrated actions ˜ a t .
Workﬂows capture this notion of similarity by specifying a set of similar actions at each time step.
Formally, a workﬂow z 1: T is a sequence of workﬂow steps, where each step z t is a function that takes a state s t and returns a constrained set z t ( s t ) of similar actions.
We use a simple compositional constraint language (Appendix A) to describe workﬂow steps.
For example, with z t = Click(Tag("img")) , the set z t ( s t ) contains click actions on any DOM element in s t with tag img .
We induce a set of workﬂows from each demonstration d
= (˜ s 1 , ˜ a 1 , . .
. , ˜ s T , ˜ a T ) as follows.
For each time step t , we enumerate a set Z t of all possible workﬂow steps z t such that ˜ a t ∈
z t (˜ s t ) .
The set of workﬂows is then the cross product Z 1 × · · · × Z T of the steps.
We can represent the induced workﬂows as paths in a workﬂow lattice as illustrated in Figure 2.
To handle noisy demonstrations where some actions are unnecessary (e.g., when the demonstrator accidentally clicks on the background), we add shortcut steps that skip certain time steps.
We also add shortcut steps for any consecutive actions that can be collapsed into a single equivalent action (e.g., collapsing two type actions on the same DOM element into a single Type step).
These shortcuts allow the lengths of the induced workﬂows to differ from the length of the demonstration.
We henceforth ignore these shortcut steps to simplify the notation.
The induced workﬂow steps are not equally effective.
For example in Figure 2, the workﬂow step
Click(Near(Text("Bob")))
(Click an element near text “Bob”) is too speciﬁc to the demonstration scenario, while Click(Tag("div"))
(Click on any <div> element) is too general and covers too many irrelevant actions.
The next section describes how the workﬂow policy π w learns which workﬂow steps to use.
[Figure]
W ORKFLOW EXPLORATION POLICY
Our workﬂow policy interacts with the environment to generate an episode in the following manner.
At the beginning of the episode, the policy conditions on the provided goal g , and selects a demonstration d that carried out a similar goal:
[Equation]
where sim ( g, g d ) measures the similarity between g and the goal g d of demonstration d .
In our tasks, we simply let sim ( g, g d ) be 1 if the structured goals share the same keys, and −∞ otherwise.
Then, at each time step t with environment state s t , we sample a workﬂow step z t according to the following distribution:
[Equation]
where each ψ z,t,d is a separate scalar parameter to be learned.
Finally, we sample an action a t uniformly from the set z t ( s t
[Equation]
The overall probability of exploring an episode e = ( s 1 , a 1 , . .
, s T , a T ) is then:
[Equation]
where p ( s t | s t − 1 , a t − 1 ) is the (unknown) state transition probability.
Note that π w ( z | d, t ) is not a function of the environment states s t at all.
Its decisions only depend on the selected demonstration and the current time t .
This environment-blindness means that the workﬂow policy uses far fewer parameters than a state-dependent policy, enabling it to learn more quickly and preventing overﬁtting.
Due to environment-blindness , the workﬂow policy cannot solve the task, but it quickly learns to certain good behaviors, which can help the neural policy learn.
To train the workﬂow policy, we use a variant of the REINFORCE algorithm (Williams, 1992; Sutton & Barto, 1998).
In particular, after rolling out an episode e = ( s 1 , a 1 , . . .
, s T , a T ) , we approximate the gradient using the unbiased estimate
[Equation]
[Equation]
where G t is the return at time step t and v d,t is a baseline term for variance reduction.
Sampled episodes from the workﬂow policy that receive a positive reward are stored in a replay buffer, which will be used for training the neural policy π n .
N EURAL POLICY
As outlined in Figure 1, the neural policy is learned using both on-policy and off-policy updates (where episodes are drawn from the replay buffer).
Both updates use A2C, the synchronous version of the advantage actor-critic algorithm (Mnih et al., 2016).
Since only episodes with reward +1 enter the replay buffer, the off-policy updates behave similarly to supervised learning on optimal trajectories.
Furthermore, successful episodes discovered during on-policy exploration are also added to the replay buffer.
Model architecture.
We propose DOM NET , a neural architecture that captures the spatial and hierarchical structure of the DOM tree.
As illustrated in Figure 5, the model ﬁrst embeds the DOM elements and the input goal, and then applies a series of attentions on the embeddings to ﬁnally produce a distribution over actions π n ( a | s ) and a value function V ( s ) , the critic.
We highlight our novel DOM embedder, and defer other details to Appendix C.
We design our DOM embedder to capture the various interactions between DOM elements, similar to recent work in graph embeddings (Kipf & Welling, 2017; Pham et al., 2017; Hamilton et al., 2017).
In particular, DOM elements that are “related” (e.g., a checkbox and its associated label) should pass their information to each other.
To embed a DOM element e , we ﬁrst compute the base embedding v e base by embedding and concatenating its attributes (tag, classes, text, etc.).
In order to capture the relationships between DOM elements, we next compute two types of neighbor embeddings :
We deﬁne spatial neighbors of e to be any element e (cid:48) within 30 pixels from e , and then sum up their base embeddings to get the spatial neighbor embedding v e spatial .
We deﬁne
depth- k tree neighbors of e to be any element e (cid:48) such that the least common ancestor of e and e (cid:48) in the DOM tree has depth at most k .
Intuitively, tree neighbors of a higher depth are more related.
For each depth k , we apply a learnable afﬁne transformation f on the base embedding of each depth- k tree neighbor e (cid:48) , and then apply max pooling to get v e tree[ k ] =
max f ( v e (cid:48) base ) .
We let the tree neighbor embedding v e tree be the concatenation of v e tree[ k ] for k = 3 , 4 , 5 , 6 .
Finally, we deﬁne the goal matching embedding v e match to be the sum of the embeddings of all words in e that also appear in the goal.
The ﬁnal embedding v e DOM of e is the concatenation of the four embeddings [ v e base ; v e spatial ; v e tree ; v e match ] .
E XPERIMENTS 6. T ASK SETUPS
We evaluate our approach on three suites of interactive web tasks:
MiniWoB : the MiniWoB benchmark of Shi et al.
MiniWoB++ : a new set of tasks that we constructed to incorporate additional challenges not present in MiniWoB, such as stochastic environments and variation in natural language.
Alaska : the mobile ﬂight booking interface for Alaska Airlines, inspired by the FormWoB benchmark of Shi et al.
We describe the common task settings of the MiniWoB and MiniWoB++ benchmarks, and defer the description of the Alaska benchmark to Section 6.3.3.
[Figure]
S u cce ss
R a t e
[Figure]
R a t e
[Figure] [Figure]
[Figure] [Figure]
Task
Description
[Figure]
WGE click-checkboxes
Click 0–6 speciﬁed checkboxes    click-checkboxes-large + ...5–12 targets    click-checkboxes-soft + ...speciﬁes synonyms of the targets    click-checkboxes-transfer + ...training data has 0-3 targets    multi-ordering +
Fill a form with varying ﬁeld orderings    multi-layout +
Fill a form with varying UIs layouts    social-media
Do an action on the speciﬁed Tweet    social-media-all + ...on all matching Tweets    social-media-some + ...on speciﬁed no. of matching Tweets    email-inbox
Perform tasks on an email inbox    email-inbox-nl + ...natural language goal
Environment.
Each task contains a 160px × 210px environment and a goal speciﬁed in text.
The majority of the tasks return a single sparse at the end of the episode either +1 (success or −(failure).
For greater consistency among tasks, we disabled all partial rewards in our experiments.
The agent has access to the environment via a Selenium web driver interface.
The public MiniWoB benchmark 1 contains 80 tasks.
We ﬁltered for the 40 tasks that only require actions in our action space, namely clicking on DOM elements and typing strings from the input goal.
Many of the excluded tasks involve somewhat specialized reasoning, such as being able to compute the angle between two lines, or solve algebra problems.
For each task, we used Amazon
Mechanical Turk to collect 10 demonstrations, which record all mouse and keyboard events along with the state of the DOM when each event occurred.
Evaluation metric.
We report success rate : the percentage of test episodes with reward +1 .
Since we have removed partial rewards, success rate is a linear scaling of the average reward, and is equivalent to the deﬁnition of success rate in Shi et al.
6. M AIN RESULTS
We compare the success rates across the MiniWoB tasks of the following approaches:
• S HI 17: the system from Shi et al.
(2017), pre-trained with behavioral cloning on 10 minutes of demonstrations (approximately 200 demonstrations on average) and ﬁne-tuned with RL.
Unlike DOM NET , this system primarily uses a pixel-based representation of the state.
• DOM NET +BC+RL: our proposed neural policy, DOM NET , but pre-trained with behavioral cloning on 10 demonstrations and ﬁne-tuned with RL, like S HI 17.
During behavioral cloning, we apply early stopping based on the reward on a validation set.
• DOM NET +WGE: our proposed neural policy, DOM NET , trained with workﬂow-guided exploration on 10 demonstrations.
For DOM NET +BC+RL and DOM NET +WGE, we report the test success rate at the time step where the success rate on a validation set reaches its maximum.
The results are shown in Figure 3.
By comparing S HI 17 with DOM NET +BC+RL, we can roughly evaluate the contribution of our new neural architecture DOM NET , since the two share the same training procedure (BC+RL).
While S HI 17 also uses the DOM tree to compute text alignment features in addition to the pixel-level input, our DOM NET uses the DOM structure more explicitly.
We ﬁnd DOM NET +BC+RL to empirically improve the success rate over S HI 17 on most tasks.
By comparing DOM NET +BC+RL and DOM NET +WGE, we ﬁnd that workﬂow-guided exploration enables DOM NET to perform even better on the more difﬁcult tasks, which we analyze in the next section.
Some of the workﬂows that the workﬂow policy π w learns are shown in Appendix B.
6. A NALYSIS 6.3. M INI W O B++ BENCHMARK
We constructed and released the MiniWoB++ benchmark of tasks to study additional challenges a web agent might encounter, including: longer time horizons (click-checkboxes-large), “soft” reasoning about natural language (click-checkboxes-soft), and stochastically varying layouts (multiorderings, multi-layouts).
Table 1 lists the tasks and their time horizons (number of steps needed for a perfect policy to carry out the longest goal) as a crude measure of task complexity.
We ﬁrst compare the performance of DOM NET trained with BC+RL (baseline) and DOM NET trained with WGE (our full approach).
The proposed WGE model outperforms the BC+RL model by an average of 42% absolute success rate.
We analyzed their behaviors and noticed two common failure modes of training with BC+RL that are mitigated by instead training with WGE:
The BC+RL model has a tendency to take actions that prematurely terminate the episode (e
g., hitting “Submit” in click-checkboxes-large before all required boxes are checked).
One likely cause is that these actions occur across all demonstrations, while other nonterminating actions (e
g., clicking different checkboxes) vary across demonstrations.
The BC+RL model occasionally gets stuck in cyclic behavior such as repeatedly checking and unchecking the same checkbox.
These failure modes stem from overﬁtting to parts of the demonstrations, which WGE avoids.
Next, we analyze the workﬂow policy π w learned by WGE.
The workﬂow policy π w by itself is too simplistic to work well at test time for several reasons:
Workﬂows ignore environment state and therefore cannot respond to the differences in the environment, such as the different layouts in multi-layouts.
The workﬂow constraint language lacks the expressivity to specify certain actions, such as clicking on synonyms of a particular word in click-checkboxes-soft.
The workﬂow policy lacks expressivity to select the correct workﬂow for a given goal.
Nonetheless the workﬂow policy π w is sufﬁciently constrained to discover reward some of the time, and the neural policy π n is able to learn the right behavior from such episodes.
As such, the neural policy can achieve high success rates even when the workﬂow policy π w performs poorly.
6.3. N ATURAL LANGUAGE INPUTS
While MiniWoB tasks provide structured goals, we can also apply our approach to natural language goals.
We collected a training dataset using the overnight data collection technique (Wang et al., 2015).
In the email-inbox-nl task, we collected natural language templates by asking annotators
[Figure]
to paraphrase the task goals (e.g., “Forward Bob’s message to Alice” → “Email Alice the email I got from Bob” ) and then abstracting out the ﬁelds ( “Email <TO> the email I got from <FROM> ” ).
During training, the workﬂow policy π w receives states with both the structured goal and the natural language utterance generated from a random template, while the neural policy π n receives only the utterance.
At test time, the neural policy is evaluated on unseen utterances.
The results in Tableshow that the WGE model can learn to understand natural language goals (93% success rate).
Note that the workﬂow policy needs access to the structured inputs only because our constraint language for workﬂow steps operates on structured inputs.
The constraint language could potentially be modiﬁed to work with utterances directly (e.g., After("to") extracts the utterance word after “to” ), but we leave this for future work.
6.3. S CALING TO REAL WORLD TASKS
We applied our approach on the Alaska benchmark, a more realistic ﬂight search task on the Alaska
Airlines mobile site inspired by the FormWoB task in Shi et al.
In this task, the agent must complete the ﬂight search form with the provided information (6–7 ﬁelds).
We ported the web page to the MiniWoB framework with a larger 375px × 667px screen, replaced the server backend with a surrogate JavaScript function, and clamped the environment date to March 1, 2017.
Following Shi et al.
(2017), we give partial reward based on the fraction of correct ﬁelds in the submitted form if all required ﬁelds are ﬁlled in.
Despite this partial reward, the reward is still extremely sparse: there are over 200 DOM elements (compared to ≈ 10–50 in MiniWoB tasks), and a typical episode requires at least 11 actions involving various types of widgets such as autocompletes and date pickers.
The probability that a random agent gets positive reward is less than 10 − 20 .
We ﬁrst performed experiments on Alaska-Shi17, a clone of the original Alaska Airlines task in Shi et al.
(2017), where the goal always speciﬁes a roundtrip ﬂight (two airports and two dates).
On their dataset, our approach, using only 1 demonstration, achieves an average reward of 0.97, compared to their best result of 0.57, which uses around 80 demonstrations.
Our success motivated us to test on a more difﬁcult version of the task which additionally requires selecting ﬂight type (a checkbox for one-way ﬂight), number of passengers (an increment-decrement counter), and seat type (hidden under an accordion).
We achieve an average reward of 0.86 usingdemonstrations.
This demonstrates our method can handle long horizons on real-world websites.
6.3. S AMPLE EFFICIENCY
To evaluate the demonstration efﬁciency of our approach, we compare DOM NET +WGE with
DOM NET +BC+RL trained on increased numbers of demonstrations.
We compare DOMNET +WGE trained on 10 demonstrations with DOM NET +BC+RL on 10 , 100 , 300 , anddemonstrations.
The test rewards 3 on several of the hardest tasks are summarized in Figure 4.
Increasing the number of demonstrations improves the performance of BC+RL, as it helps prevent overﬁtting.
However, on every evaluated task, WGE trained with only 10 demonstrations still achieves much higher test reward than BC+RL with 1000 demonstrations.
This corresponds to an over 100x sample efﬁciency improvement of our method over behavioral cloning in terms of the number of demonstrations
D ISCUSSION
Learning agents for the web.
Previous work on learning agents for web interactions falls into two main categories.
First, simple programs may be speciﬁed by the user (Yeh et al., 2009) or may be inferred from demonstrations (Allen et al., 2007).
Second, soft policies may be learned from scratch or “warm-started” from demonstrations (Shi et al., 2017).
Notably, sparse rewards prevented Shi et al.
(2017) from successfully learning, even when using a moderate number of demonstrations.
While policies have proven to be more difﬁcult to learn, they have the potential to be expressive and ﬂexible.
Our work takes a step in this direction.
Sparse rewards without prior knowledge.
Numerous works attempt to address sparse rewards without incorporating any additional prior knowledge.
Exploration methods (Osband et al., 2016;
Chentanez et al., 2005; Weber et al., 2017) help the agent better explore the state space to encounter more reward; shaping rewards (Ng et al., 1999) directly modify the reward function to encourage certain behaviors; and other works (Jaderberg et al., 2016; Andrychowicz et al., 2017) augment the reward signal with additional unsupervised reward.
However, without prior knowledge, helping the agent receive additional reward is difﬁcult in general.
Imitation learning.
Various methods have been proposed to leverage additional signals from experts.
For instance, when an expert policy is available, methods such as DA GGER (Ross et al., 2011) and A GGRE V A T E (Ross & Bagnell, 2014; Sun et al., 2017) can query the expert policy to augment the dataset for training the agent.
When only expert demonstrations are available, inverse reinforcement learning methods (Abbeel & Ng, 2004; Ziebart et al., 2008; Finn et al., 2016; Ho &
Ermon, 2016; Baram et al., 2017) infer a reward function from the demonstrations without using reinforcement signals from the environment.
The usual method for incorporating both demonstrations and reinforcement signals is to pre-train the agent with demonstrations before applying RL.
Recent work extends this technique by (1) introducing different objective functions and regularization during pre-training, and (2) mixing demonstrations and rolled-out episodes during RL updates (Hosu & Rebedea, 2016; Hester et al., 2018;
Vecerik et al., 2017; Nair et al., 2017).
Instead of training the agent on demonstrations directly, our work uses demonstrations to guide exploration .
The core idea is to explore trajectories that lie in a “neighborhood” surrounding an expert demonstration.
In our case, the neighborhood is deﬁned by a workﬂow, which only permits action sequences analogous to the demonstrated actions.
Several previous works also explore neighborhoods of demonstrations via reward shaping (Brys et al., 2015; Hussein et al., 2017) or off-policy sampling (Levine & Koltun, 2013).
One key distinction of our work is that we deﬁne neighborhoods in terms of action similarity rather than state similarity.
This distinction is particularly important for the web tasks: we can easily and intuitively describe how two actions are analogous (e.g., “they both type a username into a textbox”), while it is harder to decide if two web page states are analogous (e.g., the email inboxes of two different users will have completely different emails, but they could still be analogous, depending on the task.)
Hierarchical reinforcement learning.
Hierarchical reinforcement learning (HRL) methods decompose complex tasks into simpler subtasks that are easier to learn.
Main HRL frameworks include abstract actions (Sutton et al., 1999; Konidaris & Barto, 2007; Hauser et al., 2008), abstract partial policies (Parr & Russell, 1998), and abstract states (Roderick et al., 2017; Dietterich, 1998; Li et al., 2006).
These frameworks require varying amounts of prior knowledge.
The original formulations required programmers to manually specify the decomposition of the complex task, while Andreas et al.
(2016) only requires supervision to identify subtasks, and Bacon et al.
; Daniel et al.
(2016) learn the decomposition fully automatically, at the cost of performance.
Within the HRL methods, our work is closest to Parr & Russell (1998) and the line of work on constraints in robotics (Phillips et al., 2016; Perez-D’Arpino & Shah, 2017).
The work in Parr &
Russell (1998) speciﬁes partial policies, which constrain the set of possible actions at each state, similar to our workﬂow items.
In contrast to previous instantiations of the HAM framework (Andre, 2003; Marthi & Guestrin, 2005), which require programmers to specify these constraints manually, our work automatically induces constraints from user demonstrations, which do not require special skills to provide.
Phillips et al.
; Perez-D’Arpino & Shah (2017) also resemble our work, in learning constraints from demonstrations, but differ in the way they use the demonstrations.
Whereas our work uses the learned constraints for exploration, Phillips et al.
(2016) only uses the constraints for planning and Perez-D’Arpino & Shah (2017) build a knowledge base of constraints to use at test time.
Summary.
Our workﬂow-guided framework represents a judicious combination of demonstrations, abstractions, and expressive neural policies.
We leverage the targeted information of demonstrations and the inductive bias of workﬂows.
But this is only used for exploration, protecting the expressive neural policy from overﬁtting.
As a result, we are able to learn rather complex policies from a very sparse reward signal and very few demonstrations.
Acknowledgments.
This work was supported by NSF CAREER Award IIS-1552635.
Reproducibility.
Our code and data are available at https://github. com/stanfordnlp/wge .
Reproducible experiments are available on the
CodaLab platform at https://worksheets.codalab.org/worksheets/ 0x0f25031bd42f4aabbc17625fe1484066/ .
A C ONSTRAINT LANGUAGE FOR WORKFLOW STEPS
We try to keep the constraint language as minimal and general as possible.
The main part of the language is the object selector ( elementSet ) which selects either (1) objects that share a speciﬁed property, or (2) objects that align spatially.
These two types of constraints should be applicable in many typical RL domains such as game playing and robot navigation.
constraint ::=
Click( elementSet )
[Any click action on an element in elementSet ] |
Type( elementSet , string )
[Any type action that types string on an element in elementSet ] |
Type( elementSet ,Field(*))
[Any type action that types a goal ﬁeld value on an element in elementSet ] elementSet ::=
Tag( tag )
[Any element with HTML tag tag ] |
Text( string )
[Any element with text string ] |
Like( string )
[Any element whose text is a substring of string ] |
Near( elementSet ) [Any element that is within 30px from an element in elementSet ] |
SameRow( elementSet )
[Any element that aligns horizontally with an element in elementSet ] |
SameCol( elementSet ) [Any element that aligns vertically with an element in elementSet ] |
And( elementSet ,Class( classes ))
[Any element from elementSet matching some class name in classes ] tag ::= a valid HTML tag name string ::= a string literal |
Field( ﬁeldName )
[The value from the goal ﬁeld ﬁeldName ] classes ::= a list of valid HTML class names
To avoid combinatorial explosion of relatively useless constraints, we limit the number of nested elementSet applications to 3, where the third application must be the Class ﬁlter.
When we induce workﬂow steps from a demonstration, the valid literal values for tag , string , and classes are extracted from the demonstration state.
B E XAMPLES OF LEARNED WORKFLOWS
login-user Enter the username "ashlea" and password "k0UQp" and press login.
{username: ashlea
[Table] [Table]
C D ETAILS OF THE NEURAL MODEL ARCHITECTURE
Embeddings.
From the input state, we ﬁrst embed the DOM elements e and the goal units u , where u is a key-value pair for structured goals and a token for natural language goals.
The process for computing the embedding v e DOM of DOM elements is already described in Section 5.
For the goal unit embedding v u goal , we embed each key-value pair as the sum of word embeddings, and embed natural language goals with an LSTM.
Attentions.
After obtaining the embedding v e DOM of each DOM element e and v u goal of each goal unit u , we apply a series of attentions to relate the DOM elements with the goal:
DOM context : we applied max-pooling on v e DOM to get a query vector, and then attend over the DOM embeddings v e DOM .
The DOM context is the weighted average of the attended DOM embeddings.
Goal contexts : we use the DOM context as the query vector to attend over the goal embeddings v u goal .
We compute two goal contexts from two different attention heads.
Each head uses sentinel attention, where part of the attention can be put on a learned NULL vector, which is useful for ignoring the goal when the next action should not depend on the goal.
DOM element selection : We concatenate the DOM context and goal contexts into a query vector to attend over on the DOM embeddings v e DOM .
We use two attention heads, and combine the attention weights from the two heads based on ratio computed from the goal contexts.
The result is a distribution over the target DOM elements e . 4.
Typed string and action selection : For a given target DOM element e , we combine the goal context and the embedding v e DOM of e to get a query vector to attend over the goal embeddings v u goal .
For structured queries, we get a distribution over the goal ﬁelds, while for natural language queries, we get distributions of the start and end tokens.
The same query vector is also used to compute the distribution over the action types ( click or type ).
[Figure]