WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents
Anonymous Author(s)
AffiliationAddress email
Abstract
Most existing benchmarks for grounding language in interactive environmentseither lack realistic linguistic elements, or prove difficult to scale up due to substantial human involvement in the collection of data or feedback signals.
We develop WebShop – a simulated e-commerce website environment with 1 . 18 million realworld products and 12 , 087 crowd-sourced text instructions.
In this environment,an agent needs to navigate multiple types of webpages and issue diverse actionsto find, customize, and purchase a product given an instruction.
WebShop provides several challenges including understanding compositional instructions, query(re-)formulation, dealing with noisy text in webpages, and performing strategic exploration.
We collect over 1 , 600 human trajectories to first validate the benchmark,then train and evaluate a diverse range of agents using reinforcement learning,imitation learning, and pre-trained image and language models.
Our best modelachieves a task success rate of 29% , which significantly outperforms rule heuristicsbut is far lower than expert human performance ( 59% ).
We also analyze agent andhuman trajectories and ablate various model components to provide insights fordeveloping future agents with stronger language understanding and decision making abilities.
Finally, we show our agent trained on WebShop exhibits non-trivialsim-to-real transfer when evaluated on amazon.com , indicating the potential valueof our benchmark for developing practical web agents that can operate in the wild.
Introduction
Recent advances in natural language processing (NLP) and reinforcement learning (RL) have broughtabout several exciting developments in agents that can perform sequential decision making whilemaking use of linguistic context [20, 35, 39].
On the other hand, large-scale language models like
GPT-3 [5] and BERT [8] are excelling at traditional NLP benchmarks such as text classification,information extraction and question answering.
While the former set of tasks are limited in their setof linguistic concepts and prove difficult to scale up, the latter tasks usually contain static datasetsthat lack adequate grounding to extra-linguistic concepts [3].
In order to make further progress inbuilding grounded language models, there is a need for interactive environments that contain: (1) language elements that reflect rich, real-world usage and are collectible at scale, and (2) task feedback that is well-defined and automatically computable to facilitate scalable interactive learning, without the need for expensive feedback from humans every time. The world wide web (WWW) is a massive open-domain interactive environment that inherentlysatisfies the above requirements, through its interconnected set of pages with natural text, images [Table] C
[Table]
+bmvDGz7hMUoOSLRaFqSAmJrOvyYArZEZMLKFMcXsrYSOqKDM2m5INwVt+eZW0LqreVdVrXFZqt3kcRTiBUzgHD66hBvdQhyYwQHiGV3hzHp0X5935WLQWnHzmGP7A+fwB46WM/w==</latexit> u
Reward:
[Figure] [Figure]
and interactive elements. By being simultaneously scalable, semantic, interactive, dynamic andrealistic , the web is uniquely different from existing environments for autonomous agents like gamesor 3D navigation.
Moreover, the web also provides a practical environment to deploy trained agents,with great potential for alleviating human efforts in tedious tasks (e.g. buying products, bookingappointments).
While there has been prior work on building web-based tasks, they fall short inone of two ways.
Some benchmarks only contain either a single classification task [28, 32, 21] orinteractions containing only a handful of different pages in each episode [30].
Others propose taskswith longer horizons but are either limited to following hyperlinks for web navigation [26] or requirehuman-in-the-loop feedback due to the lack of an automated reward function [23].
In this paper, we introduce WebShop (Figure 1) – a large-scale interactive web-based environmentfor language understanding and decision making – and train autonomous agents to complete tasks onthis benchmark.
WebShop emulates the task of online shopping on an e-commerce website, wherethe agent’s goal is to understand a human instruction and purchase the best product that matches thespecifications.
To do so, the agent needs to query the website’s search engine, choose items to explorefrom search results, open and read their description and details, and select the necessary options (e.g.32 oz., red color) before clicking the ‘Buy’ button.
In order to pick the optimal product that matchesuser requirements, the agent may have to backtrack, view, and compare various products, thenpotentially perform multiple searches if the agent deems no current search result a good match.
Thebenchmark contains over one million real products scraped from amazon.com , 12 , 087 crowdsourcedinstructions, and a diverse semantic action space – all packaged into a convenient OpenAI Gym [4]environment for training and evaluating RL agents.
The environment can be rendered in two modes – HTML or simple – with different observation spaces, and rewards are automatically computed usinga combination of programmatic matching functions that consider the attributes, type, options and price of the chosen product, alleviating the need for human evaluation and providing a path to easilyscale up the benchmark in the future. WebShop is a challenging environment for autonomous agents due to several reasons: (1) the averagebranching factor (around 15 for non-search pages) is very large, making exploration hard, (2) theagent has to understand real-world text and images on webpages and generate natural language forsearch queries, (3) the textual instructions require compositional generalization, and (4) the agent hasto remember previous visited states and search queries in order to compare items before making itschoice. This makes the benchmark an ideal testbed for grounded language and visual understandingin a realistic setting (as opposed to games) while being fast enough to simulate and easily scale tomore domains and tasks (compared to simulated 3D navigation or physical robotic manipulation).
We also develop several agents for performing this task, using both reinforcement learning (RL) andimitation learning (IL) in addition to leveraging the latest pre-trained language models for representingand generating text.
Our modular architecture includes a factorized processing of state observationsand action choices using ResNets (visual) and Transformers (text), followed by an attention fusionlayer that helps the agent contextually score each action.
Our best agent achieves an average score of62 .
4 (out of 100 ) and successfully completes the task 28 .
7% of the time, significantly higher than aheuristic baseline that achieves 45 .
and 9 .
6% , respectively.
While this demonstrates the potentialfor IL and RL, the agents are still much lower than human experts that can achieve 82 .
1 and 59 .
6%on this task.
We perform several analyses and ablation studies to identify the cause of this gap andfind several avenues for agent improvement in the future including more robust search generation,explicit memory modules, and better handling of noisy web text.
Finally, we also demonstrate aninstance of ‘sim-to-real’ transfer by deploying our agent trained on WebShop to operate on the realamazon.com , and find that it is able to achieve a similar success rate ( 27% ).
This demonstrates thepractical potential of our work towards enabling agents that can operate autonomously on the realworld wide web (WWW).
Our code is available in the supplementary material.
Related Work
Reinforcement learning on the web.
Nogueira and Cho [26] introduced WikiNav as a benchmarkfor RL agents navigating pages, but the task is purely navigational with the actions restricted toeither choosing a hyperlink to follow or deciding to stop.
The World of Bits (WoB) benchmark [30]enables training of RL agents to complete tasks on webpages using pixel and Document Object
Model (DOM) observations.
Several follow-up papers have tackled MiniWoB using techniques likeworkflow-guided exploration [19], curriculum and meta-learning [9], DOM tree representation [14],adversarial environment generation [10] and large-scale behavioral cloning [13].
However, MiniWoBlacks long-range decision making across multiple different pages and does not scale easily in termsof difficulty or size due to its use of low-level mouse clicks and keystrokes as actions.
In contrast,
WebShop requires navigating longer paths with context-based action selection and backtracking, andit uses high-level search and choose actions that are more scalable and transferable to real settings.
While not directly operating on web pages, AndroidEnv [34] and MoTIF [6] provide environments totrain agents for interacting with apps and services on mobile platforms.
Non-interactive web-based tasks.
Various supervised classification tasks on webpages have beenproposed, including predicting web elements [28], generating API calls [32, 33, 36] and semanticparsing into concept-level navigation actions [21].
Perhaps most similar content-wise to our work isthe Klarna product page dataset [12] which contains over 50 , 000 product pages labeled with differentelement categories for supervised classification.
All these works only consider supervised settingswith a single decision, and may require the definition of web APIs or command templates for eachdomain.
Our benchmark, WebShop, combines webpages with realistic text and image content with arich and diverse interaction space for long-range sequential decision making.
Leveraging the web for traditional NLP tasks.
Several papers have explored the use of the web forinformation extraction [24] and retrieval [1], question answering [38, 16] and dialog [31], and traininglanguage models on webtext [2].
These approaches use search engines primarily as a knowledge retriever for gathering additional evidence for the task at hand.
Perhaps most similar to our workis WebGPT [23], which uses a web interface integrated with a search engine to train RL agents tonavigate the web and answer questions.
However, our environment has a more diverse action andobservation space (including images) and does not require human-in-the-loop evaluation.
The WebShop Environment
We create WebShop as a large-scale web-based interactive environment with over 1 . 1 million realworld products scraped from amazon.com.
In this environment, an agent needs to find and purchase aproduct according to specifications provided in a natural language instruction.
WebShop is designedin a modular fashion which disentangles the website transitions from the task-specific aspects likeinstructions and reward, allowing for easy extension to new tasks and domains.
3.1 Task Formulation
[Table]
WebShop is a partially observable Markov decision
[Table]
process (POMDP) ( S , A , T , R , U , O ) with state space
[Table] S , action space A , deterministic transition function [Table]
T : S ×
A → S , reward function R : S ×
A → [0 , 1] ,instruction space U , and a state observation space O .
State and action.
A state s ∈ S represents a web page,which falls into one of the four types – the search pagethat contains a search bar, the results page that lists a set of products returned by a search engine, theitem page that describes a product, or the item-detail page that shows further information about theproduct (Figure 1A(1-4) respectively).
We define the following notations for a product y .
We denote¯
y
to be the aggregation of the various text fields including product title, description, and overview.
We denote y price to be the price, Y opt to be a set of buying options, and I to be a set of images, eachcorresponding to a specific option.
Finally, each product is associated with Y att , a set of attributeshidden from the agent which is extracted from the title and the item-detail pages (§3.2).
The attributesare used for the automatic reward calculation.
An action a ∈ A ( s ) can either be searching a text query (e.g. search[ Red shoes ] ) or clickinga button (e.g. choose[ Size 9 ] ) as shown in Table 1.
These two action types are not availablesimultaneously – search is only allowed when the agent is at the search page; on all other pages,click is the only action choice.
The chosen action argument (button) will be clicked as a web link asopposed to the low-level mouse-click actions in previous environments such as World of Bits [30].
The transitions initiated by clicks deterministically redirect the web page to one of the four pagetypes (Table 1).
The transition initiated by search is based on a deterministic search engine (§ 3.2).
Instruction and reward.
Each natural language instruction u ∈ U contains the following information:a non-empty set of attributes U att , a set of options U opt , and a price u price .
The instruction is generatedbased on a target product y ∗ by human annotators.
The instruction collection process is lightweightand scalable (§3.2).
Concretely, U att ⊆
Y ∗ att is a subset of the product attributes, U opt ⊆
Y ∗ opt is a subsetof the product option field-value pairs, u price > y ∗ price is a price set to be higher than the target productprice.
For example, the instruction “Can you find me a pair of black-and-blue sneaker that is good inrain weather ?
I want it to have puffy soles , and price less than 90 dollars.” contains the aforementionedattributes U att
= { “waterproof” , “soft sole” } and option U opt = { “color”: “black and blue” } .
In eachepisode, the agent receives a reward r = R ( s T , a ) in the end at timestep T , where a = choose[ buy ] ,y is the product chosen by the agent in the final state s T , and Y att and Y opt are its correspondingattributes and options.
The reward is defined as:
[Equation]
where the type reward r type =
TextMatch (¯ y, ¯ y ∗ ) is based on text matching heuristics to assignlow reward when y and y ∗ have similar attributes and options but are obviously different types of products.
For example, “butter” and “plant-based meat” differ in types but may both contain attributes“cruelty-free”, “non-GMO”, and an option “size: pack of 2”.
The exact formula of TextMatch ( · ) isin the Appendix.
We report both a task score ( 100% scaled reward) and success rate (SR) – the ratioof full reward episodes – as our task metrics.
Note that it is possible to get a full score even if thefinal product is not y ∗ . Observation.
Using Flask [29] and OpenAI Gym [4], we provide two parallel observation modesto render the state and instruction S
× I → O : (1) HTML mode that contains the HTML of the webpage, allowing for interaction in a web browser(Figure 1A), and (2) simple mode which strips awayextraneous meta-data from raw HTML into a simpler format (Figure 1B).
The human performancescores in § 4.1 are collected in the HTML mode, while all models are trained and evaluated in thesimple mode.
Note that while the environment allows for training reinforcement learning agentson raw pixels in HTML mode (like in Shi et al.
[30]), we believe that it provides a very low-levelnon-semantic action space. Moreover, it is straightforward to write a translator that converts any new HTML page into simple format for use with trained agents, which enables sim-to-real transfer. 3.2 Implementation Details
Data scraping.
We use ScraperAPI [25] to scrape 1 , 181 , 436 products from amazon.com across5 categories (fashion, makeup, electronics, furniture, and food) using 113 sub-category names asqueries.
More details about product scraping is in the Appendix.
Search engine.
We use Pyserini [18] for the search engine, where indices are built offline using a
BM25 sparse retriever model with text concatenated from the product title, description, overview,and customization options.
While the search engine is deterministic in this work, which in turn easesmodel training and imitation learning, future work could consider a stochastic version to provide agreater challenge for models in terms of generalization and robustness (e.g. by adding “promoteditems” at the top of results, or effecting slight changes in item ranking).
Attribute mining and annotation.
Each product is annotated with a set of hidden attributes , whichare used to represent its latent characteristics as well as to calculate the reward as detailed in § 3.
Anattribute is a short natural language phrase that describes the property of the product (see examples in
Figure 1).
We mine the attributes by calculating TF-IDF scores for all bi-grams in the concatenatedtitles and descriptions based on each product category.
We review the top 200 bi-grams for eachcategory, remove the noisy ones by inspection (decide based on whether the bi-gram is humanunderstandable), and assign them to the products.
We consolidate a pool of 670 attributes.
Natural language instruction.
We use Amazon Mechanical Turk (AMT) to collect natural languageinstructions that specify goal products with appropriate options.
Specifically, an AMT worker ispresented with a sampled goal product, including the product title, category, attributes, and the buyingoptions, and asked to write a command to instruct an automatic shopping agent to find the target.
Workers are instructed to avoid being too specific such as including the entire title in the instruction,but stay faithful to describing the target product.
We collect a total of 12 , 087 linguistically diverseinstructions with an overall vocabulary size of 9 , 036 words and an average length of 15 .
9 words.
Weprovide the detailed annotation process and interface in the Appendix.
3.3 Verification with human studies
Human performance.
We collect trajectories from humans performing the task in HTML mode ofthe environment to understand the task difficulty for humans and analyze how humans solve thetask.
Humans achieve an overall score of 75 . 5 and a success rate of 50 . 0% .
We use qualificationtests to train and select motivated workers to perform the task.
We recruit a total of 13 workers toperform the task, and among them we also report the “expert” performance of the top 7 workers(see Appendix).
The experts achieve an overall score of 82 . 1 and a success rate of 59 .
We alsomake use of this data to train some of our models using imitation learning (§ 4.1).
These trajectoriesare further analyzed in § 5.2, which reveals research challenges in WebShop such as (1) generatinggood search queries, (2) strategic exploration for navigating through the website, (3) robust language
[Table]
+7EsLZL8+T5rda+KMW/v5eOTic1rEM27ADuxDCTziAIziBBgi4gVu4hwfvj3fnPXr/n0dL3nRnC96gBE/YTrVj</latexit> u concat understanding for reading product descriptions and overviews, then choosing the right options, and(4) long-term memory for comparing items (Figure 1). Faithfulness of reward function. We take 100 samples each of trajectories performed by bothaverage and expert humans and manually recalculate the automatic rewards for the selected productaccording to a human grading reward metric; the purpose is to determine whether the reward functionmimics a human’s judgment.
For the sample of average trajectories, the overall score was 74 . 9 andthe human based evaluation was 76 . 3 with a Pearson correlation of 0 . 856 . For expert trajectories,respective scores were 81 .
5 and 89 . 9 with a Pearson correlation of 0 . The source of error ismainly due to lexically similar or synonymous words that did not get points due to the exact matchingcriteria of the reward function.  Methods
We propose models that combine language and image pre-training with imitation learning (IL) andreinforcement learning (RL).
As a lower bound for all other models, a simple rule baseline is tosearch the exact instruction text, then click and buy the first item in the results page without choosingany options.
The heavy lifting of the lexical search engine would lead to a non-trivial attribute reward,but simple heuristic rules cannot resolve noisy natural language options, strategically explore, orlearn to generate what to search, so the total reward and task success rate should be low.
4.1 Imitation Learning (IL)
For the text generation and choice problems presented in WebShop, we propose using two pre-trainedlanguage models to separately learn how to search and choose from human demonstrations.
Imitating human search generation.
We frame searching as a sequence-to-sequence text-generationproblem: the agent generates a search action a = search[ . . .
] given an instruction u withoutconsidering any other context (e.g. past searches, visited items).
We use M = 1 , 421 instructionsearch pairs from 1 , 012 training human trajectories to construct a dataset D
= { ( u, a ) }
Mi =1 andfine-tune a BART model [17] parameterized by ϕ to perform conditional language modeling:
[Equation]
Imitating human choice.
The choice-based imitation model (Figure 2) predicts a probabilitydistribution over all the available click actions
A ( o ) in observation o and maximizes the likelihoodof the human clicked button a ∗ ∈
A ( o ) .
We construct a dataset D ′ =
{ ( o, A ( o ) , a ∗ ) } M ′ i
=1 of
M ′ = 9 , 558 samples from the training human trajectories.
We use a 12 -layer pre-trained BERTmodel [7] parameterized by θ to encode the o into an observation representation of contextualized token embeddings, and we similarly encode each action.
Each action representation is passed intoa cross-attention layer with the observation representation, then mean pooled into a single vectorand multiplied with a matrix W to obtain a scalar score S ( o, a ) .
The policy π θ ( a | o, A ( o )) is thesoftmax distribution over action scores S ( o, a ) :
[Equation]
Handling Images.
We use a pre-trained ResNet-50 [11] to pre-process images across differentproducts and options into a 512 dimensional feature vector, which is then transformed into 768dimensions with a learned linear layer and concatenated to BERT ( o ) as the observation representation.
Full pipeline.
Combining the above during environment interaction, we use the BART model inthe search page to generate the top- 5 search queries via beam search and choose a random one.
For other pages, we sample one action from π θ ( a | o, A ( o )) using the BERT model.
We find thesemethods useful to encourage diverse actions.
In contrast, an ineffective strategy that uses only thetop generated search query or the button with the highest probability might lead to limited productcandidates or being stuck (e.g. bouncing back and forth between pages).
4.2 Reinforcement Learning (RL)
We also fine-tune the choice-based IL model with online RL.
Prior work suggests that directlyfine-tuning text generation via RL might lead to language drifting [15] and deteriorated performance.
Therefore, we freeze the BART model to provide the top- 10 search generationsas a refined action space for the choice-based IL model to learn to pick – an inspiration borrowed from previous work in text games [37] and referential games [15].
We use the policy gradient method [22] with return-to-go R t =
E π [ r t + γR t
+1 ] and a learned value baseline V ( o ) =
W ⊤ v BERT ( o ; θ ) parameterized by { W v , θ } (the BERT weights are tied with thepolicy): L PG =
E π [ − ( R t − V ( o t )) log π ( a t | o t , A ( o t ))] .
The value V ( o ) is learned with an L2 loss L value =
( R t − V ( o t )) 2 .
We also add an entropy loss L entropy =
(cid:80) a ∈A ( o t )
π θ (cid:0) a t |o t , A ( o t ) (cid:1) log π θ (cid:0) a t | o t , A ( o t ) (cid:1) to prevent premature convergence.
Our full RL model minimizesthe total loss L RL = L PG + L value + L entropy .
Experiments
Setup.
We split a total of 12 , 087 instructions into an i.i.d.
distributed train / development / test splitof 10 , 587 / 1 , 000 / 500 instances for reinforcement learning (RL).
We randomly sample a subsetof the 10 , 587 training instructions, then collect 1 , 012 human demonstrations for imitation learning(IL) and a further 54 demonstrations from instances in the development set for IL hyper-parametertuning and checkpoint selection.
We also collect human trajectories for all 500 test instructions andreport human and model performances averaged across these 500 instructions.
We use two evaluationmetrics: (1) Score defined as r × 100 , which captures the total reward obtained; and (2) Success
Rate (SR) defined as the portion of instructions where r = 1 .
More setup details are in the Appendix.
5.1 Results
Main results.
From Figure 3, we observe that the rule-based model obtains a low score of 45 . 6 and avery low success rate of 10% since it cannot resolve options specified in language or explore moreproducts, empirically demonstrating the non-trivial nature of the task.
The IL model significantlyoutperforms the rule baseline on both metrics, achieving a score of 59 . 9 .
Further RL finetuningimproves the score to 62 .
4 while slightly hurting the success rate ( 29 . 1% → 28 . 7% ) (analyzed furtherin §5.2).
We also observe a significant gap between models and humans – our best model’s successrate ( 29 . 1% ) is less than half of expert humans ( 59 . 6% ) and only 60% of the average human ( 50% ). There is also a sizable gap between average and expert human performance, which indicates the taskcould be non-trivial even for humans. [Table] ✓ ✓
[Table]
✓ ✓ [Table] ✓ ✓ ✓ [Table]
✓
[Table]
✓ ✓
[Table]
✓ ✓ ✓ ✓
[Table]
[Table]
[Table]
IL ablations.
Figure 3 also contains several ablations that confirm important design choices formodels.
When the choice action model for the IL agent is randomly initialized ( IL (w/o LP Choice) ),the success rate drops by nearly two-thirds, indicating the importance of language pre-training forour task.
When the search query generator in the IL agent is replaced by a simple rule, which alwaysuses the instruction text ( IL (w/o LP Search) ), both reward and success rate drop by around 3 points.
This suggests the importance to explore by expanding the search space for exploration, but it is not ascritical as learning to choose the right options.
We experiment with incorporating history of one pastobservation and the last five actions into the model and find a slight degradation in the score from59 .
9 to 57 .
3 , suggesting more advanced techniques are needed to leverage past information.
RL ablations.
When we directly train an RL agent ( RL ) from pre-trained BERT parameters, theperformance is even worse than the rule baseline.
This suggests that IL warm-starting is critical,possibly because of the significant domain shift from traditional language tasks.
We also considera simple RL model with RNN text encoders instead of the Transformer ( RL (RNN) ), which hasa success rate more than 10% worse than the IL + RL model with a much larger variance.
Wehypothesize that RL with a more powerful architecture could help boost and stabilize the performanceif the model is initialized with better language and task priors.
5.2 Analysis
To better understand the differences between the agents and human experts, we break down theoverall score into its four sub-parts according to Eq.
(1): 1) attribute score ( | U att ∩ Y att | / | U att | ), 2)option score ( | U opt ∩ Y opt | / | U opt | ), 3) price score ( 1 [ y price ≤ u price ] ), and 4) type score ( r type ).
We alsoreport trajectory statistics such as the average number of states, unique items visited, and number ofsearches per episode (Table 2). We provide qualitative examples of the trajectories in Table 4.
Human expert vs. agents.
Human experts outperform the agents on all score sub-parts (Table 2), butthe most significant boost comes from the option score (a 28% gap), revealing that agents have troubleselecting the correct product options.
Humans also have longer trajectories, explore more items andperform more searches than the agents, with a higher variance, demonstrating their flexibility.
Table 4provides some samples trajectories.
In the first example, the human decides to search again afterremoving ‘inches’, ‘width’, ‘height’, and ‘white’ from the query since product texts often containabbreviated symbols for these terms like ‘"’, ‘w’, and ‘h’.
Thus, search generation is challenging for [Table] models since it involves reasoning and adapting to grounded environments, and ideas from queryreformulation [27, 1] could help alleviate this.
Agents also struggle to perform robust semanticmatching , which is important in choosing options that contain noisy paraphrases of instruction spans. In the second example
[Table]
the human explores several products first
[Table]
and decides to return to the firstexplored product, demonstrating long-term memory that is lacking in the IL+RL model.
Effect of RL fine-tuning after IL.
Table 2 also shows that RL fine-tuning adapts the IL model tobecome more ‘greedy’ and less ‘exploratory’, as the average trajectory length drops from 9 .
4 to 4 . 8 ,and the model explores fewer items and search queries.
As a result, the attribute, type, and pricescores all increase, but option score drops from 45 .
2 to 38 .
This points to the need for a betterbalance exploration with exploitation during RL, e.g. by using intrinsic bonuses.
5.3 Sim-to-real Transfer
Finally, we explore a ‘ sim-to-real ’ transfer scenario where our IL model trained on WebShop is testedon the real Amazon website ( amazon.com ).
We use 100 test instructions and employ the rule baselineand IL model to interact with Amazon, and manually score each episode based on Eq.
As shownin Table 3, the IL model achieves an average score of 61 . 5 , whereas directly typing instruction textinto Amazon search engine (which is much stronger than the WebShop search engine) and selectingthe first item leads to a much lower score of 45 . 8 . The performance also closely match our WebShopreported numbers in Figure 3.
The transfer only requires two minor coding additions, suggesting thatour environment is suitable for grounded agent development that to enable sim-to-real transfer.
Moredetails and example trajectories are provided in the Appendix.
Discussion
We introduced WebShop – a new web-based benchmark for sequential decision making and languagegrounding modeled on interaction with an e-commerce website.
We performed an empirical evaluationof several agents trained using imitation and reinforcement learning, identified several challengesfor future model development and also demonstrated promising results on transfer to real websites.
The modular design of WebShop allows for new tasks and domains to be easily incorporated, whichwe hope will help shape future research into grounded language agents with stronger capabilitiesfor real-world web interaction.
There are some limitations to our work – our data is restricted toproducts from one shopping website (Amazon), contains only English text and our current agents lacklong-term memory and robust reasoning capabilities. And despite being a simulated environment,
WebShop can enable training of agents that operate on the real web, which raises issues around thesafety and human control of RL agents.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] See Section 6 Discussion and
Appendix.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] SeeSection 6 Discussion and Appendix.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
(b) Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main ex-
perimental results (either in the supplemental material or as a URL)?
[Yes] Seesupplementary materials.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
[Yes] Data splits are described in the Section 5 first paragraph.
Hyperparameters and training details are in the Appendix.
(c) Did you report error bars (e
g., with respect to the random seed after running experi- ments multiple times)?
[Yes Figure
includes error bars
Table includes min max
statistics along with
averages
(d) Did you include the total amount of compute and the type of resources used (e g., type
of GPUs, internal cluster, or cloud provider)?
[Yes] In appendix training details.
If you are using existing assets (e g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] Citations include
ScraperAPI, Flask, OpenAI Gym, BERT, BART, A2C.
(b) Did you mention the license of the assets?
[Yes] Discussed in appendix.
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]In the supplementary materials.
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[Yes] Discussed in appendix, we only scrape publicly available data
from the Internet.
(e) Did you discuss whether the data you are using/curating contains personally identifiable [Table]
information or offensive content?
[Yes] Discussed in Appendix. [Table]
If you used crowdsourcing or conducted research with human subjects...
[Table] (a) Did you include the full text of instructions given to participants and screenshots, if [Table]
applicable?
[Yes] In appendix.
[Table]
(b) Did you describe any potential participant risks, with links to Institutional Review
[Table]
Board (IRB) approvals, if applicable?
[Yes] Discussed in Appendix.
[Table]
(c) Did you include the estimated hourly wage paid to participants and the total amount
[Table]
spent on participant compensation?
[Yes] Discussed in Appendix.
[Table]