U NDERSTANDING HTML WITH L ARGE
L ANGUAGE
M ODELS
Anonymous authors
Paper under double-blind review
A BSTRACT
Large language models (LLMs) have shown exceptional performance on a variety of natural language tasks.
Yet, their capabilities for HTML understanding – i.e., parsing the raw HTML of a webpage, with applications to automation of web-based tasks, crawling, and browser-assisted retrieval – have not been fully explored.
We contribute HTML understanding models (ﬁne-tuned LLMs) and an in-depth analysis of their capabilities under three tasks: (i) Semantic Classiﬁcation of HTML elements, (ii) Description Generation for HTML inputs, and (iii)
Autonomous Web Navigation of HTML pages.
While previous work has developed dedicated architectures and training procedures for HTML understanding, we show that LLMs pretrained on standard natural language corpora transfer remarkably well to HTML understanding tasks.
For instance, ﬁne-tuned LLMs are 12% more accurate at semantic classiﬁcation compared to models trained exclusively on the task dataset.
Moreover, when ﬁne-tuned on data from the MiniWoB benchmark, LLMs successfully complete 50% more tasks using 192x less data compared to the previous best supervised model.
Out of the LLMs we evaluate, we show evidence that T5-based models are ideal due to their bidirectional encoder-decoder architecture. To promote further research on LLMs for HTML understanding, we create and open-source a large-scale HTML dataset distilled and auto-labeled from CommonCrawl.
I NTRODUCTION
Web crawling (Olston et al., 2010), form-ﬁlling (Diaz et al., 2013; Gur et al., 2021), or information retrieving web agents (Nogueira & Cho, 2016) are important for both automating and assisting users in web-based tasks.
These and similar applications rely on models that can search for speciﬁc content or controls on a web page as well as navigate a website autonomously.
Since a web page in its raw form is represented as an HTML-based text sequence, the success of models for web-based tasks relies on their ability to understand HTML semantics, structure, and embedded interactions.
The predominant approach to web automation and HTML understanding is to train specialized models, i.e., gathering application-speciﬁc datasets and designing neural network (NN) architectures to leverage inductive biases of the HTML’s structure; see, e.g., Liu et al.
; Toyama et al.
Gur et al.
; Humphreys et al.
However, both dataset collection and neural architecture design are expensive, time-consuming, and require highly-specialized, domain-speciﬁc knowledge.
Meanwhile, in the natural language processing (NLP) literature, large language models (LLMs) have emerged as a solution to the difﬁculties of dataset collection and specialized NN design (Kaplan et al., 2020; Bommasani et al., 2021).
A popular paradigm in NLP is to take an off-the-shelf LLM – pretrained on a large text corpus via an unsupervised and task-agnostic learning objective – and either ﬁne-tune or prompt the LLM on a small task-speciﬁc dataset.
This paradigm has shown exceptional performance on a variety of NLP tasks (Xue et al., 2020;
Brown et al., 2020; Austin et al., 2021).
Whether LLMs can be applied to HTML understanding – especially given the much larger context and sequence lengths – remains an under-explored question.
In this paper, we investigate whether LLMs can be applied to HTML understanding to produce better-performing, more sample-efﬁcient HTML understanding models and without the need for [Figure] custom NN architecture design.
To that end, we present a suite of three benchmarking tasks for
HTML understanding that capture the essence of these applications and require understanding both structure and content.
First, we devise Semantic Classiﬁcation as a task that requires a model to classify a given HTML element into one of a set of categories, such as address, email, password etc., with application to automated form-ﬁlling.
Second, we present Description Generation , a label-extraction task where a model is given an HTML snippet and is asked to produce a natural language description.
For instance for an email ﬁeld, the description might be “Please enter your email address.”
Note that in the majority of web pages, this connection between input elements and description content is only implicit in the raw HTML code and inferring such links is a prerequisite for higher-level navigation objectives.
The third task is Autonomous Web Navigation (Shi et al., 2017).
A model is presented with an HTML page paired with a natural language command and must apply appropriate actions on a sequence of HTML pages to satisfy the command.
See Figure 1a for a simpliﬁed example of these tasks.
With these benchmark tasks in hand, we evaluate the transfer capabilities of a variety of pretrained
LLMs (Table 1), varying in architecture (encoder-only, encoder-decoder, or decoder-only), model size (from 24.6M to 62B parameters), and training data corpora (both including and excluding pretraining NLP and HTML corpus).
While prior work universally pre-parses the HTML as input to the model (Gur et al., 2021; Liu et al., 2018; Nakano et al., 2021), ours – to the best of our knowledge – is the ﬁrst work that uses raw, unprocessed HTML.
Our results show that LLMs demonstrate a remarkable level of HTML understanding across all tasks, with up to 192 × more sample-efﬁciency than models trained from scratch, and achieving a new SoTA for supervised learning on the MiniWoB benchmark suite (Shi et al., 2017).
The encoder-decoder architectures with bi-directional attention show the best performance across the board even when their pretraining does not include HTML.
In addition, we show that the performance scales sub-linearly with the model size.
The broader objective of this research is to advance the integration of LLMs with autonomous web agents. It has only been in the last year that researchers have begun to utilize LLMs outside of NLP and integrate them as core capabilities in autonomy (Lu et al. (2021); Ahn et al.
In this context, LLMs are reasoning engines for sequential decision making agents interacting with environments. The present work is the ﬁrst in the research literature to embed an LLM and train it as an agent for autonomous web navigation. This requires new implementations to adapt LLM training for behavior cloning in addition to designing interfaces for integrating text generation into a perception-compute- action cycle operating in a stateful web environment. Our implementation allows us to answer new questions regarding trade-offs among various model characteristics.
We believe these contributions expand the scope of language models and connect their unique capabilities with autonomous agents for the web. We provide a new perspective on machine learning for HTML understanding and web automation, showing that pretrained LLMs can achieve signiﬁcant performance on such tasks, reducing the need for specialized architectures and training protocols. To encourage further research in this direction, we plan to open-source our dataset for description generation and model weights for agents used in the WoB environment.
R ELATED W ORK
HTML Understanding Autonomous web navigation has been a popular application for neural network models, and a variety of works propose simulated websites for training web-based agents, with application to task fulﬁllment
(Yao et al., 2022;
Gur et al., 2021; Burns et al., 2022; Mazumder &
Riva, 2020; Shi et al., 2017; Liu et al., 2018) as well as information retrieval or question-answering (Adolphs et al., 2021; Nogueira & Cho, 2016).
Simulated websites provide an easy way to evaluate models online, and for this reason we use the existing MiniWoB benchmark (Shi et al., 2017) for our web navigation setting.
However, it is still important to have a mechanism for evaluating models on a wide variety of real-world websites.
This was the key motivation for generating our own dataset for the description generation task, which is distilled and auto-labeled from CommonCrawl and is a key contribution of our paper.
Alongside these benchmarks, many works have developed models for web navigation and related subtasks (Pasupat et al., 2018;
Bommasani et al., 2021;
He et al., 2021; Gur et al., 2021; Humphreys et al., 2022; Liu et al., 2018;
Jia et al., 2019).
These works often rely on specialized neural network architectures that leverage inductive biases of HTML structure, or on preprocessing of HTML to make it easier to input to a model (Li et al.
(2021a;b)).
In contrast, our work takes a minimalist approach, providing HTML in text form with minimal processing and using widely-adopted transformer networks.
LLMs and HTML Works that explore the intersection of LLMs and HTML generally fall into two categories.
The ﬁrst category uses LLMs to assist web navigation (Nakano et al., 2021; Yao et al., 2022), and typically relies on a custom preprocessing to map the context and structure of a web page to natural language, thus severely restricting what HTML pages the model can parse.
The second category pretrains LLMs on a large corpora of HTML text (Aghajanyan et al., 2021).
However, these works typically restrict the model evaluation to standard NLP tasks, e.g., summarization and question/answering as opposed to tasks more relevant to HTML understanding and web automation.
Our work can be thought of as the reverse: We keep the pretraining of LLMs unchanged and focus on the mechanisms for transferring the pretrained LLMs to HTML-relevant tasks.
B RIEF B ACKGROUND ON HTML AS S EMI -S TRUCTURED T EXT D ATA
HTML is a markup language, used to organize web page structure and content .
Consider the example HTML page in Figure 1a.
This web page includes two adjacent input elements, one for e-mail and another for password, with their corresponding label s on a separate branch of the page.
These input s and label s are one of many possible elements that serve as HTML building blocks.
Each element has a set of attributes – key and value pair – that describe the element’s content, such as style and human-readable text.
When rendered in a browser, these attributes will be responsible for how the element is shown and where it is positioned.
In the example in Figure 1a, the ﬁrst input has three attributes, tag="input" , type="email" , and id="uName" , that identify the element as an email input with an identiﬁer (“uName”) that can be accessed programmatically.
C ANONICAL T ASKS FOR HTML U NDERSTANDING
We devise three canonical tasks to study HTML understanding capabilities of LLM-based web agents.
These tasks require correctly interpreting both structure and content to varying degrees to make predictions, with autonomous navigation being the most challenging capability of the three.
[Table]
Dataset
Size
Input
Architecture
Output
Task Output
[Table]
Navigation
MiniWoB Demos (Shi et al., 2017) 12K
Page
Enc-Dec
Text
Dictionary Dec
[Table]
Annotated Shopping Webpages (Gur et al., 2021) 28K
Snippet
All
Text
Category
[Table]
CommonCrawl (new) 85K
Snippet
Enc-Dec
Text Text Dec Autonomous Web Navigation . This task evaluates how well a model navigates multi-page websites as a sequential decision-making problem (Shi et al., 2017; Liu et al., 2018).
At the beginning of an episode, the agent is given a natural language instruction, e.g. Enter the username “lyda” and the password “N22t” into the text ﬁelds and press login . The agent applies actions to a sequence of HTML pages, where each action is of the form function(selector, text) . The function is one of click or type , selector is an integer pointer that uniquely identiﬁes an element, and text is a text to input if the type functionality is activated. An episode terminates when either the page reaches a terminal state (e.g., the ‘sign in’ button is clicked) or the maximum number of steps is reached.
Semantic Classiﬁcation .
Many HTML understanding applications require a model that can classify HTML elements into standardized categories. For example, in automated form-ﬁlling (Diaz et al., 2013; Gur et al., 2021), it is useful to identify a ‘submit button’ across many websites (e.g., shopping, ﬂight booking, utility application) with various button representations (e.g., position, color, or text). Thus, we formulate Semantic Classiﬁcation as classifying elements into role categories.
Take the example HTML in Figure 1a which includes two input elements and a submit button .
Let’s pick the ﬁrst input as an element of interest to be classiﬁed by the system, also called a salient element .
The system should classify this element as username , since it appears on a login page and it has a label with Email Address which is typically associated with the username in form-ﬁlling applications.
To solve this, the system can aggregate information from multiple sources in the page – the label that says Enter Email Address , the input attributes ( type=“email” and id=“uName” ), or even the ordering of other elements in the page such as ‘password’ and ‘sign in’.
Description Generation .
Motivated by applications in accessibility-minded web browser control (Jorgensen & Binsted, 2005), we formulate description generation as an extractive problem where the goal is to locate the textual description of an element in the HTML and generate it as output.
For instance, the description of the salient element in Figure 1a is Enter Email Address ; when rendered, this label will appear above the ‘email’ input ﬁeld.
HTML provides a large amount of ﬂexibility, and so in general a descriptive text that appears alongside a speciﬁc element when rendered can be very far from that element when looking at the HTML plaintext.
Thus, this task evaluates a model’s ability to understand the structure of HTML as it would appear to a user, despite not having access to the rendered web page directly.
D ATASETS
Each of our canonical tasks requires a separate dataset, with the description generation task using a newly contributed, auto-labelled dataset based on CommonCrawl.
Autonomous Web Navigation .
We use the 12K demonstrations included in the publicly available
MiniWoB benchmark (Shi et al., 2017), which encompass 62 website applications ranging from email forwarding to social media interactions. Each demonstration is a sequence of (instruction,
HTML, action) tuples.
Every element in a MiniWoB demonstration is accompanied by a reference number unique within its respective pages.
This number can be used as an element selector, making the action space uniﬁed across all tasks and time steps.
For instance, the action in Figure 1a would be type(ref=5, ”username@email.com”) , where 5 refers to the index of the input when counted from
Semantic Classiﬁcation .
We use a dataset of 28K labelled examples, containing 66 different categories, of the form (HTML, element, category) , previously used in the context of environment generation (Gur et al., 2021).
The dataset consists of HTMLs from real-world shopping websites and categories relevant to form-ﬁlling during payment and checkout on these websites.
Description Generation .
For this task, we derive a dataset from CommonCrawl.
2 CommonCrawl does not include renderings or annotations that would reveal what text in the HTML is associated with which elements.
Instead, we infer descriptions of various elements by exploiting a special attribute in the HTML schema known as for .
As an example in Figure 1a, the ﬁrst label in the HTML has a for attribute with value uName , which is the id of the element described by label ; in this case, the id is that of the ﬁrst input in the page.
This annotation does not affect the rendering of the page and is typically used for accessibility purposes. We utilize the information given by these for attributes to create a large-scale dataset to study description generation.
A small sample is available in the supplemental material, while the entire dataset will be available upon publication.
Speciﬁcally, we collected 100 WARC (from April 2019) ﬁles from the CommonCrawl project and extracted all HTML label s that have a for attribute.
Removing non-Unicode and alphanumeric text in HTML label s results in a 400K example datset.
We balance the distribution of labels, effectively downsampling the dataset to 85 K samples.
Each example is represented as (HTML, element, description) , where HTML is the HTML plaintext of the page, element is the element whose id attribute matches that appearing in the label ’s for attribute, and description is the text inside the label element (see example in Figure 1a).
More details of the dataset can be found in Appendix A.1.
P RE -P ROCESSING
In treating HTML as token sequences, we minimize any HTML tree pre-processing prior to model input.
We thus provide HTML as raw text (i.e., sequences of text tokens) and only apply a snippet extraction pre-processing for pages which are too large to ﬁt into the typical LLMs context windows.
Snippet Extraction.
Real HTML pages can grow extremely large, reaching thousands of elements, far beyond the context window of the largest LLM that we studied (1920 tokens in PaLM (Chowdhery et al., 2022)).
LLMs typically truncate such long sequences, which can be detrimental to HTML understanding as HTMLs are not linearly structured.
We take an element-centric approach and extract HTML snippets (a small portion of HTML code) surrounding a salient element (Figure 5).
A simple heuristic, which controls the tree’s width and depth, guides the process: Start with a salient element and traverse its ancestors in the HTML tree until a stopping condition is satisﬁed.
As we traverse up, we estimate the height of the tree and the increased number of descendants of the new root.
We stop when either metric violates a pre-deﬁned limit and take the resulting sub-tree as the snippet.
We mark the salient element using a special attribute, called target , to distinguish it from other elements. We perform the snippet extraction for the semantic classiﬁcation and description generation datasets, and keep the full HTML pages in MiniWoB because these pages are typically much smaller than real-world HTML.
HTML un-Parsing.
We provide the models with the unparsed plaintext HTML in the form of a sequence of tokens.
This canonical representation does not require speciﬁc model architectures such as hierarchical networks (Liu et al., 2018;
Gur et al., 2021) and can be fed into any LLM.
We transform all datasets by converting every HTML page or snippet into a sequence.
For MiniWoB, we additionally concatenate (action history, instruction, HTML) tuples into a single sequence.
M ODEL T RAINING
We study a variety of transformer-based LLMs (Vaswani et al., 2017) with different sizes and architectures for HTML understanding tasks (Table 1).
In the rest of the text, we preﬁx models ﬁne-tuned for Autonomous Web Navigation , Description Generation , and Semantic Classiﬁcation with WebN- , WebD-, and WebC-, respectively.
For instance, WebD–T5-3B is the three billion parameter T model (Raffel et al., 2020)
[Figure]
Encoder-Decoder and Decoder-only Models.
We train encoder-decoder models, i.e., T5 (Raffel et al., 2020), and decoder-only models, i.e., LaMDA (Thoppilan et al., 2022) and PaLM (Chowdhery et al., 2022), with text input and text output (Figure 1b).
Inputs are raw HTML pages or snippet texts; similarly, outputs are categories, natural language descriptions, or actions represented as text.
Namely, for Semantic Classiﬁcation we use the textual representation of categories, similar to previous classiﬁcation problems in NLP (Raffel et al., 2020).
For Autonomous Web Navigation , actions are converted into text by ﬁrst converting them into key and value pairs and then concatenating the pairs.
Many websites in MiniWoB require multiple interactions, such as click-button-sequence or clickcheckboxes , where each interaction might cause a subtle change in the website state.
For instance, after clicking on a checkbox in the click-checkboxes website, its value ﬂips from positive to negative or the other way around, which is not always reﬂected in LLMs’ predictions and leads to action repetitions.
We solve this issue by augmenting tuples in the dataset with a sequence of past actions, (action history, instruction, HTML, action) , and allowing LLMs to learn from past experience.
Encoder-only Models.
We train encoder-only models, i.e., BERT (Devlin et al., 2018), with text input and categorical output.
We keep semantic categories as discrete one-hot classes.
To train encoder-only models, we add a new classiﬁcation layer after the ﬁnal encoder layer to produce a distribution over semantic categories.
In addition to the typical BERT models, we study MobileBERT (Sun et al., 2020), distilled from BERT-large with inverted bottlenecks, and Albert-XL (Lan et al., 2020), with parameter sharing and embedding split.
R ESULTS
We now present the results of ﬁne-tuned LLMs for HTML understanding.
We compare the models’ performance with the existing baselines where possible (autonomous web navigation) and against other LLM architectures and training regimes (all tasks).
Sections 8.1, 8.2, and 8.3 evaluate taskspeciﬁc performance, while Section 8.4 assesses the performance across all the tasks.
Metrics: For autonomous web navigation we evaluate models’ Success Rate , which is averaged over 100 episodes per task.
For the other tasks, we use Accuracy to measure exact match between prediction and ground truth.
In the description generation task, we additionally provide evaluations using alternative ‘soft’ text evaluation metrics, BLEU and ROUGE-1 , measuring the similarity between predicted and ground truth text.
8. A UTONOMOUS W EB N AVIGATION R ESULTS
For Autonomous Web Navigation we ﬁne-tune two WebN- encoder-decoder architectures (WebNT5-large and WebN-T5-3B) on 12k demonstrations from human-annotated real websites.
We evaluate the models on MiniWob (Liu et al., 2018) benchmark, and compare with specialized architectures trained using supervised learning (SL) on 2.4 million human expert demonstrations CC-Net [Table] models bootstrapped with SL, CC-Net (SL) (CC-Net (SL & RL) (Humphreys et al., 2022), and WGE (SL & RL) (Liu et al., 2018)).
Additionally, we compare with the decoder-only architecture (WebN-Lambda-1B) and perform an ablation study on the impact of including the action history in the input.
Comparison to SoTA.
Since previous works report success on only a subset of websites in MiniWoB, we evaluate on 48 out of 62 websites that are common across all models.
Table 8 in the Appendix reports ﬁne-grained results while Figure 2a presents results averaged over all websites.
Compared to CC-Net (SL) which is trained on all 2.4M demonstrations, WebN-T5-3B improves the success 16% while only training on 12K publicly-available demonstrations, yielding over 192x improvement in sample-efﬁciency.
We ﬁnd that all choices of LLMs outperform previous SL models.
Notably, WebN-T5-3B signiﬁcantly improves on websites requiring multiple-action sequences such as click checkboxes or websites requiring entering text such as login user (Table 8).
We observe that the performance of LLMs is only surpassed by previous works utilizing RL, which require orders of magnitude more online experience interaction.
Extending our ﬁne-tuned LLMs to an RL setting is a promising avenue for future work.
Action history ablation.
Across all LLMs we consistently observe a decrease in success, on average 6.4%, when past actions are excluded from the inputs (Figure 2a).
Action history helps with websites that require entering multiple texts, as well as understanding minor changes that could be difﬁcult to detect (e.g. click checkboxes and multi layout ).
multi layout requires entering 3 different texts in the website where the layout is randomized at each episode, yet, surprisingly, even the (relatively smaller) WebN-T5-large model without action history outperforms the CC-Net (SL) model; illustrating that incorporating action history is not the only contributing factor for the better success.
8. S EMANTIC C LASSIFICATION T ASK R ESULTS
To evaluate the Semantic Classiﬁcation task, we compare the T5 encoder-decoder architecture’s three size variants (WebC-T5-base, WebC-T5-large, and WebC-T5-3B) ﬁne-tuned on 22K real, human-labeled training websites.
We compare with a ﬁne-tuned encoder only architectures (WebC-*BERT*), three ﬁne-tuned decoder-only architectures (WebC-LaMDA and PaLM), and both encoder-decoder and decoder-only models trained on human labeled websites from scratch. Results are presented in Table-2, where we ﬁnd that all WebC-LLMs perform well and signiﬁcantly better than the same architectures without pretraining.
Accuracy per category.
In Figure 3, we present accuracy distribution of the WebC-T5-3B model on the development dataset. The ﬁne-tuned encoder-decoder model performs strongly on a majority
[Figure]
[Figure]
[Table]
of the categories (Figure 3), even on those with very few samples.
For instance, the model is 100% accurate on password new which has only 56 training examples, because the class is unambiguous.
On the other hand, unsurprisingly, the performance drops when the category is ambiguous, such as in the email category which is frequently mistaken as username .
Snippet generation ablation.
Two hyper-parameters govern snippet generation: percentage of new descendants and height of the new root.
While small variations of both parameters do not change the performance, increasing both degrades the performance signiﬁcantly (Table 4a).
With new descendants up to 500% and height up to 7, the performance drops by more than 15%.
Note that snippet generation returns the full-page HTML when both parameters increase indeﬁnitely.
Data size impact.
When varying the ﬁne-tuning training data sizes (1, 5, 10, 20, or 50 samples per class) in Figure 4b, WebC-T5-3B slightly outperforms WebC-PaLM-8B which is an order of magnitude larger.
Compared to T5-3B that is trained on all available HTML data without pretraining,
WebC-T5-3B achieves better performance while using only 3.4% of labeled data (1000 samples), thus highlighting the beneﬁt of using standard off-the-shelf pretrained LLMs for HTML understanding.
8. D ESCRIPTION G ENERATION T ASK R ESULTS
For Description Generation we split the CommonCrawl dataset based on URL top-level domains to test LLMs’ capabilities to generalize to unseen HTML.
We ﬁne-tune encoder-decoder architectures (WebD–T5*) and decoder-only models (WebD–LaMDA*), with results presented in Table 3.
We also evaluate a strong heuristic baseline which simply ﬁnds the description closest to the salient element in the HTML text (Closest Description).
Accuracy and Similarity Performance We show results of our evaluations in Table 3. All models achieve high scores across all metrics, achieving ≈ 84% on the accuracy in terms of exact match and a higher non-exact match score based on BLEU and ROUGE-1 ( ≈ 91% ).
This difference indicates that the models are capable of locating the descriptions, but not always generating the exact output.
8. HTML U NDERSTANDING LLM S P ERFORMANCE A NALYSIS A CROSS T ASKS
We now analyze our results in aggregate to derive our main conclusions. 8.4. P RETRAINING E FFECT : P RETRAINING ON L ARGE T EXT C ORPORA M ATTERS
Fine-tuned pretrained LLMs outperform LLMs trained on HTML-only data, improving the performance by more than 34.1% on the Autonomous Web Navigation (Table 2b), and 10% to 12.7% on the Semantic Classiﬁcation task (Table 2).
Since Autonomous Web Navigation is the most difﬁcult task, the improved performance is an encouraging evidence of the value of LLMs in HTML understanding tasks.
Speciﬁcally, we observe that LLMs without pretraining are comparable to ﬁne-tuned pretrained models only on websites that require simple text matching.
In contrast, for websites such as click checkboxes , text matching is harder and we ﬁnd that pretraining is key to good performance.
We also found that without pretraining, model outputs were frequently in an incorrect format such as invalid dictionaries or invalid refs with non-integer values.
This suggests that the large corpora used for pretraining helps models to learn general HTML structure.
8.4. A RCHITECTURE E FFECT : T5- BASED M ODELS P ERFORM B EST A CROSS A LL T ASKS
Encoder-decoder T5 based models perform better across all three tasks.
On the Autonomous Web
Navigation task, encoder-decoder (WebN-T5) architectures are better or comparable to WebNLaMDA-1B (Figure 2a).
On the Semantic Classiﬁcation , the smallest encoder-decoder model (WebC-T5-base) performs comparably to much larger decoder-only models (WebC-LaMDA-1B or
WebC-PaLM-8B) and the largest encoder-only model (WebC-BERT-large) which has 85M more parameters (Table 2).
We also observe that decoder-only PaLM-8B performs worse than much-smaller encoder-decoder T5-large when trained only on HTML data.
Finally, on the Description Generation encoder-decoder architecture has higher BLEU score.
One possible explanation for the strong performance of T5-based moels is the encoder-decoder architecture of these models. Namely, T5 models utilize an encoder with a bidirectional attention mechanism, not present in the LaMDA and PaLM decoders.
The bidirectional attention mechanism can process HTML pages from both ends, potentially overcoming the loss of information when tree-structured HTML pages are converted into a ﬁxed linear text sequences.
8.4. M ODEL S IZE E FFECT : S IZE (S UB - LINEARLY ) M ATTERS
Across the tasks it appears that the architecture plays an important role in the model performance.
Model size and performance are also positively correlated, although they reach diminishing returns.
For instance, the model performance is roughly O (log log n ) with respect to model size on Semantic Classiﬁcation (Figure 4b in Appendix).
On the Autonomous Web Navigation task, performance grows slowly with the model size (Table 8), while on the Description Generation it plateaus (Table 3).
8. D ISCUSSION Bi-directional attention vs training corpora
: Pretraining on large corpora matters, yielding ≤ 4.5x performance improvements.
Larger models tend to be better and we credit the bidirectional attention for T5’s best overall performance across the tasks.
PaLM and LaMDA include HTML and other code in their pretraining corpora, while BERT and T5 architectures did not, showing that pretraining on HTML is not necessary for strong performance when ﬁne-tuned for HTML understanding.
This strengthens the hypothesis behind the role of the bidirectional attention, and opens up the possibility to further improve the performance of T5 architectures by pretraining them on corpora with HTML.
Practical impact on labeling: When available, the pretrained LLMs need very little new expert data (200x and 30x reduction on the web navigation and classiﬁcation tasks, respectively).
This has a big potential impact on practical applications, reducing the data collection time and cost by orders of magnitude.
Bigger is not always better: When choosing the model size, the expected performance gains (sublinear at best and asymptotic at worst) should be considered alongside the model’s training and inference time and cost.
For instance, on the classiﬁcation task, the largest model WebC-PaLM-62B takes several days to ﬁne-tune, and evaluates at 30 Hz, while WebC-T5-large ﬁne-tunes in several hours and evaluates at 700 Hz – an order of magnitude more expensive for a single percent uplift in accuracy.
BERT models on the other hand train in minutes.
If the application does not require high precision, these might be a good choice.
Context window is a bottleneck: The major bottleneck for the HTML understanding tasks seems to be the context window length that the current LLMs support, even with models that accept 1000+ tokens.
It remains prohibitive to evaluate web navigation tasks on real websites that are orders of magnitude larger than pages in MiniWob.
Similarly, we observed that increasing the snippet size leads to major performance degradation.
This makes HTML understanding an interesting benchmark for future LLM development.
For instance, new methods may need to be developed to compress the state representation of web content for use in LLM context windows.
C ONCLUSION
We presented canonical tasks and ﬁne-tuned LLMs for HTML understanding.
The comprehensive evaluations and analyses over a range of architectures, dataset sizes, and baselines yields practical ﬁndings and highlights current limitations of these models.
We ﬁnd that a) pretraining is critical for the performance and can reduce labeled data requirements, improving sample efﬁciency up to 200x; b) model architecture is the second-most important factor, and T5 models with bidirectional attention and encoder-decoder architecture perform the best across the board; c) given a choice, model size should be evaluated in the context of the model’s training and inference performance, as the model size sub-linearly correlates with its performance.
Finally, the proposed HTML understanding tasks highlight the relatively short context window that limits current LLMs, suggesting possibilities for future research that incorporate or eliminate this constraint
A A PPENDIX A. D ATASET D ETAIL
Examining the description distribution we found the original 400 K dataset to be very skewed; only 20 descriptions (such as Email and Password ) were covering 50% of the dataset.
We sub-sampled the dataset so that each unique description has at most 10 data points.
We also found that for attributes are almost always deﬁned for HTML label s. This could cause a model to overﬁt and just ﬁnd the label element in the HTML and ignore everything else.
To avoid this sort of ‘cheating’ we replace the tags of HTML label s by randomly sampling from { div, span, a, label } .
These tags are also frequently used to inject text in HTML but they are very rarely used with for attributes.
Finally, we removed examples where there are only a single text in the HTML since models can trivially generate descriptions by ﬁnding the only text in the HTML, which biases model weights and evaluation metrics.
After this ﬁnal step, we have a total of 85 K labeled examples.
A.1. S NIPPET G ENERATION
In Figure 5, we give a high-level overview of our snippet generation procedure.
A. A DDITIONAL R ESULTS A.2. S EMANTIC C LASSIFICATION
Error Analysis.
We manually examined 50 errors of T5-3B model over the development set (Table 4) and assigned them into one of the 9 error types that we devised.
We found that 32% of the errors are due to lack of information in the HTML snippets, which is mainly the result of lost information during snippet extraction process.
Annotation errors or email/username ambiguity make up 30% of the errors.
These can’t be improved without revising the annotated data or adding extra information to resolve the ambiguity.
We also found that the model sometimes picks a more general category, or a nearby text misleads the model; the latter usually happens when the HTML snippet is long where majority of the elements are noise.
Error Type
[Table]
Not enough information in the HTML snippet Incorrect annotation (ex: ”unknown role” instead of ”organization”) Annotation tool translates user selection incorrectly
Email/Username ambiguity More general category (ex: ”header” instead of ”cart header”) Immediate neighboring
[Table]
misleads
[Table] [Table]
information
[Table]
Others Few-Shot Prompting In Table 5, we present few-shot prompting performance of a 540B PaLM model.
We probe the model using a prompt template <html> Role: <category> with 1 example per category and generate categories using greedy-decoding.
In our preliminary experiments, we found that few-shot prompting achieves only 45.6 accuracy, much lower than a model ﬁne-tuned on the same data (Figure 6).
We found two common problems – the model is not able to canonicalize predictions into categories and many of the examples are dropped due to context length.
[Table]
We developed post-processing methods to alleviate the canonicalization problem and preprocessing methods to reduce lengths of examples.
Adding a dictionary-based mapping on predictions – a manually curated paraphrase dictionary – improves the performance to 52.1.
We also tried rewriting predictions by changing the order of tokens around ” ” such as name ﬁrst to ﬁrst name which further improved the performance to 57.9.
Finally, we cleaned examples in the prompt by removing certain elements such as ”svg”, ”path”, ”img” , and ”iframe” and also removing class attribute from every element; this pre-processing step gives 64.2.
A. S AMPLE E PISODES FROM M INI W O B
See Table 6 for an example episode of web navigation inferred by a ﬁne-tuned LLM.
A. D ETAILED M INI W O B R ESULTS
See Table 7 for detailed performance of various models on MiniWob.
A. R ESOURCE R EQUIREMENTS
See Table 8.
A. S TRUCTURE D EPENDENCE A BLATION S TUDY
We conducted an ablation study to examine the sensitivity of model performance to preserving structural information.
To do so, we evaluate the model’s performance on HTML input with critical structure components removed.
We kept the order of elements and their attributes ﬁxed while corrupting the nesting structure by removing closing tags. Removing closing tags corresponds to a valid traversal (BFS) and keeps the order of elements the same as the text based input. As a simple example: [Equation]
would be converted into:
[Equation]
We evaluated the trained WebN-T5-3B model on the same set of synthetic websites from the
MiniWoB benchmark with this aspect of structure removed from the HTML pages.
WebN-T53B achieves a 45.4% success rate, 6% lower than before, suggesting that WebN-T5-3B is at least partially dependent on the DOM topology.
A. T ASK - SPECIFIC M ODELS
An alternative to LLMs is to adapt bespoke task-speciﬁc architectures tailored towards processing of structured documents and HTML (Li et al.
(2021b;a)).
StructuralLM
(Li et al.
(2021a)) is an approach speciﬁcally tailored for document understanding (i.e., combinations of images and text), and thus makes several simplifying assumptions for its model that limit its applicability to HTML understanding (i.e., trees of elements with a richer structure and functionality).
It is trained only on the textual content of a document - the markup information is ignored.
For example, any input ﬁeld or dropdown in a document would be missing from the model inputs.
All of the tasks we study require knowledge of this information.
For example, in autonomous navigation the model needs to interact with input elements (e.g. text, checkboxes, dropdowns) such as username and password in the login-user task in MiniWoB.
Typically, a “type” action with a reference to an element and a text argument is generated by the model.
Without knowing which input elements are available in the page, it is impossible to generate a reference to any input element.
While MarkupLM (Li et al.
(2021b)) is better tailored for understanding HTML pages, it has similar drawbacks as StructuralLM in that it focuses solely on text while ignoring the markup. It is evaluated on NLP-like tasks such as QA or entity classiﬁcation where understanding page content is paramount, whereas we focus on HTML understanding tasks such as autonomous navigation where both content and the page’s layout structure need to be understood. We perform a quantitative evaluation of MarkupLM on our tasks to understand how signiﬁcant these limitations are. We ﬁne-tune the MarkupLM-base model on the semantic classiﬁcation task, using the same setup as other WebC models but with the suggested hyperparameters from (Li et al.
(2021b)).
On development and test sets, MarkupLM-base achieves 65% and 66% accuracy, respectively.
These results are more than 16% lower compared to similar size WebC-BERT-base results that we report in our work.
This suggests that although domain speciﬁc models may be suitable for processing HTML for NLP tasks, the generality, ﬂexibility, and sample efﬁciency LLMs provide advantages for autonomous navigation tasks.
[Table]
[Table]
[Table]
[Table]
[Figure]
n/a
[Figure]
n/a n/a n/a n/a n/a n/a n/a n/
[Figure]
n/a n/a n/a n/a n/a n/a n/a n/a
[Figure]
n/a n/a
[Figure]
/a n/a n/a n/a n/a n/a n/
[Figure]
n/a
[Figure]
n
[Figure]
n
[Figure]
a n/a n/a n
[Figure]
a n [Figure] n
[Figure]
a n/
[Figure]
n/a
[Figure]
n/a
[Figure]
n [Figure] [Figure] n/a
[Figure]
/a [Figure] n/
[Figure]
n/
[Figure]
n/
[Figure]
n/a n/a
[Figure]
n/a n/a n/a n/a n/a
[Figure]
[Figure]
/a
[Figure]
n/
[Figure]
n/a n/a n/a
[Figure]
[Figure] [Figure]
/a
[Figure]
n/a n/a n/a
[Figure]
n/a n/a
[Figure]
n/a n/a n/a n/a n/a n/a n/a n/
[Figure]
/a n/a n/a n/a n/a n/a n/a n/
[Figure]
/a n/a
[Figure]
n/a n/a n/a n/a n/a n/a n/a n/a
[Figure]
n/a
[Figure]
n/a n/a n/a n/a n/a n/a n/a n/a
[Figure]
[Figure]
n/a
[Figure]
n/a n/a
[Figure]
n/a 0.
[Figure]
n/a n/a
[Figure]
n/a n/a n/a n/a n/a n/a n/a n/
[Figure]
n
[Figure]
a n/a  n/a n/a n/a n/a n/a n/a n/a n/a
[Figure]
n/a n/a
[Figure]
n/a n/a n/a n/a n/a n/a n/a n/a
[Figure]
n/a
[Figure]
n
[Figure] [Figure]
n/a
[Figure]
[Figure]
a
[Figure]
/a
[Figure]
n/a
[Figure]
/a
[Figure]
/a n/a n/a
[Figure] [Figure]
[Figure]
n/a  n/a
[Figure]
[Figure]
[Figure]
n/a n/a 0. n/a 0.
[Figure]
/a n/a 0.
[Figure]
n/a n/a n/a n/a n/a 0.
[Figure]
[Figure]
[Figure]
/a
[Figure]
/a n/a n/a
[Figure]
n
[Figure]
n
[Figure] n/ [Figure] n/
[Figure]
n
[Figure]
n/
[Figure]
[Figure] [Figure] [Table]