Understanding HTML with Large Language Models (2023)

Abstract: Large language models (LLMs) have shown exceptional performance on a variety of natural language tasks. Yet, their capabilities for HTML understanding – i.e., parsing the raw HTML of a webpage, with applications to automation of web-based tasks, crawling, and browser-assisted retrieval – have not been fully explored. We contribute HTML understanding models (fine-tuned LLMs) and an in-depth analysis of their capabilities under three tasks: (i) Semantic Classification of HTML elements, (ii) Description Generation for HTML inputs, and (iii) Autonomous Web Navigation of HTML pages. While previous work has developed dedicated architectures and training procedures for HTML understanding, we show that LLMs pretrained on standard natural language corpora transfer remarkably well to HTML understanding tasks. For instance, when fine-tuned on data from the MiniWoB benchmark, LLMs successfully complete 50% more tasks using 192x less data compared to the previous best supervised model. We create and open-source a large-scale HTML dataset distilled and auto-labeled from CommonCrawl

## introduction web crawling @xcite , formfilling @xcite , or information retrieving web agents @xcite are important for both automating and assisting users in web-based tasks. these and similar applications rely on models that can search for specific content or controls on a web page as well as navigate a website autonomously. since a web page in its raw form is represented as an html-based text sequence, the success of models for web-based tasks relies on their ability to understand html semantics, structure, and embedded interactions. the predominant approach to web automation and html understanding is to train specialized models, i.e., gathering application-specific datasets and designing neural network (nn) architectures to leverage inductive biases of the html's structure; see, e.g., @xcite ; @xcite ; @xcite ; @xcite . however, both dataset collection and neural architecture design are expensive, time-consuming, and require highly-specialized, domain-specific knowledge. meanwhile, in the natural language processing (nlp) literature, large language models (llms) have emerged as a solution to the difficulties of dataset collection and specialized nn design @xcite . a popular paradigm in nlp is to take an off-the-shelf llm -pretrained on a large text corpus via an unsupervised and task-agnostic learning objective -and either fine-tune or prompt the llm on a small taskspecific dataset. this paradigm has shown exceptional performance on a variety of nlp tasks @xcite @xcite . whether llms can be applied to html understanding -especially given the much larger context and sequence lengths -remains an under-explored question. in this paper, we investigate whether llms can be applied to html understanding to produce better-performing, more sample-efficient html understanding models and without the need for custom nn architecture design. to that end, we present a suite of three benchmarking tasks for html understanding that capture the essence of these applications and require understanding both structure and content. first, we devise semantic classification as a task that requires a model to classify a given html element into one of a set of categories, such as address, email, password etc., with application to automated form-filling. second, we present description generation, a label-@xmath0: </@xmath1. </@xmath2: html example page with a highlighted salient element, an element of interest (dashed box). all canonical tasks evaluate a distinct interaction with this element, either by classifying it as one of a set of categories, generating a text description of its purpose, or applying an action as part of a sequential navigation of a multi-page website. with these benchmark tasks in hand, we evaluate the transfer capabilities of a variety of pretrained llms ( the broader objective of this research is to advance the integration of llms with autonomous web agents. it has only been in the last year that researchers have begun to utilize llms outside of nlp and integrate them as core capabilities in autonomy @xcite ). in this context, llms are reasoning engines for sequential decision making agents interacting with environments. we believe these contributions expand the scope of language models and connect their unique capabilities with autonomous agents for the web. we provide a new perspective on machine learning for html understanding and web automation, showing that pretrained llms can achieve significant performance on such tasks, reducing the need for specialized architectures and training protocols. to encourage further research in this direction, we open sourced foot_1 model weights for agents used in the wob environment and our dataset for description generation. alongside these benchmarks, many works have developed models for web navigation and related subtasks @xcite @xcite @xcite @xcite . these works often rely on specialized neural network architectures that leverage inductive biases of html structure, or on preprocessing of html to make it easier to input to a model ( @xcite ). in contrast, our work takes a minimalist approach, providing html in text form with minimal processing and using widely-adopted transformer networks. llms and html works that explore the intersection of llms and html generally fall into two categories. the first category uses llms to assist web navigation @xcite , and typically relies on a custom preprocessing to map the context and structure of a web page to natural language, thus severely restricting what html pages the model can parse. the second category pretrains llms on a large corpora of html text @xcite . however, these works typically restrict the model evaluation to standard nlp tasks, e.g., summarization and question/answering as opposed to tasks more relevant to html understanding and web automation. our work can be thought of as the reverse: we keep the pretraining of llms unchanged and focus on the mechanisms for transferring the pretrained llms to html-relevant tasks. ## canonical tasks for html understanding we devise three canonical tasks to study html understanding capabilities of llm-based web agents. these tasks require correctly interpreting both structure and content to varying degrees to make predictions, with autonomous navigation being the most challenging capability of the three. autonomous web navigation. this task evaluates how well a model navigates multi-page websites as a sequential decision-making problem @xcite . at the beginning of an episode, the agent is given a natural language instruction, e.g. enter the username "lyda" and the password "n22t" into the text fields and press login. the agent applies actions to a sequence of html pages, where each action is of the form function(selector, text). the function is one of click or type, selector is an integer pointer that uniquely identifies an element, and text is a text to input if the type functionality is activated. an episode terminates when either the page reaches a terminal state (e.g., the 'sign in' button is clicked) or the maximum number of steps is reached. ## datasets each of our canonical tasks requires a separate dataset, with the description generation task using a newly contributed, auto-labelled dataset based on commoncrawl. autonomous web navigation. we use the 12k demonstrations included in the publicly available miniwob benchmark @xcite , which encompass 62 website applications ranging from email forwarding to social media interactions. each demonstration is a sequence of (instruction, html, action) tuples. every element in a mini-wob demonstration is accompanied by a reference number unique within its respective pages. this number can be used as an element selector, making the action space unified across all tasks and time steps. for instance, the action in figure 1 would be type(@xmath3, "username@email.com"), where 5 refers to the index of the input when counted from top-to-bottom. as model input, we concatenate the natural language instruction and html into a single text input sequence. similarly, we treat the action as a text sequence for the model to predict. ## pre-processing in treating html as token sequences, we minimize any html tree pre-processing prior to model input. we thus provide html as raw text (i.e., sequences of text tokens) and only apply a snippet extraction pre-processing for pages which are too large to fit into the typical llms context windows. ## model training we study a variety of transformer-based llms @xcite with different sizes and architectures for html understanding tasks ( encoder-decoder and decoder-only models. we train encoder-decoder models, i.e., t5 (raffel et al., 2020), and decoder-only models, i.e., lamda @xcite and palm @xcite , with text input and text output. inputs are raw html pages or snippet texts; similarly, outputs are categories, natural language descriptions, or actions represented as text. namely, for semantic classification we use the textual representation of categories, similar to previous classification problems in nlp @xcite . for autonomous web navigation, actions are converted into text by first converting them into key and value pairs and then concatenating the pairs. many websites in miniwob require multiple interactions, such as click-button-sequence or clickcheckboxes, where each interaction might cause a subtle change in the website state. for instance, after clicking on a checkbox in the click-checkboxes website, its value flips from positive to negative or the other way around, which is not always reflected in llms' predictions and leads to action repetitions. we solve this issue by augmenting tuples in the dataset with a sequence of past actions, (action history, instruction, html, action), and allowing llms to learn from past experience. encoder-only models. we train encoder-only models, i.e., bert @xcite , with text input and categorical output. we keep semantic categories as discrete one-hot classes. to train encoder-only models, we add a new classification layer after the final encoder layer to produce a distribution over semantic categories. in addition to the typical bert models, we study mobilebert (sun et al., 2020), distilled from bert-large with inverted bottlenecks, and albert-xl @xcite , with parameter sharing and embedding split. ## results we now present the results of fine-tuned llms for html understanding. we compare the models' performance with the existing baselines where possible (autonomous web navigation) and against other llm architectures and training regimes (all tasks). sections 7.1, 7.2, and 7.3 evaluate taskspecific performance, while section 7.4 assesses the performance across all the tasks. metrics: for autonomous web navigation we evaluate models' success rate, which is averaged over 100 episodes per task. for the other tasks, we use accuracy to measure exact match between prediction and ground truth. in the description generation task, we additionally provide evaluations using alternative 'soft' text evaluation metrics, bleu and rouge-1, measuring the similarity between predicted and ground truth text. ## conclusion we presented canonical tasks and fine-tuned llms for html understanding. the comprehensive evaluations and analyses over a range of architectures, dataset sizes, and baselines yields practical findings and highlights current limitations of these mod-els. we find that a) pretraining is critical for the performance and can reduce labeled data requirements, improving sample efficiency up to 200x; b) model architecture is the second-most important factor, and t5 models with bidirectional attention and encoder-decoder architecture perform the best across the board; c) given a choice, model size should be evaluated in the context of the model's training and inference performance, as the model size sub-linearly correlates with its performance. finally, the proposed html understanding tasks highlight the relatively short context window that limits current llms, suggesting possibilities for future research that incorporate or eliminate this constraint.