Turbocharging Web Automation: The Impact of Compressed History States (2025)

Abstract: Language models have led to leap forward in web automation. The current web automation approaches take the current web state, history actions, and language instruction as inputs to predict the next action, overlooking the importance of history states. However, the highly verbose nature of web page states can result in long input sequence and sparse information, hampering the effective utilization of history states. In this paper, we propose a novel web history compressor approach to turbocharge web automation using history states. Our approach employs a history compressor module that distills the most task-relevant information from each history state into a fixed-length short representation, mitigating the challenges posed by the highly verbose history states. Experiments are conducted on the Mind2Web and WebLINX datasets to evaluate the effectiveness of our approach. Results show that our approach obtains 1.2-5.4% absolute accuracy improvements compared to the baseline approach without history inputs.

## introduction the task of web automation involves performing a sequence of actions to accomplish given tasks on any website, guided by language instructions @xcite @xcite @xcite . driven by advances in language models, web automation has attracted a lot of attention in recent years @xcite @xcite @xcite . these approaches leverage the current web state (i.e., web html and/or screenshot), history actions, and language instruction to predict the next action, obtaining promising web automation accuracy. * the work was done when xiyue zhu was an intern at amazon. * * peng tang and srikar appalaraju are the corresponding authors. however, the existing approaches do not consume history states, ignoring the fact that the history states are crucial to accomplish some web automation tasks, which leads to sub-optimal accuracy, see figure 1 . this fact motivates us to explore techniques that leverage history states to improve the accuracy of web automation. the most straightforward way to leverage history states is to concatenate history states with other inputs (i.e., the current state, history actions, and language instruction) and feed the concatenated inputs into models. but the state of real-world web pages could be very verbose @xcite , posing several challenges to benefiting web automation models from the straightforward approach: 1) long input sequence. concatenating verbose history and current states results in a long input sequence, leading to high gpu memory cost and inference latency. 2) sparse information in history states. compared to the current state, the information that is relevant to the next action is much sparser in history states. failing to effectively distilling the sparse information from history states could adversely impact the accuracy. ## approach figure 2 (a) shows the architecture of our model. our model takes the current input and n history inputs as input, where each input consists of state (i.e., web html and/or screenshot), history actions, and language instruction describing the task. a history compressor module is employed to compress each history input into a fixed-length representation, see section 2.1. here, the history compressors for different history inputs share the same model weights. the compressed representations of history inputs are concatenated with the current inputs, forming the input to a transformer model. the transformer model then predicts the next action based on the inputs. we apply a history compressor to history inputs only, because the current input is highly relevant to the task so we want to keep as much information in the current input as we can. ## conclusion in this paper, we propose a novel web history compressor approach to improve web automation using history states. our approach trains a history compressor module to compress each highly verbose history state to a fixed-length short representation, maintaining the most task-relevant information meanwhile. experimental results show that our approach surpasses the baseline approach of without history inputs by 1.2-5.4% on the mind2web and weblinx datasets. in the future, we will apply our approach to stronger transformer models and models that can take different modalities (e.g., image, image + text) as inputs, and apply our approach to more web automation datasets.