GEM : Gestalt Enhanced Markup Language Model for Web Understanding via Render Tree (2023)

Abstract: Inexhaustible web content carries abundant perceptible information beyond text. Unfortunately, most prior efforts in pre-trained Language Models (LMs) ignore such cyber-richness, while few of them only employ plain HTMLs, and crucial information in the rendered web, such as visual, layout, and style, are excluded. Intuitively, those perceptible web information can provide essential intelligence to facilitate content understanding tasks. This study presents an innovative Gestalt Enhanced Markup (GEM) Language Model inspired by Gestalt psychological theory for hosting heterogeneous visual information from the render tree into the language model without requiring additional visual input. Comprehensive experiments on multiple downstream tasks, i.e., web question answering and web information extraction, validate GEM superiority.

## introduction web pages serve as crucial carriers for humans to acquire and perceive information. due to the significant wealth of these documents, long-standing efforts have been undertaken to address web understanding tasks @xcite @xcite @xcite . however, understanding web pages can be challenging for automated systems compared to humans, as the design of layout and visual style caters specifically to human perceptual patterns, thereby facilitating comprehension. recently, the pre-trained language models (lms) @xcite advance web understanding and demonstrate superior performance on various related tasks by jointly pretraining on text and markup information. nevertheless, these models oversimplify web pages as plain html 1 (hypertext markup language) sequences confuse html code rendered page li2 li3 li1 li7 li6 li5 li4 focus question: what is the key performance of xperia? answer visual view ul li1 li2 li7 li3 li4 li5 li6 li2+li3 li4+li5 +li6+li7 hierarchy answer code view ul li1 li2 li7 li3 li4 li5 li6 sibling figure 1 : from the html code view, the page consists of seven @xmath0, which can be divided into three groups from the visual perception view (represented by differently colored boxes). by taking hierarchical relationships of the visual view into account, the system can answer the question "what is the key performance of the xperia?" better than only considering the sibling relationship. while neglecting the advantageous visual information of rendered pages, such as style and layout, which is essential for web understanding. several recent works investigate web pages as pictures of screenshots @xcite @xcite , emphasizing the importance of the fixed layout of the input image. however, the dynamic nature of web page rendering can lead to significant appearance variations across devices and browsers, resulting in the limited applicability of such models. additionally, these methods require an ocr foot_0 system to extract text with coordinates, which leads to extra overhead and neglect the hierarchical information provided by the markup language. regardless of render conditions, humans are capable of rapidly comprehending the content of rendered web pages, which indicates that human visual perception is an effective way to consume web semantics @xcite . to leverage this advancement, we propose to encode visual information of web pages into the lm flexibly via render-tree-enabled pre-training tasks. it is inspired by human perceptive patterns rather than taking visual features as input directly. gestalt psychological theory @xcite , a prominent cognitive model, explains the human perceptive processes that elements with similar visual styles and proximate locations are commonly regarded to be of similar semantic functions. for instance, as figure 1 depicts, while the seven nodes of a given web page are siblings in the markup language, their different rendering appearances and positions categorize them into three semantic blocks. without visual awareness (i.e., only using html tags), it is challenging to answer downstream questions like "what is the key performance of xperia?". however, when considering the visual cues of the web page, particularly the answer is comprised of nodes in the same row and color, this question becomes rather achievable. in this paper, we propose gestalt enhanced markup (gem) language model designed to host visual knowledge into the language model without additional visual input requirement. the gem utilizes visual information of rendered web pages solely during the pre-training, while during finetuning or inference, the rendering is unnecessary. to incorporate visual perception into language models, two further pre-training objectives are proposed, each relating to one gestalt law. the one is same textual style prediction (stsp), which is based on the gestalt laws of similarity that humans tend to perceive objects with similar appearances as a group. this task enlightens the language model with an appearance perception to learn semantics. on the other hand, the proximate nodes prediction (pnp) enables the model to understand the relationships of elements from visual positions, not only from the dom 3 (document object model) tree. the pnp task is proposed on the gestalt proximity law, according to which humans consider objects close to each other perceptively coherent. our corpora are built from render trees of rendered 3 https://en.wikipedia.org/wiki/document_objec t_model web pages, which combine the dom and cssom foot_1 (css object model) that contain comprehensive information about rendered web pages, including textual, structural, and visual information. in practice, gem can learn a strengthened representation of markup language with the enhancement of visual prior knowledge. to validate gem superiority, we conduct experiments on two downstream tasks, i.e., web question answering and web information extraction. gem consistently surpasses several strong baselines. moreover, we verify the model architecture adaptability of gestalt objectives. in addition, we compare the performance of gem and large language models (llms) @xcite @xcite . our main contributions are as follows: ## related work web pages contain rich perceptible information beyond text. extracting this information is crucial for web understanding but also challenging. previous works on this task mainly use rule-based modules @xcite @xcite , which are not robust to different websites. representation learning for plain text documents has been well-studied. pre-trained language models (plms), which use text encoders with selfsupervised objectives, achieve remarkable performance on numerous nlp tasks @xcite @xcite . some recent works extend these methods to web pages by taking html documents as input and encoding semantic information with specific pre-training tasks @xcite . however, these models ignore the visual features of web pages, which are essential for understanding them. some other works address this issue by treating web pages as images of screenshots @xcite , but they lost the hierarchical structure of html. @xcite and @xcite incorporate coordinates of web elements into the input along with the html. however, they all rely on the fixed layout of the input image, which could vary with different devices and browsers. therefore, these models have limited applicability. our work diverges from prior attempts in two crucial areas: first, we model web pages by leveraging both html documents and render trees, providing the lm with visual perception. additionally, we harness knowledge from web page visual aspects by encoding it into the text representation rather than using visual information as input. recent developments in llms @xcite @xcite ushered in a new age of ai applications. while these advances have demonstrated a remarkable capacity, there remains an acute need for a comprehensive evaluation of their application to web understanding. ## conclusion this paper presents gem, a gestalt enhanced markup (gem) language model that leverages gestalt psychological theory. gem innovatively enriches the language model with heterogeneous visual information from render trees of web pages without requiring visual modality input. as part of this innovation, two distinctive gestalt pre-training objectives are formulated in order to codify visual qualities such as style and position within the language model. evaluations of different downstream tasks and backbones show that gem can learn a stronger representation of markup language with visual knowledge enhancement.