PLA t E : A Large-scale Dataset for List Page Web Extraction (2023)

Abstract: Recently, neural models have been leveraged to significantly improve the performance of information extraction from semi-structured websites. However, a barrier for continued progress is the small number of datasets large enough to train these models. In this work, we introduce the PLAtE (Pages of Lists Attribute Extraction) benchmark dataset as a challenging new web extraction task. PLAtE focuses on shopping data, specifically extractions from product review pages with multiple items encompassing the tasks of: (1) finding product list segmentation boundaries and (2) extracting attributes for each product. PLAtE is composed of 52,898 items collected from 6,694 pages and 156,014 attributes, making it the first large-scale list page web extraction dataset. We use a multi-stage approach to collect and annotate the dataset and adapt three state-of-the-art web extraction models to the two tasks comparing their strengths and weaknesses both quantitatively and qualitatively.

## introduction semi-structured data extraction, i.e., web extraction, is the task of extracting data found in templated text fields from html pages. once extracted, the structured data can be utilized in various downstream tasks such as information retrieval, recommendation, and question answering. while recent work has shown the potential of neural approaches for web extraction @xcite @xcite , there are very few publicly available large-scale datasets suitable for training and evaluation of these approaches, limiting progress in this area. additionally, most existing datasets @xcite figure 1 : a single item (i.e. product) from a list page for a board game (from gamesradar.com). @xcite focus on one subset of the problem; namely detail page extraction. in this paper, we introduce the plate (pages of lists attribute extraction) dataset foot_0 , that specifically targets the task of list page extraction and focuses on product review pages in the shopping vertical, i.e., multiproduct review pages. to elaborate, item pages can be broadly categorized into two classes: detail pages and list pages. detail pages provide detailed information about a single item. list pages comprise a list of items with abridged detail, organized under a single theme, e.g. "best board games". this organization facilitates direct comparison of each item and allows for the extracted data to be easily integrated into recommender, question answering, or dialogue systems powering digital assistants @xcite @xcite . extracted product data can be utilized by both content creators (publishers) as well as customers looking to make purchase decisions. 2 because plate is built from list (multi-item) pages, we can evaluate two tasks: segmentation and attribute extraction. in the segmentation task, the model should determine the boundaries of each product, i.e. where the information for each product begins and ends. in the attribute extraction task, each node in the html dom tree can be assigned any number of three labels: product name, product review text, and product link. the dataset is comprised of 52, 898 products from 6, 694 pages split into: training, development, and held-out test sets. to evaluate the dataset, we adapt two stateof-the-art neural web extraction models; marku-plm @xcite and dom-lm @xcite , as well as explore a text-based transformer model, roberta @xcite . we achieve an f1-score of 0.787 and 0.735 for segmentation and attribute extraction tasks respectively. we evaluate the potential of multi-task learning to improve performance and find that multi-task learning improves recall but slightly decreases precision, for both tasks. to summarize, our contributions are: (1) creating the first large-scale list page web extraction dataset, (2) adapting state-of-the-art neural web extraction approaches to the segmentation and attribute extraction tasks, and (3) qualitatively and quantitatively comparing the performance of different models on the two presented tasks; segmentation and attribute extraction. ## related work the vast majority of previous web extraction datasets (e.g., swde @xcite , the klarna product page dataset @xcite , and wdc @xcite ) are composed of single item pages. multiple item or list page datasets are much less common. @xcite created a dataset of 771 list pages, while @xcite created a small dataset (bscm) of about 30 pages from 4 verticals: books, seminars, cs faculty and mlconfs. @xcite built a dataset of 431 pages from two verticals: uk used car dealer websites collected from a used car aggregator website and uk real-estate websites collected from the yellow pages. to the best of our knowledge, plate is the largest multi-item web extraction dataset. additionally, most previous web extraction datasets assume a single dom node has at most a single label. however, this assumption does not hold true in many product pages. for example, for many products, the product name's text-span is also a link to the product. our dataset does not have this assumption; we allow a node to have multiple labels. finally, most existing list page datasets are not publicly available. from the methods side, web extraction has first been tackled using wrapper induction methods that create a set of rules (wrappers) to transform unstructured input into structured output @xcite @xcite @xcite @xcite . recently, a number of advances have been made by utilizing neural-based approaches to construct a representation for each html node for the extraction task @xcite @xcite @xcite . other tasks related to semi-structured information extraction include boilerplate removal @xcite , extraction from html tables @xcite @xcite , and segmenting pages, e.g., using clustering followed by string alignment @xcite , optimization based on divide and conquer @xcite , and hierarchical conditional random fields (crfs) @xcite . ## plate benchmark in this work, we tackle two tasks: (1) segmentation, i.e., identifying the boundaries of the individual products in a given page and (2) attribute extraction, i.e., identifying the individual attributes for each identified product. for each given product, we extract the following three attributes: (1) product name: this refers to the name of the product, e.g. "iphone 11", (2) review text: this is generally a high-level review or general description of the product, and (3) purchase link: a link (or button) to a merchant's website (e.g., amazon, ebay). we commonly see generic text such as "buy here", the name of the merchant such as "amazon", or the name of the product. similar to prior work, we perform classification on the leaf nodes; i.e., only nodes that contain text are passed to the classification models. ## models we evaluate the performance of three recent neural models: roberta @xcite , dom-lm @xcite and markuplm @xcite on plate. roberta is a transformer-based @xcite model pre-trained with natural language texts sourced from the bookcorpus @xcite and wikipedia. the pre-training task is masked language modeling. in our experiments, the input to roberta is a sequence of text tokens from the dom tree. consequently, it does not utilize other types of information from the dom tree such as xpaths or html tags. ## experiments for both segmentation and attribute extraction tasks, we report the precision, recall, and f1score. results are macro-averaged over the different classes. in addition, for segmentation, we report clustering metrics following @xcite , where the attributes in the same segment are considered to be in the same cluster. in our dataset, when looking at two adjacent products in a page, there is a single node which we label as a "segmentation boundary". in reality, there can be multiple nodes which appear between two adjacent products and split the products apart. if multiple valid segmentation boundaries for a product are possible, f1-score will penalize the model for picking any segmentation boundary which is not labelled as such. on the other hand, the clustering metrics provide a relaxation which checks that product attributes are assigned to the correct product. for clustering metrics, we report adjusted rand index (ari) @xcite and normalized mutual information (nmi) @xcite where a higher number indicates better performance. 6,7 overall, we observe that markuplm performs well on the segmentation task yielding scores of 0.787, and 0.771 and 0.870 for f1, ari and nmi respectively while dom-lm performs worse on this task with scores of 0.722 and 0.716 and 0.764 for f1, ari and nmi respectively. this can likely be attributed to the fact that markuplm was pretrained on the task of relation prediction between nodes. identifying if two nodes have a parent-child, sibling, or ancestor-descendant relation could help the model distinguish nodes within the same product from nodes of different products, and conse- the attribute extraction task shows a different trend from segmentation where roberta performs the best, outperforming markuplm and dom-lm. we look into the reason behind this in section 6. detailed results for both tasks are presented in for review text, the higher performance is not surprising given that it normally has a more consistent style than product links. to better understand the different factors affecting the performance of the attribute extraction task, we (1) break down the scores by site in our test set to see whether some sites are more challenging than others, (2) analyze the relationship between the number of products in a page and attribute extraction performance (3) explore whether the index of the product (i.e., where it appears in the page) affects the performance, (4) study the effect of adding more pages (versus sites) to the training data, and (5) explore whether a multi-task model that jointly tackles both segmentation and attribute extraction tasks can yield better performance through utilizing the complementary signals between both tasks. as can be seen in figure 3 , extraction performance varies greatly by site. we suspect that this is due to the diverse page layouts and styles of different sites. when grouping pages by number of products, we find that for all three models, as the number of products increases, the f1-score improves. we believe this is due to the increased ease of the model to learn patterns within a page given more examples in a page. when grouping the scores by product index, we find that product index does not have a significant impact on the performance of attribute extraction; i.e. performance is relatively uniform across different locations in the page. 8 to study the value of annotating more sites compared to annotating more pages, we sampled two subsets of pages from our original training set. we collected all pages from 14 randomly sampled sites from the training set for a total of 1514 pages, then sampled 1514 random pages across all sites in the training set. the 14 sites achieved an @xmath0.595 compared to sampling from all sites achieving an @xmath1.681. from this, we can conclude there is indeed value in annotating a greater sites instead of simply annotating more pages from the same site. we suspect the model is better able to generalize to the test set, when provided with a more diverse training set of different sites. finally, we look into whether multi-task learning improves model performance on plate. to this end, we train a marku-plm model with a shared encoder, but distinct loss functions and output layers for the segmentation and attribute extraction tasks. 9 we find that multitask learning improves the recall of both tasks -as shown in ## discussion contrary to our expectations, roberta outperformed dom-lm, a model specially designed to model webpages, on the attribute extraction task. as can be seen in we perform additional analysis of the outputs from the markuplm model, to identify areas for improvement for the attribute extraction task. specifically, we sampled up to 5 false positive (fp) and 5 false negative error examples for each attribute from each site in the test set, and ended up with a total of 257 examples. 10 for product review text, we find that many of the false negatives are very short textspans such as "these", "as well", and "require" representing a single text leaf node within a larger paragraph. we suspect that the model has not been trained with enough examples of varied text length. in the future, we could also consider training the model to classify the parent nodes representing a whole paragraph instead of classifying at the leaf node level. for false positives, we find 4 examples of user-written comments (as opposed to reviewer-written text). while these do not represent the official review text, they are semantically similar to the official review text hence are easily confused by the model. one such example is a user's comment mentioning that "the design and functionality of these cookers is top-notch" which has a similar style to text which could have been written by a reviewer. for product link, 17 false positives are hyperlinks that link to a homepage rather than to a specific product which indicates that the model is over-fitting to @xmath3. training the model using contrastive learning with positive and negative product @xmath4. next we explore the effect of threshold-tuning on the performance of attribute extraction. in the presence of an oracle that specifies whether or not a node should be tagged with an attribute, we can use the argmax of the logits of the different attribute classes in order to guarantee that the node is tagged with one of the three attributes. based on the very high agreement between the argmax and the actual label for product name (89.4%) and product review text (91.8%), it is clear that some of the challenges in tagging nodes in plate stem from needing to determine whether or not a node should be tagged with any attribute, on top of determining what the correct attribute tag is. finally, we analyze the errors in the segmentation task, collecting all pages with an ari and @xmath5.5 within test set. we find that in 97% of these pages the number of gold segmentation boundaries is higher than the number of predicted segmentation boundaries. this means that when ari and nmi are very low, the model is failing to split the page into enough segments. to improve performance, we could consider utilizing a structured prediction to model segmentation interdependencies e.g. the model could explicitly model that it is unlikely that two segmentation boundaries appear directly next to one another. ## conclusion in this work, we introduce plate, the first largescale list page dataset for web extraction. we describe our methodology in creating the dataset, from pre-processing to crowd-sourced annotation collection. we evaluate the performance of three strong web extraction baselines and achieve an f1score of 0.787 and 0.735 on the segmentation and attribute extraction tasks respectively. while we focus on shopping domain due to its importance to several downstream applications, in the future we intend to extend our work to other verticals to facilitate further research studying model generalization and domain adaptation.