W eb DP : Understanding Discourse Structures in Semi-Structured Web Documents (2023)

Abstract: Web documents have become rich data resources in current era, and understanding their discourse structure will potentially benefit various downstream document processing applications. Unfortunately, current discourse analysis and document intelligence research mostly focus on either discourse structure of plain text or superficial visual structures in document, which cannot accurately describe discourse structure of highly free-styled and semi-structured web documents. To promote discourse studies on web documents, in this paper we introduced a benchmark – WebDP, orienting a new task named Web Document Discourse Parsing. Specifically, a web document discourse structure representation schema is proposed by extending classical discourse theories and adding special features to well represent discourse characteristics of web documents. Then, a manually annotated web document dataset – WEBDOCS is developed to facilitate the study of this parsing task. We compared current neural models on WEBDOCS and experimental results show that WebDP is feasible but also challenging for current models.

## introduction with the rapid development of internet during past decades, web documents have become one of the most primary and the biggest data resources in current era. as a result, understanding their discourse structure -how different components in a document semantically interact with each other to form a cognitive entirety -will greatly benefit many downstream applications, as previous works have demonstrated the virtue of structure information in other types of documents @xcite @xcite @xcite . the free-styled, semi-structured nature of web documents gives them characteristics different from traditional forms of documents, providing abundant opportunities and challenges for discourse research. on one hand, web documents exhibit more free-styled discourse organization. for instance, it is common for web documents to encompass multiple topics @xcite , where different blocks 2 within the document are loosely connected by implicit semantic relevance or even describe independent topics (figure 1 ). the free-styled nature allows discourse with loose structures, multiple topics and weak coherence, in contrast to the compactly-structured, single-topic and strongly-coherent nature of traditional documents. on the other hand, web documents are semi-structured in respect of their html markup language and layout of blocks. the content of web documents tend to be organized under multiple hierarchies, usually reflected by the html markup hierarchies @xcite . however, the semistructured information provided by html markup and layout structures is merely superficial, since it does not consistently align with the underlying semantic relations in the discourse structure (figure 1 ), to which, analogous phenomenon has also been found in plain texts @xcite . in fact, the inconsistency between 2 "blocks" @xcite , or "physical objects" @xcite more formally, are customary concepts used in previous document image processing research. @xcite defined "blocks" as "a set of text lines with the same typeface font and a constant line interval" within a document. in this paper, we study discourse connections among "blocks" in web documents and use this term as a short name for elementary discourse units in the proposed web document discourse schema. a formal definition of these units is presented in § 3.4. these superficial structures and the underlying discourse structure is prevalent in web documents due to their free-styled writing process. unfortunately, previous research can not fully fulfill the requirements of discourse representation proposed by the free-styled, semi-structured characteristics of web documents. this limitation can be attributed to two distinct lines of research: discourse analysis @xcite and document intelligence @xcite . as for discourse analysis research, traditional studies focus on classical plain texts with compact, single-topic and strong-coherent discourses, leaving a gap to freestyled web documents with loose discourse organization, multiple topics and weak coherence. furthermore, they mainly study clause-level discourse phenomena, rather than blocks in semi-structured web documents. as for document intelligence research, although they recognize various kinds of semi-structured and multi-modal information carried in documents, including visual, layout, and html markup features, most studies only examine superficial structures directly derived from these multi-modal features. as demonstrated in figure 1 , the inconsistency between superficial structures and underlying discourse structure in web documents impairs the power of this line of works in downstream applications that require a deep understanding of semantics. in this paper, we introduce a new benchmark webdp, based on a well-designed discourse representation schema for web documents named webdp structure. the aim is to promote studies on web document discourse analysis and fill the gap between underlying discourse structure and superficial layout structure in documents. to accurately represent the free-styled discourse structure of web document, we design the discourse schema by extending previous linguistic theories of @xcite and @xcite to take into account characteristics of discourse organization found in web documents. specifically, webdp structure adopts two kinds of rhetorical relations to connect blocks in document and form a hierarchical structured discourse representation: subordinating relations depict semantic functions between blocks across different levels of the hierarchy and coordinating relations depict coherence properties between blocks at the same-level hierarchy. then, we collect web documents and manually annotate a dataset webdocs to support further research of webdp. additionally, to verify the feasibility of the proposed task and reveal its challenges to current methods, we re-implement representative neural models from related fields and conduct systematic experimental analyses on the dataset. contribution of this work can be concluded as: ## related work previous researches approach the structures within documents from two orthogonal research lines, i.e., discourse analysis @xcite and document intelligence @xcite . our proposed schema, webdp structure, makes good complement to them on modeling the discourse of web documents (see ## designing discourse representation schema for web documents in this section, we describe the proposed web document discourse schema -webdp structure. we first summarize main characteristics of web documents and the consequent requirements for a welldesigned discourse schema ( § 3.1). then we briefly task name text genre basic unit modeling of semantics single topic document discourse parsing rst @xcite plain, short text clause not constrained pdtb @xcite plain, short text clause yes discourse dependency parsing @xcite plain, short text clause yes macro-level dp @xcite plain text paragraph yes dialogue dp @xcite multi-turn dialogue utterance turn yes ## webdp: a new benchmark for web document discourse parsing to facilitate web document discourse analysis research, we present a new task named web document discourse parsing (webdp). the goal of the task is to automatically convert linear list of edus in the input web document into the hierarchical discourse structure defined in webdp structure. to benchmark the task, we construct a new dataset webdocs and introduce its evaluation metrics. in virtue of webdocs, we can benchmark further studies on webdp, reveal the key challenges and difficulties on web document discourse analysis, assess the effectiveness and diagnose the defect of different parsing algorithms. ## conclusion in this paper, we inspect web documents from a discourse linguistic perspective to reveal their underlying discourse structure. inspired by linguistic theories of rhetorical relations and sdrt, we build a web document discourse schema which simultaneously models subordinating relations and coordinating relations. based on the schema we propose a new task webdp and contribute a dataset to promote the discourse analysis researches on web documents. experimental results of recent neural models exhibits the challenge of webdp and detailed analyses provide insights for future studies. we believe the web document discourse schema is prospective in facilitating document-level natural language processing research by explicitly modeling discourse structure for web documents.