Leveraging Collection-Wide Similarities for Unsupervised Document Structure Extraction (2024)

Abstract: Document collections of various domains, e.g., legal, medical, or financial, often share some underlying collection-wide structure, which captures information that can aid both human users and structure-aware models.We propose to identify the typical structure of document within a collection, which requires to capture recurring topics across the collection, while abstracting over arbitrary header paraphrases, and ground each topic to respective document locations. These requirements pose several challenges: headers that mark recurring topics frequently differ in phrasing, certain section headers are unique to individual documents and do not reflect the typical structure, and the order of topics can vary between documents. Subsequently, we develop an unsupervised graph-based method which leverages both inter- and intra-document similarities, to extract the underlying collection-wide structure. Our evaluations on three diverse domains in both English and Hebrew indicate that our method extracts meaningful collection-wide structure, and we hope that future work will leverage our method for multi-document applications and structure-aware models.

## introduction knowing the structure of a typical document within a collection can be useful in various use cases across different domains. for example, consider the legal domain, where lawyers seek to analyze corpora of legal proceedings, looking for trends and patterns over time. in a retrieval scenario, they may look for punishment trends over many verdict decisions @xcite . while each verdict decision normally includes a dedicated punishment section, it is often hard to locate it, because it is not consistently marked -different verdicts may use different headers ("punishment", "sentencing decision", or "incurred penalty"), and position the section in different document locations, requiring scholars to scan vast amounts of text. in an exploratory use case, lawyers may not have a welldefined apriori question, but instead interested in emerging collection-wide trends, for example, analyzing what judges take into account in verdict. furthermore, a collection-wide signal about document structure can also be leveraged by structureaware models for multi-document downstream applications, e.g., infusing the document structure as part of the transformer architecture, to improve multi-document applications @xcite . to support both human users and structure-aware models, we propose to identify the typical document structure within the collection ( §2). this requires to capture recurring topics across the collection, while abstracting over arbitrary header paraphrases, and ground each topic to respective document locations. for example, in figure 1 we want to identify "case summary", "evidence presented" and "verdict" as the recurring and prominent topics of a typical document within the collection of legal verdict decisions, as opposed to "software development standards implications", which is document specific. this also requires to identify that "verdict" and "judgement decision" represent the same topic, i.e., abstract over header paraphrases. the colored bounding boxes in figure 1 represent the grounding unto each document. automatically extracting the typical document structure is challenging. while topic boundaries are often loosely defined by explicit headers, it is hard to use them directly to understand collection-level properties, as headers indicating the same information often vary in phrasing, for example "verdict", "judgement outcome" and "decision". moreover, some section headers are local to individual document, and do not participate in the global structure. for example, "software development standards implication" in figure 1 . finally, while section order provides some signal, it is often inconsistent, and may vary across different documents. the challenge is then align section headers across the collection, while being flexible enough to discard sections that do not reflect a global shared structure. following, we devise an unsupervised method using a collection-wide signal to perform the structure extraction ( §3). to achieve this, we represent the document collection using a complete undirected weighted graph, whose nodes represent topic boundary candidates, and the weight of edges between each pair of nodes represents their semantic similarity. such formulation allows us to model relations both within document, as well as across documents. for example, set large edge weight between "case overview" and "introduction to the case", as they convey semantically similar topics. finally, we find communities within the graph, where each community is a group of similar nodes that form a single component of the collection-wide structure, and filter the communities that best outline the typical document structure, which we call the collection-wide table of contents (toc). to illustrate the robustness of our method across various domains and languages, we curate three distinct datasets ( §4). these include two english datasets from different domains (financial and legal), and a hebrew dataset, composed of legal documents. this variety shows that our method is adaptive and effective under different linguistic and domain-specific contexts. to evaluate our model, we propose three distinct metrics ( §5). first is the "header intrusion" human evaluation, which is adapted from the popular "word intrusion" metric for clustering evaluation @xcite , to evaluate the quality of the representation of the collection. second, an automatic evaluation for the document-level grounding, to evaluate the predicted grounding coverage. last, a qualitative analysis over the predicted toc entries, manually exploring how meaningful are the produced entries, comparing them to an existing toc of the financial documents collection. we find that our method extracts meaningful typical document structure, capable of retrieving both the overall collection structure, while also mapping it unto individual documents, and that it is robust to varying domains and languages, with no supervision and little domain-specific adaptation. detailed error analysis shows that while it is capable of mapping most topics unto documents, it struggles with identifying the exact topic boundaries. our released code allows future work to leverage collection-wide structure to enrich both user-facing applications, and to integrate document structure within llms to cope with multiple documents. our main contributions are: (1) we formally define a novel task of identifying the typical document structure from a document collection, (2) we curate three datasets for the task from diverse domains and languages, and (3) we develop an unsupervised approach for the task, by leveraging collection-wide signal in a community detection algorithm. ## task definition given a collection of documents which share an underlying latent structure, we aim to recover a figure 2 : a high-level description of the three steps in our approach for extracting structure from a collection of documents. first, we identify topic boundaries in individual documents, based on lexical cues ( §3.1). second, we build a collection representation using a complete graph whose nodes are the topics identified in the previous step, and weighted edges represent semantic similarity between them ( §3.2) finally, we extract the typical document structure using unsupervised community detection, which we filter based on coverage maximization ( §3.3). formally, we denote a collection of documents @xmath0, . . . , d n }, where a model is required to output a tuple (t, m ). @xmath1, . . . , t k } is a set of k most prominent topics across d. in the scope of this work, each topic t i should appear as a continuous text span in some document d j . for example, in figure 1 , @xmath2, "evidence presented", "verdict"}. m : (t × d) → (n + × n + ) grounds topics to individual documents, mapping each topic to start and end indices. m is exemplified with colored bounding boxes in figure 1 . ## method we describe our method for generating the toc and grounding, which captures the typical document structure within a given document collection, and composed of three steps as outlined in figure 2 . ## data curation to evaluate our method, we curate three document collections. as presented in ## results we evaluate three different aspects of our method, including human evaluation of the communities we produce ( §5.2), document-level evaluation of the toc grounding ( §5.3), and a qualitative analysis of the predicted toc ( §5.4). ## related work a seminal theory of a pragmatic meaning is rhetorical structure theory @xcite , that inspired much follow-up work, mostly approaching document structure extraction as a supervised task @xcite @xcite . the need for supervision narrows the application of such tools only to high-resource languages and small number of domains. our method is unsupervised, does not require labeling, and is applicable to many languages and domains. another line of work suggest to approach structure extraction as an unsupervised task, to extend its applicability @xcite @xcite , but they still approach this as a single-document task. in our work we take the entire collection as our input, leveraging signal regarding the underlying collection-wide structure. xing et al. ( 2022 ) discovered that injecting external (above-sentence) information can help with identifying discourse dependency structure, but they only look at individual documents. we extend this observation into modeling relations across an entire collection, to inject external information about the structure components of the collection. finally, theories like cst @xcite and cast (altmami and menai, 2020), or @xcite , suggest to look at the entire data collection to extract the main components that capture a typical structure layout of a document within the collection, but, their approach too relies on supervision and labeled data. ## conclusion in this paper we suggested an unsupervised method to extract the typical document structure within a collection, leveraging similarities across documents that come from the same collection with some shared underlying latent structure. we evaluated the extracted typical document structure through three different evaluation metrics, over three different datasets, showing our method's robustness to different domains and languages. the evaluations indicated that our method succeeds in identifying the different topics consisting the typical document structure, while showing room for improvement in its accuracy to detect the exact location of where a topic starts and where it ends. our method can be leveraged by users of multidocument applications, allowing them more focused browsing over a collection of documents, which can be useful for downstream tasks like retrieval and summarization. as for models, future work can utilize our method and inject the collection-wide structure to structure-aware mod-els, to improve downstream applications like multidocument summarization or cross document information extraction. kaitao song, xu tan, tao qin, jianfeng lu, and tie-yan liu. 2020. mpnet: masked and permuted pretraining for language understanding. in advances in neural information processing systems, volume 33, pages 16857-16867. curran associates, inc. teemu vahtola, mathias creutz, and jörg tiedemann. 2022. it is not easy to detect paraphrases: analysing semantic similarity with antonyms and negation using the new semantoneg benchmark. in proceedings of the fifth blackboxnlp workshop on analyzing and interpreting neural networks for nlp, pages 249-262, abu dhabi, united arab emirates (hybrid). association for computational linguistics. mohr wenger, tom kalir, noga berger, carmit klar chalamish, renana keydar, and gabriel stanovsky. 2021. automated extraction of sentencing decisions from court cases in the hebrew language. in proceedings of the natural legal language processing workshop 2021, pages 36-45, punta cana, dominican republic. association for computational linguistics. jinxiong xia, cao liu, jiansong chen, yuchen li, fan yang, xunliang cai, guanglu wan, and houfeng wang. 2022. dialogue topic segmentation via parallel extraction network with neighbor smoothing. in proceedings of the 45th international acm sigir conference on research and development in information retrieval, sigir '22, page 2126-2131, new york, ny, usa. association for computing machinery. linzi xing and giuseppe carenini. 2021. improving unsupervised dialogue topic segmentation with utterance-pair coherence scoring. in proceedings of the 22nd annual meeting of the special interest group on discourse and dialogue, pages 167-177, singapore and online. association for computational linguistics. linzi xing, patrick huber, and giuseppe carenini. 2022. improving topic segmentation by injecting discourse dependencies. in proceedings of the 3rd workshop on computational approaches to discourse, pages 7-18, gyeongju, republic of korea and online. international conference on computational linguistics. zixuan zhang, heba elfardy, markus dreyer, kevin small, heng ji, and mohit bansal. 2023. enhancing multi-document summarization with cross-document graph-based information extraction. in proceedings of the 17th conference of the european chapter of the association for computational linguistics, pages 1696-1707, dubrovnik, croatia. association for computational linguistics.