G raph LSS : Integrating Lexical, Structural, and Semantic Features for Long Document Extractive Summarization (2025)

Abstract: Heterogeneous graph neural networks have recently gained attention for long document summarization, modeling the extraction as a node classification task. Although effective, these models often require external tools or additional machine learning models to define graph components, producing highly complex and less intuitive structures. We present GraphLSS, a heterogeneous graph construction for long document extractive summarization, incorporating Lexical, Structural, and Semantic features. It defines two levels of information (words and sentences) and four types of edges (sentence semantic similarity, sentence occurrence order, word in sentence, and word semantic similarity) without any need for auxiliary learning models. Experiments on two benchmark datasets show that GraphLSS is competitive with top-performing graph-based methods, outperforming recent non-graph models. We release our code on GitHub.

## introduction extractive document summarization condenses documents into summaries by selecting only the most relevant sentences. one intuitive approach is to model cross-sentence relationships using graph structures, which offer unique advantages over traditional sequence-based models. graph-based methods provide flexibility in handling varying document lengths and explicitly capture multigranularity text relationships. this structured representation enhances document analysis, enabling improved contextual understanding and deeper insights into document structure @xcite @xcite . while prior work considered homogeneous graphs @xcite , recent heterogeneous graph proposals have shown high effectiveness @xcite , as they define complex relationships between multiple semantic units and capture long-distance dependencies. despite their success in summarizing long documents such as scientific papers, many efforts have been made to devise more effective graph constructions. these vary in their definitions of nodes, often requiring external tools or additional machine learning models @xcite , and of edges, which despite being effective, may lead to complex structures that reduce the intuitiveness of the resulting graphs @xcite . this paper introduces graphlss, a graph construction that avoids the need for external learning models to define nodes or edges. graphlss utilizes lexical, structural, and semantic features, incorporating two types of nodes (sentences and words) and four types of edges (sentence order, sentences semantic similarity, words semantic similarity, and word-sentence associations). we limit word nodes to nouns, verbs, and adjectives for their high semantic richness @xcite . our document graphs are processed with gat @xcite ) models on two summary benchmarks, pubmed and arxiv, which are preprocessed and labeled by us. our contributions are: i. a novel heterogeneous graph construction using lexical, structural, and semantic features, ii. state-of-the-art results on both benchmarks compared to previous graph strategies and recent non-graph methods, iii. we share our code, including calculated extractive labels and graph-data creation pipeline, on github foot_0 for reproducibility and collaboration. ## previous work graph structure developing an effective graph structure for summarization has been challenging, leading to a proliferation of diverse approaches. @xcite proposed connecting sentence nodes to word nodes by establishing undirected as-sociations with the contained words. subsequently, @xcite extended this by introducing named entity nodes and three other edge types: directed edges for tracking subsequent named entities and words in a sentence, directed edges for entities and words within a sentence, and undirected edges for sentence pairs with trigram overlap. topic-graphsum @xcite was one of the first attempts to apply graph strategies to long document extractive summarization. it integrated a joint neural topic model to discover latent topics in a document, defining these as intermediate nodes to capture inter-sentence relationships across various genres and lengths. ssn @xcite defined a sliding selector network with dynamic memory. ssn splits a given document into multiple segments, encodes them with bert @xcite , and selects salient sentences. instead of representing the document as a graph, it uses a graph-based memory module, updated iteratively with a gat @xcite , to allow information to flow across different windows. heter-graphlongsum @xcite utilized words, sentences, and passages as nodes, while considering undirected edges for words in sentences, and directed edges for words in passages and passage to sentences. instead of pre-trained embeddings, it used cnns and bidirectional lstms for node encoding, yielding outstanding results. mtgnn-sum @xcite achieved similar results by capturing both inter and intra-sentence information when combining a homogeneous graph of sentence nodes with a heterogeneous graph of words and sentences, as in @xcite . recent studies underscore the importance of structural information in long document summarization. hegel @xcite modeled documents as hypergraphs, with edges capturing keyword coreference, section structure, and latent topics. changes @xcite introduced a sentence-section hierarchical graph, creating fully connected subgraphs for sentences and sections, and linking sentences to their sections. sentence labeling there is no consensus on generating extractive ground truth labels. most previous work @xcite @xcite used the @xcite greedy approach without specifying the rouge n-gram level, which significantly impacts sentence classifier performance. some methods @xcite @xcite se-figure 1: graphlss construction. sentence order edges are unweighted to preserve document structure, word in sentence edges are weighted using tf-idf to reflect word importance, and similarity edges between words and between sentences are determined using cosine similarity. ## graphlss graph construction we propose a heterogeneous model that represents documents as undirected graphs, @xmath0, e). we use sentences and words as nodes, @xmath1, and four edge types to capture lexical, structural, and semantic features, as @xmath2, e ss , e ws , e ww }. here, v s corresponds to the n sentences in the document, and v w denotes the set of m unique words of the document, limited to the most semantically rich ones, i.e., nouns, verbs, and adjectives foot_1 as in @xcite . for connections between nodes, boolean unweighted edges e ns indicate the sequential order of sentences within a document, while e ss includes sentence pair edges weighted by cosine similarity within a predefined window size. this constraint preserves local similarity and prevents dense graphs. to ensure that only strongly correlated sentences are connected, edges are established only when the cosine similarity surpasses a predefined threshold. additionally, e ws denotes words in sentence edges weighted by tf-idf scores, and e ww represents word pair edges using cosine similarity. the construction of graphlss is illustrated in figure 1 . adaptive class weights our graphs are processed by a heterogeneous gat @xcite followed by a sentence node classifier to conduct the extractive summarization. since the extractive ground truth labels for long documents are highly imbalanced, we optimize the model using weighted cross-entropy loss. we assign initial class weights to relevant and irrelevant sentences, employing adaptive class weights for the relevant class and static weights for non-summary sentences: with τ the portion of sentences predicted as relevant for the summary over all the existing sentences. extractive labels extractive labels are obtained by greedily optimizing the rouge-1 score, an intuitive and widely used method that allows us to label more sentences as relevant than alternative strategies. although we adopted the same labeling approach, we identified substantial sentence tokenization errors in the dataset from @xcite . hence, we independently preprocessed and labeled the data, removing duplicates, empty samples, and instances where abstracts exceeded source document lengths. we also replaced special characters (e.g., \, . . . , », "", \n) with blanks. we applied sentence tokenization using nltk and merged particularly short sentences with their preceding ones (cf. appendix a). for word node definitions, we converted sentence text to lowercase, removing non-ascii characters, punctuation, and stopwords. the resulting graph datasets are described in comparison methods for a more detailed comparative analysis with the models that achieved the best benchmark results (topic-graphsum, ssn, and hetergraphlongsum), we also executed our model using the preprocessed data and sentence-level relevance labels provided by @xcite . we also include results from recent non-graph extractive summarizers in ## results & analysis ## conclusions we introduced graphlss, a heterogeneous graph for long document extractive summarization incorporating lexical, structural, and semantic features. experiments on pubmed and arxiv highlight the impact of extractive labels due to their inherent imbalance. graphlss proves competitive with topperforming graph-based methods and outperforms recent non-graph models by using a greedy labeling strategy and adaptive weights during training. future work will focus on integrating an abstractive summarization model built upon our extractive results, while also investigating alternative methods to optimize storage and improve scalability.