Improving Graph-Based Text Representations with Character and Word Level N-grams (2022)

Abstract: Graph-based text representation focuses on how text documents are represented as graphs for exploiting dependency information between tokens and documents within a corpus. Despite the increasing interest in graph representation learning, there is limited research in exploring new ways for graph-based text representation, which is important in downstream natural language processing tasks. In this paper, we first propose a new heterogeneous word-character text graph that combines word and character n-gram nodes together with document nodes, allowing us to better learn dependencies among these entities. Additionally, we propose two new graph-based neural models, WCTextGCN and WCTextGAT, for modeling our proposed text graph. Extensive experiments in text classification and automatic text summarization benchmarks demonstrate that our proposed models consistently outperform competitive baselines and state-of-the-art graph-based models.

## introduction state-of-the art graph neural network (gnn) architectures @xcite such as graph convolutional networks (gcns) @xcite and graph attention networks (gats) @xcite have been successfully applied to various natural language processing (nlp) tasks such as text classification @xcite @xcite and automatic summarization @xcite . the success of gnns in nlp tasks highly depends on how effectively the text is represented as a graph. a simple and widely adopted way to construct a graph from text is to represent documents and words as graph nodes and encode their dependencies as edges (i.e., word-document graph). a 1 code is available here: https://github.com/ graphforai/textgraph given text is converted into a heterogeneous graph where nodes representing documents are connected to nodes representing words if the document contains that particular word @xcite . edges among words are typically weighted using word co-occurrence statistics that quantify the association between two words, as shown in figure 1 (left). however, word-document graphs have several drawbacks. simply connecting individual word nodes to document nodes ignores the ordering of words in the document which is important in understanding the semantic meaning of text. moreover, such graphs cannot deal effectively with word sparsity. most of the words in a corpus only appear a few times that results in inaccurate representations of word nodes using gnns. this limitation is especially true for languages with large vocabularies and many rare words, as noted by @xcite . current word-document graphs also ignore explicit document relations i.e., connections created from pair-wise document similarity, that may play an important role for learning better document representations @xcite . ## methodology given a corpus as a list of text documents @xmath0, ..., d n }, our goal is to learn an embedding h i for each document d i using gnns. this representation can subsequently be used in different downstream tasks such as text classification and summarization. ## experiments and results we conduct experiments on two nlp tasks, i.e., text classification and extractive summarization. the latter one can be also viewed as a classification problem for each sentence level (i.e., to be included in the summary or not). ## conclusion in this paper, we proposed a new text graph representation by incorporating word and character level information. gnn models trained using our text graph provide superior performance in text classification and single-document summarization