Which Sentence Representation is More Informative: An Analysis on Text Classification (2023)

Abstract: Text classification is a popular and well-studied problem in Natural Language Processing. Most previous work on text classification has focused on deep neural networks such as LSTMs and CNNs. However, text classification studies using syntactic and semantic information are very limited in the literature. In this study, we propose a model using Graph Attention Network (GAT) that incorporates semantic and syntactic information as input for the text classification task. The semantic representations of UCCA and AMR are used as semantic information and the dependency tree is used as syntactic information. Extensive experimental results and in-depth analysis show that UCCA-GAT model, which is a semantic-aware model outperforms the AMR-GAT and DEP-GAT, which are semantic and syntax-aware models respectively. We also provide a comprehensive analysis of the proposed model to understand the limitations of the representations for the problem.

## introduction the text classification problem has been widely studied in the literature @xcite in the field of natural language processing @xcite . the text classification problem has been recently used as a downstream task in senteval @xcite , a toolkit for evaluating sentence representations. in the literature, studies on semantic textual similarity (sts) @xcite have used the text classification to evaluate the sentence embeddings learned by their proposed models using the datasets provided by the senteval toolkit @xcite . for text classification, traditional deep learning models such as long short-term memory (lstm) networks @xcite and convolutional neural networks (cnn) @xcite have been adopted. these deep learning models capture the local semantic and syntactic information by using the input as a sequence of words but they ignore the semantic and syntactic information of the input @xcite . recently, graph neural networks (gnns) @xcite have been used for text classification @xcite , sequence labeling @xcite , and question answering @xcite . in dependency parsing the aim is to find a tree that represents dependencies between words in a sentence. on the contrary, semantic parsing maps a text to its formal representation that provides an abstraction of its meaning. there has been a recent increase in the studies that propose various neural network architectures such as tree-lstm @xcite , heterogeneous graph transformer @xcite , and transformer @xcite that integrate semantic and syntactic information. gnn models that integrate external representations into deep learning models referred to as semantic and syntax-aware models, are the well-studied models in the literature for various nlp problems such as neural machine translation (nmt) @xcite and text classification @xcite . these models have gained attention because they are capable of capturing information over long distances, especially between discontinuous constituents @xcite . in this study, we analyzed the impact of semantic and syntactic representations within graph attention networks (gat), particularly for the text classification problem. we used the dataset provided by senteval toolkit @xcite . we constructed the gat model by integrating abstract meaning representation (amr) @xcite and the universal conceptual cognitive annotation (ucca) @xcite as graph-based semantic representations and the dependency tree as syntactic representation. since the size of the datasets in senteval toolkit @xcite is different, we evaluated the results of our proposed model with the studies that use the senteval toolkit @xcite . 1 the rest of the paper is organized as follows. section 2 reviews similar semantic and syntax-aware models. section 3 describes our methodology for addressing the text classification problem using semantic and syntactic parser models. section 4 presents our experimental results along with a detailed analysis of the proposed models. finally, section 5 concludes the paper with insights on the impact of the semantic and syntactic information on the classification problem. ## related work in addition to the traditional neural networks that simply rely on neural language models, semantic and syntax-aware models have been recently used effectively in nlp problems such as text classification @xcite @xcite , natural language generation @xcite , question answering (schlichtkrull et al., 2020), semantic role labeling (srl) @xcite , reading comprehension (sachan and xing, 2016; galitsky, 2020), text summarization @xcite , language modelling @xcite , and machine translation @xcite @xcite . dependency trees usually provide sufficient syntactic information in various nlp tasks @xcite @xcite and improve the performance of the models considerably. as for the external resource of semantic information, the most popular semantic representation is the amr (hardy and vlachos, 2018; elbasani and @xcite . in particular, gnns @xcite @xcite @xcite have been used as models into which syntactic and semantic information are easily integrated. in addition to gnns, transformers have also been used to integrate such external resources such as syntax-aware word representation (sawr) @xcite , syntax-aware local at-tention (sla) @xcite , syntax-graph guided self-attention (sgsa) @xcite , scene-aware self-attention (sasa), and scene-aware cross-attention (sacra) head @xcite . last but not least, the heterogeneous graph transformer @xcite , a customized version of the transformer @xcite , has been recently introduced as a model with semantic amr information @xcite . ## methodology in this section, we describe the proposed semanticand syntax-aware gat models that integrate semantic and syntactic information as external resources into the model. first, we explain the preprocessing step that is performed to convert the text into the required form to be processed by the gat model. ## conclusion semantic and syntax-aware models have recently been proposed for various nlp problems, that especially require long-distance information, especially between discontinuous constituents, in addition to the local information captured by sequential models. in this paper, we propose a graph neural network model that incorporates semantic and syntactic information for the text classification task. to the best of our knowledge, this is the first study that compares semantic and syntactic information used in a graph neural network, specifically for the task of text classification. we present a detailed analysis of the results, showing that the ucca semantic information improves the performance of the classification model compared to syntactic information (i.e. dependency tree). however, we were not able to obtain similar results with the model using the amr semantic representation. this shows that the preprocessing step to convert the graph into adjacency and feature matrices is a very important step in gnn models. as future work, we plan to improve the preprocessing step to obtain more informative adjacency and feature matrices.