Investigating the Effectiveness of Graph-based Algorithm for B angla Text Classification (2023)

Abstract: In this study, we examine and analyze the behavior of several graph-based models for Bangla text classification tasks. Graph-based algorithms create heterogeneous graphs from text data. Each node represents either a word or a document, and each edge indicates relationship between any two words or word and document. We applied the BERT model and different graph-based models including TextGCN, GAT, BertGAT, and BertGCN on five different datasets including SentNoB, Sarcasm detection, BanFakeNews, Hate speech detection, and Emotion detection datasets for Bangla text. BERT’s model bested the TextGCN and the GAT models by a large difference in terms of accuracy, Macro F1 score, and weighted F1 score. BertGCN and BertGAT are shown to outperform standalone graph models and BERT model. BertGAT excelled in the Emotion detection dataset and achieved a 1%-2% performance boost in Sarcasm detection, Hate speech detection, and BanFakeNews datasets from BERT’s performance. Whereas, BertGCN outperformed BertGAT by 1% for SetNoB, and BanFakeNews datasets while beating BertGAT by 2% for Sarcasm detection, Hate Speech, and Emotion detection datasets. We also examined different variations in graph structure and analyzed their effects.

## introduction natural language processing (nlp) has become very significant in recent years and text classification is one of the most crucial tasks in this domain. text classification is the process of classifying text based on specific labels utilized in document categorization. it has applications in many diverse problems, including hate speech detection, spam detection, sentiment analysis, topic modeling, question answering, intent recognition, medical text analysis, legal document classification, social media analysis, fake news detection, and many more. @xcite graph algorithms can capture complex relationships and dependencies in various text structures and represent better semantic and syntactic relationships. thus graph algorithms help create a more accurate understanding and interpretation of the text. @xcite moreover, these models can identify the grammatical relationship and contextual information between words and documents. by enabling these models to understand how the meaning of words changes based on context, for example, bertgcn performed better than bidirectional encoder representations (bert) @xcite and roberta on different english datasets because of its contextual understanding. graph-based models @xcite can also identify sentiment-related relationships between words in sentiment analysis and can produce more accurate predictions based on sentiments. furthermore, graph algorithms @xcite can capture user interactions, mentions, and relationships during social media text analysis, hate speech detection, sentiment analysis, influence identification, and community recognition tasks. @xcite in recent times, extensive research has been conducted on bangla text using different machine learning and deep learning models @xcite . however, graph-based bangla text classification has remained largely unexplored. on the contrary, numerous graph-based models and structures have been implemented worldwide, particularly in english. the research works @xcite motivate us to conduct our study on how different graph-based models with state-of-art models perform on bangla datasets used for various tasks bert. in this study, the previous best-performing models for each dataset were also observed alongside the implementation of various graph-based methods. graph methods include graph convolutional networks for text (textgcn) @xcite , graph attention networks(gat) @xcite , bertgcn @xcite , and bertgat @xcite . these algorithms were applied to five datasets containing sentnob, sarcasm detection, banfakenews, hate speech detection, and emotion detection datasets. in this study, bert outperformed the previously leading models and demonstrated significantly superior performance compared to gcn and gat across all the datasets. bertgat surpassed bert's performance by 1%-5% for all datasets except sentnob. however, bertgcn achieved better performance by 1%-2% over bert-gat for all datasets, which was attributed to bertgcn's superior ability to comprehend local contextual and global semantic relationships @xcite . as part of the ablation study, using bert embeddings as node features showed better performance for gat and gcn, while onehot embeddings performed better for the integrated models. two types of edge sets: document-word only (d2w) and documenttoword + wordtoword (d2w+w2w) were utilized for all the graph-based models. the edge set d2w+w2w outperformed all the models by 1%-2%. bertgcn outperformed all other models in this study. this research also aimed to find the right balance between graph models and bert. bertgcn exhibited the highest performance when λ ranged from 0.3 to 0.7. bertgat showed better performance when λ ranged from 0.1 to 0.5. ultimately, this study was conducted to demonstrate that graph-based models can outperform traditional models for bangla text classification and pave the way for future research in this domain. our main contributions are: ## related work text classification is one of the classical problems for nlp. naive bayes, svm, and other orthodox approaches for text classifications faced challenges in effectively learning meaningful text representations. addressing these constraints, the application of deep learning models such as convolutional neural networks (cnns) @xcite and recurrent neural networks (rnns) @xcite , materialized. these models demonstrated the ability to capture intricate features from text data. bert from transformers achieves superior performance on sentence-level and token-level tasks @xcite . global structure refers to the information of the whole document, and graphbased models utilize an adjacency matrix to capture this information. to address the constraint of bert models, researchers explored the usage of graph neural networks (gnns) and graph embeddings. some of the most used gnns are graph convolutional network (gcn)(kipf and welling, 2016), gat, graph sample and aggregated @xcite , and, monet @xcite . one such model used for text classification is the textgcn. textgcn focuses on global word co-occurrence information. however, textgcn has significant drawbacks, including a small receptive field, a lack of edge characteristics, over-smoothing, and an inability to accommodate varying neighborhoods. gat overcame these restrictions by utilizing self-attentional layers. the limitations associated with pretraining in gcn and gat are noteworthy. to address this, an innovative approach known as bert-gcn has been developed, strategically amalgamating the advantages of bert and gcn. bertgcn has the power of large-scale pretraining on enormous unrefined data. in addition, by spreading label influence through graph convolution, transductive learning concurrently learns representations for training data and unlabeled test data. @xcite in recent years, there has been a significant amount of research conducted on bangla text classification because of its importance. inverse class frequency along with tf-idf @xcite was proposed for bangla text classification. machine learning algorithms such as naive bayes, j48, knn, and svm were also used for bangla texts @xcite . alam and islam (2018) used logistic regression, svm, liblinear, and neural networks for a massive volume of data with 300k samples in datasets. transformer was used on six different datasets in bangla by alam et al. (2020). ahmed et al. (2022); rahman and chakraborty (2021) implemented deep learning rnn attention layer and rnn with bilstm. furthermore, the bert and electra (efficiently learning an encoder that classifies token replacements accurately) models were tested by rahman et al. (2020). however, graph-based models are very rarely used in bangla text classification, which motivated our investigation to check how graph-based algorithms perform on different bangla text datasets. ## methodology graph-based models use graphs as the representation of textual information. these models build their graphs based on specific criteria. usually, in these text-derived graphs, individual words and entire documents serve as nodes, while connections (edges) represent co-occurrences, semantic similarity, grammatical dependencies, or any other relevant relationship among the nodes. ## result & discussion in this section, we measured performance metrics across all five models for each of the five distinct datasets, facilitating a comprehensive comparative analysis. we evaluated edge features, where we assessed the effects of d2w-only relationships exclusively, as well as the d2w+w2w relationships. subsequently, we turned our attention to a thorough investigation into the utilization of both one-hot embeddings and bert embeddings, with a focus on how these variations influenced the overall model performance. finally, we embarked on the quest to identify the optimal values for the parameter λ, particularly within the bertgcn and bertgat models. our pursuit aimed to unravel the intricacies of their behavior and performance under varying λ values. ## conclusion this study evaluates various graph-based models for bangla text classification and assesses their performance. textgcn and gat exhibit comparatively lower performance when compared to bert. however, the integration of these models with bert yields superior results in comparison to other models for classification tasks. bertgcn incorporates bert's largescale pretraining and fine-tuning, enhanced by transductive learning. bertgcn and bertgat exhibit improved comprehension of local semantics through their integration with bert. we advocate for the adoption of graph-based models, particularly bertgcn and bertgat, for bangla text classification, given their comparatively heightened predictive accuracy when contrasted with traditional text classification models. in conclusion, it's noteworthy that the domain of bangla text remains relatively unexplored in the context of graph-based algorithms and concepts. numerous unexplored avenues including knowledge graphs and alternative graph models beyond gcn and gat demand further exploration. however, it's essential to acknowledge that graph-based models entail significant computational resources, leading us to consider these avenues for future research endeavors.