Understanding Survey Paper Taxonomy about Large Language Models via Graph Representation Learning (2024)

Abstract: As new research on Large Language Models (LLMs) continues, it is difficult to keep up with new research and models. To help researchers synthesize the new research many have written survey papers, but even those have become numerous. In this paper, we develop a method to automatically assign survey papers to a taxonomy. We collect the metadata of 144 LLM survey papers and explore three paradigms to classify papers within the taxonomy. Our work indicates that leveraging graph structure information on co-category graphs can significantly outperform the language models in two paradigms; pre-trained language models’ fine-tuning and zero-shot/few-shot classifications using LLMs. We find that our model surpasses an average human recognition level and that fine-tuning LLMs using weak labels generated by a smaller model, such as the GCN in this study, can be more effective than using ground-truth labels, revealing the potential of weak-to-strong generalization in the taxonomy classification task.

## introduction collective attention in natural language processing (nlp)-and the wider public-has turned to large language models (llms) @xcite @xcite . it has become so difficult to keep up with the proliferation of new models that many researchers have written survey papers to help synthesize research progress. survey papers are often crucial for newcomers to gain an in-depth understanding of the research evolution. however, volumes of survey papers have become unruly for researchers-especially newcomers-to sift through. as illustrated in figure 1 , the number of survey papers has been increasing significantly. this leads to our research question: is it possible to automatically reduce the barriers for newcomers in a way that can keep up with the constant influx of new information? this question underscores one of the challenges in scholarly document processing @xcite . to answer our research questions, we investigate three types of attributed graphs: text graphs, co-author graphs, and co-category graphs. extensive experiments indicate that leveraging graph structure information of co-category graphs can help better classify the survey papers to the corresponding categories in the proposed taxonomy. moreover, we validate that graph representation learning (grl) can outperform language models in two paradigms, fine-tuning pre-trained language models and zero-shot/few-shot classifications using llms. inspired by a recent study, which indicates that leveraging weak labels, which are generated by smaller (weaker) models, may help enhance the performance of larger (stronger) models @xcite , we further examine whether using weak labels, which are generated by gnns in this study, in the fine-tuning paradigm can help the pre-trained language models. the experiments demonstrate that fine-tuning using weak labels can exceed that using ground-truth labels. for the latter paradigm, we use the results of human recognition as the baseline. the analysis demonstrates that grl achieves higher accuracy and f1 scores, and even surpasses the average human recognition level by a substantial margin. overall, our primary contributions can be summarized as follows: taxonomy classification conventional taxonomy classification is a subset of automatic taxonomy generation (atg), which aims to generate or augment taxonomy for processing scholarly documents @xcite @xcite @xcite . the main challenge in atg is to cluster the unlabeled topics into different hierarchical structures. thus, most existing methods in atg are clustering-based methods. @xcite design a mechanism, grouper, that dynamically groups and la-bels the search results. @xcite propose a model to generate hierarchical clusters. @xcite discover the relationship among words to generate concept hierarchies. within these methods, a subset, called co-clustering, clusters keywords and documents simultaneously @xcite . different from atg, we classify survey papers into corresponding categories in the proposed taxonomy on small and class-imbalanced datasets with highly similar terminologies. graph representation learning graph representation learning (grl) is a powerful approach for learning the representation in graph-structure data @xcite , whereas most recent works achieve this goal using graph neural networks (gnns) @xcite @xcite @xcite . @xcite first introduce a significant advancement in convolution operations applied to graph data using both spatial method and spectral methods. to improve efficiency, @xcite simplify graph convolutions to a first-order polynomial while yielding superior performance in semi-supervised learning. @xcite propose an inductive-learning approach that aggregates node features from corresponding fixed-size local neighbors. regarding atg via gnns, several works attempt to generate or expand taxonomies using new proposed graph models @xcite . compared to model-based approaches, we first improve the performance of taxonomy classification from data-driven perspectives. ## methodology in this section, we first introduce the procedure of data collection and then explore the metadata. we further explain the process of constructing three types of attributed graphs and how we learn graph representation via graph neural networks. ## experiment in this section, we evaluate the graph representation learning (grl)'s effectiveness compared with two paradigms using language models. experimental settings to examine the generalization of our method on various graph structures, we investigate three types of attributed graphs: text graphs, co-author graphs, and cocategory graphs, and compare the classification performance of grl with that of fine-tuning pretrained language models across three subsets of our data. both data n ov23 and data jan24 contain survey papers collected at the end of corresponding months @xcite . data jan24 includes a new category, hardware architecture. we further construct the third subset data subset by removing some proposed categories with fewer instances in data jan24 ; these categories are law, finance, education, hardware architecture, and others. the motivation for constructing three subsets is to validate the generalization of our method across different subsets since the classification performance may significantly change on small datasets. also, new categories may emerge at any period because research on llms is developing rapidly, and so are related survey papers. for example, a new category, hardware architecture, emerges in data jan24 . the change of categories may affect the performance as well. therefore, we investigate our method on three subsets that contain different categories. the statistics of our dataset and corresponding attributed graphs are presented in to evaluate our model, we split the train, validation, and test data as 60%, 20%, and 20%. due to the potential for random splits to result in an easier task for our model, we ran the experiments five times using random seed ids from 0 to 4 and reported the mean values with corresponding standard deviations, mean (std). we evaluate the classification performance by accuracy and weighted f1 score. accuracy is a common metric on classification tasks, whereas the weighted f1 score provides a balanced measure of the class-imbalanced dataset. ## conclusion in this work, we aim to develop a method to automatically assign survey papers about large language models (llms) to a taxonomy. to achieve this goal, we first collected the metadata of 144 llm survey papers and proposed a new taxonomy for these papers. we further explored three paradigms to classify survey papers into the categories in the proposed taxonomy. after investigating three types of attributed graphs, we observed that leveraging graph structure information on co-category graphs can significantly help the taxonomy classification. furthermore, our analysis validates that graph representation learning outperforms pre-trained language models' finetuning, zero-shot/few-shot classifications using llms, and even surpasses an average human recognition level. last but not least, our experiments indicate that fine-tuning pre-trained language models using weak labels, which are generated by a weaker model, such as gcn, can be more effective than using ground-truth labels, revealing the potential for weak-to-strong generalization in the taxonomy classification task. limitations & future work constructing a graph structure may encounter certain constraints. for instance, we build co-category graphs based on the arxiv categories. when papers come from distinct fields, such as biology, physics, and computer science, the graph structure may be very sparse, weakening the effectiveness of grl. in the future, our primary motivation extended from this study is to tailor gpt-based applications to assist readers in understanding survey papers more effectively. also, we could consider different taxonomy designs, such as adopting the computer science ontology or assigning multiple classes to an instance.