Exploring and Verbalizing Academic Ideas by Concept Co-occurrence (2023)

Abstract: Researchers usually come up with new ideas only after thoroughly comprehending vast quantities of literature. The difficulty of this procedure is exacerbated by the fact that the number of academic publications is growing exponentially. In this study, we devise a framework based on concept co-occurrence for academic idea inspiration, which has been integrated into a research assistant system. From our perspective, the emergence of a new idea can be regarded as the fusion of two concepts that co-occur in an academic paper. We construct evolving concept graphs according to the co-occurrence relationship of concepts from 20 disciplines or topics. Then we design a temporal link prediction method based on masked language model to explore potential connections between different concepts. To verbalize the newly discovered connections, we also utilize the pretrained language model to generate a description of an idea based on a new data structure called co-occurrence citation quintuple. We evaluate our proposed system using both automatic metrics and human assessment. The results demonstrate that our system has broad prospects and can assist researchers in expediting the process of discovering new ideas.

## introduction academic publications have witnessed the evolution and advancement of human civilization. in modern society, out-of-box and interdisciplinary scientific work can get more attention from science funders, industry, and the public @xcite , where a good idea is the cornerstone of academic research. however, for most researchers, it takes a lot of time to put forward new ideas. for one thing, the number of academic publications is increasing exponentially, and it is difficult for an independent researcher to understand these papers thoroughly. besides, researchers often focus on their specialized but narrow fields, which makes it a challenge to discover underlying connections beyond their familiar areas @xcite . in this work, our purpose is to unveil the profound connections between different academic concepts and ignite researchers' exploration of potential academic ideas while expediting the research process. the two primary goals are idea exploration and verbalization. for the first goal, we need to understand how new ideas originate. generally speaking, the emergence of a simple idea is often formed by the interaction between two different concepts rather than from scratch. for example, the combination of convolution and graph neural network contributes to graph convolutional network @xcite . this understanding of idea as connection and combination inspires us to model the process of idea exploration as a link prediction task based on the evolving co-occurrence graph of concepts. such graphs are constructed according to the cooccurrence relationship of concepts in the papers published in different years. it should be highlighted that there exist numerous factors leading to new ideas in the real world. we provide a possible way as a preliminary exploration. the second goal, idea verbalization, is carried out after idea exploration to generate fluent and reasonable texts describing an idea, which usually comprises new contents derived from the combination of two different concepts. we retrieve sentences pertaining to concepts from existing publications and then verbalize ideas using the technique of natural language generation. specifically, we propose a new data structure called co-occurrence citation quintuple (figure 1 ), which stores two concepts, their corresponding sentences of papers, and idea texts. the definition is given in section 3.1. the quintuple is an extension of edges in the evolving concept co-occurrence graph and indicates where an idea comes from. we use such quintuples to train a sequence-to-sequence text generation model. in our application scenario, there are various types of disciplines. each of them has distinct characteristics and concepts. existing methods of link prediction and text generation @xcite @xcite @xcite are mostly trained on one dataset by optimizing a set of parameters. owing to the fact that different datasets require specific training configurations and hyperparameters, such models cannot be transferred to other datasets. particularly, link prediction models need to set the scale of graphs before training, such as the number of nodes. moreover, in the field of natural language generation, some works @xcite tend to construct domain knowledge bases as external information to generate texts. however, building large knowledge bases for each discipline takes tremendous resources, which is unrealistic. to this end, it is preferable to design general and informative models which can be applied to numerous disciplines. thanks to the abundant training corpus of pretrained language models (plms) such as bert @xcite , t5 @xcite , bart @xcite , and gpt @xcite , plm can be regarded as an implicit knowledge graph @xcite , which has the ability of extrapolation. in this work, we integrate the whole academic information into the same representation space by leveraging the capability of plm to break through disciplinary barriers. for idea exploration, we devise a plm-based link prediction method, which only needs to train one set of model parameters. for idea verbalization, we use another sequence-to-sequence-based plm endowed with academic knowledge from millions of highly-cited papers via unsupervised denoising training. subsequently, we re-train the denoised plm with co-occurrence citation quintuples in a supervised way. our contributions are summarized as follows: ## conclusion we model the emergence of a new idea as two sequential processes: temporal link prediction for exploration and text generation for verbalization. to achieve the objectives, we first construct and release two datasets with new data structures, including evolving concept co-occurrence graph and co-occurrence citation quintuple. then, we devise a new temporal link prediction method based on the masked language model, which can be applied to various evolving concept co-occurrence graphs of different disciplines. finally, we finetune a plm to verbalize ideas using the released quintuples. the pipeline has been integrated into a system free for researchers to obtain inspiration. from the experiments and the feedback of users, our system can provide useful information for idea discovery. in the future, we will release an academic oriented language model with the paradigm of prompt learning and instruction tuning to tackle both link prediction and text generation.