Artificial Intuition: Efficient Classification of Scientific Abstracts (2024)

Abstract: It is desirable to coarsely classify short scientific texts, such as grant or publication abstracts, for strategic insight or research portfolio management. These texts efficiently transmit dense information to experts possessing a rich body of knowledge to aid interpretation. Yet this task is remarkably difficult to automate because of brevity and the absence of context. To address this gap, we have developed a novel approach to generate and appropriately assign coarse domain-specific labels. We show that a Large Language Model (LLM) can provide metadata essential to the task, in a process akin to the augmentation of supplemental knowledge representing human intuition, and propose a workflow. As a pilot study, we use a corpus of award abstracts from the National Aeronautics and Space Administration (NASA). We develop new assessment tools in concert with established performance metrics.

## introduction analyzing technical documents is a crucial strategic task, enabling the management of research portfolios, tracking investment trends, and exploring scientific advancement. on a more tactical level, it can aid the preliminary screening of scientific abstracts in systematic reviews @xcite @xcite . several approaches are possible. first, authors can label their own work, but this presents several challenges: (1) authors that self-label their own texts may make idiosyncratic decisions, (2) authors in close disciplines may use different terms for related concepts, such as "robotics" and "autonomy", and (3) multidisciplinary projects may require novel or multiple labels. a second method is to impose an external ontology. however, these schemes often have both fineand coarse-granularities (e.g., "networks" versus "ad-hoc networks"). another concern is that the scheme simply lacks appropriate labels, especially for emerging fields. automated processes do exist. those with a large number of parameters are now customizable at lower computational cost @xcite . although dedicated pretrained models can yield robust results, they incur significant expenses in manual annotation due to reliance on supervised learning @xcite @xcite . in summary, we face two distinct needs in the analysis of scientific documents: (1) a unified, coarse-grained, non-overlapping taxonomy, tailored to uniquely classify a set of documents; and in manual labeling, an expert's rapid progress often hinges on integrating prior knowledge, crucial for effective comprehension @xcite . in so doing, the expert rapidly identifies the phrases conveying the most information and uses those for classification. importantly, this process is not a simple frequency or statistical analysis; indeed, the most important phrase may appear only once. moreover, multigrams carrying high semantic value may not appear systematically in the same place in a sentence or paragraph. here we describe "artificial intuition," a method mimicking the expert's process to execute two objectives: generating an optimal label space and producing accurate predictions within this new space. we integrate tools into a novel workflow to identify important terms, augment them with relevant background information, then aggregate these enhanced documents into clusters for classification purposes. as a pilot case to evaluate our methodology, we analyze award abstracts of federally funded projects from the national aeronautics and space administration (nasa) small business innova-tion research (sbir) program. we obtain domain knowledge by extracting and ranking the abstract's keywords / keyphrases (which we will collectively refer to as "keywords"). we generate metadata for these keywords in a zero-shot setting and derive embeddings for the keyword-metadata concatenations using a pre-trained sentence transformer. for label space generation, we implement a clustering process that represents the task of organizing awards into funding themes. this method not only clarifies the thematic organization of the documents but also reveals the hierarchical relationships between different topics. we introduce a novel evaluation scheme to assess whether the label set comprehensively spans the document space and can serve as a set of basis vectors. to predict labels, we reinterpret the multilabelclassification problem as a semantic matching challenge wherein the document space is characterized by the keyword-metadata concatenation and the label space is described by the element closest to the centroid for each cluster. this retrieval-based perspective allows for flexibility in adapting to new label spaces without the need for retraining. this framework accommodates various levels of parsimony, which we explore extensively in our experiments. finally, using our test sample, we demonstrate the efficacy of our prediction methodology and quantify the performance. ## related work various methodologies have been proposed for text classification. bayesian approaches @xcite classify the text by extracting features. one method is to first select document features with discriminative power, then compute the semantic similarity between features and documents @xcite , but this becomes more difficult as the number of features grows. support vector machines (svms) can be used for document classification @xcite . however, these approaches are constrained by the requirement for manual feature engineering, limiting their ability to capture the complexity of natural language. new deep learning techniques have advanced scientific document classification. neural network-based architectures (lee and dernoncourt, 2016), particularly convolutional neural networks (cnns) @xcite and recurrent neural networks (rnns) @xcite , outperform some traditional machine learn-ing methods. these models automatically learn feature representations from data, capturing both the semantic and syntactic nuances of text. these methods presume that documents are related to only one label. newer approaches (e.g., @xcite @xcite @xcite ) classify documents with multiple labels, and one alternative attempts to map 10,000 fine-grained labels for scientific documents @xcite although most methods consider 10-50 coarse labels. these models are incompletely validated because many real-world datasets will have limited or poorly labeled data. weakly supervised learning and zero-shot learning (zsl) models do not use annotated data. some pre-trained language models demonstrate impressive performance in zero-shot document classification @xcite @xcite and can be used to assign multiple labels to a given document @xcite . on the other hand, hierarchical multi-class methods can use just class names -without training examples -as supervision @xcite . large language models trained on scientific data, such as galactica @xcite and scincl @xcite , can be used to assign labels to a scientific document. many approaches use metadata, such as generic descriptions, as supervision for further classification @xcite . however, these methods are still potentially subject to noise. here we describe a method to identify keywords and derive context-specific metadata to improve classification accuracy, particularly for short abstracts. ## conclusion for labeling short scientific documents, such as abstracts, pre-existing domain-specific taxonomies are ambiguous. defining a label space spanning the set of documents is an important task that humans execute easily. in this paper, we demonstrate that the text of the documents is insufficient to either define the label space or predict the labels. we present evidence that an llm can provide critical metadata to address this gap, forming the basis for artificial intuition. additionally, we propose both an architecture to address this and two novel measures to evaluate the constructed label spaces. testing our model with a corpus of nasa award abstracts, we demonstrate a workflow that integrates the llm's supplemental data successfully.