Beyond Retrieval: Topic-based Alignment of Scientific Papers to Research Proposal (2024)

Abstract: The inception of a research agenda typically commences with the creation of a comprehensive research proposal. The efficacy of the proposal often hinges on its ability to connect with the existing scientific literature that supports its ideas. To effectively assess the relevance of existing articles to a research proposal, it is imperative to categorize these articles into high-level thematic groups, referred to as topics, that align with the proposal. This paper introduces a novel task of aligning scientific articles, relevant to a proposal, with researcher-provided proposal topics. Additionally, we construct a dataset to serve as a benchmark for this task. We establish human and Large Language Model (LLM) baselines and propose a novel three-stage approach to address this challenge. We synthesize and use pseudo-labels that map proposal topics to text spans from cited articles to train Language Models (LMs) for two purposes: (i) as a retriever, to extract relevant text spans from cited articles for each topic, and (ii) as a classifier, to categorize the articles into the proposal topics. Our retriever-classifier pipeline, which employs very small open-source LMs fine-tuned with our constructed dataset, achieves results comparable to a vanilla paid LLM-based classifier, demonstrating its efficacy. However, a notable gap of 23.57 F1 score between our approach and the human baseline highlights the complexity of this task and emphasizes the need for further research.

## introduction researchers frequently draft research proposals to present new ideas, define research agendas and seek funding grants. an integral part of the proposal writing process is reviewing relevant literature and relating it to different aspects of the proposal. several existing approaches designed for automatic * equal contribution retrieval of scientific articles can be applied to identify articles relevant to a proposal, with a detailed description (abstract) of the proposal serving as the query @xcite . however, there is often a further need to formulate high-level thematic categories (henceforth referred to as topics) relevant to the proposal and map retrieved relevant scientific articles to these categories for fine-grained contextualization. such fine-grained mapping can further facilitate motivating the research problem, identifying its novelty, establishing baselines, synthesizing methods, and automatic literature review generation. given a set of reference article abstracts relevant to a proposal, @xcite ; @xcite auto-generates the thematic categories in a hierarchical form (termed as a catalogue) and organize references. however, the results demonstrate that the auto-generated catalogue does not match with the original-author-defined catalogue, leading to discrepancies in downstream literature review generation. this is due to inability of stateof-the-art lms as well as subjectivity of the task. as opposed to this, we consider a more realistic setting, where we assume the availability of not only the reference papers retrieved to be relevant to a proposal; but also high-level topics provided by the researcher, for further literature categorization. with this assumption, we focus on the novel task of alignment of these reference papers, to one or more of these topics, offering a comprehensive understanding of its distinct contributions to the target proposal. this ensures a more personalized approach, aligning the cataloguing process closely with the researcher's unique perspective representing their understanding of the field. figure 1 illustrates an example, showcasing proposal topics, relevant reference papers and their mappings to the topics, with distinct text spans from the reference paper representing the relevant context (henceforth termed as the reference text figure 1 : a sample from the dataset illustrating the many-to-many relationship between the proposal topics and reference papers. reference papers p i and p l are cited under topic t k , while reference paper p i is cited by both topics t j and t k with distinct reference text spans from the paper. the highlighted texts emphasize the similarities among the topic, citation text, and reference text spans. span). thus, a reference can be cited in more than one proposal topic with distinct context (reference text spans) @xcite . considering the token lengths of scientific articles, vanilla paid llm-based approach for this task would be expensive. we consider the requirement of using smaller open-source lms without compromising on the performance of llms. hence, there is a need for a dataset to fine-tune domain-specific lms for the task. as existing datasets for scientific document understanding @xcite @xcite @xcite are not well-suited for this task, we extend the unarxiv dataset @xcite , originally designed for literature review generation. we establish human and vanilla llm baselines and define a novel three-stage approach for topic-based alignment of papers. since ground truth labels for the reference text spans relevant to topics are not available, we augment the data by retrieving the reference text spans using the text around the article citation within the proposal topic (citation text), which is assumed to be available only for training. we use the synthesized pseudo pairs to train language models (lm) for retrieval of text spans from articles for a topic, and for classification of the retrieved articles to the topic. as opposed to all the prior approaches (hoang and kan, 2010; @xcite @xcite @xcite , considering the realistic need of initial proposal writing stage, we neither assume the availability of citation text for the cited papers nor detailed description of the topics during inference. our primary contributions are: ## task definition consider a research proposal with its title and abstract r and with a set of topics forming the catalogue denoted as t @xmath0, ..., t k . we assume a corpus of reference research papers p @xmath1, ..., p n retrieved using r via existing retrieval methods ## dataset construction figure 2 demonstrates the steps we have followed for the construction of our dataset. we utilize the papers in the unarxiv dataset @xcite , with their titles and abstracts as target proposals r. we leverage the arxiv tags of these papers to obtain their domains and focus on papers in ai and computer vision. this was to ensure high-quality annotations since the human annotator who facilitated the establishment of the baseline (section 6.2) had expertise in these domains. ## approach we break down our approach into three stages ( figure 2 ): (i) augmenting the dataset with positive and negative reference text spans for a proposal topic, retrieved from papers relevant to the proposal, with the citation text as the query, using a state-of-the-art retriever model (sr) (ii) training the topic based reference text span retriever (t r) and reference paper classifier (t c) using the augmented data (iii) using tc to classify a reference paper for its relevance to a proposal topic in context of the proposal and the reference text span retrieved from the paper relevant to the topic using t r . ## conclusion and future work we have defined a novel task of mapping relevant scientific articles to research proposal topics as a precursor to the literature review generation task for a new research proposal. we introduce a large-scale dataset for the task and establishment of competitive baselines by an expert and an llm, underscoring task feasibility. we define a novel approach for the task and are the first ones to simulate the real-life scenario, at the early stage of proposal writing, of unavailability of citation text or detailed topic descriptions to retrieve topic-wise reference text spans from relevant articles. we came up with a novel strategy of using citation text (available for the training data) as a link between the topic and the text spans to create pseudo-labels for training a retriever. our pipeline using much smaller open-source lms trained with pseudo-labels yields comparable performance to that of the paid llm baseline, demonstrating the efficacy of constructed data and designed pipeline. in contrast, a noticeable gap of 23.57 f1 score between our approach and the human baseline underscores the task's complexity, demanding more sophisticated solutions. while organizing reference articles by topic, the hierarchy among these topics (catalogue), is crucial for a comprehensive understanding. however, we found it difficult to capture this hierarchy due to the pdf parsing challenges, treating every topic as a standalone entity. in future, with more sophisticated pdf parsing and semantic paragraph segmentation techniques, we plan to capture the topic hierarchy for task completeness. the primary objective is to gain a profound understanding of the challenges inherent in the task. the identified 121 erroneous annotations provided by human are categorized into the following groups: • subjectivity: these errors, constituting 77.69% of total erroneous cases, stem from a misalignment between the thought processes of the author and the annotator. as shown in figure 4 (a), the reference paper is not cited by the author under the given topic, however human annotator tends to map the reference paper to the topic due to the match between the semantics of the citation text span and the topic. in these instances, the annotator correctly deems the reference article relevant to the topic, while the author holds a contrasting view. this emphasizes the inherent subjectivity embedded in the task, where interpretations of relevance can diverge between individuals. the systematic categorization of these errors showcases the prevalence of subjectivity, arising from differences in the author's writing style and thought process which constitutes the majority of mis-classifications by human annotators.