An Interactive Co-Pilot for Accelerated Research Ideation (2024)

Abstract: In the realm of research support tools, there exists a notable void in resources tailored specifically for aiding researchers during the crucial ideation phase of the research life-cycle. We address this gap by introducing ‘Acceleron’, a ‘Co-Pilot’ for researchers, designed specifically to accelerate the ideation phase of the research life-cycle. Leveraging the reasoning and domain-specific skills of Large Language Models (LLMs) within an agent-based architecture with distinct personas, Acceleron aids researchers through the formulation of a comprehensive research proposals. It emulates the ideation process, engaging researchers in an interactive fashion to validate the novelty of the proposal and generate plausible set-of hypotheses. Notably, it addresses challenges inherent in LLMs, such as hallucinations, implements a two-stage aspect-based retrieval to manage precision-recall trade-offs, and tackles issues of unanswerability. Our observations and end-user evaluations illustrate the efficacy of Acceleron as an enhancer of researcher’s productivity.

## introduction with fast-paced research happening in every field, we are witnessing an exponential growth in the number of scientific articles and research papers on the web. it is difficult for an individual researcher or a small research team to keep abreast of the relevant advances amidst this information explosion. this has a downstream impact on the ability to be consistently appraised and ensure novelty of a proposed solution at various stages of the research life cycle. thus there is an urgent need for a tools that can aid researchers to 1) understand, evaluate and incorporate the latest developments in the literature and 2) formulate/modify the current proposed solution accordingly to ensure novelty and impact. most of the existing tools focus on notifying and recommending researchers with relevant literature, facilitate exploration of existing literature and/or writing research manuscripts. researchers have also proposed learning representations for retrieval of relevant scientific articles @xcite @xcite , literature review generation @xcite @xcite , question answering over scientific articles @xcite @xcite , scientific document summarization @xcite , citation recommendation @xcite @xcite citation intent detection @xcite @xcite , critical review and rebuttal generation @xcite @xcite @xcite , etc. however, to the best of our knowledge, no tool or no approach in the literature facilitates a researcher during the most arduous ideation stage of the research life-cycle. @xcite attempts ideation in completely automated fashion. however, their results demonstrate ∼40% gap in the generation of ideas 'helpful' from the novelty perspective. most of the tasks involved in research require domain expertise and complex reasoning skills. the recent advancement in large language models (llms) has made it possible to partially automate some of these tasks @xcite @xcite @xcite . however complete automation of these tasks may not yield qualitative outcomes. in this work, we propose 'acceleron' (figure 1 ), a tool to accelerate the research life cycle. the tool exploits the reasoning and domain specific skills of llm based agents not to replace researchers but to assist them for research ideation. with llm powered mentor and colleague agents, acceleron provides relevant inputs to researchers in an interactive fashion via a user-friendly interface. thus, it aids the researcher to develop the research proposal consisting of a validated motivation, a well-defined research problem focusing of research gaps in the literature, a proposed approach selected from a set-of plausible synthesized methods and possible set-of experiments to be conducted to evaluate the approach for the research problem. to the best of our knowledge, we are the first ones to mimic the research ideation process using llms and execute it using human-machine interaction ensuring accelerated as well as qualitative outcomes, in terms of novel ideas. ## system architecture acceleron provides a web-based interface for researchers to interact. the system architecture is illustrated in figure 2 . we define an llm agent based architecture @xcite , with agents of two distinct types of profiles or personas. a colleague persona foot_0 performs less complex tasks including extraction of relevant information from user inputs, generation of relevant questions from extracted information or retrieval of relevant data from scientific documents. whereas, mentor persona 2 performs more complex tasks requiring reasoning such as understanding the limitations or gaps of the existing work, identifying problems similar to the problem discussed in the proposal, identifying sub-tasks of the problem being solved in the proposal, solving similar problems and/or sub-tasks to synthesize a solution to the proposed problem and re-write the proposal given a plausible set-of approaches or possible limitations of related work. the architecture is flexible such that the llm agents can interact with (i) llms like gpt-3.5-turbo 3 , cohere 4 and gemini 5 using api calls or (ii) open-source llms like llama-2 6 , zephyr 7 , mixtral 8 which reside on an internal hosting server. we expect to have a global repository which is a vector store of domain specific scientific articles 9 which are indexed by the specter embeddings (cohan et al., 2020) produced using the paper's title and abstract. we also have a user specific corpus which has chunks of all the retrieved papers relevant to the current proposal the researcher is working on. the paper chunks are created with our in-house parser 10 treating paragraphs as semantic segments. if a paragraph does not fit into the the maximum token length of llm agents, while chunking it is further split to fit into the maximum token length. the chunks are further converted to vector embeddings and indexed for efficient retrieval based on semantic similarity with a query. this user corpus acts as a shared 'memory' for the llm agents. ## approach the acceleron ideation simulation involves interaction between a researcher and the llm agents, where the llm agents perform actions based on the feedback received by the researcher or another agent. the process takes a proposal as an input from a researcher with a research problem description specified at a high level along with the motiva2 openapi's gpt4 model 3 https://platform.openai.com/docs/models/gpt-3-5-turbo 4 https://cohere.com/ 5 https://gemini.google.com/ 6 https://llama.meta.com/ 7 https://huggingface.co/huggingfaceh4/zephyr-7b-beta 8 https://mistral.ai/news/mixtral-of-experts/ 9 we use more than 2 million scientific articles in semantic scholar fetched using s2orc dataset @xcite as the global repository 10 we built a pdf parser using pdfminer global corpus of scientific articles colleague llm agent mentor llm agent researcher extract motivation and generate questions to validate the motivation proposal title and abstract user corpus (retrieved paper chuncks) retriever top-k similar articles parser and indexer faiss index questions provide edit articles which addresses the motivation extract the limitations or gaps of the work research gaps re-write the proposal taking the gaps into consideration updated proposal title and abstract update 1 2 3 4 6 5 7 8 9 query retrieve 10 11 12 query 13 14 15 16 17 18 19 20 21 15 tion behind the problem. the process involves: (i) analyzing the existing literature to critically evaluate the motivation behind the research problem a researcher is trying to address to ensure that the mentioned research gap(s) still exist(s), (ii) reformulating the proposed research problem and objectives based on the validation stage output and re-identification of research gaps, (iii) identifying analogous research problems or sub-problems addressed in the literature and utilizing their solutions, available in the literature, to derive a set-of approaches or synthesizing a set-of plausible methods as a solution to the problem, (iv) designing experimentation strategy for the given problem and selected methodology. the output of the ideation process is the updated proposal with a (i) a research problem with validated motivation (ii) plausible methods to address the research problem. the overall ideation task is split into two pipelines: (i) motivation validation and (ii) method synthesis. the detailed prompts for the steps in each of the pipeline are illustrated in the appendix section a.2. ## novel components with acceleron our aim is to bridge human-computer interaction and natural language processing using an interactive tool infused with the best of nlp and goodness of hci. we created several novel components within acceleron that fixes known shortcomings of nlp based systems using hci inspired ideas. ## qualitative analysis of the workflows in the absence of an appropriate dataset for the tasks relevant to the ideation process, we evaluate our workflows by user-studies. we allow re-searchers working in distinct domains like computer science, material science and life science, to use acceleron for ideation of their research problems. for computer science domain , we use semantic scholar data fetched using s2orc dataset @xcite as our global repository. whereas, for material science and life science domain we use our repository of papers downloaded from 'science direct' 11 and 'pubmed' foot_2 , respectively. we utilize the logging functionality of 'acceleron' to keep track of the interactions between the researcher and the llm agents. for space constraints and data confidentiality preservation of unpublished work, here, we provide a qualitative analysis of the workflows with 2 proposals from distinct researchers, specifically in the domain of artificial intelligence (ai), machine learning (ml) and natural language processing (nlp). the topics of these proposals are: (i) topic-based citation retrieval for research proposal and (ii) reference-free evaluation metric for retrieval augmented question answering. we receive an input from a researcher with a proposal titled 'topic-based citation retrieval for research proposal' and the corresponding abstract 'retrieval of research articles pertinent to a given query represents a thoroughly investigated research challenge. typically, queries take the form of a title and abstract of a research article, or a specific sentence or paragraph from an existing research article requiring citation. however, existing approaches presuppose the availability of a well-constructed manuscript, an assumption that is inappropriate during the initial research proposal writing stage. at this initial phase, researchers seek pertinent literature for citing in their proposals, often focusing on specific topics or intents and further build the proposal. in this work, we aim to tackle the issue of topic-based citation retrieval for research proposals. we anticipate researchers providing the title and abstract of their research proposals, encompassing elements such as the research gap, problem statement, and a high-level overview of the proposed methodology and experiments. additionally, researchers will furnish a list of topics for which relevant scientific articles need to be retrieved. our proposed algorithm intends not only to fetch research articles pertinent to the given proposal from a corpus, but also to establish a crucial many-to-many mapping between these articles and the specified topics.' the colleague llm agent generates the following questions for validation of the motivation: 1. "is the research paper specifically addressing the retrieval of research articles relevant to a topic of a research proposal?" and 2. "is the research paper developing a technique to map research articles to specified topics in research proposals?". out of top-50 research articles used to validate the motivation of the proposal by posing the above mentioned questions, four (appendix a.1.1) got retrieved to be answering as 'yes' to at the least one of the above questions, and thus invalidating the motivation behind the proposal. however, the justifications provided for these papers highlight that paper no. 1 and 3 introduce an approach for citation recommendations during the writing phase of the target manuscripts and not at the proposal writing stage. also, scientific article 2 leverages contents of a target paper and citation graph to extract scientific information. the outcome of the scientific article 4 is a dataset which can be useful for the proposal, but does not address the task of 'topic-based citation retrieval for research proposal'. thus, we observe that after evaluating the retrieved scientific articles claimed to be invalidating the proposal, the researcher disagrees with the justifications provided for each of the retrieved articles for addressing the motivation behind the proposal, hence validating the novelty of the proposal. this exemplifies the need as well as the effectiveness of this human computer interaction facility provided by the tool for the workflow. this example demonstrates acceleration of motivation validation stage of the research-life cycle (∼8x for this proposal as stated by the researcher), by eliminating the need for the researcher to manually go through multiple relevant research articles retrieved by generic or academic search engines to ensure that the literature does not have a solution for the specific problem the researcher is trying to address, leading to a time consuming process. we receive input from another researcher with the proposal titled 'reference-free evaluation metric for retrieval augmented question answering task' and the abstract 'we observe that questions with long answers on long documents do not have unique reference evidences (relevant paragraphs from the document) and answers. rather, there is a distribution over reference answers, making expert based evaluation expensive and existing unique reference-based evaluation metrics inadequate. we also do not find any reference-free evaluation met-ric designed for evaluating retrieval augmented question answering task. hence, this this work we propose to define this metric.'. the colleague llm agent generates the following question to validate the motivation of the proposal: "is the research paper proposes a reference-free evaluation metric designed for evaluating retrieval augmented question answering tasks?". we observe that out of top-50 retrieved scientific articles relevant to the proposal, none of the articles provides answer as 'yes' to the question, leading to retrieval of zero relevant paper hence invalidating the motivation of the proposal. manual analysis of the top-50 retrieved articles performed by the researcher (as well as other relevant articles manually visited by the researcher) to evaluate this outcome of the workflow, substantiates the results. for the next workflow of method synthesis for the above proposal, the mentor llm agent generates following set of research problems similar to the problem defined in the proposal: 1. "evaluating complex tasks where there is no unique correct answer or reference", 2. "designing evaluation metrics for tasks that involve retrieval and interpretation of large amounts of data", 3. "creating reference-free evaluation metrics for tasks where reference-based metrics are inadequate or impractical", 4. "assessing the quality of answers in tasks where the answers can be long and drawn from extensive documents". the mentor llm agent also generates the following sub-tasks for the problem defined in the proposal: 1. "defining a new metric that can effectively evaluate retrieval augmented question answering tasks" and 2. "overcoming the inadequacy of existing unique reference-based evaluation metrics for questions with long answers on long documents". with these similar and subproblems as queries, the colleague llm agent first retrieves top-10 similar scientific articles per statement (total 40 articles with some overlap as same article may be retrieved for multiple queries) and then poses the question that 'if the article provides a methodology or an approach to solve the above defined problem or sub-task'. the researcher receives total 17 scientific articles which answers 'yes' to the question along with a description of the methodology implemented for each of the above problems or task. out of these papers, the researcher accepts 11 (appendix a.1.2) scientific articles, finding them to be more relevant to the problem the researcher is trying to address. based on approaches discussed in these scientific articles to solve the above enlisted similar problems and/or sub-tasks, the mentor llm agent suggests the following 10 plausible methodologies for the proposal. theses suggestions with the exact verbatim provided by the agent are as follows: 1. ava (automatic evaluation approach): this approach uses transformer-based language models to encode the question, answer, and reference text. by measuring the similarity between the reference and an automatic answer, biased towards the question semantics, ava can estimate the system accuracy. this approach can be used to evaluate retrieval augmented question answering tasks and overcome the inadequacy of existing unique reference-based evaluation metrics. 2. informedness and markedness measures: these new probabilistic and information theoretic measures can be used to evaluate complex tasks where there is no unique correct answer or reference. they can also be used to design evaluation metrics for tasks that involve retrieval and interpretation of large amounts of data. 3. large language models (llms): llms can be used as automated metrics for evaluating tasks that involve retrieval and interpretation of large amounts of data. they can also be used to assess the quality of answers in tasks where the answers can be long and drawn from extensive documents. this output showcases the quality of method recommendations provided by the tool for the given proposal. though mentioned at high-level, the researcher agrees that most of these methods are well-suited as a plausible approach for the given proposal. though there is a need for further work to finalize the most appropriate plausible method for proposal, the researcher finds this first cut of output provided by the tool to be relevant and the overall process to be significantly more efficient than the regular process followed by the researcher for constructing a plausible set-of approaches for the given problem, by searching through the relevant literature from scratch. these examples illustrating the outcomes of the motivation validation and method synthesis phases of the ideation workflow of the tool, demonstrates the efficacy of the tool, in terms of providing relevant outputs at each stage of the workflow. the observations made in terms of time saved by the researchers with the tool usage for the respective tasks demonstrates the power of the tool with regards to time efficiency gains. ## conclusion in this work, we have demonstrated a tool called 'acceleron', developed to accelerate the ideation phase of the research life-cycle. to the best of our knowledge this is the first tool which addresses the tasks involved in the ideation stage. to emulate the ideation process, we use llm agents with colleague and mentor personas to execute two workflows, viz. motivation validation and method synthesis, which engage researchers in an interactive fashion to develop the research proposal. our workflow involves novel components to (i) alleviate the hallucinations of llms through user interaction, (ii) ensure relevant outcomes by two-stage aspect based retrieval, where first stage introduces higher recall reducing false negatives and false positives are corrected by user interaction and second stage of more precise fine-grained aspect-based retrieval, (iii) introduction of unanswerability and (iv) moderation of generativeai via human interaction acting as a pseudo first layer of moderation increases user involvement in the final task specific outcome. the qualitative analysis performed with proposals from researchers in distinct domains, demonstrates qualitative outcomes for various stages in the workflow with ∼7.5x gains in the time efficiency for various stages of the ideation phase. most importantly, expert-interaction avoids error propagation through the stages of workflows yielding qualitative outputs in terms of generation of novel and diverse ideas.