Towards a Unified Framework for Reference Retrieval and Related Work Generation (2023)

Abstract: The task of related work generation aims to generate a comprehensive survey of related research topics automatically, saving time and effort for authors. Existing methods simplify this task by using human-annotated references in a large-scale scientific corpus as information sources, which is time- and cost-intensive. To this end, we propose a Unified Reference Retrieval and Related Work Generation Model (UR3WG), which combines reference retrieval and related work generation processes in a unified framework based on the large language model (LLM). Specifically, UR3WG first leverages the world knowledge of LLM to extend the abstract and generate the query for the subsequent retrieval stage. Then a lexicon-enhanced dense retrieval is proposed to search relevant references, where an importance-aware representation of the lexicon is introduced. We also propose multi-granularity contrastive learning to optimize our retriever. Since this task is not simply summarizing the main points in references, it should analyze the complex relationships and present them logically. We propose an instruction-tuning method to leverage LLM to generate related work. Extensive experiments on two wide-applied datasets demonstrate that our model outperforms the state-of-the-art baselines in both generation and retrieval metrics.

## introduction the automatic related work generation (rwg) system, which aims at generating a related work section for a target paper, may help the readers go through the cutting-edge research progress @xcite . two main adjacent stages are considered fundamental disciplines in rwg task: an intuitive retrieval practice in rwg is taking the abstract as a query and employing dense retrieval methods to obtain the related papers according to semantic similarity. however, the abstract given by the user is too vague to be a clear intention for reference. some related topics are inherently correlated with the paper but are not explicitly mentioned in the abstract, indicating semantic mismatches. moreover, the presence of rare proper nouns, e.g., specialized terminologies in scientific papers, makes it difficult for existing dense retrieval methods to capture the semantic nuances. to generate the related work section, a commonused alternative is leveraging the summarization models @xcite . nevertheless, an ideal related work section should reason intricate relations among various references, instead of simply enumerating their main contributions. therefore, training the model to grasp the concept of rwg while producing a high-quality related work section remains challenging. in this work, we leverage the large language model (llm) to enhance retrieval and generation stages in the rwg task to tackle the above challenges. as shown in figure 2 , we propose an unified reference retrieval and related work generation framework (ur 3 wg). specifically, we leverage the llm to extend the vague abstract and generate a query containing more relevant background information since the llm has shown strong knowledge association ability @xcite . the generated query is used to subsequent retrieval stage. to mitigate the diminished performance of dense semantic retrieval methods, we propose a lexicon-enhanced dense retrieval (ler) method to integrate the advantages of lexical retrieval, where a learnable lexical text-matching algorithm is introduced. we also propose a multigranularity contrastive learning method to optimize our retriever, including group-wise and pairwise contrast. since the related work section should comprehensively introduce the relationship between references, instead of simply summarizing the main points, we propose an instruction-tuning method to guide llm to generate a high-quality related work section. extensive experiments conducted on two benchmark datasets show that our model significantly outperforms all the strong baselines, e.g., pushing the rouge-1 to 31.59 (8.4% relative improvement) and bertscore to 0.70 (6.9% relative improvement). our contributions are as follows: (1) we propose the ur 3 wg, the first unified reference retrieval and related work generation model in the rwg task. (2) we propose a lexicon-enhanced dense retrieval method with an llm-based query extension method to retrieve highly related reference papers, supervised by multi-granularity contrastive learning. (3) we propose an instruction-tuning method to incorporate the references and guide the llm to generate related work logically. (4) experimental results on two benchmark datasets show the superi-ority of our proposed model. ## task formulation in this paper, we generate the related work section of the target paper and only use the abstract x as the input without the ground-truth references r * . specifically, we propose a unified reference retrieval and text generation framework, which consists of three steps: (1) leverage the knowledge of llms to extend the input abstract and generate the query used for the subsequent retrieval stage. ## discussion variant model for reference retrieval. for each retrieval, we take the abstract, keywords of abstract, and extended abstract as the query to search relevant documents. specifically, the keywords are extracted via keybert @xcite , and extended abstract is generated via the proposed ur 3 wg. as shown in the impact of lexicon representation group size. ## conclusion in this paper, we propose a ur 3 wg, which fuses the reference retrieval and related work generation into a unified framework based on large language models (llms). concretely, ur 3 wg first extend the input abstract and generate the query used for next retrieval stage. then a lexicon-enhanced dense retrieval (ler) is proposed to retrieve the relevant document at the semantic and lexicon level. the ler is optimized via multi-granularity contrastive learning, including group-and pair-wise contrasts. since rwg is different from simple text summarization tasks, we employ an instruction-tuning method, enabling the llm to understand the definition of rwg task and generate a high-quality related work section. experiments on two datasets demonstrate that our model establishes a new stateof-the-art with automatically retrieved references.