This Reference Does Not Exist: An Exploration of LLM Citation Accuracy and Relevance (2024)

Abstract: Citations are a fundamental and indispensable part of research writing. They provide support and lend credibility to research findings. Recent GPT-fueled interest in large language models (LLMs) has shone a spotlight on the capabilities and limitations of these models when generating relevant citations for a document. Recent work has focused largely on title and author accuracy. We underline this effort and expand on it with a preliminary exploration in relevance of model-recommended citations. We define three citation-recommendation tasks. We also collect and annotate a dataset of model-recommended citations for those tasks. We find that GPT-4 largely outperforms earlier models on both author and title accuracy in two markedly different CS venues, but may not recommend references that are more relevant than those recommended by the earlier models. The two venues we compare are CHI and EMNLP. All models appear to perform better at recommending EMNLP papers than CHI papers.

## introduction citations are a common feature of research writing. they lend credibility to claims and can help identify gaps in prior research. they can also provide a chain of ideas from prior work to a research task. the last year has seen a drastic increase of interest about large language models (llms). chatgpt @xcite has opened the eyes of the general public to the potential of llms. chatgpt and its related gpt-x llms are being applied to a growing array of tasks @xcite @xcite . one task that has drawn both interest and ire is that of using llms to identify citations for a topic. several recent blog posts and articles have warned of chatgpt's hallucinated references @xcite @xcite . we build on recent work to assess the problem. ## related works various citation recommendation systems exist, relying on an array of nlp and information retrieval (ir) approaches. farber and jatowt (2020) offer a thorough survey of automated citation recommendation approaches. more recently, use of llms has been explored, leading to discussion of the tendency llms have to hallucinate output. @xcite offered an early exploration of hallucinated references by chatgpt. they assessed references output by chatgpt based on accuracy of journal name, volume, issue and page number and found the model incapable of generating any valid references. on the other hand, @xcite found chatgpt capable of some accuracy when answering questions about clinical radiological sources. a letter of warning from mcgowan et al. (2023) discussed fabricated references from both chatgpt and google's bard @xcite in psychiatry literature. they found real authors are often included, even when a paper title is fabricated. they also raised the alarm on the possibility of fake references entering into automated indexes. @xcite found chatgpt output in response to medical reference questions was of limited quality, but that references offered by the model were deceptively realistic. orduna-malea and cabezas-clavijo (2023) compared chatgpt and bard 2.0 citations in english, spanish, and italian. they explored reasons for fabricated citations and steps to address the issue. @xcite fine-tuned their own llm, galactica, and assessed it on three citation generation tasks. they found llm accuracy for citation generation appears to improve with scale. finally, @xcite found llms tend to hallucinate different authors of fabricated references in multiple independent query sessions, but consistently hallucinate authors in the same session. they compared accuracy on gpt text-davinci-003, chatgpt, and gpt-4. previous work has primarily focused on metrics related to accuracy of information. while understanding accuracy is important, accurate citations that are irrelevant will still be of little use to researchers. in this work we still assess accuracy, but we also offer a preliminary assessment of the relevance of citations identified by three models. ## methods we define three citation recommendation tasks, intended to model aspects of academic writing that could be supplemented by use of llms. ## results accuracy results can be found in tables 1 and 3 , while relevance results can be found in ## conclusion we evaluated gpt-3, gpt-3.5, and gpt-4 on three different citation recommendation tasks and compared them across two research disciplines. we found contrasts in terms of relevance and accuracy between those disciplines. this is important because individuals outside of nlp are beginning to use these models in their research. it is important for researchers from other disciplines to recognize these models' limitations for their disciplines. finally, while gpt-4 typically outperforms previous models on accuracy, it does not clearly perform better in terms of relevance.