Evaluating and Training Long-Context Large Language Models for Question Answering on Scientific Papers (2024)

Abstract: With the number of scientific papers published every year growing and current large language models (LLMs) showing state-of-the-art performance on natural language processing (NLP) tasks, we ask the question if LLMs could be utilized to answer questions on scientific papers.We investigate how well state-of-the-art large language models (LLMs) can answer questions on scientific paper by experimenting with long-context versions of the LLaMA 2 model and evaluating and training on the Qasper dataset.We analyze how well the LLMs handle longer papers and questions that can only be answered by accessing information from far out paragraphs. During our experiments, we see that the performance of these LLMs drops with growing length and position of relevant information.We employ different measures from simple prompts to chain-of-thought prompts and zero-shot usage to fine-tuning with QLoRA.While we still observe a performance loss with increased context length, our measures reduce the effects of this flaw, and we can achieve F_{1} scores similar to bigger models like GPT-4.

## introduction the number of scientific papers published every year is growing exponentially @xcite . this creates a problem for scientists but also the general public to keep up with the developments in science. a natural language processing (nlp) system that can reliably answer questions on scientific papers could help in this situation. question answering (qa) systems often rely on task-specific machine learning models that can only be used for this purpose. large language models (llms) are a newer type of deep learning model trained to be general-purpose models for nlp. current commercial and open-source llms are often used in an intuitive, conversational manner as chatbots. they offer the ability to answer follow-up questions and have an intuitive interface for most users. they show state-of-the-art (sota) nlp performance @xcite and even display some reasoning capabilities @xcite @xcite and would be one contender for the core of a qa system focused on scientific papers. scientific papers present great challenges as context for qa when using llms for multiple reasons: their text part is typically about around 4,000 to 13,333 tokens long assuming that one word amounts to around 1.3 tokens @xcite . the base versions of newer commercial models like gpt-3.5 and gpt-4 have context windows of 4,096 @xcite and @xcite @xcite tokens while open-source llms like llama 2 @xcite offer a 4,096 long context window. also, scientific papers consist of long unstructured (except sectioning etc.) raw text making, it hard to determine which part is important to answer the question. the answer type is also not clear as the question could be about explaining some concepts presented in the paper, simple facts or even yes or no questions, or the question could be unanswerable. the unstructuredness and length of the context is especially problematic even for long-context llms as @xcite found: for multi-document qa where the llm has to select the relevant context part from multiple options, the performance curve has a u-shape with respect to the position of the documents as the ones at the beginning and the end are better retrieved than those in the middle. in this paper, we evaluate how well a small opensource llm can perform as a qa system for scientific papers if used in a zero-shot manner -especially regarding long papers (>4k tokens) and those questions whose relevant paragraphs are far out token-wise. to do this, we bin the papers per length and the questions per position of the relevant paragraphs. we try to improve the performance using recent llm adaptation techniques (prompting, parameter-efficient fine-tuning). we also investigate what weaknesses (e.g., instruction-following, long-context understanding) of the models specifically the fine-tuning improves. finally, we compare our best model with bigger ones. we observe that increased context length and position of the relevant paragraphs result in worse performance even for long-context llms. while more sophisticated prompting does not help, fine-tuning increases overall performance significantly but mostly by improving the instruction-following of the llms. ## related work large language models the foundation for most current large language models (llms) like the gemini @xcite , @xcite , llama @xcite , and mistral @xcite families is pre-training transformer decoder-only models with billions of parameters on internet-scale data enabling them to perform tasks they were not explicitly trained on. as we want to experiment on llms themselves which includes fine-tuning and modifying them, we utilized available open-source models: large language model meta ai (meta ai, 2023) (llama) @xcite . following the work on training-compute-optimal llms @xcite , the authors of llama focus on training smaller models with more data (and more compute) to achieve better inference-compute efficiency. vicuna is a collection of fine-tuned llama models @xcite . on top of models fine-tuned for better chatbot performance, there are models with longer context windows than the original llama model (version 2: 4k ( touvron et al., 2023b) question answering task the type of context for question answering (qa) can differ as it may be present as knowledge or as harder to manage raw text. modern qa system mostly use deep learning-based models like (fine-tuned) bert-or gpt-style models. datasets that cover the topic of scientific papers focus on various aspects. many focus on the review process which yields different artifacts. these enable different tasks: (meta-) review generation @xcite , acceptance prediction / paper rating @xcite , argument pair extraction from reviews and corresponding rebuttals @xcite , and multi-document summarization on reviews @xcite . but there are also datasets specifically for question answering on scientific papers @xcite . ## methodology to improve the performance of the general-purpose llms on the task of qa on scientific papers, we apply different prompting techniques and fine-tuning. we list all our prompt templates in appendix a. ## conclusion we wanted to investigate how well llms can handle scientific papers and how we can improve their performance. we observe that the (unmodified) small open-source long-context llms we tested are able to process scientific papers with up to about 16k tokens from the qasper dataset but fall short of commercial llms. additionally, the performance drops after the context exceeds the original context window -especially if the relevant information to answer to question lies in that region of the paper. when we employ the current techniques for efficient training qlora and flashattention, we can fine-tune the models on papers with a length of up to 8k tokens on a single datacenter gpu that is available to a university student for research. the performance of our fine-tuned model still increases for even longer papers without being trained on these lengths. experiments with models that we only trained on extracted paragraphs without providing the model the full paper suggest that our training primarily improves instruction following but also improves the models' ability to determine if a question is unanswerable as it has to learn the connection between the absence of relevant information and the unanswerability of the question. when comparing our results against baselines, we saw that our best approach reaches or surpasses the result of the original gpt-4.