Evaluating Large Language Models on W ikipedia-Style Survey Generation (2024)

Abstract: Educational materials such as survey articles in specialized fields like computer science traditionally require tremendous expert inputs and are therefore expensive to create and update. Recently, Large Language Models (LLMs) have achieved significant success across various general tasks. However, their effectiveness and limitations in the education domain are yet to be fully explored. In this work, we examine the proficiency of LLMs in generating succinct survey articles specific to the niche field of NLP in computer science, focusing on a curated list of 99 topics. Automated benchmarks reveal that GPT-4 surpasses its predecessors, inluding GPT-3.5, PaLM2, and LLaMa2 by margins ranging from 2% to 20% in comparison to the established ground truth. We compare both human and GPT-based evaluation scores and provide in-depth analysis. While our findings suggest that GPT-created surveys are more contemporary and accessible than human-authored ones, certain limitations were observed. Notably, GPT-4, despite often delivering outstanding content, occasionally exhibited lapses like missing details or factual errors. At last, we compared the rating behavior between humans and GPT-4 and found systematic bias in using GPT evaluation.

## introduction recently, large language models (llms) have attracted significant attention due to their strong performance on general natural language processing (nlp) tasks @xcite . especially, the gpt family @xcite shows great ability in various applications. while it has been demonstrated that they perform well in many general tasks, their effectiveness in domainspecific tasks continues to be under scrutiny @xcite . specifically, the text produced by llms can sometimes exhibit issues like creating false information and hallucination @xcite . in the context of scientific education, automatic survey generation aims to employ machine learning or nlp techniques to create a structured overview of a specific concept @xcite @xcite . automating this process not only alleviates the manual effort but also ensures timely updates at a reduced cost. a common approach involves an initial information retrieval phase to select pertinent documents or sentences based on the query topic. this is followed by a summarization or simplification phase to produce the final survey @xcite . while llms have the potential to be an alternative method for writing scientific surveys, their effectiveness and limitations are not yet thoroughly investigated. existing work focuses on applying llms to similar scenarios, including aiding scientific writing @xcite , questionanswering with scientific papers @xcite , writing paper reviews @xcite , and answering quiz or exam questions @xcite . this study pushes the boundary of this research area as the first to evaluate the capability of llms in generating education surveys within the scientific domain of nlp @xcite . our primary objective is to understand whether llms can be used to explain concepts in a more structured manner. to this end, we aim to answer the following research questions (rqs): ## analysis in this section, we provide an in-depth analysis of the llms' internal knowledge on survey writing ability, and compare the evaluation scores of human and llm assessments. error types we have shown that both automated and manual evaluations demonstrated that llms excel in crafting survey articles on scientific concepts. we analyze the best setting, gpt-4 osp, assessing errors identified by two experts, and summarize error types and distributions in fig. 2 . we classify these errors into four categories: verbose, wrong fact, missing information, and no error (indicating flawless content). it shows that most errors are missing information, followed by verbosity and factual inaccuracies. furthermore, the history and introduction sections of the generated articles contained the highest number of errors, while the application section exhibited the best. ## discussion and conclusion in this work, we evaluate the ability of llms to write surveys on nlp concepts. we find that llms, particularly gpt-4, can author surveys following specific guidelines that rival the quality of human experts, even though there are shortcomings such as incomplete information. our findings also indicate that gpt-4 may not be a perfect replacement for human judgment when evaluating human-composed texts, and certain biases exist when asking it to rate machine-generated texts. nevertheless, the results imply that these advanced generative llms could play a transformative role in the realm of education. they hold the promise of effectively structuring domain-specific knowledge tailored to general learners. this adaptability could potentially lead to a more interactive and personalized learning experience, enabling students to engage in query-driven studies that cater directly to their unique curiosities and learning objectives.