S yllabus QA : A Course Logistics Question Answering Dataset (2024)

Abstract: Automated teaching assistants and chatbots have significant potential to reduce the workload of human instructors, especially for logistics-related question answering, which is important to students yet repetitive for instructors. However, due to privacy concerns, there is a lack of publicly available datasets. We introduce SyllabusQA, an open-source dataset with 63 real course syllabi covering 36 majors, containing 5,078 open-ended course logistics-related question-answer pairs that are diverse in both question types and answer formats. Since many logistics-related questions contain critical information like the date of an exam, it is important to evaluate the factuality of answers. We benchmark several strong baselines on this task, from large language model prompting to retrieval-augmented generation. We introduce Fact-QA, an LLM-based (GPT-4) evaluation metric to evaluate the factuality of predicted answers. We find that despite performing close to humans on traditional metrics of textual similarity, there remains a significant gap between automated approaches and humans in terms of fact precision.

## introduction in educational applications, artificial intelligence (ai) approaches have shown significant promise in improving learning outcomes @xcite , by automatically providing feedback to students or engaging in tutoring dialogues with them. the key idea is to use ai to create an ondemand virtual teaching assistant to interact with many students simultaneously; see, e.g., khamigo from khan academy @xcite . these approaches can scale up the effort of expert human teachers and tutors, and relieve them from doing repetitive tasks so that they can focus on providing personalized feedback or designing new learn-course name: biol 151: introductory biology length: 5864 tokens, 16 pages formatting: tables, lists, schedules, images syllabus text: [page 5] . . . the two mid-term exams are not cumulative . . . the final exam is cumulative . . . exam replacement: if you score higher on the final exam than on one of your mid-term exams, the final exam score will replace that mid-term exam score. this is only true if you take both. . . . @xcite . . . topics that will be covered in the order shown below and we will spend roughly a week on each topic. 1. the cell . . . detailed topic list 1) the cell a) organization b) membranes . . . ## syllabusqa data collection we now detail how we construct syllabusqa to address some of the limitations in existing datasets. ## syllabusqa data analysis statistics we randomly split the dataset into trainval-test splits with a proportion of roughly 60%-20%-20% in terms of qa pairs. we ensure no overlap in syllabi across splits to test the generalization ability of qa approaches to answer questions based on unseen new syllabi. we show detailed statistics of syllabusqa in ## results, analysis, and discussion we report the performance on both surface textual similarity and factuality metrics for all approaches in models are good at capturing surface textual similarity the best-performing approach is llama-2-13b with sft combined with rag and chain-of-thought (llama-2-13b+sft+rag+cot). this approach performs similarly to humans on surface-level textual similarity metrics. this observation suggests that training automated teaching assistant chatbots has some promise towards reducing human workload for course logistics-related qa. however, zero-shot approaches perform much worse than sft on the same underlying llm, which suggests that training on actual qa pairs is necessary to adapt to the diverse language styles among real instructors. we note that human performance is underestimated since we asked annotators to use ## future work and conclusions in this work, we introduced syllabusqa, a diverse dataset consisting of real-world course syllabi and corresponding course logistics-related qa pairs. we also benchmarked the performance of several strong, llm-based automated qa baselines on this dataset. results show that finetuning and retrieval-augmented generation are helpful but there remains a significant gap between the performance of llm-based approaches and that of humans on answer factuality. we make the syllabusqa dataset, our code, and evaluation metrics publicly available to facilitate future work on automated teaching assistant development. there are plenty of avenues for future work. first, we can leverage question meta information to improve the performance of llm-based qa approaches, and use overgenerate-and-rank to improve their robustness. second, we can collect human labels on answer factuality and further improve qa approaches using methods such as direct policy optimization @xcite .