Exploring the Potential of Large Language Models in Generating Code-Tracing Questions for Introductory Programming Courses (2023)

Abstract: In this paper, we explore the application of large language models (LLMs) for generating code-tracing questions in introductory programming courses. We designed targeted prompts for GPT4, guiding it to generate code-tracing questions based on code snippets and descriptions. We established a set of human evaluation metrics to assess the quality of questions produced by the model compared to those created by human experts. Our analysis provides insights into the capabilities and potential of LLMs in generating diverse code-tracing questions. Additionally, we present a unique dataset of human and LLM-generated tracing questions, serving as a valuable resource for both the education and NLP research communities. This work contributes to the ongoing dialogue on the potential uses of LLMs in educational settings.

## introduction and background the teaching of introductory programming courses continues to be a challenging endeavor, despite the global uptake and popularity of such courses. high enrollment rates often result in diverse student populations, with a wide range of programming experience from those just starting their journey to others with prior exposure @xcite . ensuring an effective learning experience that accommodates this wide disparity presents a daunting task, making the teaching of these courses complex. one critical component in teaching introductory programming is the focus on code tracing, a skill identified as instrumental in enhancing code writing abilities @xcite @xcite . current educational methodologies encourage code tracing through a variety of means, such as practice questionnaires @xcite , direct teaching strategies @xcite figure 1 : we aim to assess the large language models' (llms') capability to generate code tracing questions, pivotal in computer science education. the accompanying illustration outlines our approach, where gpt4 is employed to generate questions based on given code snippets and descriptions. subsequent comparative analysis with human-created questions aids in exploring critical aspects, such as the quality and diversity of generated questions, discernibility between human and ai authors, and the relative superiority in question quality. 2018), and tracing quizzes @xcite . these strategies consistently utilize codetracing questions aimed at fostering and developing a student's understanding and skills. however, the preparation of code-tracing questions poses challenges. manual question creation by instructors @xcite is time-consuming and lacks scalability. automatic generation using program analysis saves time, yet is limited by the analyzer's capabilities and lacks question diversity @xcite @xcite @xcite . in light of the increasing potential of large language models (llms) in sectors like code summarization and explanation @xcite , the question arises: can llms generate high-quality code-tracing questions? our study explores this query using gpt4 (openai, 2023), leveraging prompts to guide its question generation based on given code snippets and descriptions. to assess the llm's capability in this pivotal aspect of computer science education, we devised a set of human evaluation metrics. this allowed for an objective appraisal of the llm-generated questions, and, through a comparative analysis with human-created counterparts, critical aspects such as question quality, diversity, discernibility between human and ai authors, and relative superiority in quality were explored (figure 1 ). these analyses have enhanced our understanding of the potential roles of llms in computer science education. this investigation provides a foundation for considering the potential inclusion of llms in learning platforms, which could offer new possibilities for enhancing the learning experience in introductory programming courses. given these advancements, our study contributes to the field as follows: code llms for cs education: recent advances in code large language models (llms) @xcite @xcite have enabled various downstream applications, including code completion, retrieval, summarization, explanation, and unit test generation @xcite @xcite . studies have showcased the llms' ability to generate novice programming content comparable to humans @xcite . llms have been utilized in classroom environments @xcite , to generate coding exercises and explanations @xcite , and to create counterfactual questions @xcite . our study represents the first exploration of llms for code tracing question generation, a critical component of cs education, thus underscoring the potential of these models for generating educational content. ## analyses and results this section details our analysis and highlights the results, encompassing quality ratings, expert perceptions, and textual similarities in question generation. ## conclusion this study explored the capability of gpt-4 in generating code-tracing questions that rival the quality of those crafted by human educators. the findings illuminate the potential of llms to bolster programming education, marking a significant stride in the domain of code-tracing question generation and llm application. this sheds light on scalable, high-quality automated question generation.