On the Effectiveness of Prompt-Moderated LLM s for Math Tutoring at the Tertiary Level (2025)

Abstract: Large Language Models (LLMs) have been studied intensively in the context of education, yielding heterogeneous results. Nowadays, these models are also deployed in formal education institutes. While specialized models exist, using prompt-moderated LLMs is widespread. In this study, we therefore investigate the effectiveness of prompt-moderated LLMs for math tutoring at a tertiary-level. We conduct a three-phase study with students (N=49) first receiving a review of the topics, then solving exercises, and finally writing an exam. During the exercises, they are presented with different types of assistance. We analyze the effect of LLM usage on the students’ performance, their engagement with the LLM, and their conversation strategies. Our results show that the prompt-moderation had a negative influence when compared to an unmoderated LLM. However, when the assistance was removed again, both LLM groups performed better than the control group, contradicting concerns about shallow learning. We publish the annotated conversations as a dataset to foster future research.

## introduction the publication of @xcite and the following developments around other llms have allowed easy, informal access to language models even without technical knowledge. the combination of the large amount of training data, model size, and instruction-following ability has led to llms becoming not only a subject of research but also a tool used by the broader population due to its ease of use and perceived usefulness and novelty @xcite . however, many members of the general public equate llms to artificial intelligence and can lack an understanding of the limitations of these models. problems such as hallucinations and bias can be 1 https://github.com/sebastian-steindl/gmade_ data can a strictly monotonically increasing sequence be limited upwards that's an interesting question! let's go through it step by step. 1. definition of strictly monotonically increasing: [...] how to calculate the sequence limit there are various methods for determining the limit value of a sequence [...] if you have a specific sequence in mind, we could go through this process together! what is the specific sequence for which you want to find the limit? lim_{x->\inf}\frac{n^{2}(n+sin(n))}{4*n^{3}+2n+1} * to calculate the limit ( \lim_{n \to \infty} \frac{n^{2}(n + \sin(n))}{4n^{3} + 2n + 1} ), let's go through the steps. obfuscated by the very strong natural language understanding and natural language generation capabilities of current state-of-the-art models. this can lead to a naive, unreflected usage of llms for knowledge tasks @xcite . in this work, we extend previous research into the effectiveness of llms in education by providing empirical results for teaching math in formal, tertiary education. to facilitate real-world, largescale adoption of llms in educational institutions, we deem two factors as necessary: ease-of-use and low cost. we therefore use a realistic technical setup to perform our experiment on students' performance with and without llm assistance. the students use the llm gpt-4o-mini to solve challenging math assignments of different formats, incorporating the topics sequences, continuity, and differential calculus (dc). we test an out-of-thebox, unmoderated llm and its prompt-moderated version. prompt-moderation in this case means that the model was prompted to not give away the solution directly, but to act as a tutor performing scaffolding. we provide the resulting conversations as the german math assistant dataset for llms in education (gmade), which, to the best of our knowledge, is the first of its kind. an exemplary excerpt of such a conversation can be found in fig. 1 . our main contributions are, firstly, the investigation into the effect of llms, prompt-moderated and unmoderated, in formal tertiary education. we thereby uncover shortcomings of their tutoring competence. secondly, we perform an in-depth analysis of the students' test performance. lastly, we publish gmade to enable future work into understanding and hopefully improving llms in education, especially regarding math tutoring. our results show the prompt-moderation to be ineffective and thus align with @xcite while contradicting results by @xcite . this indicates that future work needs to be done on llm-tutors making use of scaffolding to be efficient in educational settings. ## experimental design our experiment aims to measure the effectiveness of llm assistance for math tutoring. the setup of our study is analogous to @xcite , who found that prompt-moderation improved the tutoring. we conducted the experiment with voluntary participants from the first semester mathematics for engineers and computer scientists courses. to reduce selection bias, the study was conducted during the last lecture of the semester and also served as a preparation for the exam. this resulted in @xmath0, which equals roughly 70% of the eligible students. the experiment was divided into three phases: review, exercise, and exam phase -each lasting 40 minutes with a break of 20 minutes in between. in the review phase, the lecturer presented topics that were relevant to the following phases to all participants. since these subjects had already been discussed during the normal lecture period, this served as a review for the participants and aligned their prior knowledge. in the exercise and exam phases, the students were presented with a number of mathematical questions. the assignments were printed on paper, and the students were required to answer on paper. this avoids the naive copy & paste behavior that @xcite observed. ## method in the following section, we describe the methods and interventions used in our experiment, including the setup of the two llm-based tutoring systems and the reference group. we also outline the procedures for data collection and evaluation. ## analysis of llm assistant usage to better understand the strengths and weaknesses of prompt-moderated llm tutoring, we perform an in-depth analysis focusing on the user engagement and investigate this with regard to the question's topic and type. moreover, we study the frustration among the users and conversation strategies employed by the participants. since the students only had llm assistance during the exercise phase, all assignment numbers in the following reference the tasks from this phase (cf. appendix f). ## analysis of student performance and engagement in the following, we will discuss the effect of (prompt-moderated) llm tutoring on student performance in our study. first, we consider the student performance and engagement with the chatbot as a whole, then on an assignment-level, and lastly, we give special consideration to transfer tasks. we measure the student performance with regard to their point score as well as the number of tasks completed. the results in ## the german math assistant dataset for education we publish the annotated conversations between the students and the llm assistants as gmade to foster future research into tutoring llms. it includes 32 multi-turn german conversations between students and gpt-4o-mini that occurred over the span of 40 minutes, accumulating in roughly 900 messages. the students interact with the llm to solve a series of mathematical questions. to the best of our knowledge, gmade is the first dataset of its kind and could be useful for studies that currently rely on fully synthetic data (e.g., @xcite . each user message is annotated as either chitchat, direct, abstract, malicious, follow-up, or other. additionally, the annotation includes a binary field marking if the message suggests frustration and a mapping to the assignment number (if applicable). lastly, we provide the timestamps at which the message was sent, allowing for analysis of processing time. the statistics of our dataset are presented in gmade shows that the unmoderated llm did not only answer the user's question but also, unsolicitedly, provided additional information such as definitions, examples, and conclusions. this can be attributed to the llm being trained as a helpful assistant and humans preferring the more detailed responses during reinforcement learning from human feedback @xcite . ## conclusion this study investigates the impact of pretrained llms as learning assistants in a challenging mathematical qs scenario. the questions are on the level of tertiary education, span multiple topics, and test different competencies. we compare the effect of students being provided with an unmoderated llm, a prompt-moderated llm, and no assistance in a three-phase experiment. our results indicate that both during the assisted exercise phase and the unassisted exam, the cohort using the unmoderated llm achieved the highest score on average. moreover, on average, they also had the largest proportion of tasks worked on during both phases. while the tutoring group had a lower mean score in the exercise than non, their average score in the exam was higher, even though they had worked on fewer assignments. one possible reason why the tut group performed worse in total than gpt is information overload caused by the long responses and frustration stemming from the tutor refusing to answer the question directly, which was similarly observed by @xcite . the investigation into the student engagement with the assistants shows that the usage is not governed by llm helpfulness or topic, but by question length and how easy it is to enter it into the chat. however, when considering the actual effect the engagement had on the qs, we see that this varies by question type and topic. a more detailed analysis of the conversations reveals that the students utilizing the tutor llm got frustrated with its evasive responses and tried to extract a direct answer. additionally, they received longer responses, which increased the time between two user messages and probably led to them working on fewer assignments than the other two groups during the exercise. this indicates that llm assistance, even when not optimal, can still have a beneficial effect on learning compared to no assistance, and refutes concerns regarding shallow learning. to conclude, the llm assistance had a positive effect on the students performance, even when unassisted. this suggests that concerns of shallow learning might be exaggerated and stresses the need for further studies into llm tutors. however, prompt-moderation with the intention of having the llm act as a tutor, could actually be harmful to the learning effect. given our results, we believe future work can focus on how prompt-moderation can become beneficial, how to best enhance traditional lectures with llm assistance and the role of the subject or domain in this context.