Be My Mate: Simulating Virtual Students for collaboration using Large Language Models (2024)

Abstract: Advancements in machine learning, particularly Large Language Models (LLMs), offer new opportunities for enhancing education through personalized assistance. We introduce “Be My Mate,” an agent that leverages LLMs to simulate virtual peer students in online collaborative education. The system includes a subscription module for real-time updates and a conversational module for generating supportive interactions. Key challenges include creating temporally realistic interactions and credible error generation. The initial demonstration shows promise in enhancing student engagement and learning outcomes.

## introduction in recent years, advancements in machine learning have impacted the educational landscape, particularly with the emergence of large language models (llms). these technologies offer new opportunities to enhance the learning experience and provide personalised assistance. however, the possibility of using llms to create a collaborative learning environment has not been explored. currently, llms have passed the turing test and it is difficult to tell llm-generated text content apart from human-generated text. this, along with the emergent abilities in llms @xcite , shows the possibility of having llm impersonate a peer student. in the context of online collaborative education, it is often difficult to pair students with peers, or obtain a large enough pool of students to group individuals. for this reason, we propose the use of llms disguised as students to generate interaction with other students as their peers. this would enable benefits associated with collaborative learning, without the difficulties of pairing students. however, the implementation of these systems is not straightforward. we offer a first demo of be my mate foot_0 , an initial attempt to simulate virtual students for collaboration using large language models. by leveraging the capabilities of llms, we aim to create virtual entities that can interact with human students, participate in group discussions, and contribute to collaborative projects. ## agent implementation we start from @xcite , an existing system that supports the collaborative resolution of math word problems, while providing active feedback. the system utilises a websocket-based queue notification protocol, stomp, for publishing updates about the state of the solution and the conversation of a given solution instance (room). the proposed agent must subscribe to these updates to receive the current status and produce relevant contributions to the problem-solving. in order to do so, we split the system in two main modules, as shown in figure 1 : subscription module: establishes the connection with the intelligent tutoring system (its) and subscribes for updates on the generated rooms. every time a new room is created, the module will listen to the user's inputs and modifications on the solution status (i.e. notes or equations) to propose new solutions or a conversational message to the student. conversational module: its main goal is to provide relevant messages for supporting the student. the module has to decide whether to send a message to the student or propose a new letter, or equation. consistency with the current context is ensured with two components that prepare, or discard, the output of the model to conform with the expected outputs in our system. in this direction, we check if the user has directly spoken to his peer (the agent), and therefore will be expecting a response, or, on the contrary, the system can freely propose new steps, such as letter assignation or equation definition. alternatively, in case of a message to advance the resolution status, we must ensure the consistency of such message in terms of the expected message by the its. these validations aim to ensure the messages are credible. for example, ensuring that the provided equations are correctly structured for a mathematical context or that the letters used in the proposed equations by the agent have been previously defined. this validation must not prevent the agent from failing, but from making mistakes that may not be credible. ## technical and scientific challenges during the analysis phase, we have identified temporisation as a main factor that must be addressed to convince alumni that they are interacting with peers. in the context of our its, humans tend to take more than a couple of seconds to write a chat message or propose a new definition. the time taken to write a message is influenced by a variety of factors, such as how advanced the problem is (at the beginning of a problem, humans try to understand it before writing the first message), familiarity with the system, age, or others. this time should be modelled and depends on the specifics of the system to be integrated in. at this stage and after manually inspecting a variety of values, we decided to include a uniform random wait time between 2 and 6 seconds for generating a message to the user, and between 5 and 10 seconds for generating the next step. another relevant factor that has been found relevant, is the type of mistakes the llm can produce. in our context (an arithmetic, and algebra its) human mistakes are often related to the identification of relations between the problem quantities. the impersonated student should commit mistakes as if it were a regular student. this would create a perception of interacting with a peer. however, the errors should be credible. moreover, although our system is capable of sending direct messages to the students, the agent should be able to decide to directly talk to the student to explain a concept or ask questions, as a peer would do. finally, regarding data privacy and collection, taking into account the sensitivity of educationalrelated data, authors should only capture the required samples to evaluate the system while preserving the student's anonymity. to this end, a new challenge arises for deciding if a student utterance is safe or not (directly leaks any personal data), correct it if possible, or directly reject it. moreover, and appropiate storage scheme must be implemented to ensure that only individuals with research access can access the confidential data. ## conclusions the demonstrated work foot_1 is a proof of concept of what we are aiming for in the be my mate agent. although some aspects of our demo require further refinement, it demonstrates that llm-based agents are a feasible option for providing collaborative learning environments. while significant progress must be made before agents can truly emulate humans and pass the turing test -such as timing the agent's interventions to enable a realistic conversational flow-, this demo highlights the potential of using llms to exploit the advantages of collaborative learning without the need for multiple students to be connected simultaneously. this first stone opens many research opportunities such as is addressing the bias these models might contain in the educational context. moreover, the usefulness of these models, although theoretically sound must be empirically validated, either by studying the agent-student interaction by experts, in terms of learning compared to a control group, or by directly surveying the students with relevant tests. in addition, the use of llms replacing real students can provide further benefits in this type of setting. in particular, it allows the system to rely on a single student model to make decisions, such as selecting the next activity, determining the optimal level of scaffolding, and tailoring the content and