Empowering Teachers with Usability-Oriented LLM -Based Tools for Digital Pedagogy (2024)

Abstract: We present our work on two LLM-based tools that utilize artificial intelligence and creative technology to improve education. The first tool is a Moodle AI plugin, which helps teachers manage their course content more efficiently using AI-driven analysis, content generation, and an interactive chatbot. The second one is a curriculum planning tool that provides insight into the sustainability, work-life relevance, and workload of each course. Both of these tools have the common goal of integrating sustainable development goals (UN SDGs) into teaching, among other things. We will describe the usability-focused and user-centric approach we have embraced when developing these tools.

## introduction in recent years, there has been an increase in the use of artificial intelligence (ai) in education that offers novel ways to support teachers and improve students' learning experiences (see cleland silva @xcite . this shift has been particularly transformative in the field of digital humanities, where ai tools are being used to analyze vast datasets of historical texts @xcite , digitize and interpret cultural artifacts @xcite and even create interactive learning environments @xcite . ai-driven platforms now enable students to explore complex topics like literature, history, and philosophy through personalized learning paths (see @xcite . large language models (llms) in particular are now in the forefront of enabling the use of ai from a pedagogically insightful perspective (see cleland silva and hämäläinen 2024). however, their full potential in education is yet to be capitalized as there are many obstacles such as a lack of understanding of the potential of llms (see @xcite and legal issues such as gdpr concerns that hinder universities from embracing them (see @xcite . in this article, we present two llm powered tools that show some of the ways llms can be used in educational settings. the first tool is a moodle ai plugin, which integrates llm capabilities within the moodle foot_0 educational platform to assist educators in course creation and management. the second tool aims for curriculum planners to analyze the contents of course descriptions and degrees to better align them with the university's strategic goals. sustainability is an overarching theme for both of the tools as un sustainable development goals (sdgs) foot_1 are strategically important to the university. recently, there has been a rise in interest in sdgs both in digital humanities @xcite @xcite and nlp @xcite @xcite . the convergence of digital humanities and nlp in this context not only supports the advancement of sustainable development but also enriches the academic discourse around the role of technology in addressing global challenges. this paper will provide a comprehensive overview of these tools, discussing their design and functionality. furthermore, we have gathered user feedback through user studies and workshops. in the later sections of the paper, we describe the new features that we implemented based on the feedback. when building tools for educators, it is important to listen to user needs so that the tools will end up as useful as possible. this design philosophy is by no means new to the field of nlp (see @xcite ). ## related work ever since the emergence of chatgpt (openai, 2022), there has been an ever growing body of research on applying llms on every field of science. digital humanities and digital pedagogy have not gone unaffected by this hype of the transformational new technology. in this section, we describe some of the llm-based tools developed to empower teachers and curriculum planners. this proliferation of education data, encompassing unstructured and structured text, relational databases etc. presents a complex challenge for constructing knowledge graphs (kgs). recent research @xcite has proposed methods for automatically building kgs from these heterogeneous data sources and explored the application of these kgs in conjunction with llms for improving the accuracy and relevance of responses in educational question-answering tasks. recent research @xcite has begun to explore the role of llms in instructional design, particularly within the context of mathematics education. the study focused on assessing the instructional design capabilities of llms, using an instructional framework based on mathematical problem chains. the results indicated that llms, particularly gpt-4, performed well in various aspects of instructional design, such as setting instructional objectives, organizing teaching activities and selecting appropriate methods and strategies. codeaid @xcite is an llmbased assistant designed to provide support without directly revealing code solutions. the deployment of codeaid in a programming course with 700 students revealed key design considerations for future educational ai assistants, emphasizing the importance of leveraging ai's unique capabilities, promoting cognitive engagement through simplified query formulation, avoiding direct answers to encourage deeper learning and ensuring transparency to enable students to guide and assess ai responses. another contemporary research @xcite has explored the potential of llms as aibased tutors in university courses. the study used standard similarity metrics such as bleu-4, rouge, and bertscore, alongside a human evaluation of helpfulness and trustworthiness, to assess the performance of the models. the findings suggest that combining rag with prompt engineering significantly improves the factual accuracy of llm-generated responses, making it an ideal ap-proach for educational settings where additional course material can be leveraged to enrich model input. ## moodle ai plugin the purpose of the tool is to help teachers incorporate sdgs into their teaching. the functionalities include analysis of relevant sdgs, assignment suggestions and an interactive chatbot. the plugin can also create an executive summary of a course and highlight the most important skills and learning outcomes. the idea behind developing a moodle plugin instead of a separate system was motivated by the simple design principle of reducing the required effort from the user's perspective. when teachers can use the ai tool directly in a familiar platform, they do not need to learn an entirely new system, but instead interact with a new functionality in a more accessible way. the plugin is implemented as injected javascript, php, and css within moodle. this architecture allows for seamless integration with moodle's existing infrastructure. when a user interacts with the plugin for the first time, a background process is triggered. specifically, a post request is sent to a flask server, where data scraping, preprocessing, and ai interactions take place. during this initial interaction, a "super user" with managerial rights in moodle scrapes the necessary data, such as slides. the data undergoes preprocessing stages that include language detection, removal of private information, and content refinement using the uralicnlp library (hämäläinen, 2019). this preprocessing ensures that no personally identifiable information (pii) is passed to the ai model, ensuring full gdpr compliance. the removal of urls, email addresses, and references, alongside the detection and elimination of named entities, guarantees that the data processed by the model is entirely anonymized, therefore the plugin strictly adheres to gdpr regulations and prioritizes data privacy throughout the process. the processed information is then stored in the chroma vector database foot_2 as embeddings. these embeddings serve as the basis for various ai functionalities within the plugin. lastly, the plugin uses the preprocessed slides and respective queries to prompt google's palm 2 model @xcite figure 1 : moodle ai plugin main screen over vertex ai foot_3 . this model was chosen for its strong performance, cost and gdpr compliance, which aligns with the university's data privacy policies and its use of google cloud infrastructure. the plugin interacts with the model primarily through prompt-based requests, which enable the generation of summaries, related sdgs, suggestions of assigments or quizzes based on course content or conversations with the chatbot. to optimize performance, the plugin employs a caching mechanism using a mysql database. once the slides for a course have been processed, the results are stored in the database, allowing the plugin to serve future requests without contacting the flask server again. the moodle ai plugin currently supports english and finnish. we implemented localization, allowing the plugin to automatically switch languages based on moodle's language settings. for instance, the string $string['pluginname'] is defined as 'ai summarizer' in english and 'ai yhteenveto' in finnish. this logic ensures that every text element in the plugin is fully translatable. future language expansion will be straightforward, requiring only the addition of translations, as the localization system is already in place. the plugin's first screen includes a summary of slides and the sdgs related to that content. when the user clicks on one sdg, a new window opens with more information about that sdg containing an explanation of how it relates to the slides, assignment suggestions and a draft speech that would help teachers mention the sustainability aspect during their lectures. another functionality included in the plugin is the chatbot, which makes use of the embeddings stored on the chroma database along with vertex ai to implement a retrieval-augmented generation (rag) system @xcite . the rag system pulls relevant content from the chroma vector database, which stores embeddings of course slides and related metadata, and combines this with ai-generated responses. langchain foot_4 is used to manage conversational history and memory, ensuring a structured conversation flow that enables the chatbot to provide context-aware, accurate answers across multiple interactions. this integration allows the chatbot to assist teachers by offering personalized, relevant responses that are directly grounded in the specific course material. although finetuning has not yet been implemented in this iteration of the chatbot, future improvements, as outlined in @xcite , might involve fine-tuning the model to enhance its performance for specific educational tasks. additionally, the plugin includes the functionality of analyzing all the slides of the same course. this system relies on metadata stored in the figure 3: moodle ai plugin chatbot chroma vector database, where each slide is linked to a specific course through its course id. the retrieved content is then fed into the ai model to generate a summary, key points, main skills, and related sdgs. the overall course functionality also includes a general chatbot that is designed to answer queries related to the course. the chatbot, like other parts of the plugin, utilizes a rag system, but it focuses exclusively on the content from the whole course. this ensures that the chatbot's responses are always relevant to the specific course material, providing a personalized experience. 4 curriculum planning tool the curriculum planning tool is a separate tool that operates on a different level of abstraction to the moodle plugin. this tool works directly on course descriptions and degree structures. the functionalities include sdg analysis, analysis of arene competences foot_5 and work life relevance analysis. on the technical side, this tool uses palm 2 as well. the tool's architecture separates the ai processing, which is handled server-side through flask and vertex ai, from the sensitive curriculum data and interface, managed in mariadb and react. this division between local and server-side components also allows the tool to offload computationally intensive tasks to the server while maintaining a responsive user interface. the server-side components, including the palm 2 model accessed through vertex ai, handle the core analysis tasks, ensuring that sensitive curriculum and course data remain secure. this allows for powerful ai analysis while keeping the human user in control of the interpretation and application of results. the course-level sdg analysis feature directly addresses a critical gap in current curriculum planning tools: the lack of systematic methods for integrating sustainability principles across educational programs. while many institutions recognize the importance of sustainability education, there has been a dearth of practical tools to support this integration at the course level @xcite . this tool provides educators with actionable insights into how individual courses contribute to sustainability goals, facilitating more informed curriculum design decisions. the methodology employed in degree-level sdg analysis typically involves aggregating and synthesizing data from individual course assessments to create a program-wide overview. integration of sdgs into this mapping process represents a novel extension of traditional curriculum mapping techniques. the visual representation of sdg alignment across all the courses of any one degree, using charts and color-coded matrices, offers an intuitive interface for understanding complex relationships between course content and sustainability objectives. this visual approach addresses the need for more accessible and interpretable data in curriculum planning, a gap that has been noted in educational technology research. the incorporation of work-life competencies analysis in curriculum planning tools addresses the growing demand for alignment between higher education outcomes and labor market needs. this feature responds to the call for more employabilityfocused curriculum design, as emphasized by @xcite in their stakeholder analysis of university curricula and employability. ## workshop and feedback we organized three workshops, each lasting 2 hours, to gather feedback from educators on the usability and features of the moodle plugin. the number of participants varied between 5 and 10 for each workshop, with attendees representing different departments within the university. two of the workshops were held in person in different campuses, whereas one workshop was organized online over teams. furthermore, we interviewed study planners on their needs for the curriculum tool (see huovinen 2024). as a result of the workshops, several usability problems were identified such as teachers losing track of where they were in the plugin, some functionalities not working on firefox, our regular expressions failed parsing the output of the llm and so on. additionally, teachers were not happy with the quality of the generated assignments as they wanted more control over how broad the assignments were as well as being able to generate quizzes automatically. most importantly, the teachers emphasized the need for a tool whose focus is assisting with the teaching experience in general and not specifically just with sustainability. in response to this feedback, we shifted our approach from having sustainability as the primary focus to creating a more versatile tool that supports teaching in a broader context. while sustainability remains an integral part of the plugin, we have expanded its functionalities to better assist teachers in various aspects of their courses. the updated design, featuring a hamburger menu and separate windows for different functionalities, reflects this shift. now, the plugin offers tools for generating quizzes, assignments, and interactive content, alongside its original sustainability features and chatbot capabilities. in the updated plugin, each functionality is separated, ensuring that teachers don't lose track of where they are in the process. the new layout not only enhances usability but also ensures that the plugin's functionalities are more accessible. the new quizzes functionality allows teachers to automatically generate quizzes based on the content of their slides. teachers can select the number of questions and the question type, such as multiple-choice or true/false. additionally, teachers can input specific guidelines or constraints for the quiz generation, ensuring that the generated quizzes align with their teaching goals. the quizzes window interacts with the moodle backend to create actual quizzes within the moodle platform, making it easy for teachers to integrate the quizzes into their courses. the ai takes into consideration the teacher's suggestions, ensuring that the generated content meets their expectations. additionally, the new assignment generator functionality allows teachers to create assignments tailored to their course content. teachers can select various parameters such as the type of assessment (e.g., projects, essays, presentations), the difficulty level, and the content focus. similar to the quizzes functionality, teachers can provide specific instructions or constraints to guide the assignment generation process. the ai uses the input provided by teachers to generate assignments that align with the course content and the teacher's objectives. this feature was designed to give teachers more control over the generated content, addressing the feedback we received during the workshops about the need for more customizable assignments. as for the curriculum planner, there was a demand for a workload assessment functionality and quality assessment functionality. these functionalities require more future work on integrating the moodle ai plugin with the curriculum tool. as this requires a more systematic approach to analysing the material of all of our teachers, this deeper integration has reached a state of hiatus due to organizational resistance. ## conclusions in this paper, we have presented our hands-on tools that bring llms closer to educators to empower a digital transformation of pedagogy at our university. by integrating these tools into the existing educational framework, we aim to enhance the teaching and learning experience, making it more interactive, personalized and efficient for all parties involved. furthermore, we have organized workshops to help instructors effectively utilize these technologies and to gather their feedback. in addition to transforming pedagogy, our llmbased tools pave the road for digital humanities research. by leveraging the capabilities of llms, educators and researchers can explore novel methods for analyzing and interpreting vast amounts of learning data. this can be used to further understand the interplay of llms, teachers and students in educational contexts.