Learn With Martian: A Tool For Creating Assignments That Can Write And Re-Write Themselves (2023)

Abstract: In this paper, we propose Learn, a unified, easy-to-use tool to apply question generation and selection in classrooms. The tool lets instructors and TAs create assignments that can write and re-write themselves. Given existing course materials, for example a reference textbook, Learn can generate questions, select the highest quality questions, show the questions to students, adapt question difficulty to student knowledge, and generate new questions based on how effectively old questions help students learn. The modular, composable nature of the tools for handling each sub-task allow instructors to use only the parts of the tool necessary to the course, allowing for integration in a large number of courses with varied teaching styles. We also report on the adoption of the tool in classes at the University of Pennsylvania with over 1000 students. Learn is publicly released at https://learn.withmartian.com .

## introduction advances in natural language processing, particularly through large langauge models, will enable the creation of powerful applications in many fields @xcite . one of the most positive and most promising may be education, where rapid improvement has been achieved in fields like question generation @xcite @xcite . in this paper, we propose learn, a unified, easyto-use tool to apply question generation in classrooms. the tool allows instructors to create assignments that can write and re-write themselves. instructors provide existing course materials, and the platform is able to generate new questions, select the best questions, show those questions to students, provide analytics, and improve as it collects more data on student performance. we also report on a case study of the successful adoption of learn in classes at the university of pennsylvania. figure 1 : a diagram of steps, sub-steps, and features in learn. learn is a tool for creating assignments that write and re-write themselves. to write new assignments, users upload material and the platform generates questions. to re-write assignments and improve them, the questions are first shown to students, then the data from student interactions is analyzed to update the questions. we hope this tool can be used by course staff to save time and improve education, and that learn can serve as an example of a high-quality socialgood tool which inspires the development of further applications. ## tool tour and design we begin with a brief tour. generating questions is done simply by uploading the materials from which the questions should be generated. uploaded materials can be text, code, pdf, or audio/video. in the case of pdf, they are converted to text through ocr. in the case of audio/video, they are converted to text through asr. once the materials are uploaded and text is extracted, they are used to generate questions. the generated questions are then shown to the course staff. members of the course staff can accept, reject, or edit any question. accepting a question adds it either to the general question bank for the course or to a particular assignment. after questions are added to an assignment, the assignment can be released to students, who can then complete the assignment. when the student is completing the assignment, their answers are stored, alongside relevant metadata such as the amount of time to complete the question and whether the student got the question correct. students can also mark questions as being "really good" for studying, or as "not helpful". the data collected in this way is then analyzed, both to inform instructors and to improve future question generation. that data is used to provide suggestions to course staff. note 1: uploading material. users have multiple means of uploading material to learn. although they can upload all their content manually, that is typically not necessary. instead, if they have an existing course website, they can point a scraper built into the learn tool at that website, and it will find the relevant material (e.g. the course textbook, lecture notes, youtube links, etc.) and only upload the material which cannot be found on their course website. we also have an lti integration, allowing us to pull data from existing learning management systems (like canvas and moodle), as well as pushing data (like generated questions or student grades) to those systems. these alternative methods of uploading material decrease effort required to adopt the platform, making it something which can be used in minutes. note 2: rich question formats. in most tools for creating assignments, questions are purely text. this is not the case for learn. instead, all questions are javascript functions of the type () => ({question: html, answer: html}) if a user wants to create a question which acts like text, we provide an interface which lets them create such questions without writing any code, just as they would in a traditional assignment-creation software. however, we also give users access to a coding environment which they can use to create questions. this lets them -or our question generation models -generate questions in rich formats. that includes questions with images, latex, or interactive components like coding challenges, animations, or full-fledged games (see figure 8 ). note 3: composability. courses can use the various parts of the tool independently. for example, if a course largely uses written assignments and does not want to switch to online assignments, the tool can be used purely for question generation by sending all questions to the question bank and then copying them into the physical exams. a course which has existing questions can bring them onto the platform, display them to students, then get analytics and have them re-written to be more effective. courses with an lti or which use other tools like gradescope can display all questions through those platforms, link them to learn, and get analytics using their existing data. this allows many different kinds of courses to use the tool, and also for gradual adoption of those parts of the tool which are most useful to any given course. complete flow an instructor uploads the material for their course, either manually, by pointing our scraper to their course website, or through an lti integration. we then generate richly-formatted questions which can include interactive components through code. those questions are placed into assignments which can be completed by students. once students complete the assignments, the results are used to re-write the assignments and improve future question generation. ## how natural language processing is being used to augment education although the most visible application of nlp in learn is that questions are generated automatically, many aspects of writing and re-writing assignments are enhanced by nlp research. in this section, we detail the features of the platform enabled by nlp. ## case study learn has been integrated in multiple courses at the university of pennsylvania, and over 1000 students have used the platform. the different classes used the tool in different ways, verifying its composability. in many of those classes, for example, learn was a supplementary tool, whereas the introduction to artificial intelligence class used it as a quizzing platform for reading quizzes. in the latter class, we were able to collect significant data on the efficacy of the platform, which we report here. ## conclusion learn is a unified, easy-to-use tool to apply question generation in classrooms. the tool is able to create assignments which write and re-write themselves. it achieves this through a number of features enabled by recent advances in nlp. having been tested in multiple classes with over 1000 students, it has been demonstrated to improve test scores and is prefered by students to traditional alternatives.