Personalized Multimodal Feedback Generation in Education (2020)

Abstract: The automatic feedback of school assignments is an important application of AI in education. In this work, we focus on the task of personalized multimodal feedback generation, which aims to generate personalized feedback for teachers to evaluate students’ assignments involving multimodal inputs such as images, audios, and texts. This task involves the representation and fusion of multimodal information and natural language generation, which presents the challenges from three aspects: (1) how to encode and integrate multimodal inputs; (2) how to generate feedback specific to each modality; and (3) how to fulfill personalized feedback generation. In this paper, we propose a novel Personalized Multimodal Feedback Generation Network (PMFGN) armed with a modality gate mechanism and a personalized bias mechanism to address these challenges. Extensive experiments on real-world K-12 education data show that our model significantly outperforms baselines by generating more accurate and diverse feedback. In addition, detailed ablation experiments are conducted to deepen our understanding of the proposed framework.

## introduction in recent years, oral presentations have become a popular form of assignments in online k-12 education @xcite ). an oral presentation assignment requires students to answer a question or explain a concept verbally. it is able to test students' oral expression skills, language organization abilities, and understanding of the topic itself simultaneously @xcite . oral assignments are usually submitted in video format, which involves multimodal information. for example, given a math question "please describe the procedure of finding the greatest common divisor of two integers", a student is asked to record a video in which he or she presents the answer to teachers. the video contains information on the modality of image (pictures of the video), audio (voice of the student), and text (transcribed speech). to evaluate such oral presentation assignments, teachers on the online education platform provide short textual feedback. an example is shown in figure 1 . feedback may involve different aspects of modalities, such as the clarity of video, the fluency of voice, and the relevance between the answers and the questions. meanwhile, different teachers tend to write feedback on their own language styles. in order to lighten the workloads of teachers and improve the efficiency of online teaching, in this paper, we aim to automatically generate personalized feedback from multimodal oral presentation assignments based on the language styles of various teachers. the task, known as personalized multimodal feedback generation, is an important but rarely touched application of ai in education @xcite . one traditional solution of this task is to construct a pipeline-based system. firstly, raw information from different modalities is passed into a series of pre-trained models for image recognition, speech fluency evaluation, and text relevance assessment. then, according to some manually designed strategies, a piece of artificial feedback is retrieved from a repository. the main limitations of the pipeline methods are: (1) the terminal supervisory signals (the feedback texts in our case) cannot be propagated to upstream you did a great job, but the fluency should be improved. modules. thus we have to use extra labeled data to train them, also, domain knowledge may be required; to address the above challenges, in this work, we propose a novel deep learning architecture named personalized multimodal feedback generation network (pmfgn) which can be trained in an end-to-end manner. although there exist some end-to-end multimodal-based text generation approaches @xcite @xcite , most of them cannot handle inputs with the form of image, audio, and text simultaneously. to the best of our knowledge, our model with a novel network structure is the first one designed for the task of multimodal feedback generation. in our proposed framework, we introduce a modality gate with a hierarchical attention mechanism that enables the model to generate different parts of the feedback based on the evaluation of different modalities. meanwhile, we design a personalized bias mechanism to encourage the model to generate personalized feedback. experiments have been conducted on a real-world k-12 oral presentation dataset. the results have verified the superiority of our proposed model according to various evaluation metrics. compared with several baselines, our method can provide more precise, reasonable, and diverse feedback. we summarize our major contributions as follows: the rest of the paper is organized as follows. first, we introduce the task definition of personalized multimodal feedback generation in section 2. afterward, we present our pmfgn framework in section 3. next, section 4 carries out our experimental setup and results with discussions. then, we review related works in section 5. finally, section 6 concludes the work with possible future research directions. ## task definition typically the feedback given by teachers is mainly concerned about the clarity of the video, the fluency of the voice, and the relevance between the answer and the given topic. thus, given a question q and a video submission, we extract raw information from three modalities: image, audio, and text. since images don't change much from beginning to end when a student giving an oral presentation, we just take a screenshot of the video, i.e. an image i, as the signals of image modality. we separate the sound from the video as image h t-1 h t h t+1 p gen (y t ) ey t-1 p(y t ) = r ⋅ p gen (y t ) + (1r) ⋅ p per (y t ) r h t text encoder ey t-1 ey t-2 p per (y t ) input layer hidden layer 1 (personalized) hidden layer 2 audio encoder image encoder question answer text audio figure 2 : the overview of the framework. the illustration shows an example where the audio information is selected to pass through the modality gate at the current decoding step. the audio signals a. besides, we transcribe the audio by automated speech recognition (asr) tools into texts t. both the question content q and the asr transcription t compose the signals of textual modality. we formulate the task of personalized multimodal feedback generation as follows. given a corpus @xmath0, a, t, q, p, y)} n @xmath1, where each instance contains an image i, a piece of audio a, a speech text t, a question text q, a teacher identifier p and the corresponding feedback text y written by that teacher, we seek to train a model which can generate personalized feedback for the teacher p given a set of the above multimodal inputs (i, a, t, q). ## experiment in this section, we conduct extensive experiments to evaluate our proposed framework based on a realworld dataset collected from an online education platform. through the experiments, we try to answer two questions: (1) does our model achieve better performances than representative baselines? and (2) how does each component in our proposed framework contribute to the performance? ## related work to the best of our knowledge, there is no previous work on the problem of automatic feedback generation for oral presentation assignments. however, there exist related works about text generation based on unimodal or multimodal information. the works @xcite @xcite investigate the problem of image captioning, which takes an image as input to generate a textual description. video description is studied in the works @xcite and @xcite . the first two take only dynamic images as input while the last one integrates the images and audio via a hierarchical attention mechanism to generate the description. the works more similar to our work in the application are those which generate reviews for products on online shopping platforms. in the work @xcite , a seq2seq model is adopted to encode attributes (user, product, rating) into vectors and then generate the review with an rnn decoder. an attention layer is added between encoder and decoder to learn the alignments between attributes and generated words. the work @xcite generates reviews for a given user, item, several related phrases such as product titles, and aspect-aware knowledge. in the work @xcite , the authors use an image of a product and a predetermined rating as evidence, to generate a review. for a given pair of user and item, the work @xcite first predicts a rating that the user would give to the item, and then integrate the multimodal information of the user, the item, the predicted rating, and an image to generate a review. in net2text @xcite , the authors first construct a graph with users and items as nodes and then predict a review of a user towards an item in a language model conditioned on learned node embeddings. personalized text generation is also a hot problem in many nlp tasks, especially dialogue generation. the most common solution is to introduce persona embeddings to model the personality of speakers @xcite . besides, the work @xcite proposes to build personalized dialogue models via multi-task learning. the authors train a seq2seq model on common conversation data and an autoencoder on personal non-conversation data with the parameters in their decoders sharing. in addition, domain adaptation @xcite and transfer learning @xcite methods are also proposed for personalized dialogue generation. ## conclusion in this paper, we study the problem of automated multimodal feedback generation for oral presentation assignments in k-12 education. as a pioneering work for this task, we propose a novel pmfgn that learns to produce personalized feedback for oral presentations in an end-to-end manner. equipped with a modality gate mechanism and a personalized bias mechanism, the proposed framework encodes and fuses multimodal information in an effective way and achieves personalized feedback generation. the performance of the proposed model is demonstrated by experiments conducted on a real-world k-12 education dataset according to various evaluation metrics. this work focuses on evaluating an oral presentation assignment based on its intrinsic quality (clarity, fluency, relevance, etc.), but doesn't consider judging whether an answer is right or wrong since the automated correction of assignments is another problem under study. in the future, we plan to integrate this aspect into feedback generation.