Re:Member: Emotional Question Generation from Personal Memories (2025)

Abstract: We present Re:Member, a system that explores how emotionally expressive, memory-grounded interaction can support more engaging second language (L2) learning. By drawing on users’ personal videos and generating stylized spoken questions in the target language, Re:Member is designed to encourage affective recall and conversational engagement. The system aligns emotional tone with visual context, using expressive speech styles such as whispers or late-night tones to evoke specific moods. It combines WhisperX-based transcript alignment, 3-frame visual sampling, and Style-BERT-VITS2 for emotional synthesis within a modular generation pipeline. Designed as a stylized interaction probe, Re:Member highlights the role of affect and personal media in learner-centered educational technologies.

## introduction as language learning technologies evolve, there is growing interest in systems that go beyond rote vocabulary drills or disembodied text. research in human-computer interaction (hci) and natural language processing (nlp) has shown that social presence, emotional involvement, and personal relevance significantly improve learning outcomes, especially for the acquisition of second languages (l2). however, most existing tools are based on generic or de-contextualized content, limiting their potential to tap into the emotional and mnemonic power of a learner's lived experiences. a growing body of hci research has explored how large language models (llms) can support learners and designers through agent-assisted creativity. systems such as ideationweb @xcite and promptify @xcite scaffold user interaction with generative models, enabling iterative refinement and analogical exploration. in language learning, voiced chatbot interfaces, such as conversational characters and stress-free conversational partners @xcite , have shown how conversational systems can support learners by tailoring responses to their needs. related work in agentassisted creativity and co-design highlights the importance of aligning model outputs with user intent and emotional framework @xcite . from an nlp perspective, recent work on question generation has moved toward more contextsensitive and user-aligned output. newer methods leverage llms for conversational foresight @xcite and empathetic dialogue @xcite where the user's inferred state shapes responses for empathy and engagement @xcite . our work contributes to this work by combining environmental-aware inference from sequential visual frames with llm-based question generation in a real-time learner interface. the system supports reflective learning by surfacing system-generated, context-sensitive questions that adapt to the learner's evolving affective and attentional state, a goal aligned with broader calls for emotionally intelligent educational technologies @xcite . this bridges recent work in hci and nlp on responsive, learneraware systems for mixed-initiative interaction. our system builds on previous work by grounding llm-generated questions in video-based emotion cues, enabling emotionally responsive interactions that match the learner's current context. specifically, we present re:member 1 , an opensource system that turns videos of personal memories, also known as episodes, such as casual recordings of travel, family, or everyday life, into emotionally voiced, interactive prompts for language learning. by combining recent advances in large language models (llms), expressive speech synthesis, figure 1 : example interface frame from video (1), showing (from top to bottom) three frames of sequential visual context, the generated emotion, the generated system question, a playable text-to-speech box, the name of the video file, the "analyze video" button, and "previous" and "next" buttons to navigate between sequential moments. and vision-language processing, re:member analyzes short user-uploaded videos, extracts scenerelevant transcripts and images, and generates stylized spoken questions in the learner's target language. these questions are voiced in emotional speaking styles (e.g., playful, whispered, drowsy), selected to match the tone and atmosphere of the scene. the core idea behind re:member is that emotionally salient, personally meaningful content may create deeper engagement for language learners, especially when paired with stylized voice output that mimics familiar social dynamics (e.g., a whisper from a friend or an excited exclamation). rather than relying on synthetic neutrality, our system embraces affective richness as an instructional tool. this paper introduces the design of re:member and demonstrates its capabilities as an emotional question-generation companion. we detail our architecture, design rationale, and sample outputs, and reflect on the broader implications for language education, affective computing, and memory-centered interaction. ## system overview the goal of re:member is to generate emotionally expressive questions from personal memory videos for l2 (second language) conversational practice. these questions are designed to support language learning by connecting spoken language, visual content, and emotional speech. ## illustrative outputs we demonstrate the system with two sample videos: (1) a video of a walk along tokyo's sumida river with the commentary playing the role of a language teacher, and (2) a video of the user boarding a train in japan with spoken instructions for boarding the train. both videos were recorded with meta ray-ban glasses, and (1) is 1 minute and 31 seconds in length while (2) is 31 seconds in length. for video (1), the system segmented and analyzed 13 moments, generating an emotion, a student question, and text-to-speech for each moment. for each of the 13 segments in video (1), the system generated a natural japanese-language question grounded in both the visual scene and the transcript. these questions reflect a consistent student-like curiosity, such as asking what kinds of boats travel through the river or how tall the tokyo skytree is. the selected emotion styles were well-matched to the riverfront setting, with a majority in the gentle voiced whisper style, interspersed with more upbeat cheerful and late-night relaxed tones. all five available emotion styles appeared at least once, showing that the variation mechanism functioned appropriately given the consistent environment. the visual frames used as context were sampled from before, during, and after each utterance, helping the language model infer motion and visual focus --such as when the user points to a boat or approaches a bridge. each segment resulted in synchronized audio narration with emotional speech, allowing for immersive and pedagogically meaningful playback. a select moment from video (1), as seen in figure 1 shows the user pointing their finger to tokyo sky tree, a tall tower on the other side of the river. for this moment, the system generated the question 東京スカイツリーの高さはどのくら いですか？ translation: about how tall is tokyo sky tree? with the cheerful emotion (るんるん). for video (2), the system identified and processed three distinct segments, each aligned with the user's spoken instructions for boarding a train in japan. the generated questions reflect a studentlike curiosity about practical aspects of the scene, such as the convenience of using trains near event venues or the layout of the train interior. emotion styles were chosen to match the focused, informational tone of the video: a balance of voiced whisper, silent whisper, and neutral speech was used across the three questions. though the short duration of the video limited the range of styles, the variation mechanism successfully avoided repetition and produced a tone consistent with the setting. visual context was drawn from three-frame windows centered on each utterance, allowing the language model to reference specific spatial cues -such as when the user physically steps onto the train. as with video (1), the result is synchronized emotional narration paired with visually grounded, pedagogically meaningful questions. the generated questions and associated emotion styles are shown below: question: (silent whisper) 試合が行わ れている場所での電車の利用はどの ように便利ですか？ translation: how is using the train convenient near where the event is being held? question: (neutral) この電車の車内 はどのように見えますか？ translation: what does the inside of this train look like? question: (voiced whisper)この電車 の車両には特別な座席やスペースが ありますか？ translation: does this train car have special seats or areas?