A Memory Model for Question Answering from Streaming Data Supported by Rehearsal and Anticipation of Coreference Information (2023)

Abstract: Existing question answering methods often assume that the input content (e.g., documents or videos) is always accessible to solve the task. Alternatively, memory networks were introduced to mimic the human process of incremental comprehension and compression of the information in a fixed-capacity memory. However, these models only learn how to maintain memory by backpropagating errors in the answers through the entire network. Instead, it has been suggested that humans have effective mechanisms to boost their memorization capacities, such as rehearsal and anticipation. Drawing inspiration from these, we propose a memory model that performs rehearsal and anticipation while processing inputs to memorize important information for solving question answering tasks from streaming data. The proposed mechanisms are applied self-supervised during training through masked modeling tasks focused on coreference information. We validate our model on a short-sequence (bAbI) dataset as well as large-sequence textual (NarrativeQA) and video (ActivityNet-QA) question answering datasets, where it achieves substantial improvements over previous memory network approaches. Furthermore, our ablation study confirms the proposed mechanisms’ importance for memory models.

## introduction the question answering (qa) task is one of the most important and challenging tasks in natural language processing (nlp). a significant advance has been seen in this subject thanks to models based on deep learning @xcite @xcite . however, these models assume that the whole input (e.g., sentences, paragraphs, etc.) can always be accessed while answering the question. this is a reasonable and practical approach if the input sequence is short but becomes less effective and efficient as the input length grows. these models are also cognitively implausible, lacking incremental human language processing @xcite (or visual @xcite ), which could enable the models for online tasks where data input is a stream, such as discourse, dialog or video processing. memory-augmented neural networks (manns) @xcite offer a solution to this problem as they introduce mechanisms to compress and remember the input contents. manns have shown effectiveness in various tasks @xcite @xcite , including question answering @xcite @xcite . in the latter, given a data sequence, the models process the input incrementally, capturing relevant information in the memory to answer a given query. this is akin to the daily behavior of human beings. for instance, our mental state is updated when we encounter a new text segment while reading @xcite . later, we can use the memorized information to solve a specific task @xcite . despite the success of manns and their similarity to human memory processes, they still have one significant limitation. most of these models rely on learning a single task to maintain their memory; for example, a mann for qa must learn what to store in memory while learning to answer questions. instead, some studies have suggested that human beings are endowed with specific mechanisms for memorization. on the one hand, rehearsal @xcite is a mechanism humans use to memorize information more effectively by repeating information over and over to be remembered. on the other hand, anticipation @xcite @xcite suggests that our memory could lead to predictions of upcoming material. furthermore, both processes are potentially guided by coreference information @xcite , an integral part of discourse comprehension. in this work, we propose a rehearsal and anticipatory memory model (ram) consisting of a memory module that uses a fusion module to integrate incoming information from a data stream to solve the qa task. this model is supported by two pretext tasks respectively inspired by human rehearsal and anticipation processes to enhance memory representation. these tasks are based on masked modeling @xcite of coreference information, allowing the model to anticipate a coreferent and link it to the past through memory and rehearsal. we validate our proposed model with datasets of short synthetic text sequences, long realistic text sequences, and video question answering. results show that our model significantly outperforms previous well-known memory approaches due to the enhanced memory representation achieved by using the pretext tasks. ## proposed method in this section, we introduce our model, which processes inputs and builds memory representations incrementally from a data stream, used later to provide an answer to a question (figure 1 ). ram leverages attention and gating mechanisms along with masked modeling-based self-supervision to create a simple but effective method to improve memory representation and memorization. we first describe the problem formulation of qa with manns. we then present our basic memory architecture for encoding input and decoding an answer. finally, we introduce our novel self-supervised mechanisms inspired by rehearsal and anticipation guided by coreference information. ## conclusion this article presents ram, a qa memory model supported by two self-supervised pretext tasks to improve memory presentation. it uses masked modeling to mimic the human processes of rehearsal and anticipation of coreference information. our experiments on short sequence qa, long sequence qa, and video qa show that our model substantially outperforms previous memory models. our further experimentation demonstrates that the rehearsal and anticipation tasks help to improve memorization and that the model leverages coreference information to support the processes.