P e T ra: A S parsely S upervised M emory M odel for P eople T racking (2020)

Abstract: We propose PeTra, a memory-augmented neural network designed to track entities in its memory slots. PeTra is trained using sparse annotation from the GAP pronoun resolution dataset and outperforms a prior memory model on the task while using a simpler architecture. We empirically compare key modeling choices, finding that we can simplify several aspects of the design of the memory module while retaining strong performance. To measure the people tracking capability of memory models, we (a) propose a new diagnostic evaluation based on counting the number of unique entities in text, and (b) conduct a small scale human evaluation to compare evidence of people tracking in the memory logs of PeTra relative to a previous approach. PeTra is highly effective in both evaluations, demonstrating its ability to track people in its memory despite being trained with limited annotation.

## introduction understanding text narratives requires maintaining and resolving entity references over arbitrary-length spans. current approaches for coreference resolution @xcite @xcite scale quadratically (without heuristics) with length of text, and hence are impractical for long narratives. these models are also cognitively implausible, lacking the incrementality of human language processing @xcite . memory models with finite memory and online/quasi-online entity resolution have linear runtime complexity, offering more scalability, cognitive plausibility, and interpretability. memory models can be viewed as general problem solvers with external memory mimicking a turing tape @xcite . some of the earliest applications of memory networks in language understanding were for question answering, where the external memory simply stored all of the word/sentence embeddings for a document @xcite . to endow more structure and interpretability to memory, key-value memory networks were introduced by @xcite . the key-value architecture has since been used for narrative understanding and other tasks where the memory is intended to learn to track entities while being guided by varying degrees of supervision @xcite @xcite . we propose a new memory model, petra, for entity tracking and coreference resolution, inspired by the recent referential reader model @xcite but substantially simpler. experiments on the gap @xcite pronoun resolution task show that petra outperforms the referential reader with fewer parameters and simpler architecture. importantly, while referential reader performance degrades with larger memory, petra improves with increase in memory capacity (before saturation), which should enable tracking of a larger number of entities. we conduct experiments to assess various memory architecture decisions, such as learning of memory initialization and separation of memory slots into key/value pairs. to test interpretability of memory models' entity tracking, we propose a new diagnostic evaluation based on entity counting-a task that the models are not explicitly trained for-using a small amount of annotated data. additionally, we conduct a small scale human evaluation to assess quality of people tracking based on model memory logs. petra substantially outperforms referential reader on both measures, indicating better and more interpretable tracking of people. foot_0 ## model figure 2 depicts petra, which consists of three components: an input encoder that given the tokens generates the token embeddings, a memory module that tracks information about the entities present in the text, and a controller network that acts as an interface between the encoder and the memory. ## related work there are several strands of related work, including prior work in developing neural models with external memory as well as variants that focus on modeling entities and entity relations, and neural models for coreference resolution. memory-augmented models. neural network architectures with external memory include memory networks @xcite , neural turing machines @xcite , and differentiable neural computers @xcite . this paper focuses on models with inductive biases that produce particular structures in the memory, specifically those related to entities. ## conclusion and future work we propose a new memory model for entity tracking, which is trained using sparse coreference resolution supervision. the proposed model outperforms a previous approach with far fewer parameters and a simpler architecture. we propose a new diagnostic evaluation and conduct a human evaluation to test the interpretability of the model, and find that our model again does better on this evaluation. in future work, we plan to extend this work to longer documents such as the recently released dataset of @xcite .