SEEN : Structured Event Enhancement Network for Explainable Need Detection of Information Recall Assistance (2022)

Abstract: When recalling life experiences, people often forget or confuse life events, which necessitates information recall services. Previous work on information recall focuses on providing such assistance reactively, i.e., by retrieving the life event of a given query. Proactively detecting the need for information recall services is rarely discussed. In this paper, we use a human-annotated life experience retelling dataset to detect the right time to trigger the information recall service. We propose a pilot model—structured event enhancement network (SEEN) that detects life event inconsistency, additional information in life events, and forgotten events. A fusing mechanism is also proposed to incorporate event graphs of stories and enhance the textual representations. To explain the need detection results, SEEN simultaneously provides support evidence by selecting the related nodes from the event graph. Experimental results show that SEEN achieves promising performance in detecting information needs. In addition, the extracted evidence can be served as complementary information to remind users what events they may want to recall.

## introduction people have to deal with many events in their daily life. as time passes, they might forget details about their past experiences. forgetting the exact name of people or places or things and mixing up life events is a common occurrence. this explains the importance of an information recall system that helps people bring to mind what they are trying to recall. we propose reactive and proactive service modes for an information recall system @xcite . in reactive mode, users directly ask the system about their life events, whereas in proactive mode, the system attempts to automatically detect whether users need memory recall assistance and then provides the information they seek to recall. for reactive mode, studies have been done on visual lifelog recall @xcite @xcite @xcite , which focuses on the construction of a multimodal retrieval model that enables users to search through photos using textual queries. we propose an information recall system @xcite to answer questions about life experiences over a personal knowledge base. in contrast to reactively receiving users' requests, proactive mode, which detects the right time to trigger the information recall service, is still little explored. in this paper, we further propose a pilot study to proactively detect the user's need for information recall assistance. one common use case of memory recall assistance occurs in human conversation. to identify whether people have difficulties in recalling past experiences, @xcite propose a model to detect speech hesitation. here, we focus on detecting the need for information recall support in people's narratives. specifically, we seek to detect the following four situations in narratives to determine whether to trigger the service: to the best of our knowledge, no dataset is available for this purpose. for this reason, we extended the hippocorpus dataset @xcite with new life event annotations as cases where users encounter problems and require recall assistance. @xcite invited crowd-workers to write stories about their life experiences, and asked them to write those stories again a few months later. as such, the nature of hippocorpus meets our requirement. in hippocorpus, life experiences written the first and the second times are referred to here as pre-retold and post-retold stories, respectively. the need for information recall is detected by comparing the pre-retold and post-retold stories. in this paper, we propose a model to identify the event types in post-retold and pre-retold stories. the model is referred to as structured event enhancement network (seen). a transformer-based language model is used for encoding textual data. to encode the structured information of event description in stories, we construct an event graph by utilizing life event triples. to further capture the relations between events, the results of coreference resolution are incorporated into the event graph. the graph is encoded by the graph attention network (gat) @xcite and fused with the language model for integrating textual and structured information. in addition, our model will extract the relevant events in a story pair as support evidence to explain the decision of the prediction. in this way, the user will easily recall the forgotten events. in sum, the contributions of our work are threefold: (1) we introduce the task of detecting the need for information recall in a narrative and providing the related information as the support evidence. (2) we present the nir dataset, foot_0 a human-annotated life experience retelling dataset for detecting the needs of information recall. additional: this is extra information about a life event that is not previously recorded in the collected lifelogs. the event in sentence (6) is additional due to the lack of similar event in the preretold story. forgotten: the life events that have been forgotten, i.e., are not mentioned here. as the event in sentence (3) does not relate to other events in the post-retold story, it is a forgotten event. unforgotten: in contrast to forgotten, the life events in the pre-retold story and also mentioned in the post-retold story belong to unforgotten events. as the events in sentence (2) are also mentioned in the sentence (4) and sentence (5) in the post-retold story, they are unforgotten events. in our dataset, each life event in the pre-retold and post-retold story stories is labeled with one of five event types. the annotation of relevant events within another story of the story pair is also included to denote as support evidence of the event type. that is, we annotate event types and the corresponding support evidence in the pre-retold and post-retold stories. the construction of the dataset is described in section 3. ## structured event enhancement network although the pre-trained language models have shown great success on various nlp tasks, some works @xcite @xcite also suggest that the structured information can enhance token representations. we construct an event graph based on life event triples. the event graph is incorporated into our model for capturing fine-grained information of life event relations within a document. inspired by greaselm @xcite , which incorporates the language model with the external knowledge graph, we initialize the node representations by using the language model. specifically, we extract the hidden states from different encoder layers of the language model as the node representations. a gat model is employed to propagate the structured information of the event graph. then the updated node representations are used for enhancing the token representations in the language model by our fusion mechanism. figure 3 shows an overview of our proposed structured event enhancement network (seen). the details are described as follows. ## comparison with natural language inference to identify event types, we propose a pilot model to determine the relations between e d ′ and d. this is different from simply comparing the relation between two sentences in a natural language inference (nli) task @xcite @xcite . identifying the event types in narratives involves two main challenges. firstly, the event type of the event in d ′ must be determined by identifying the event pair relations with all life events in d, since the discourse structures in d and d ′ are often different. secondly, the granularity of event descriptions between the stories in a pair can differ. hence, to determine the event type, we must infer the relevant details of the described events in both stories. to investigate the difference between nli and event type identification in information recall assistance, we experiment with the impact of introducing the nli task into our model. we find that pre-training the language model on the nli task and fine-tuning the model on our task will improve the performance. however, the label definitions in the nli task are different from our task. the details are discussed in appendix d.2. ## related work recently, more and more works show their interests in lifelogging. some works have investigated lifelogging applications on lifestyle understanding @xcite , diet monitoring @xcite , and contact tracing @xcite ). in addition, several studies have worked on the reactive information recall service. @xcite @xcite introduce visual lifelog retrieval tasks that aims at querying specific moments in a lifelogger's life. @xcite and @xcite construct a multimodal retrieval model that enables users to search their photos with textual queries. @xcite propose a system to answer the questions about personal life experiences over personal knowledge base. in this work, we focus on detecting the need for a proactive information recall service along with the support evidences. the structured information, such as dependency parsing results, has proved the effectiveness in capturing the contextual interactions. for instance, the model proposed by @xcite , bert4gcn @xcite , and sgnet @xcite integrate the dependency relations to leverage syntactic information. greaselm @xcite and luke @xcite integrate the external knowledge base by fusing token representations and entity representations from the language model and the additional embeddings, respectively. here, we intro-duce an event graph into our proposed model to capture the relations of the life events. ## conclusion information recall has attracted much attention in recent years. in contrast to previous studies, we present the task of proactive information recall support and construct nir, the first human-annotated dataset, to investigate the need for information recall. in this work, we seek to detect event relations between life experiences retold at different times, and identify five event types to determine the time to trigger information recall. to identify the event types for information recall assistance, a pilot model-structured event enhancement network (seen) is proposed. we construct an integration layer to fuse the structured information from the event graph into textual representations. in addition, seen provides the support evidence to the events by selecting the related nodes in the event graph. users can consult the explanation to recall their past experiences. however, identifying inconsistent events is still challenging; this is left as future work. we also plan to construct an endto-end system to extract life events in narratives and provide proactive information recall support. besides, at the current stage, we utilize the gold event graph of each story in our experiments. in the future, we will explore the method to extract personal life events from document to construct a personal knowledge graph. as time passes, many events continuously happen in our daily life. consulting only one document that describes personal life experiences is not enough to identify the need for information recall assistance in the real-world application. however, the dataset that can be applied to investigate the issue of detecting information recall needs is hard to collect. we extend the hippocorpus dataset, whose nature is in line with our work, to construct the nir dataset. on the other hand, although our nir dataset provides two versions of stories of the same events written at different times, we still cannot confirm which story, the previous one or the latter, is correct when contradictory. in this work, we propose a pilot exploration of proactively information recall assistance. to this end, we simply postulate that the story written at the previous time was correct when the user was still deeply impressed by the life events, so the story written at that time was used as a reference story. in addition, the number of inconsistent events is relatively lower in our dataset due to the human writing habit of avoiding uncertain events. in other words, when writing a diary, we always write the ones we exactly remember, which leads to difficulty collecting inconsistent events.