T imeline QA : A Benchmark for Question Answering over Timelines (2023)

Abstract: Lifelogs are descriptions of experiences that a person had during their life. Lifelogs are created by fusing data from the multitude of digital services, such as online photos, maps, shopping and content streaming services. Question answering over lifelogs can offer personal assistants a critical resource when they try to provide advice in context. However, obtaining answers to questions over lifelogs is beyond the current state of the art of question answering techniques for a variety of reasons, the most pronounced of which is that lifelogs combine free text with some degree of structure such as temporal and geographical information. We create and publicly release TimelineQA, a benchmark for accelerating progress on querying lifelogs. TimelineQA generates lifelogs of imaginary people. The episodes in the lifelog range from major life episodes such as high school graduation to those that occur on a daily basis such as going for a run. We describe a set of experiments on TimelineQA with several state-of-the-art QA models. Our experiments reveal that for atomic queries, an extractive QA system significantly out-performs a state-of-the-art retrieval-augmented QA system. For multi-hop queries involving aggregates, we show that the best result is obtained with a state-of-the-art table QA technique, assuming the ground truth set of episodes for deriving the answer is available.

## introduction the promise of augmented reality (ar) glasses has renewed interest in building personal assistants that are capable of being with us at all times of the day. in order for such assistants to be useful, they need to have detailed knowledge about the user, including their past experiences, preferences, habits and goals in the spirit of systems such as memex @xcite and mylifebits @xcite . a lot of that knowledge already is implicitly present in the digital data that people generate by interacting with a myriad of online services such as photos, maps, health apps, shopping and content streaming. a lifelog is a private and secure database that contains a set of episodes from the user's past that are gleaned from these data sources and in the future from smart glasses. the lifelog is completely under the control of the user, and only they can decide if and when to share fragments of it as they see beneficial. for example, they may share past dining experiences with an assistant when trying to choose an item from a menu, or past movie preferences with a friend when trying to decide which movie to watch together. in addition to issues relating to privacy, lifelogs raise two main classes of challenges. the first is to infer meaningful episodes from the raw data. for example, such an inference module would take as input a set of photos and output an episode such as visited venice for 7 days, or celebrated a birthday party with friends. the second challenge, which is the subject of this paper is to answer questions over the lifelog, such as when did i go to tokyo, what did i eat on my second night in paris, or how many times did i go to the dentist last year. question answering is challenging because the data contains a combination of text and structure. the episodes themselves are described as text (and may also contain images and video), but each episode is associated with a time and location. for example, in order to answer a query such as where did i take my mom when she visited seattle, the system first needs to figure out when mom visited seattle and then look for episodes within that time interval. other questions may require counting or reasoning over sets of episodes, similar to challenges raised in @xcite . this paper describes timelineqa, a benchmark for querying lifelogs. the benchmark includes a generator that produces lifelogs for imaginary people with different personas (e.g., age, gender, education and family status). each lifelog includes episodes drawn from a variety of activities, ranging from significant activities (e.g., going on a trip or getting married) to more daily activities (e.g., cooking dinner or going to the doctor). for each lifelog, the benchmark creates a set of question/answer pairs, specified in english. naturally, real lifelogs are complex and extremely diverse and are challenging to generate synthetically. our main contribution is a benchmark for qa systems over lifelog data of different sizes. the goal of the benchmark is not to represent people's lives in their full complexity or diversity, but to offer a sufficiently rich set of lifelogs that already exposes the challenges involved in question answering (qa). we show some snippets of our generated lifelogs section 4.1. as our qa techniques improve, the benchmark will be enriched to include more real life complexities. we describe a set of experiments demonstrating that current sota qa techniques fall short of adequate performance on lifelogs. we experimented with extractive @xcite and rag @xcite qa systems on atomic queries. somewhat surprisingly, even after fine-tuning, the generative rag qa system still lags behind the extractive system for question-answering. in addition, we ran a tapex @xcite , a table qa model and bart @xcite for complex queries over timelineqa. our experiments reveal that the best performing system, tapex, only scores 59.0%, assuming that the subset of episodes needed to compute the answer is known. ## related work the idea of creating a repository that captures all the knowledge about a person's life dates back to vannevar bush's vision of the memex system @xcite . @xcite describes the mylifebits system that implemented the vision with the technology available in the late 1990's, and they used simple keyword search with the help of an sql database to query its contents. @xcite describes a more recent project on creating lifelogs, and the solid project @xcite ) takes an even more radical approach, suggesting that all of the user's data be stored in a data pod and that applications be redesigned to access it from the pod. since the early years, the promise of personal agents has increased since data storage has become cheaper and ubiquitous, we anyway generate many more digital breadcrumbs with services we use on a daily basis, and ai techniques have become much better at analyzing text and image content. the design of our benchmark was inspired by the clevr benchmark for evaluating visual query answering systems @xcite . like clevr, we design a space of possible questions that can be asked and then generate synthetic datasets where we know the answer to each questions posed. there is a rich body of work on query answering. the ones closest to our work are on multi-hop queries @xcite and neural databases @xcite . in addition to queries that can be answered from a single episode in a lifelog, timelineqa includes more complex queries that require combining information from multiple episodes in a lifelog. this is similar to work on qa over long documents @xcite . however, the length of a lifelog is typically much greater than any existing benchmark or experimental dataset to the best of our knowledge. a typical lifelog can contain between 15m to 78m entries on average, where each entry contains about 8-9 tokens on average. furthermore, timelineqa queries can also contain aggregates (e.g., max, sum, average). neural databases considers the problem of answering aggregate queries over text data of arbitrary size, but it does not address the temporal aspects that are critical to queries over lifelogs. ## lifelogs a lifelog includes any kind of experience that a user recorded digitally (see figure 1 ). we model experiences as episodes in the lifelog, and every episode is associated with a start/end time and start/end location, if those are known. episodes are captured via photos or videos, smart watches (e.g., exercise and sleep tracking), mapping services (e.g., routes and visits), documents that have been explicitly stored (e.g., passport), or notes that the user takes describing their subjective experiences. a lifelog is completely private and accessible only to the user. she can share slices of her lifelog if and when there's value in doing so (e.g., getting better service from a sales person). episodes are typically activities that the user was involved in, such as celebrating a holiday, going on a trip, going for a run or a bike ride, physi-cal therapy, seeing fireworks or watching a movie. episodes in the lifelog can either be done by the owner of the lifelog or by someone in their family or circle of acquaintances, e.g., mom moving to seattle, sister getting married, having one's airconditioning fixed, or being told something by a friend. in addition to time and location, episodes may have attributes, such as who was involved, the distance and speed of a run, or the name of a product that was purchased. some of these attributes may be modeled explicitly in the lifelog if they're easy to extract, and others may remain in the raw text or image and found at query time. lifelogs are meant to be built with as little friction as possible from users. hence, as shown in figure 1 , the data is imported from the external services into the lifelog as raw data. some raw data already describes episodes (e.g., purchase or content consumption episodes). other episodes are then inferred by analyzing and fusing multiple pieces of raw data (e.g., a trip, or a meal with friends). of course, the inference step is a besteffort one, which means that some questions may still be impossible to answer and in some cases the qa system will point the user back to data that contains the answer (e.g., what did we eat on my daughter's birthday). questions are answered based on the text and structured data describing all the episodes in the lifelog. our work concerns question answering after the inference of episodes has been done. hence, formally a lifelog is a collection of episodes, each one associated with their start/end time and location: time-and-space boxed episodes. each episode contains some text and possibly pointers to external raw data. note that episodes can be nested within other episodes. ## creating lifelogs in timelineqa since we believe that timelineqa is the first in a series of lifelog benchmarks, we explain here in some detail how it is built. a lifelog is a set of episodes in the life of a person. our goal is to create lifelogs that contain a good range of experiences that a person may have in life and sufficient to begin benchmarking the performance of qa systems on lifelogs. to collect a broad set of typical episodes, we started with a detailed set of episode categories described in coelition foot_1 , a site that provides technology and expert advice for data collected about people on the internet of things, and distilled them into the categories shown in creating episodes once a persona is created, we begin creating episodes starting from the day the person was 18 years old to the present year. we first create episodes in the lifelog for life events, such as birth, educational phases, starting and ending jobs, marriage(s) and having children. we then proceed to generate episodes at different levels of granularity based on the timescales (annual, monthly, weekly, daily) as shown in some of the episodes we create are super episodes, which involves sub-episodes that depict events of finer granularity. for example, a multiday travel or trip episode will be broken down to movements between different destinations, and the itinerary for every single day and special episodes that happened in each day. the descriptions of episodes are generated by instantiating templates that we specify. every episode is associated with a set of alternative templates and a template is randomly picked and instantiated for a given episode to be created. since the templates are fixed, the descriptions generated may not offer the variety in descriptions we expect from a general population. we are in the process of incorporating the use of language models to generate episode descriptions as yet another alternative. however, it is interesting to understand what limitations on qa such a benchmark already exposes with templated descriptions. more variations in the episode activities can be added to the lifelog generator to more closely reflect the categories we find in the coelition and also what we crowdsourced ( consistency through constraints: to ensure more consistency, we keep track of the attributes of every single day in one's life. for example, the probability of certain episodes can change drastically if a person is on a trip or in the process of getting married. in timelineqa, constraints can be specified to prevent inconsistencies from occurring. for example, since it is much less likely that one bakes or has an annual dental checkup while traveling, we can explicitly state that these episodes should be mutually exclusive in timelineqa. if an episode is to be created on a certain day, timelineqa checks that it is mutually exclusive to any existing episode applicable to that day before creating the new episode. generating questions and answers: every lifelog, d, in timelineqa is associated with a set of question/answer pairs (q, a), where q is a natural language question over d and a is the correct answer to it. in order to ensure that we can create a variety of questions that are meaningful on a particular lifelog d and that we know the correct answers to them, the process of creating begins by creating a logical representation of the episodes in the lifelog and of the questions and the answers, and then turning them into natural language. the natural language of the questions and answers are created by instantiating a few templates for every episode type. because we use templates, timelineqa clearly lacks the richness of linguistic variation, but as noted previously, dealing with linguistic variation is not the focus of this benchmark. we generate questions and answers for each lifelog in two steps: atomic questions and complex questions. since atomic questions are ones whose answer is contained in a single episode in the lifelog, we can create them at the same time the episode is created. for example, if the episode is i went to a japanese restaurant with sarah on october 7th and ate sushi, then we would generate questions of the form: when did i have japanese food?, when did i meet sarah?, and where did i eat on october 7th? for each single episode, we create what, where, when and who questions as appropriate along with the corresponding answers. complex questions are ones that either rely on a set of facts in the lifelog, such as, how many times did i go to london? and where did i spend the first night in tokyo? or require combining multiple facts as in multi-hop questions such as, which restaurants did i go to during my trip to new york? to create such question/answer pairs easily, we create a database of the logical representation of all the episodes in the lifelog. we then consider a set of query templates and check whether the template can be instantiated on that database. examples of templates we consider are: since we have all the episodes, we can compute the answers to these questions correctly. size and density: lifelogs of different sizes can be created with timelineqa. the user specifies a year and duration parameter, and this will determine the length of the lifelog to generate. for example, if the year is 2023 and the duration is 5, then 5 years of episodes from 2018 to 2023, including lifetime episodes, will be created. lifetime episodes such as birth and college education may occur outside those 5 years. the user can also specify the density (sparse, medium, or dense) of episodes to generate in the lifelog. the variations in density are used to mimic that different users log their life events at different frequencies. for example, if the generator is called with the "sparse" parameter, then the probabilities of generating daily/weekly/monthly episodes will be much lower than the case when the generator is called with the "dense" parameter. ## conclusions we presented timelineqa, a benchmark for generating lifelogs of imaginary people. our experiments, with state-of-the-art qa models, showed that there is still significant room for improving qa over lifelog data. specifically, while extractive systems can achieve impressive performance on timelineqa for atomic queries, the best performing qa system for multi-hop queries scores only 59.0% in the perfect setting where the ground truth set of episodes are available. we view the current state of timelineqa as a first version that will be enhanced in several ways as the qa technology improves. in future enhancements the episodes can be made more realistic and varied to also include events such as driving one's children to practices, or car breakdowns, to more unexpected events such as experiencing an earthquake etc. in addition, episodes can be enhanced to include different modalities, such as photos or videos of the episodes and more complicated queries can be included such as "how many times did i swim in the month before i traveled to machu picchu?". ideally, with appropriate obfuscations to preserve privacy, a future version can mirror precisely the lifelogs of real people.