Recursive Question Understanding for Complex Question Answering over Heterogeneous Personal Data (2025)

Abstract: Question answering over mixed sources, like text and tables, has been advanced by verbalizing all contents and encoding it with a language model. A prominent case of such heterogeneous data is personal information: user devices log vast amounts of data every day, such as calendar entries, workout statistics, shopping records, streaming history, and more. Information needs range from simple look-ups to queries of analytical nature. The challenge is to provide humans with convenient access with small footprint, so that all personal data stays on the user devices. We present ReQAP, a novel method that creates an executable operator tree for a given question, via recursive decomposition. Operators are designed to enable seamless integration of structured and unstructured sources, and the execution of the operator tree yields a traceable answer. We further release the PerQA benchmark, with persona-based data and questions, covering a diverse spectrum of realistic user needs.

## introduction 1.motivation and problem statement question answering (qa) over combinations of structured and textual data has been substantially advanced by verbalizing all contents and encoding it with a large language model (llm) @xcite @xcite . a prominent case of such heterogeneous content, of growing importance, is personal information management @xcite . user devices collect vast amounts of personal data on a daily basis. each interaction with digital services, such as amazon marketplace, netflix or spotify, is stored for the specific user, and services running on smartphones or smart watches track data without explicit user actions, such as travel routes or workouts. users want convenient tools to query their data, as easy as with an llm-powered ai figure 1 : excerpt of relevant user data for the question q 3 , stored in structured (workouts), semi-structured (calendar) and unstructured (social media, mails) sources. co-pilot. at the same time, data protection is a crucial requirement, as full control over sensitive content is a top priority for users @xcite . ideally, all data and its processing resides locally on the owner's devices (mobile, tablet or pc), tightly limiting the available computational resources. reconciling these goals is a challenge @xcite . users' needs range from look-ups of events to queries of analytical nature @xcite : such questions require fine-grained understanding of the user intent, and entail complex operations like grouping, aggregations, joins or matching temporal conditions, often with hundreds of relevant information pieces from different sources. the goal of this work is to devise a qa system for personal data that (i) supports such complex information needs, (ii) is user-friendly and intuitive for lay people, and (iii) has low footprint regarding computation, memory and energy, suitable for running directly on user devices. ## the reqap data model we treat all data sources as temporally ordered lists of events, where each event is a dictionary with a set of key-value pairs. for example, each mail is an event with keys date, time, sender, recipients, subject and body; for music streams, each event would have keys like date, start_time, end_time, title, artists. example events are shown in the bottom right of figure 2 (workout in light green and social media in yellow). our current implementation covers the following event types, accessed via data exports: calendar entries, notes, mails and social media posts with semi-structured and unstructured data, and workouts, streams (movies, tv series, music), and online purchases with structured data. note that event types can be added easily with this agile data model, which treats all contents as key-value pairs. ## the reqap methodology reqap processes questions in two stages: (i) the question understanding and decomposition (qud) for constructing the operator tree, and (ii) the operator tree execution (otx) stage for deriving the answer with the corresponding events. figure 2 shows a characteristic example that illustrates the two stages of reqap. ## experimental setup configurations. we experiment with three different options for qud: using gpt4o ("gpt-4o") or llama3.3 foot_2 with 8 icl examples, or a 1b model foot_3 fine-tuned as described in section 3.1 (sft). to initialize the extract operator, we use bart-base foot_4 , and the retrieve operator uses splade foot_5 and a cross-encoder pre-trained on msmarco foot_6 . further details for model training are in appendix c. baselines. we compare reqap against rag and codegen: metrics. we measure hit@1 as our main metric for personal qa, where users expect a single concise answer. as some questions aggregate over hundreds of events, we also measure a relaxed metric rlx-hit@1 which allows @xmath0. both metrics are micro-averaged over all questions in the test set. statistical significance of improvements by reqap compared to the best baseline (for each model) is measured via mcnemar's test with @xmath1.05, and is indicated by an asterisk(*). rag fails on perqa. while the baselines based on codegen remain competitive, rag performs poorly. it cannot cope with questions that need to aggregate (sum, avg, ordering, etc.) over hundreds of events. in some cases, the input data exceeds the maximum context size of the lm (128k tokens). ## evaluation via user study the perqa benchmark is designed with synthetic personas to avoid privacy issues. the goal of this user study is two-fold: (i) analyze whether the questions in perqa are comparable with real user questions, and (ii) investigate whether a small-scale variant of reqap, run on end-user devices, can deal with such realistic information needs. ## conclusion this paper introduced a new approach, reqap, for qa over heterogeneous personal data, spanning structured tables and unstructured text. reqap recursively decomposes a user question into an operator tree, which can be efficiently executed on user devices. in our experiments, reqap outperforms methods based on verbalization (rag) or translation (codegen), and our small-model configurations achieve competitive performance, thus reconciling the goals of expressive qa and light-weight footprint.