Backtracing: Retrieving the Cause of the Query (2024)

Abstract: Many online content portals allow users to ask questions to supplement their understanding (e.g., of lectures). While information retrieval (IR) systems may provide answers for such user queries, they do not directly assist content creators—such as lecturers who want to improve their content—identify segments that caused a user to ask those questions.We introduce the task of backtracing, in which systems retrieve the text segment that most likely caused a user query.We formalize three real-world domains for which backtracing is important in improving content delivery and communication: understanding the cause of (a) student confusion in the Lecture domain, (b) reader curiosity in the News Article domain, and (c) user emotion in the Conversation domain.We evaluate the zero-shot performance of popular information retrieval methods and language modeling methods, including bi-encoder, re-ranking and likelihood-based methods and ChatGPT.While traditional IR systems retrieve semantically relevant information (e.g., details on “projection matrices” for a query “does projecting multiple times still lead to the same point?”), they often miss the causally relevant context (e.g., the lecturer states “projecting twice gets me the same answer as one projection”). Our results show that there is room for improvement on backtracing and it requires new retrieval approaches.We hope our benchmark serves to improve future retrieval systems for backtracing, spawning systems that refine content generation and identify linguistic triggers influencing user queries.

## introduction content creators and communicators, such as lecturers, greatly value feedback on their content to address confusion and enhance its quality @xcite . for example, 1 our code is opensourced: https://github.com/ rosewang2008/backtracing . ok, guys, we're almost ready to make this lecture immortal. ok. are we on? all right. this is an important lecture. it's about projection. and i'll, let me start by just projecting a vector b down on a vector a. so just to, so you see what the geometry looks like in, when i'm in, in just two dimensions. i'd like to find the point along this line. so that, that line through a is a one-dimensional subspace, so i'm starting with one dimension. i'd like to find the point on that line closest to b. can i just take that problem first and then i'll explain why i want to do it and why i want to project on other subspaces. so where, where's the point closest to b that's on that line? it's somewhere there. and let me connect that. and, and what's the whole point of my picture now? what, what's the, where does orthogonality come into this picture? the whole point is that this, this best point, that's the projection, p, of b onto the line, where's orthogonality? it's the fact that that's a right angle. that this, the error, this is like how much i'm wrong by. this is the difference between b and p. the whole point is that that, that that's perpendicular to a. that's got to give us the equation. that's got to tell us, that's the one fact we know, that's got to tell us where that projection is. let me also say, look, i, i've drawn a triangle there. so if we were doing trigonometry, we would do like, we would have angles theta and distances that would involve sine theta and cos theta. that leads to lousy formulas compared to linear algebra. the, the, the formula that we want comes out nicely. and what's the, what do we know? we know that p, this projection, is some multiple of a, right? it's on that line. so we know it's, it's in that one-dimensional subspace. it's some multiple, let me call that multiple x, of a. so really it's that number x i'd like to find. so this is going to be simple in 1d, so let's just carry it through and then see how it goes in high dimension. ok. the key fact is, so the, the key, the key to everything is that perpendicular, the fact that, that a is perpendicular to, a is perpendicular to e, which is b minus a x, xa. i don't care if i, xa. that that equals zero. you see that as the central equation? that's saying that this a is perpendicular to this correction. that's going to tell us what x is. let me just raise the board and simplify that and out will come x. ok. so if i simplify that, let's see, i'll move one to, one term to one side, the other term will be on the other side. it looks to me like x times a transpose a is equal to a transpose b. right? i have a transpose b is one term, a transpose a is the other, so right away, here's my a transpose a, but it's just a number now, and i divide by it, and i get the answer. x is a transpose b over a transpose a. and p, the projection i wanted, is that's the right multiple. that's got a cosine theta built in, but we don't need to look at angles. we've just got vectors here. and the projection is p is a times that x. or x times that a, but i'm really going to, eventually, i'm going to want that x coming on the right-hand side. so you see that i've got two of the three formulas already, right here. i've got the, the equation, that's the equation that, that, that leads me to the answer. here's the answer for x, and here's the projection. ok. can i do, add just one more thing to this one-dimensional problem? one more, like, lift it up into linear algebra, into matrices. here's the last thing i want to do with those, but don't forget those formulas. a transpose b over a transpose a. actually, let's look at that for a moment first. suppose, and a, well, then i'll, i'll let me, i'll, i'll, let me take this next step. so p is a times x. so can i write that, then? p is a times this neat number, a transpose b over a transpose a. that's our projection. can i ask a couple of questions about it, just while we look, get that, digest that formula? suppose b is doubled. suppose i change b so p squared equals p. so sure enough, those two propen this same lecture. so that'll give me a chance to recap the formulas and there they are, and recap the ideas. so let me start the problem today. i'm given a bunch of data points. and they lie close to a line but not on a line. let me take that. say at t equal to one, two, and three, i have one and two and two again. so my data points are, this is the, like, the time direction, and this is like, well, let me call that b or y or something. i'm given these three points and i want to fit them by a line, by the best straight line. so the problem is fit the points, one, one is the first corpus (e.g., lecture transcript) x i have a question, if i project the projection again that's the same point that is p^2=p. but if i keep doing such it should tell p^3=p^4=p^@xmath0, and this property holds for identity matrix. is my logic correct? backtracing: given the corpus and query , retrieve the sentence that most likely caused the query. ## related works the task of information retrieval (ir) aims to retrieve relevant documents or passages that satisfy the information need of a user @xcite . prior ir techniques involve neural retrieval methods like ranking models @xcite ; khattab and zaharia, 2020) and representation-focused language models @xcite @xcite . recent works also use plms for ranking texts in performing retrieval @xcite @xcite ; an advantage of using plms is not requiring any domain-or task-specific training, which is useful for settings where there is not enough data for training new models. these approaches have made significant advancements in assisting information seekers in accessing information on a range of tasks. examples of these tasks include recommending news articles to read for a user in the context of the current article they're reading @xcite , retrieving relevant bio-medical articles to satisfy health-related concerns @xcite @xcite , finding relevant academic articles to accelerate a researcher's literature search @xcite , or extracting answers from texts to address questions @xcite @xcite . however, the converse needs of content creators have received less exploration. for instance, understanding what aspects of a lecture cause students to be confused remains under-explored and marks areas for improvement for content creators. backtracing is related to work on predicting search intents from previous user browsing behavior for understanding why users issue queries in the first place and what trigger their information needs @xcite @xcite . the key difference between our approach and prior works is the nature of the input data and prediction task. while previous methods rely on observable user browsing patterns (e.g., visited urls and click behaviors) for ranking future search results, our backtracing framework leverages the language in the content itself as the context for the user query and the output space for prediction. this shift in perspective allows content creators to get granular insights into specific contextual, linguistic triggers that influence user queries, as opposed to behavioral patterns. another related task is question generation, which also has applications to education @xcite @xcite . while question generation settings assume the answer can be identified in the source document, backtracing is interested in the triggers for the questions rather than the answers themselves. in many cases, including our domains, the answer to the question may exist outside of the provided that caused the query q. one example is the lecture domain where the corpus is a lecture transcript and the query is a student question. the lecturer only discusses about projecting twice and the student further extends that idea to something not raised in the lecture, namely into projecting a matrix an arbitrary n times. ## backtracing formally, we define backtracing as: given corpus of n sentences @xmath1, . . . , x n } and query q, backtracing selects our work explores the backtracing task in a "zero-shot" manner across a variety of domains, similar to @xcite . we focus on a restricted definition of zero-shot in which validation on a small development set is permitted, but not updating model weights. this mirrors many emerging real-world scenarios in which some data-driven interventions can be applied but not enough data is present for training new models. completely blind zero-shot testing is notoriously hard to conduct within a reusable benchmark @xcite and is much less conducive to developing different methods, and thus lies outside our scope. ## backtracing benchmark domains we use a diverse set of domains to establish a benchmark for backtracing, highlighting both its broad applicability and the shared challenges inherent to the task. this section first describes the domain datasets and then describes the dataset statistics with respect to the backtracing task. ## methods we evaluate a suite of existing, state-of-the-art retrieval methods and report their top-1 and top-3 accuracies (i.e., whether the top 1 and 3 candidate sentences include the ground-truth sentences). reporting top-k accuracy is a standard metric in the retrieval setting. the methods can be broadly categorized into similarity-based (i.e., using sentence similarity) and likelihood-based retrieval methods. similar to @xcite , the likelihoodbased retrieval methods use plms to measure the probability of the query conditioned on variations of the corpus and can be more expressive than the similarity-based retrieval methods; we describe these variations in detail below. we use gpt-2 @xcite , gpt-j @xcite , and opt-6.7b @xcite as the plms. we additionally evaluate with gpt-3.5-turbo-16k, a new model that has a long context window ideal for long text settings like sight. however, because this model does not output probability scores, we cast only report its top 1 accuracy. random. this method randomly retrieves a sentence from the corpus. ## results the model results are summarized in the best-performing models achieve modest accuracies. for example, on the lecture domain with many distractor sentences, the bestperforming model only achieves top-3 43% accuracy. on the conversation domain with few distractor sentences, the best-performing model only achieves top-3 65% accuracy. this underscores that measuring causal relevance is challenging and markedly different from existing retrieval tasks. ## conclusion in this paper, we introduce the novel task of backtracing, which aims to retrieve the text segment that most likely provokes a query. this task addresses the information need of content creators who want to improve their content, in light of queries from information seekers. we introduce a benchmark that covers a variety of domains, such as the news article and lecture setting. we evaluate a series of methods including popular ir methods, likelihoodbased retrieval methods and gpt-3.5-turbo-16k. our results indicate that there is room for improvement across existing retrieval methods. these results suggest that backtracing is a challenging task that requires new retrieval approaches with better contextual understanding and reasoning about causal relevance. we hope our benchmark serves as a foundation for improving future retrieval systems for backtracing, and ultimately, spawns systems that empower content creators to understand user queries, refine their content and provide users with better experiences.