Decomposing Complex Queries for Tip-of-the-tongue Retrieval (2023)

Abstract: When re-finding items, users who forget or are uncertain about identifying details often rely on creative strategies for expressing their information needs—complex queries that describe content elements (e.g., book characters or events), information beyond the document text (e.g., descriptions of book covers), or personal context (e.g., when they read a book). Standard retrieval models that rely on lexical or semantic overlap between query and document text are challenged in such retrieval settings, known as tip-of-the-tongue (TOT) retrieval. We introduce a simple but effective framework for handling such complex queries by decomposing the query with an LLM into individual clues routing those as subqueries to specialized retrievers, and ensembling the results. Our approach takes advantage of off-the-shelf retrievers (e.g., CLIP for retrieving images of book covers) or incorporate retriever-specific logic (e.g., date constraints). We show that our framework incorporating query decomposition into retrievers can improve gold book recall up to 6% absolute gain for Recall@5 on a new collection of 14,441 real-world query-book pairs from an online community for resolving TOT inquiries.

## introduction tip of the tongue (tot) refers to the retrieval setting in which a user is unable to formulate a precise query that identifies a sought item, even if the user knows they've encountered this item before. for example, users searching for movies they watched or books they read long ago often resort to complex and creative queries that employ a diverse set of strategies to express information relevant to the sought item-high-level categories (e.g., topic, genre), content details from the movie or book (e.g., events, characters), references to personal context figure 1 : to resolve tip-of-the-tongue queries about books, our approach decomposes long, complex queries into subqueries routed to specific retrieval experts, each trained to handle a particular aspect of a book. a key challenge in tot retrieval is that queries are not just longer and more complex than those in popular retrieval datasets, but resolving them requires an enriched document collection since query-document relevance can't necessarily be established from document content alone (see in this work, we present a simple yet effective technique for improving tot retrieval: first, we decompose queries into individual subqueries or clues that each capture a single aspect of the target document. then, we route these subqueries to expert retrievers that can be trained individually. finally, we combine their results with those from a base retriever that receives the original query. experiments show improvement in gold book recall over description-only retrieval baselines on a set of 14,441 real-world query-book pairs collected from an online forum for resolving tot inquiries, complete with cover images and metadata. ## decomposing queries in this section, we describe our a simple but effective method for tackling long, complex tomt queries. given a collection of documents d 1 , . . . , d n and a textual query q, the tot retrieval task aims to identify the sought document d * . the input (raw) documents are semi-structured; each document d contains metadata fields d (1) , . . . , d (k) . in our the whatsthatbook dataset, the documents are books that have the fields that correspond to plot, its publication year, an image of its book cover, title, genre, and author etc. missing elements take on a default value (e.g., blank image, earliest publish date in overall book collection). ## datasets whatsthatbook we introduce the whatsthat-book dataset consisting of query-book pairs collected from a public online forum on goodreads for resolving tot inquiries about books. 4 on this forum, users post inquiries describing their sought book and community members reply with links to books on goodreads as proposed answers. 5 if the searcher accepts a book as the correct answer, the post is manually tagged as solved and a link to the found book is pinned to the thread. for these solved threads, we take the original inquiry as our query q and the linked book as gold d * . at the end, whatsthatbook contains 14,441 query-book pairs. each query corresponds to a unique book. finally, these books are associated with pages on goodreads, which we used to obtain publication year metadata and images of book covers. we further collect 172,422 negative books (books that do not correspond to a gold query in our dataset) to make the setting more realistic, for a total of 186,863 books. to collect negative books, crawl books authored by crawling books by the authors of the positive books. query-document clue-field clue-field clue-field query: i think i saw this in a used store once and i remember saying to my new husband "my daughter use to read that book to her little brother" and it's funny because on the outside cover is a little girl reading a book to her little brother. it's called.....my book, or my story, or something simple like that. it would be about 15 or more years old. the girl was blond and the boy brunet....i think!!!! inside was the cutest little sentences and my kids use to do what each page said...thank you!! extracted title: it's called.....my book, or my story, or something simple like that. extracted date clue: 15 years old (2006 or earlier) extracted cover: the outside cover is a little girl reading a book to her little brother. description: glossy pictorial hardcover no dust jacket. 2001 7.75x9.13x25. guide for par-ents with pictures, how to teaching children reading. actual title: my first book actual date first published september 1, 1984 actual cover: query: i read a book in 2008 or 2009 that was part of a series for young adults. it was fantasy, and about several families of witches and warlockes. the main character was named holly. extracted genre: for young adults, fantasy predicted plot: holly is a young witch who comes from a long line of magical families... predicted cover: young woman with long, curly hair holding a wand and surrounded by swirling colors and symbols description: holly cathers's world shatters when her parents are killed in a rafting accident. she is wrenched from her home in san francisco and sent to seattle to live with her aunt, marie-claire, and her twin cousins, amanda and nicole... actual genre: fantasy, young adult, paranormal, romance, fiction actual plot: (see description) actual cover: query: written in the 1990s, by [probably] one of those romance novelists (not nora roberts though). the cover was just a snowed-in cabin on a mountain side at night. in a nut shell -an escaped prisoner takes a woman captive and steals her car with her in it!...any ideas what this might be called? i haven't forgotten it after all these years. extracted author: by [probably] one of those romance novelists (not nora roberts though) predicted cover: dark, mysterious cabin obscured by falling snow with the silhouette... extracted cover: the cover was just a snowedin cabin on a mountain side at night. description: a rootless foster child, julie mathison had blossomed under the love showered upon her by her adoptive family. now a lovely and vivacious young woman... ## error analysis we sample 50 predictions where the top performing model where the model fails to get the book correct in the top 5 and categorize the types of errors made by the model in known-item and tot retrieval. tip of the tongue (tot) is a form of known-item retrieval @xcite , a long-studied area in the library and information sciences. yet, lack of large-scale public datasets has made development of retrieval methods for this task difficult. prior work on known-item retrieval focused on constructing synthetic datasets @xcite @xcite . for example, @xcite released a dataset of 2,755 query-item pairs from yahoo! answers and injected query inaccuracies via hired annotators to simulate the phenomenon of false memories @xcite , a common property of tot settings. the emergence of large, online communities for resolving tot queries has enabled the curation of realistic datasets. arguello et al. ( 2021 ) categorized the types of information referred to in tot queries from the website i remember this movie. 9 most recently, @xcite collected queries from the tip of the tongue community on reddit foot_9 and evaluated bm25 and dpr baselines. our work expands on their work in a key way: we introduce a new method for retrieval inspired by long, complex tot queries. in order to test our method on a large dataset of tot queries, we collected a new dataset of resolved tot queries such that we also had access to metadata and book cover images, which were not part of @xcite 's dataset. ## conclusion we study tip of the tongue retrieval, a real-world information-seeking setting in which users issue long, complex queries for re-finding items despite being unable to articulate identifying details about those items. we introduce whatsthatbook, a large challenging dataset of real-world tot queries for books. we also introduce a simple but effective approach to handling these complex queries that decomposes them into subqueries that are routed to expert retrievers for specialized scoring. our simple framework allows for modular composition of different retrievers and leveraging of pretrained models for specific modalities such as clip for document images. we experiment with different subquery generation strategies and find that generating predictions of document fields is more effective. we observe improvements up to 6% absolute gain over state-of-the-art dense retrievers for recall@5 when incorporating query decomposition into existing retrievers.