QUEST : A Retrieval Dataset of Entity-Seeking Queries with Implicit Set Operations (2023)

Abstract: Formulating selective information needs results in queries that implicitly specify set operations, such as intersection, union, and difference. For instance, one might search for “shorebirds that are not sandpipers” or “science-fiction films shot in England”. To study the ability of retrieval systems to meet such information needs, we construct QUEST, a dataset of 3357 natural language queries with implicit set operations, that map to a set of entities corresponding to Wikipedia documents. The dataset challenges models to match multiple constraints mentioned in queries with corresponding evidence in documents and correctly perform various set operations. The dataset is constructed semi-automatically using Wikipedia category names. Queries are automatically composed from individual categories, then paraphrased and further validated for naturalness and fluency by crowdworkers. Crowdworkers also assess the relevance of entities based on their documents and highlight attribution of query constraints to spans of document text. We analyze several modern retrieval systems, finding that they often struggle on such queries. Queries involving negation and conjunction are particularly challenging and systems are further challenged with combinations of these operations.

## introduction people often express their information needs with multiple preferences or constraints. queries corresponding to such needs typically implicitly express set operations such as intersection, difference, and union. for example, a movie-goer might be looking for a science-fiction film from the 90s which does not feature aliens and a reader might be interested in a historical fiction novel set in france. similarly, first, (1) we sample wikipedia category names and find their corresponding set of relevant entities. (2) then, we compose a query with set operations and have this query paraphrased by crowdworkers. (3) these queries are then validated for fluency and naturalness. (4) finally, crowdworkers mark the entities' relevance by highlighting attributable spans in their documents. a botanist attempting to identify a species based on their recollection might search for shrubs that are evergreen and found in panama. further, if the set of entities that satisfy the constraints is relatively small, a reader may like to see and explore an exhaustive list of these entities. in addition, to verify and trust a system's recommendations, users benefit from being shown evidence from trusted sources @xcite . addressing such queries has been primarily studied in the context of question answering with structured knowledge bases (kbs), where query constraints are grounded to predefined predicates and symbolically executed. however, kbs can be incomplete and expensive to curate and maintain. meanwhile, advances in information retrieval may enable developing systems that can address such queries without relying on structured kbs, by matching query constraints directly to supporting evidence in text documents. however, queries that combine multiple constraints with implicit set operations are not well represented in existing retrieval benchmarks such as msmarco @xcite and natural questions @xcite . also, such datasets do not focus on retrieving an exhaustive document set, instead limiting annotation to the top few results of a baseline information retrieval system. to analyze retrieval system performance on such queries, we present quest, a dataset with natural language queries from four domains, that are mapped to relatively comprehensive sets of entities corresponding to wikipedia pages. we use categories and their mapping to entities in wikipedia as a building block for our dataset construction approach, but do not allow access to this semistructured data source at inference time, to simulate text-based retrieval. wikipedia categories represent a broad set of natural language descriptions of entity properties and often correspond to selective information need queries that could be plausibly issued by a search engine user. the relationship between property names and document text is often subtle and requires sophisticated reasoning to determine, representing the natural language inference challenge inherent in the task. our dataset construction process is outlined in figure 1 . the base queries are semi-automatically generated using wikipedia category names. to construct complex queries, we sample category names and compose them by using pre-defined templates (for example, a ∩ b \ c). next, we ask crowdworkers to paraphrase these automatically generated queries, while ensuring that the paraphrased queries are fluent and clearly describe what a user could be looking for. these are then validated for naturalness and fluency by a different set of crowdworkers, and filtered according to those criteria. finally, for a large subset of the data, we collect scalar relevance labels based on the entity documents and fine-grained textual attributions mapping query constraints to spans of document text. such annotation could aid the development of systems that can make precise inferences from trusted sources. performing well on this dataset requires systems that can match query constraints with corresponding evidence in documents and handle set operations implicitly specified by the query (see query: supernatural horror films set in nevada entity: willy's wonderland document text: willy's wonderland is a 2021 american action comedy horror film directed by kevin lewis from a screenplay by g. o. parsons. the film stars nicolas cage, who also [...] it follows a quiet drifter who is tricked into cleaning up an abandoned family entertainment center haunted by eight murderous animatronic characters. the project was announced in october 2019, [...] when his car catches a flat tire on a remote country road, a quiet drifter ends up stranded outside of hayesville, nevada. mechanic jed love picks [...] responding evidence in documents and handle set 112 operations implicitly specified by the query (see 113 figure 2), while also efficiently scaling to large 114 collections of entities. we evaluate several re-115 trieval systems by finetuning pretrained models on 116 our dataset. systems are trained to retrieve multi-117 answer sets given a query. we find that current dual 118 encoder and cross-attention models up to the size 119 of t5-large (raffel et al., 2020) are largely not 120 effective at performing retrieval for queries with 121 set operations. queries with conjunctions and nega-122 tions prove to be especially challenging for models 123 and systems are further challenged with combina-124 tions of set operations. our error analysis reveals 125 that non-relevant false positive entities are often 126 caused by the model ignoring negated constraints, 127 or ignoring the conjunctive constraints in a query. @xcite and natural questions @xcite . also, such datasets do not focus on retrieving an exhaustive document set, instead limiting annotation to the top few results of a baseline information retrieval system. to analyze how well retrieval systems handle such queries, we present quest, a dataset with natural language queries from four domains, that are mapped to relatively comprehensive sets of entities corresponding to wikipedia pages. we use wikipedia categories and their mapping to entities in wikipedia as a building block for our dataset construction approach, but do not allow access to this semi-structured data source at inference time, to simulate text-based retrieval. wikipedia categories represent a broad set of natural language descriptions of entity properties and often correspond to selective information need queries that could be plausibly issued by a search engine user ([at least 90% of the time based on our filtering?]). the correspondence between property names and document text is also often subtle and requires sophisticated reasoning to determine relevance, representing the natural language inference challenge inherent in the task, while the knowledge of category membership allows us to construct relatively comprehensive sets of candidate entities for atomic categories and their combinations. our dataset construction process is outlined in responding evidence in documents and handle set 112 operations implicitly specified by the query (see 113 figure 2), while also efficiently scaling to large 114 collections of entities. we evaluate several re-115 trieval systems by finetuning pretrained models on 116 our dataset. systems are trained to retrieve multi-117 answer sets given a query. we find that current dual 118 encoder and cross-attention models up to the size 119 of t5-large (raffel et al., 2020) are largely not 120 effective at performing retrieval for queries with 121 set operations. queries with conjunctions and nega-122 tions prove to be especially challenging for models 123 and systems are further challenged with combina-124 tions of set operations. our error analysis reveals 125 that non-relevant false positive entities are often 126 caused by the model ignoring negated constraints, 127 or ignoring the conjunctive constraints in a query. figure 2 ), while also efficiently scaling to large collections of entities. we evaluate several retrieval systems by finetuning pretrained models on our dataset. systems are trained to retrieve multidocument sets given a query. we find that current dual encoder and cross-attention models up to the size of t5-large @xcite are largely not effective at performing retrieval for queries with set operations. queries with conjunctions and negations prove to be especially challenging for models and systems are further challenged with combinations of set operations. our error analysis reveals that non-relevant false positive entities are often caused by the model ignoring negated constraints, or ignoring the conjunctive constraints in a query. ## related work previous work in question answering and information retrieval has focused on qa over knowledge bases as well as open-domain qa and retrieval over a set of entities or documents. we highlight how these relate to our work below. knowledge base qa several datasets have been proposed for question answering over knowledge bases @xcite @xcite @xcite . these benchmarks require retrieval of a set of entities that exist as nodes or relations in an accompanying knowledge base. questions are optionally supplemented with logical forms. @xcite provide a comprehensive survey of complex kbqa datasets. previous work has simultaneously noted that large curated kbs are incomplete @xcite . notably, kbqa systems operate over a constrained answer schema, which limits the types of queries they can handle. further, these schema are expensive to construct and maintain. for this reason, our work focuses on a setting where we do not assume access to a kb. we note that kbqa datasets have also been adapted to settings where a kb is incomplete or unavailable @xcite . this was done by either removing some subset of the data from the kb or ignoring the kb entirely. a key difference from these datasets is also that we do not focus on multihop reasoning over multiple documents. instead, the relevance of an entity can be determined solely based on its document. open-domain qa and retrieval many opendomain qa benchmarks, which consider qa over unstructured text corpora, have been proposed in prior work. some of these, such as trec @xcite , msmarco @xcite and natural questions @xcite are constructed using "found data", using real user queries on search engines. @xcite present a benchmark where they consider many such existing datasets. datasets such as hot-potqa @xcite , and multirc @xcite have focused on multi-hop question answering. other work has explored e-commerce datasets (for example, @xcite ), but these have not been released publicly. notably, the focus of these datasets differs from ours as we focus on queries that contain implicit set operations over exhaustive answer sets. such queries are not well represented in existing datasets because they occur in the tail of the query distributions considered. multi-answer retrieval related work @xcite also studies the problem of multi-answer retrieval, where systems are required to predict multiple distinct answers for a query. @xcite adapt existing datasets (for example, webquestionssp @xcite ) to study this setting and propose a new metric, mrecall@k, to evaluate exhaustive recall of multiple answers. we also consider the problem of multi-answer set retrieval, but consider queries that implicitly contain set constraints. in concurrent work, romqa (zhong et al., 2022) proposes an open-domain qa dataset, focusing on combinations of constraints extracted from wikidata. romqa shares our motivation to enable answering queries with multiple constraints, which have possibly large answer sets. to make attribution to evidence feasible without human annotation, romqa focuses on questions whose component constraints can be verified from single entity-linked sentences from wikipedia abstracts, annotated with relations automatically through distant supervision, with high precision but possibly low recall (t-rex corpus). in quest, we broaden the scope of queryevidence matching operations by allowing for attribution through more global, document-level inference. to make human annotation for attribution feasible, we limit the answer set size and the evidence for an answer to a single document. ## dataset generation quest consists of 3357 queries paired with up to 20 corresponding entities. each entity has an associated document derived from its wikipedia page. the dataset is divided into 1307 queries for training, 323 for validation, and 1727 for testing. the task for a system is to return the correct set of entities for a given query. additionally, as the collection contains 325,505 entities, the task requires retrieval systems that can scale efficiently. we do not allow systems to access additional information outside of the text descriptions of entities at inference time. category labels are omitted from all entity documents. ## experimental setup we evaluate modern retrieval systems to establish baseline performances. we also perform extensive error analysis to understand patterns of model errors and the quality of the labels in quest. ## results and analysis we report the performance of our baseline systems on the test set in dual encoders outperform bm25. as shown in ## conclusion we present quest, a new benchmark of queries which contain implicit set operations with corresponding sets of relevant entity documents. our experiments indicate that such queries present a challenge for modern retrieval systems. future work could consider approaches that have better inductive biases for handling set operations in natural language expressions (for example, @xcite ). the attributions in quest can be leveraged for building systems that can provide finegrained attributions at inference time. the potential of pretrained generative lms and multi-evidence aggregation methods to answer set-seeking selective queries, while providing attribution to sources, can also be investigated.