Numbers Matter! Bringing Quantity-awareness to Retrieval Systems (2024)

Abstract: Quantitative information plays a crucial role in understanding and interpreting the content of documents. Many user queries contain quantities and cannot be resolved without understanding their semantics, e.g., “car that costs less than $10k”. Yet, modern search engines apply the same ranking mechanisms for both words and quantities, overlooking magnitude and unit information. In this paper, we introduce two quantity-aware ranking techniques designed to rank both the quantity and textual content either jointly or independently. These techniques incorporate quantity information in available retrieval systems and can address queries with numerical conditions equal, greater than, and less than. To evaluate the effectiveness of our proposed models, we introduce two novel quantity-aware benchmark datasets in the domains of finance and medicine and compare our method against various lexical and neural models. The code and data are available under https://github.com/satya77/QuantityAwareRankers .

## introduction despite advances in semantic search and sophisticated neural network architectures, handling quantitative information in text remains challenging. specifically hard are quantity-centric queries in which the query contains a quantity and a numerical condition, e.g., "bmw with more than 530hp". the reason for this is that information retrieval (ir) systems are not aware of numbers and their semantics, such as proximity, in particular in combination with units. numbers and units are treated in the same way as any other text token that is subject to subsequent processing, e.g., indexing or embedding. what complicates treating numbers and units in a proper way is that these objects can also have different surface forms (e.g., 6k vs 6,000 and mph vs miles per hour) and require standardization @xcite . while there are approaches that specifically focus on numbers in text, e.g., extracting quantities for entities @xcite , linking quantities in tables @xcite , or numerical reasoning @xcite , they are tailored to specific tasks and not semantic search in general. this applies to neural models supporting ir, which are trained on general-purpose data without the focus on the quantity semantics. language models (lm), forming the basis for neural models, exhibit a limited understanding of number scales and proximity @xcite . despite recent work on numerical language models @xcite , these architectures are very specific and require changes in the architecture of popularly used language models in ir, which indicates an expensive pre-training. moreover, the lack of accessible quantity-centric benchmarks for training or comparing systems exacerbates the issue. in this paper, we present two strategies to enhance the quantity understanding of current ir systems. we aim for a general-purpose model that is not specific to quantity ranking but is also capable of textual ranking. the two approaches differ in their integration of quantity ranking with textual ranking. the first employs a disjoint combination, while the second focuses on the joint ranking of quantities within the context of textual content. the disjoint approach is based on a complete separation of quantities from other textual tokens, from tokenization to the definition of relevance. it is an unsupervised and heuristic method, utilizing an index structure, compatible with various lexical and semantic ir models. on the other hand, in joint ranking, we aim to learn quantity-aware document and query representations through task-specific fine-tuning of neural ir models, without specific architectural changes. further, we introduce two novel quantitycentric benchmarks, focusing on queries involving numerical conditions in the domains finance and medicine. we evaluate our proposed approaches with various lexical and neural models and show significant improvements over the baselines. ## related work related work for quantity-centric search is limited. @xcite focus on quantity search for named entities, using a deep neural network for extracting quantity-centric tuples from text and query and matching based on context similarity. their pipeline involves semantic role labeling and named entity extraction, both resource-intensive and reliant on sparsely available annotated data for quantities. further, focusing on named entities limits the applicability to real-world scenarios. qfinder @xcite integrates numerical and lexical indexes to enhance numerical understanding in a lexical ir system. our disjoint model utilizes qfinder's heuristic ranking function, but we extend the qfinder ranker to include neural models and go beyond the limited query language, allowing users to provide queries in plain text. mqsearch @xcite extracts quantities with a set of regular expressions to create a rule-based system for finding documents containing certain keywords and ranges of values. loosely related to ir, @xcite and @xcite perform numerical summarization on unstructured text in the form of plots and graphs. in the area of databases, there has been some work focusing on building numerical indices for queries that contain numerical restrictions @xcite @xcite . however, the main focus of such systems is the efficiency of the index structure and filtering out irrelevant numbers from the results with hard constraints rather than ranking. unlike quantity-aware ir, investigating numeracy in lms is well-established. @xcite are among the first to highlight the limitations of embedding models when handling numbers. subsequent efforts have led to dedicated embeddings and lms for understanding scales, basic arithmetic, and numerical common sense knowledge @xcite @xcite @xcite @xcite . these lms are specific to numeracy and not ir in general. while using them can enhance performance, we focus on improving quantity understanding in current ir models without architectural changes or training an lm from scratch. ## quantity-aware model a quantity-centric query contains a numerical condition, a value, and a unit, e.g., "iphone xs with price under $1500 ". queries like "what is the price of iphone xs?" are not considered quantitycentric as they do not require an understanding of scales and units. in the following, we assume a document collection where each document consists of a sequence of sentences. following previous work @xcite , we focus on sentences as retrieval units. a sentence s i := (t i , q i ) is a sequence of tokens t @xmath0, ..., t l ) and quantities ## evaluation given the absence of task-specific models, we assess our quantity-aware models against general domain lexical and neural models. lexical models include a bm25 and a bm25 f ilter variant. bm25 f ilter has a separate numerical index to eliminate the results of bm25 where the query condition is not met. this method resembles numerical indices from databases as it works by filtering rather than by ranking. neural models include the trained checkpoints of splade and colbert as well as cohere v3 foot_1 . cohere v3 shows that even industry-level models trained on extensive data lack quantity awareness. ## conclusion and ongoing work in this work, we (1) examined the shortcomings of prominent ir models concerning quantity-centric queries, (2) proposed two methods, joint and disjoint quantity-aware models, to integrate quantity understanding into both classical ir models as well recent neural architectures, (3) introduced two novel benchmark datasets containing quantitycentric queries, which to our knowledge are the first of their kind, and (4) demonstrated significant improvements in quantity understanding over the baselines for both proposed techniques. our joint and disjoint approaches enable the integration of quantity understanding without altering existing architectures and with minimal overhead in query latency. we further highlight the strengths and weaknesses of both approaches, arguing that the choice of method should depend on the specific use case scenario. the disjoint approach, which relies on a quantity index for ranking, consistently outperforms joint models across various domains. its unsupervised and heuristic nature also makes it more flexible than the joint rankers. however, despite the lower performance, the joint approach eliminates the need for an external index and the associated latency increase. due to the lack of existing quantity-aware ir models, most of our baselines are general-purpose, but we hope that our systems can serve as baseline for future work in this direction and that our benchmarks encourage the researchers to work on this task. in the future, we plan to explore the impact of dedicated numerical embeddings and lms in information retrieval.