Moving Beyond Downstream Task Accuracy for Information Retrieval Benchmarking (2023)

Abstract: Neural information retrieval (IR) systems have progressed rapidly in recent years, in large part due to the release of publicly available benchmarking tasks. Unfortunately, some dimensions of this progress are illusory: the majority of the popular IR benchmarks today focus exclusively on downstream task accuracy and thus conceal the costs incurred by systems that trade away efficiency for quality. Latency, hardware cost, and other efficiency considerations are paramount to the deployment of IR systems in user-facing settings. We propose that IR benchmarks structure their evaluation methodology to include not only metrics of accuracy, but also efficiency considerations such as a query latency and the corresponding cost budget for a reproducible hardware setting. For the popular IR benchmarks MS MARCO and XOR-TyDi, we show how the best choice of IR system varies according to how these efficiency considerations are chosen and weighed. We hope that future benchmarks will adopt these guidelines toward more holistic IR evaluation.

## introduction benchmark datasets have helped to drive rapid progress in neural information retrieval (ir). when the ms marco @xcite passage ranking leaderboard began in 2018, the best performing systems had mrr@10 scores around 0.20; the latest entries have since increased accuracy past 0.44. similarly, the xor tydi multilingual question answering (qa) dataset @xcite was released in 2021 and has seen improvements in recall scores from 0.45 to well past 0.70. the leaderboards for these datasets are defined by a particular set of accuracy-based metrics, and progress on these metrics can easily become synonymous in people's minds with progress in genthe trade-offs evident here are common in real-world applications of ir technologies. these submissions do not represent "optimal" implementations of each respective approach, but rather reflect existing reported implementations and hardware configurations in the literature. including cost and other efficiency considerations on our leaderboards would lead to more thorough exploration of possible system designs and, in turn, to more meaningful progress. for hardware specifications, we show the precise resources given as the running environment in the paper, even if not all resources were available to the model or the resources were over-provisioned for the particular task. in this work, we contribute to the growing literature advocating for multidimensional leaderboards that can inform different values and goals @xcite @xcite @xcite . our proposal is that researchers should report orthogonal dimensions of performance such as query latency and overall cost, in addition to accuracy-based metrics. our argument has two main parts. in part 1 ( §2), we create a post-hoc ms marco leaderboard from published papers ( in part 2 ( §3), we systematically explore four prominent systems: bm25, dense passage retriever (dpr; @xcite ), bt-splade-l @xcite , and plaid colbertv2 @xcite . these experiments begin to provide a fuller picture of the overall performance of these systems. we close by discussing practical considerations relating to the multidimensional leaderboards that the field requires. here, we argue that the dynascore metric developed by @xcite is a promising basis for leaderboards that aim to (1) measure systems along multiple dimensions and (2) provide a single full ranking of systems. dynascores allow the leaderboard creator to weight different assessment dimensions (e.g., to make cost more important than latency). these weightings transparently reflect a particular set of values, and ## a post-hoc leaderboard while existing ir benchmarks facilitate progress on accuracy metrics, the lack of a unified methodology for measuring latency, memory usage, and hardware cost makes it challenging to understand the trade-offs between systems. to illustrate this challenge, we constructed a post-hoc leaderboard for the ms marco passage ranking benchmark ( ## experiments with representative retrievers as more sophisticated versions may achieve better accuracy @xcite , though often with trade-offs along other dimensions @xcite . for evidence that simple bm25 models often perform best in their class, see @xcite . dpr @xcite ) a dense singlevector neural ir model. dpr separately encodes queries and documents into vectors and scores them using fast dot-product-based comparisons. bt-splade-l (lassance and clinchant, 2022) splade @xcite is a sparse neural model. the bt-splade variants are highly optimized versions of this model designed to achieve low latency and reduce the overall computational demands of the original model. to the best of our knowledge, only the large configuration, bt-splade-l, is publicly available. plaid colbertv2 @xcite the colbert retrieval model @xcite encodes queries and documents into sequences of output states, one per input token, and scoring is done based on the maximum similarity values obtained for each query token. col-bertv2 @xcite improves supervision and reduces the space footprint of the index, and the plaid engine focuses on achieving low latency. the parameter k to the model dictates the initial candidate passages that are scored by the model. larger k thus leads to higher latency but generally more accurate search. in our initial experiments, we noticed that higher k led to better out-of-domain performance, and thus we evaluated the recommended settings from @xcite , namely, @xmath0, 100, 1000}. to distinguish these configurations from the number of passages evaluated by the mrr or success metric (also referred to as k), we refer to these configurations as the '-s', '-m', and '-l' variants of colbertv2, respectively. we chose these models as representatives of key ir model archetypes: lexical models (bm25), dense single-vector models (dpr), sparse neural models (splade), and late-interaction models (colbert). the three colbert variants provide a glimpse of how model configuration choices can interact with our metrics. we use two retrieval datasets: ms marco @xcite and xor-tydi @xcite . all neural models in our analysis are trained on ms marco data. we evaluate on xor-tydi without further fine-tuning to test out-of-domain evaluation (see appendix a for more details). our goal is to understand how the relative performance of these models changes depending on the available resources and evaluation criteria. our approach differs from the post-hoc leaderboard detailed in §2 in two key ways: (1) we fix the underlying hardware platform across all models, and (2) we evaluate each model across a broad range of hardware configurations (aws instance types), ensuring that we capture an extensive space of compute environments. furthermore, in addition to quality, we also report the average per-query latency and the corresponding cost of running 1 million queries given the latency and the choice of instance type. this approach therefore enables a more principled and holistic comparison between the models. we use the open-source primeqa framework, foot_3 which provides a uniform interface to implementations of bm25, dpr, and plaid colbertv2. for splade, we use the open-source implementation maintained by the paper authors. 4 for each model we retrieve the top 10 most relevant passages. we report the average latency of running a fixed sample of 1000 queries from each dataset as measured across 5 trials. see appendix a for more details about the evaluation environments and model configurations. overall, bm25 is the least expensive model when selecting the minimum viable instance type: only bm25 is able to run with 4 gb memory. however, its accuracy scores are low enough to essentially remove it from contention. on both datasets, we find that bt-splade-l and the plaid colbertv2 variants are the most accurate models by considerable margins. on ms marco, all the colbertv2 variants outperform bt-splade-l in mrr@10 and success@10 respectively, while bt-splade-l offers faster and cheaper scenarios than colbertv2 for applications that permit a moderate loss in quality. in the out-of-domain xor-tydi evaluation, bt-splade-l outperforms the colbertv2-s variant, which sets @xmath1. we hypothesize this loss in quality is an artifact of the approximations employed by the default configuration. hence, we also test the more computationally-intensive configurations mentioned above: colbertv2-m (@xmath2. these tests reveals that colbertv2-l solidly outperforms bt-splade-l in mrr@10 and success@10, while allowing bt-splade-l to expand its edge in latency and cost. interestingly, despite per-instance costs being higher for certain instances, selecting the more expensive instance can actually reduce cost depending on the model. for example, the c7g.4xlarge instance is 3.5× more expensive than x2gd.large, but colbertv2-s runs 4× faster with 16 cpu threads and therefore is cheaper to execute on the c7g.4xlarge. these findings further reveal the rich space of trade-offs when it comes to model configurations, efficiency, and accuracy. ## discussion and recommendations in this section, we highlight several considerations for future ir leaderboards and offer recommendations for key design decisions. ## conclusion we argued that current benchmarks for information retrieval should adopt multidimensional leaderboards that assess systems based on latency and cost as well as standard accuracy-style metrics. such leaderboards would likely have the effect of spurring innovation, and lead to more thorough experimentation and more detailed reporting of results in the literature. as a proof of concept, we conducted experiments with four representative ir systems, measuring latency, cost, and accuracy, and showed that this reveals important differences between these systems that are hidden if only accuracy is reported. finally, we tentatively proposed dynascoring as a simple, flexible method for creating multidimensional leaderboards in this space.