Fréchet Distance for Offline Evaluation of Information Retrieval Systems with Sparse Labels (2024)

Abstract: The rapid advancement of natural language processing, information retrieval (IR), computer vision, and other technologies has presented significant challenges in evaluating the performance of these systems. One of the main challenges is the scarcity of human-labeled data, which hinders the fair and accurate assessment of these systems. In this work, we specifically focus on evaluating IR systems with sparse labels, borrowing from recent research on evaluating computer vision tasks.taking inspiration from the success of using Fréchet Inception Distance (FID) in assessing text-to-image generation systems. We propose leveraging the Fréchet Distance to measure the distance between the distributions of relevant judged items and retrieved results. Our experimental results on MS MARCO V1 dataset and TREC Deep Learning Tracks query sets demonstrate the effectiveness of the Fréchet Distance as a metric for evaluating IR systems, particularly in settings where a few labels are available.This approach contributes to the advancement of evaluation methodologies in real-world scenarios such as the assessment of generative IR systems.

## introduction with the rapid advancement of technologies in fields such as natural language processing, natural language generation, computer vision, and information retrieval (ir), evaluating the performance of these systems is becoming increasingly challenging @xcite @xcite . we must develop new metrics, benchmarks, and evaluation protocols that are specifically tailored to the unique characteristics of the systems considering the rapid changes in system architecture, training data, and model configurations @xcite . in many cases, obtaining high-quality labeled data that accurately represents the complexity of real-world scenarios can be expensive, time-consuming, or even impractical. this scarcity of labeled data adds to the limitations of conducting extensive evaluations and may lead to biased or incomplete assessments @xcite . offline evaluation poses a significant challenge due to the sparsity of labeled data @xcite @xcite . this challenge is particularly prominent in datasets like ms marco, a widely used benchmark for ad hoc retrieval reserach @xcite @xcite @xcite in which, the majority of queries are annotated with only one relevant judged document. however, to suit the dataset for effective traininig of deep learning models, a high number of queries are judged, resulting in sparse labels. consequently, most queries have only one relevant judgment, while the relevance of the remaining documents remains unknown. other researchers have shown that there are potentially relevant documents that are as good as, or even better than, the judged queries @xcite . given the sparsity of ground truth labels, it is crucial to recognize the challenges involved in distinguishing between rankers when the differences in performance are small @xcite . the limited labeled data for retrieved documents introduces noise, making it challenging to definitively determine which ranker is performing better @xcite . the incomplete judgments can introduce problems in evaluations, as they do not capture the full range of relevant documents @xcite . this issue becomes even more pronounced in generative-based tasks. it is impractical to reassess the generated results, such as images or text, with each system run due to their non-deterministic nature @xcite . evaluating a generative system's performance based on the similarity of generated content to sparsely labeled data remains one of the most effective approaches in many generativebased nlp and computer vision benchmarks and tasks @xcite @xcite @xcite . particularly in the evaluation of text-to-image generation task, the fréchet inception distance (fid), has gained recognition for showing high robustness and correlation with human judgements @xcite @xcite . fid compares the distribution of generated images across a set of prompts to the distribution of target images across the same set of prompts. to compute fid, features of ground truth images and generated images are extracted from both sets, and multivariate gaussian distributions are fitted to these features. the fréchet distance (fd), which quantifies the similarity between two probability distributions, is then computed based on the fitted gaussian distributions. a lower fid score indicates a higher similarity between the distributions, indicating that the generated images closely match the real images in terms of their visual features. in this paper, we shed light on how evaluating generated results is similar to assessing the quality of retrieved results with sparse labels in an ad hoc retrieval setting. most benchmarks for both tasks have quite sparse labels i.e., not all the items are judged and while there are a few annotations available for some of the candidates, there can be other unjudged relevant items available. while labelling more data is expensive for both tasks, there could be more than one correct answer in both tasks. in this work, we mimic an information retrieval system with sparse relevance judgements as a generation task where the ground truth targets are sparse. due to the success of fid in evaluating the quality of generated images, especially for generative adversarial networks @xcite @xcite @xcite , we explore if we can quantify the quality of retrieved documents in an ad hoc retrieval system through fréchet distance. in the context of ir evaluation, we can analogously consider the relevant judged items as the ground truth set and the retrieved items as the set of generated items. our objective is to extract features from both sets, the relevant judged items and the retrieved results, and investigate whether metrics such as the fréchet distance can effectively capture the quality of the retrieved results with respect to the ground truth labels in ir systems. we study the following research questions: we conduct our experiments by assessing different retrieval pipelines on the ms marco v1 dev dataset, which has extremely sparse labels, as well as the trec deep learning track 2019 and 2020 datasets, which have more complete labels @xcite @xcite . our study demonstrates the effectiveness of the fréchet distance as a metric for quantifying the performance of ir systems especially when the ground truth labels are sparse. ## experimental setup in this section, we describe the general settings of our experiments including datasets, the traditional ir metrics, retrieval methods and the pre-trained language models we used to embed the documents. ## assessment with sparse labels we are interested in investigating how fd can assess the performance of different retrievers when there are only sparse labels available i.e., on 6980 queries from ms marco small dev set. we present the performance of the 12 retrieval methods, including the sparse to dense retrievers, sparse retrievers with learned representations, and hybrid retrievers that were introduced in section 3.2 in terms of mrr@10 as well as measuring the fréchet distance between two sets of retrieved items and relevant judged items on the cut-offs of 1 and 10 in to examine the robustness of the fd in the context of ir assessment, and to evaluate the generalizability of the method across different subsets of queries, we employ a bootstrap sampling @xcite from the msmarco dev set for @xmath0. this would allows us to investigate whether the results obtained in the previous section were influenced by the data or if they can be reliable. the results are visualized in figure 1 , in which we present the mean and empirical 0.95% confidence interval for each retriever across the 1000 query sets in terms of mrr@10 and fd@10. it is important to note that for the mrr plot, a higher position on the plot indicates better performance, while for the fd plot, a lower position indicates better performance. the findings confirm that despite considering different sample sets, we observe a consistent pattern and similarity in the performance trends. ## assessing with comprehensive labels in this section, we investigate the performance of the fréchet distance in evaluating ir systems when the labels are not sparse and we have more complete labels. we conduct experiments using the runs submitted to trec dl 2019 (37 runs) and trec dl 2020 (59 runs). unlike the ms marco dev set which on average each query has 1.06 judged documents, the queries in trec dl tracks on average have over 210 judged documents per query assessed with four different levels of relevance including "not relevant", "related", "highly relevant", and "perfectly relevant" @xcite . we notice that the number of judged relevant items per query in these benchmarks varies a lot. due to the trec-style judgment criteria, only the top few retrieved items from all submitted runs were judged. depending on the overlap between the top retrieved items from different runs, the number of relevant judged items per query may vary. when applying fd with an imbalanced number of relevant judged items per query, it can introduce biases in the ground truth distribution and potentially lead to problems in evaluation. to address this issue, we balanced the number of relevant judged items per query by limiting them to a maximum of 1, 5, and 10 relevant judged items per query i.e., we randomly select k relevant items from the pool of relevant judged documents for the query of interest. we first randomly select from the most relevant level i.e., level 3 which are perfectly relevant documents and then when there is not a sufficient number of perfectly relevant documents, we move on to highly relevant level and randomly choose from that grade. this experiment also allows us to examine how the sparsification of judgments affects the performance of evaluation metrics. we note that these modifications in relevance judgements are only applied for measuring fd and ndcg@10 is measured with all the judged documents without any modification. we plotted the ndcg@10 on the x-axis and the fd with balanced and sparsified judgments on the y-axis of each sub-figure in figure 2 , for all the runs submitted to trec dl19 (first row) and trec dl20 (second row). consistent with our previous experiments, we observe a highly linear relationship between the two metrics. we also provide the kendall τ correlation under each sub-figure. for instance, when sparsifying the labels and considering only one relevant judged item per query, we obtain a kendall τ correlation of -0.836 for trec dl2019 and -0.867 for trec dl2020, between ndcg@10 and fd@10 of each dataset. the experiments on the trec dl datasets highlight two key points. first, unlike using the fréchet inception distance to evaluate the quality of generated images in text-to-image generation tasks, where a large number of data points (in the order of thousands) are required for the evaluation to be valid, we demonstrated that even with a smaller number of queries (around 40-50), fd is capable of distinguishing the performance of different rankers @xcite . second, fd is not sensitive to the sparsity of the ground truth labels and it performs well with both sparse and more complete labels. it is not affected by the number of judgments, as evidenced by the fact that the performance did not differ greatly when increasing the number of relevant judged items. however, for trec dl2019, we observed a small drop in correlation by increasing the number of relevant judgments. further exploration revealed that a higher number of relevant judgments in trec 2019 resulted in a higher usage of level 2 relevance judgments (highlight relevant) instead of level 3 judgments (perfectly relevant). consequently, we suggest that fd may be more sensitive to the quality of relevant judged items rather than the quantity. overall, in response to rq2, we find that fd works well when using comprehensive labels, and consistent with the findings in section 4, sparsifying the labels does not compromise the quality of assessment. ## assessing unlabeled retrieved results here, we undertake an evaluation of different ir systems under an extremely challenging case of assessing unlabeled retrieved results. this scenario presents a situation where each query is assumed to have mostly only one relevant item, and the relevant judged items are not included in the top-k results. our objective is to investigate the effectiveness of the fréchet distance in assessing the top-k unlabeled retrieved results (urr) when no judgments are available for any of the top-k retrieved items. this is particularly valuable considering the high cost and limited availability of labeled data, which often exhibit sparsity. previous research has demonstrated that as rankers improve in performance, they tend to retrieve previously unseen content that may be highly relevant to the original query @xcite . if fréchet distance is capable of evaluating the retrieved results in such cases, it would be a valuable tool for assessing the relevance of unlabeled data and even beyond that, for evaluating generative-based responses. we measure the fd between one set consisting of the relevant judged items per query and the other set consisting of the top-k unjudged retrieved item for each query. in other words, we scan down the ranked list and retain the first k unjudged document to assess. this is an interesting aspect to study because traditional ir metrics such as mrr, ndcg, and map rely on the presence of relevant items in the retrieved list and would assign a performance score of zero in cases where no relevant items are retrieved. they do not account for unjudged documents. we argue that by utilizing the fd metric, we can capture the similarity between unjudged retrieved items and the limited set of judged examples and measure the performance of the retriever based on this value. the results of this experiment are reported in ## conclusion and future work in this paper, we leverage fréchet distance to address the challenges of evaluating ir systems with sparse labels. we measure the similarities between the embedded representation of retrieved results as well as the limited available relevant judged documents using fréchet distance. through experiments conducted on datasets with sparse and more complete ground truth labels, including the ms marco dev dataset and the trec deep learning track datasets , we demonstrated the effectiveness of the fréchet distance in evaluating ir systems. our findings suggest that the fréchet distance has significant implications for evaluating ir systems in real-world settings where obtaining comprehensive ground truth labels can be challenging and expensive. we believe that future research could utilize the fréchet distance to evaluate different generative models, expanding the scope of evaluation in ir systems. as such, it allows for having the generated results compared with the retrieved results in the same playground.