Q 2 R : A Query-to-Resolution System for Natural-Language Queries (2022)

Abstract: We present a system for document retrieval that combines direct classification with standard content-based retrieval approaches to significantly improve the relevance of the retrieved documents. Our system exploits the availability of an imperfect but sizable amount of labeled data from past queries. For domains such as technical support, the proposed approach enhances the system’s ability to retrieve documents that are otherwise ranked very low based on content alone. The system is easy to implement and can make use of existing text ranking methods, augmenting them through the novel Q2R orchestration framework. Q2R has been extensively tested and is in use at IBM.

## introduction a document retrieval system typically solves a text ranking problem defined as follows: given a query x, a relevance score s(x, y) is computed for each document y in the target collection d. thus, the text ranking problem can be equivalently cast as a relevance-based binary classification problem @xcite , where for each (x, y) pair, the label is either "relevant" or "not-relevant". a learned probabilistic model can be used to provide the score where s(x, y) ∝ pr(relevant|x, y). typically, computing the relevance score s(x, y) involves using the content of each document y. for example, the keyword-based approach bm25 @xcite employs sparse bag-of-words representations of the query x and the content y, f q (x) and f d (y), and then s(x, y) is given by the inner product f q (x), f d (y) . modern deep learning approaches learn a parametric classifier s θ (x, y) that takes as input the concatenated content of x and y. such approaches may be computationally costly since s θ (x, y) needs to be evaluated for every @xmath0. a two-stage approach is typically employed where a small d ⊂ d is first retrieved through a fast keyword-based method, then re-ranked with s θ (x, y). an alternative to this approach is to learn a dense-representation (e.g. @xcite ) for both f q and f d and compute s(x, y) = ξ(f q (x), f d (y)) where ξ is easy to compute (e.g. the dot product) and f d (y) can be pre-computed for every @xmath1. for simple ξ, the top-scoring documents can be easily retrieved via approximate nearest-neighbors (ann) techniques. the latter is appealing for real-world applications due to its computational advantage. while content-based methods have proven effective for general-purpose document ranking and are in widespread use, there are circumstances where using the content of the target documents is less effective. a primary example is when the document content is technical and queries are structurally and linguistically different. consider the medical domain where documents concern medical treatments. queries may describe symptoms experienced, which need not be included in a database of treatments. the availability of labeled examples that map symptoms to treatment plans motivates mapping queries from the labeled pairs to the best treatment documents. the same occurs in other domains such as information technology (it), law, etc. for these technical domains, where such curated historical data often exists, we propose an approach based on direct classification that relies on learning a classifier from the queries themselves to the identity of the target document, without the need for the content of the target documents. the proposed method is complementary to content-based approaches. hence, to cover potential new queries where similar labeled examples do not exist, we provide an ensemble paradigm, called the q2r orchestrator -q2r stands for "query-to-resolution" -to obtain the best of both worlds. in the proposed approach, each document @xmath2. we thus have a multiclass classification problem with |d| classes. in prac-tice, |d| can be huge and is likely to increase with time. therefore, a parametric method for learning a classifier may not be a good fit. we thus propose a nonparametric approach based on kernel k-nearest-neighbors (knn), which readily handles a growing set of documents, d, and requires only occasional re-tuning. the knn approach takes the similarity between a new query x and past queries x in the training set, rather than the contents of the documents y. in addition to bypassing the problem of using long document content, this approach allows retrieving documents not reachable through content alone. this ability is valuable in application domains such as technical support, where the content may not be well-represented in pretrained language models. the effectiveness of the knn approach is, however, limited by the availability of labeled training examples and their coverage in terms of the "reachable" documents in d. the q2r orchestrator thus combines highly accurate results from knn for queries where labeled training data is sufficiently similar with a standard content-based retrieval system, for non-similar queries, through a learned orchestrator. empirically we show that the resulting system benefits from both components. the three main contributions of q2r are as follows: (i) q2r adds a direct classification component to document retrieval based on kernel knn that enhances the ability to retrieve relevant documents. (ii) q2r makes use of a labeled data set to train a symmetric query-to-query similarity metric for the kernel knn, which enhances considerably the system performance, and (iii) q2r blends the results from the knn and content-based retrieval methods through an optimized orchestrator. ## related work for a survey on text ranking, especially modern transformer-based approaches, we refer the reader to @xcite . the majority of text-ranking approaches, driven by publicly available datasets such as those from trec @xcite and more recently ms marco @xcite , are content-based. these approaches range from keyword-based, such as bm25 @xcite , to the recent bert-based @xcite models such as re-ranking @xcite @xcite and full-ranking with dense-representations (reimers and gurevych, 2019; @xcite @xcite . in terms of ensembling multiple document retrieval approaches, the recent focus has been on the computational cost, where faster techniques are used to pre-filter the large document pool, to be re-ranked by computationally more expensive but more accurate techniques. a good example is the work by @xcite , which combines a series of traditional ir techniques with neural approaches. while content-based approaches benefit greatly from models pre-trained with large corpora, they are at a disadvantage in specialized domains involving technical support documents. in such domains, the "resolution" documents given a query need not have a high relevance score based on the content alone. document expansion techniques can play a role but often fall short as compared to direct classification, as we demonstrate in this work. to the best of our knowledge, there are no existing works that combine a content-based approach with direct classification as proposed in this work. the proposed q2r orchestrator learns a separate classifier to choose results from either the contentbased or the direct classification approaches. traditional fusion techniques @xcite @xcite can be used here and in some settings may further improve the retrieval performance. we leave this as possible future work. ## conclusion we presented the q2r system aimed at providing relevant documents in response to technical queries in natural language. the key novelty in this system is its use of both content-based document retrieval techniques as well as the proposed kernel knn approach, which taps into the available labeled data from historical queries. our experimental results show that content-based document retrieval and the kernel knn approach complement each other; q2r is able to take advantage of both. the system has been deployed at ibm on various applications that involve natural language queries and has shown encouraging performance improvement over existing systems. potential future enhancements include more sophisticated sampling procedures for the metric-learning, as well as new fusion approaches for the orchestrator. ellen m. voorhees. 2004. overview of the trec 2004 robust retrieval track. in in proceedings of the thirteenth text retrieval conference (trec2004), page 13. kilian q. weinberger and lawrence k. saul. 2009. distance metric learning for large margin nearest neighbor classification. j. mach. learn. res., 10:207-244. lee xiong, chenyan xiong, ye li, kwok-fung tang, jialin liu, paul n. bennett, junaid ahmed, and arnold overwijk. 2021. approximate nearest neighbor negative contrastive learning for dense text retrieval. in international conference on learning representations.