Dense Retrieval as Indirect Supervision for Large-space Decision Making (2023)

Abstract: Many discriminative natural language understanding (NLU) tasks have large label spaces. Learning such a process of large-space decision making is particularly challenging due to the lack of training instances per label and the difficulty of selection among many fine-grained labels. Inspired by dense retrieval methods for passage finding in open-domain QA, we propose a reformulation of large-space discriminative NLU tasks as a learning-to-retrieve task, leading to a novel solution named Dense Decision Retrieval (DDR). Instead of predicting fine-grained decisions as logits, DDR adopts a dual-encoder architecture that learns to predict by retrieving from a decision thesaurus. This approach not only leverages rich indirect supervision signals from easy-to-consume learning resources for dense retrieval, it also leads to enhanced prediction generalizability with a semantically meaningful representation of the large decision space. When evaluated on tasks with decision spaces ranging from hundreds to hundred-thousand scales, DDR outperforms strong baselines greatly by 27.54% in P @1 on two extreme multi-label classification tasks, 1.17% in F1 score ultra-fine entity typing, and 1.26% in accuracy on three few-shot intent classification tasks on average.

## introduction many discriminative natural language understanding (nlu) tasks require making fine-grained decisions from a large candidate decision space. for example, a task-oriented dialogue system, when responding to users' requests, needs to frequently detect their intents from hundreds of options @xcite . description-based recommendation in e-commerce needs to search from millions of products in response to users' descriptions @xcite . previous studies for such nlu tasks still train classifiers as the solution @xcite @xcite . however, we argue that this straightforward approach is less practical for large-space decision making for several reasons. first, more decision labels naturally lead to data scarcity, since collecting sufficient training data for all labels will need significantly more cost. this issue is also accompanied by the issue of rare labels in the long tail of highly skewed distributions suffering severely from the lack of sufficient training instances @xcite , leading to overgeneralization bias where frequent labels are more likely to be predicted compared with rare ones @xcite . second, the non-semantic logit-based representation for decision labels in a classifier also makes the model hard to generalize to rarely seen labels, and not adaptable to unseen ones in training. this issue also impairs the applicability of the decision-making model to real-world scenarios, such as recommendation @xcite and task-oriented parsing @xcite , where the decision space may expand rapidly and crucial labels may be absent in training. on the contrary, motivated by semantic similarity of examples annotated with identical labels, recent studies propose contrastive learning schemes that leverage siamese encoding architectures to maximize similarity scores of representation for positive example pairs @xcite @xcite @xcite . meanwhile, inductive bias from nlu tasks such as masked language modeling @xcite and natural language inference (nli; @xcite has shown beneficial for learning on rare and unseen labels via indirect supervision @xcite . however, when dealing with very large decision spaces, existing methods still face critical trade-offs between generalizability and efficiency of prediction. 2 inspired by the recent success of dense retrieval methods that learn to select answer-descriptive passages from millions of candidate documents for open-domain qa @xcite @xcite , we propose an indirectly supervised solution named dense decision retrieval @xcite . ddr provides a general reformulation of large-space decision making as learning to retrieve from a semantically meaningful decision thesaurus constructed based on taskrelevant resources. the model adopts the dualencoder architecture from dense retrieval models to embed input texts and label descriptions, and learns to predict by retrieving from the informative decision thesaurus instead of predicting fine-grained decisions as logits. in this way, ddr not only leverages rich indirect supervision signals from other easy-to-consume learning resources for dense retrieval. it also leads to enhanced prediction performance and generalizability with a semantically meaningful representation of the large decision space. we evaluate ddr on large decision spaces ranging from hundreds for few-shot intent detection, ten-thousand for ultra-fine entity typing, to hundred-thousand scales for extreme multi-label classification. ddr obtains state-of-the-art performance on 6 benchmark datasets in multiple fewshot settings, improving the most competitive baselines by 27.54% in p@1 for extreme classification, 1.17% in f1 for entity typing and 1.26% in accuracy for intent detection on average. ablation studies show that both the constructed informative label thesaurus and indirect supervision from dense retrieval contribute to performance gain. the technical contributions of this work are three-fold. first, we present a novel and strong solution, ddr , for nlu tasks with large-space decision making that leverages indirect supervision from dense retrieval. second, we provide semantically meaningful decision thesaurus construction that further improves the decision-making ability of ddr . third, we comprehensively verify the effectiveness of ddr on tasks of fine-grained text classification, semantic typing and intent detection where the size of decision spaces range from hundreds to hundreds of thousands. however, the aforementioned studies are not designed for discriminative tasks with very large decision spaces, and do not apply directly due to issues such as high inference costs (nli) and requiring decisions to be inclusive to the input (qa). we instead propose dense retriever that naturally serves as a proper and efficient form of indirect supervision for large-space decision making. nlu with large decision spaces many concrete nlu tasks deal with large decision spaces, including description-based recommendation in ecommerce @xcite and web search @xcite , user intent detection in task-oriented dialog systems @xcite , and fine-grained semantic typing @xcite , etc. previous studies for such tasks either train classifiers @xcite or rely on contrastive learning from scratch @xcite , which are generally impaired by insufficiency of training data and hard to generalize to rarely seen and unseen labels. these challenges motivate us to explore a practical solution with indirect supervision from a dense retriever. thesaurus construction intent detection: "my refund is not here yet" ultra-fine entity typing: he retired by the lower house in 1890. extreme multi-label classification abraham lincoln (/ˈlɪŋkən/ link-ən; february 12, 1809 -april 15, 1865) was an american lawyer, politician, and statesman who served… input text encoder e x label thesaurus entry encoder e p sim(x,p) linconia was the name of a proposed… lincoln (film) is a 2012 american biographical histor lincoln tower are two undergraduat grace bedel was was an american … where is my refund? a refund for a recent return …waiting for a refund to show up… administration manage the affairs… government: governing authority… organization a group of people… input text x label thesaurus p dpr trained for open-domain qa indirect supervision initialize initialize figure 1 : overview of ddr that reformulates general large-space decision making tasks as dense retrieval. label thesaurus is constructed with detailed descriptions from publicly available resources and the dual-encoder learns to maximize similarity between embeddings of input and label thesaurus entry with indirect supervision from a pre-trained dense retriever. ## method we first describe the reformulation of large-space decision making task as dense retrieval ( §3.1). we then introduce several automatic ways to construct label thesaurus of high quality ( §3.2). lastly, we demonstrate the dual-encoder architecture of ddr (illustrated in fig. 1 ) to retrieve decisions ( §3.3). ## experiments we evaluate ddr on three nlu tasks with large deicsion spaces in two criteria, 1) a minority of label space is observed during training: extreme multi-label classification ( §4.1) and entity typing ( §4.2), or 2) limited amounts of examples are available per label: intent classification ( §4.3). we investigate decision spaces of up to hundreds of thousands. dataset statistics are shown in appx. tab. 1. ## conclusions in this paper, we focus on discriminative nlu tasks with large decision spaces. by reformulating these tasks as learning-to-retrieve tasks, we are able to leverage rich indirect supervision signals from dense retrieval. moreover, by representing decision spaces with thesaurus, we provide rich semantic knowledge of each label to improve the understanding of labels for dense decision retrievers. experiments on 6 benchmark datasets show the effectiveness of our method on decision spaces scaling from hundreds of candidates to hundreds of thousands of candidates. future work can extend our method to more large-space decision making tasks, especially in the low-resource setting.