Slice-Aware Neural Ranking (2020)

Abstract: Understanding when and why neural ranking models fail for an IR task via error analysis is an important part of the research cycle. Here we focus on the challenges of (i) identifying categories of difficult instances (a pair of question and response candidates) for which a neural ranker is ineffective and (ii) improving neural ranking for such instances. To address both challenges we resort to slice-based learning for which the goal is to improve effectiveness of neural models for slices (subsets) of data. We address challenge (i) by proposing different slicing functions (SFs) that select slices of the dataset—based on prior work we heuristically capture different failures of neural rankers. Then, for challenge (ii) we adapt a neural ranking model to learn slice-aware representations, i.e. the adapted model learns to represent the question and responses differently based on the model’s prediction of which slices they belong to. Our experimental results (the source code and data are available at https://github.com/Guzpenha/slice_based_learning ) across three different ranking tasks and four corpora show that slice-based learning improves the effectiveness by an average of 2% over a neural ranker that is not slice-aware.

## introduction retrieving text for a given information need is a fundamental task in information retrieval (ir). for a long time neural networks failed to convincingly outperform traditional term matching approaches with pseudo-relevance feedback, e.g. rm3 @xcite , for text retrieval tasks including the classic adhoc retrieval task @xcite . however, with recent breakthroughs in natural language processing (nlp), neural approachesprominently bert @xcite -are achieving state-of-the-art effectiveness across a 1 the source code and data are available at https:// github.com/guzpenha/slice_based_learning . understanding when and why retrieval models fail is an important part of the research cycle. even tough we have clues about the failures of neural rankers-obtained for instance by the study of question performance prediction @xcite , diagnostic datasets @xcite ) and error analysis @xcite )-automatically identifying difficult instances (tuples of question and response list) and improving the effectiveness of models for such difficult instances are still open challenges. we consider here difficult instances to be question and responses for which a given neural ranker retrieval effectiveness is below the average. a recent approach, referred to as slice-based learning @xcite , has been proposed to identify and improve the effectiveness of subsets of data (so-called slices), as opposed to focusing on all data equally. the core idea is that a slice-aware neural model will represent instances differently depending on the slices of data they come from. slice-based learning has been applied to computer vision and nlp tasks, with overall effectiveness improvements up to 3.5% @xcite over a model that is not slice-aware. in this paper we focus on the challenges of (i) detecting difficult instances for neural rankers and (ii) improving the retrieval effectiveness for such instances. we address the challenges by (i) creating slicing functions (sfs), i.e., functions that define whether an instance belongs to a slice which heuristically capture different errors of rankers (cf. figure 2 for examples of sfs); and (ii) employing a slice-aware neural ranker, i.e., a neural ranker that learns to represent each instance differently based on its prediction of which slice the input belongs to (cf. figure 2 for a diagram of the sliceaware neural ranker). our main research questions are the following two. rq1: to what extent can slice-based learning improve neural ranking models? rq2: what are the underlying reasons for the effectiveness of slice-based learning? our experimental results on three different conversational tasks show that slice-based learning is beneficial to ir, showing positive evidence for rq1. the gains are observed for both overall effectiveness and the effectiveness for slices of the data. concerning rq2, we evaluate to which extent the effectiveness gains observed for the sliceaware model come from the effect of ensemble learning @xcite , a direction not explored empirically by previous work @xcite . we find that, when using random sfs we can also significantly improve upon a non sliceaware neural ranker. we note though that not all improvements of slice-based learning can be attributed to the effect of ensemble learning, and carefully implementing sfs is indeed advantageous. ## slice-based learning slice-based learning @xcite is an approach based on the engineering of sfs that capture slices of data. the sfs all follow the same format: they receive the instance as input (in our case a question and a list of candidate responses) and return a boolean variable indicating whether the instance belongs to the slice. based on the sfs a neural model is adapted to improve the effectiveness of such slices of data, for example, by having a different set of weights for each slice. training a different model for each slice, and combining their predictions is inefficient: training and maintaining a different neural ranking model for each slicing function amounts to a large number of parameters and an increased prediction time. as an efficient solution, @xcite al. (2019 for each sf we define we have a sram module to learn slice-expert representations, that are then combined with an attention mechanism into a slice-aware representation. ## slice-based learning for ir we first introduce the sfs we defined to heuristically capture subsets of data containing different categories of errors, for which the effectiveness is lower than average, based on intuitions drawn from prior work (rq1). we then introduce the random sfs we deploy to study the effect of ensemble learning in slice-based learning (rq2). finally we describe the slice-aware neural ranker. ## experimental setup we employ four datasets and three retrieval tasks: msdialog @xcite and mantis @xcite for conversation response ranking, quora @xcite for similar question retrieval and antique @xcite for non-factoid question answering. we use the official train, validation and test sets provided by the datasets' creators. as a strong neural ranking baseline model we fine-tune bert foot_2 for sentence classification, using the cls token to predict whether the concatenation of a question and response is relevant or not, following recent research in ir @xcite . using 512 input tokens (larger inputs are truncated) and a batch size of 8 we train each model for 5 epochs. when employing srams @xcite ) with a bert backbone for neural ranking using both the question-based and question-responses based sfs we refer to the model as bert-sa. when using random sfs we refer to the model as bert-sa-r. for the sfs that have a threshold value (e.g., ql), we choose thresholds that select less than 50% of the data to avoid selecting the majority of the training instances in each slice. for sfs that include a categorical value, e.g., question category (qc) physics, we add one slice per category in the dataset. for the random sfs we create 10 different slices foot_3 for which 50% of randomly chosen instances from the training data belong to 6 . we train each model 5 times with different random seeds and report the test set effectiveness using mean average precision (map). ∆map indicates the difference between bert-sa(-r) and bert ## results let us first consider rq1. we observe in to better understand which features of a slice correlate the most with the observed gains from bert-sa, we study how three properties of the slices correlate with the slice ∆map (i.e., the improvement over bert): we consider (1) the size of the slice, (2) the classification accuracy of the slice-aware model to predict slice membership, and, (3) the bert model effectiveness for each slice. the only property that has a statistically significant pearson correlation (0.504 average for the different datasets) with map gains is the bert baseline performance , suggesting that focusing on failures of neural ranking models (slices for which bert has low effectiveness) when implementing sfs is effective. to provide insights into the underlying reasons of the effectiveness of slice-based learning (rq2), we replace the sfs that capture error categories with random sfs, i.e. bert-sa-r. we find that this model also has a significantly better effectiveness than the bert baseline, with the exception of quora. this indicates that part of the gains provided by slice-based learning could be attributed to the effect of ensemble learning, since each slice-aware representation is trained on random parts of the data and are then combined foot_5 . we note however that the slice gains of bert-sa are higher than bert-sa-r for antique and quora with statistical significance. this indicates that not all improvements of slice-based learning can be attributed to the effect of ensemble learning and carefully implementing sfs is advantageous. ## conclusion in this paper we demonstrated that a slice-aware neural ranker is an effective approach to ir, increasing the effectiveness of rankers by margins up to 3.8% overall and up to 13% per slice in terms of map. as future work we plan to study slice-aware neural rankers that do listwise optimization-such a ranker could learn better representations particularly for sfs that uses several responses as input.