How Does BERT Rerank Passages? An Attribution Analysis with Information Bottlenecks (2021)

Abstract: Fine-tuned pre-trained transformers achieve the state of the art in passage reranking. Unfortunately, how they make their predictions remains vastly unexplained, especially at the end-to-end, input-to-output level. Little known is how tokens, layers, and passages precisely contribute to the final prediction. In this paper, we address this gap by leveraging the recently developed information bottlenecks for attribution (IBA) framework. On BERT-based models for passage reranking, we quantitatively demonstrate the framework’s veracity in extracting attribution maps, from which we perform detailed, token-wise analysis about how predictions are made. Overall, we find that BERT still cares about exact token matching for reranking; the [CLS] token mainly gathers information for predictions at the last layer; top-ranked passages are robust to token removal; and BERT fine-tuned on MSMARCO has positional bias towards the start of the passage.

## introduction pre-trained language models like bert @xcite have achieved prominent improvements in both information retrieval (ir) and natural language processing (nlp). concurrently, researchers have raised wide awareness about the difficulty of explaining such deep learning models @xcite @xcite . recently, many papers scrutinize bert's behaviors in various tasks @xcite @xcite @xcite . when it comes to token-wise analysis, most of the work study intra-layer self-attention and how it relates to various linguistic characteristics. although these analyses yield unique insights on layer-local behavior across pairs of tokens, they do not take a global perspective of how token-wise representations exactly relate to the prediction. this is crucial for answering a fundamental question in interpretability: what hidden features and tokens contribute the most to the prediction? to faithfully compute such feature-prediction attribution maps, @xcite and @xcite propose to apply information bottlenecks. in this paper, we leverage this model-agnostic method to analyze passage reranking for pre-trained transformers. we first introduce the information bottleneck for attribution (iba) method @xcite in general and elaborate its use in interpreting passage reranking. afterwards, we compare it with two other widely adopted attribution methods to demonstrate its credibility and justify our choice. we then carry out detailed analyses on the inner mechanisms of passage reranking. bert reranking @xcite starts a new chapter in information retrieval, as it combines the dual advantages of the speed of sparse representation (bm25) and the deep contextualization of dense representation. to be specific, given a query , bm25 returns top-1,000 passages . the label is 1 if a @xmath0, and 0 otherwise. for bert, the input is [cls] q @xcite , and the output label is . after fine-tuning, we rerank based on the output probabilities of relevance. this setting is different from most nlp tasks, where positive and negative labels are provided by the dataset, and only one pair of (input, probability) is required for the final output. we use iba to generate attribution maps for bert-large @xcite fine-tuned on the msmarco dataset @xcite in this paper. with the attribution maps, we investigate the following questions: q1. what are the similarities and differences between bert and bm25? for the two-stage pipeline, we wonder how bert's ranking mechanism is similar to bm25 and what it provides that bm25 doesn't. through crosspassage examination, we find that bert still regards lexical matching as important to some extent, similar to bm25. bert, furthermore, manages to capture deeper-contextualized relationships between the query and the relevant passage. ## related work generally speaking, interpretability methods are either model specific, applying to only a single architectural family, or model agnostic, covering a broad spectrum of supervised models. since pretrained transformers represent the state of the art in nlp, for model-specific techniques we discuss those for bert, the prototypical, most-interpreted transformer model. as this work specifically explores passage reranking, we also provide the necessary literature about recent progress. ## experiments and analyses figure 1c shows an example of important tokens in the query and the passage identified by iba. aside from token matching like "ph" and "water", deeper semantic relatedness like "acid" and "neutral" are also stressed. more examples and analysis are shown in appendix g. given the attribution maps, we are now able to study which tokens bert looks at for reranking. to be specific, we exploit iba to extract the top-20 most important tokens m for each ( , ), ∈ , ∈ , where and represent the query list and the passage list. we carry out our experiment under two different settings: 1. consists of 1,000 randomly selected queries from the entire msmarco passage reranking dev set. is composed of the human annotated relevant passages. we then apply iba to all 24 layers to get top-20 tokens m for each ( , ). for setting 1, we aim at cross-layer analysis for relevant passages. specifically, we identify if lower layers show different focus from higher layers. this setting is similar to glue-like @xcite classification tasks where we want to find general patterns about bert. the reason for using the top 20 is that, in our sampled instances, the average tokenized query length is 9.2, and we also want to see the emphasized tokens in passages. for setting 2, we perform cross-passage analysis to investigate different patterns between higher-ranked passages and lower-ranked passages. the choice of the top-50 cutoff is due to frugality: the recall@50 (0.817) is comparable to the recall@1000 (0.848), with much less computation. ## conclusions in this work, we leverage iba to examine bert for reranking. we compare ranking mechanisms between bm25 and bert, finding that bert still values token matching, and it also learns deeper relevance between queries and passages. we further analyze special tokens across layers and demonstrate patterns that [cls] aggregate evidence. we then investigate the robustness of top-ranked passages. finally, we find that bert fine-tuned on msmarco has positional bias towards the start of the passage. in summary, attribution maps can explain models' predictions and serve well as an observation tool that helps us visualize patterns, resulting in improved hypothesis formulation and experimental design.