Improving Passage Re-Ranking with Word N-Gram Aware Coattention Encoder (2020)

Abstract: In text matching applications, coattentions have proved to be highly effective attention mechanisms. Coattention enables the learning to attend based on computing word level affinity scores between two texts. In this paper, we propose two improvements to coattention mechanism in the context of passage ranking (re-ranking). First, we extend the coattention mechanism by applying it across all word n-grams of query and passage. We show that these word n-gram coattentions can capture local context in query and passage to better judge the relevance between them. Second, we further improve the model performance by proposing a query based attention pooling on passage encodings. We evaluate these two methods on MSMARCO passage re-ranking task. The experiment results shows that these two methods resulted in a relative increase of 8.04% in Mean Reciprocal Rank @10 (MRR@10) compared to the naive coattention mechanism. At the time of writing this paper, our methods are the best non transformer model on MS MARCO passage re-ranking task and are competitive to BERT base while only having less than 10% of the parameters.

## introduction passage ranking (or re-ranking) is a key information retrieval (ir) task in which a model has to rank (or re-rank) set of passages based on how relevant they are to a given query. it is an integral part of conversational search systems, automated question answering systems (qa), etc., the typical answer extraction process in these systems consists of two main phases. the first phase is ranking passages from the collection that most likely contain the answers. the second phase is extracting answers from these passages. the performance of first phase significantly impact the performance of extracting answers and the performance of the overall system. thus it is important for a qa system to effectively rank passages. attention mechanisms have shown tremendous improvements in the deep learning based nlp models @xcite @xcite @xcite . attention allows the model to dynamically focus only on certain parts of the input that helps in performing the task at hand effectively. coattentions @xcite are class of attention mechanisms which can be applied on text matching problems. they proved to be highly effective as they enables the learning to attend based on computing word level affinity scores between two texts thus helping in effectively deciding the relevance between them. this paper builds on previous work on coattention mechanism (alaparthi, 2019) (we call it as naive coattention encoder) to tackle the problem of passage re-ranking. we recall that the coattention encoder attends across query and passage by computing the word-level affinity scores. similar to @xcite , we argue that attending at word-level limits the ability to capture local context in the query-passage interactions. as an example (which we later explain in section 5.3), for a query: what is january birthstone color, the naive coattention encoder can relate the passages describing passages such as november birthstone color, april birthstone color, etc. this is likely because of common matching terms birthstone and color and semantically similar words january, november, april, etc. we demonstrate that extending the coattentions to words and n-grams can improve the matching signals, which will contribute to final relevance scores. in the naive coattention encoder, max-pooling was applied on the coattention encodings to obtain the co-dependent representation of the passage, which forms the base in deciding the relevance. we argue that using max-pooling limits the ability to compose complex co-dependent representations. intuitively, we can leverage query to supervise the co-dependent representation from coattention encodings of the passage. with this intuition, we propose a simple query-based attention pooling. we show that query-based attention pooling can pick the appropriate clauses which are distributed across the coattention encodings. this allows the model to only focus on relevant parts of passage coattention encodings, which are appropriate in judging the relevance. additionally, the final passage representation is supervised by the query, which helps the model to better judge the relevance. to solve these challenges, we first extend the coattention encoder to words and phrases by applying it across all word n-grams of query and passage. for this purpose, similar to c-lstm @xcite and conv-knrm @xcite , we generate the word n-gram representations from the word embeddings of query and passage using the convolutional layers of different heights and multiple filters. we then encode these n-gram sequences using a bilstm to capture the long term dependencies into n-gram encodings. a coattention encoder is then applied on these n-gram encodings of query and passage to get the coattention encoding of the passage. the coattention encoder first generates the co-dependent representations of query and passage by attending across all word n-grams from query and passage. these co-dependent representations of passage are then fused with the n-gram encodings of the passage using a bilstm to get the coattention encoding of the passage. we show that this coattention encoding can better capture the local context between query and passage, thus improving the overall judging power of the model. to get the final representation of the passage, alaparthi (2019) applied max-pooling over time on the coattention encoding of the passage (note that coattention encoding is the outputs of bilstm). this final representation forms the base in deciding the relevance between query and passage. in this paper, we apply a query based attention pooling on the coattention encoding instead of max-pooling to pick the appropriate clauses which are distributed across the coattention encoding. we argue that, query based attention pooling allows the model to only focus on relevant parts of passage coattention encoding which are appropriate in judging the relevance. additionally, the final passage representation is supervised by query, which helps the model to better judge the relevance. we experimented our methods on ms marco passage re-ranking task foot_0 @xcite . making the coattention encoder n-gram aware (uni,bigrams) has increased the mean reciprocal rank @10 (mrr@10) from 28.6 to 29.9 (+4.5% relative increase) when compared to the naive coattention encoder. replacing the max pooling layer with the query based attention pooling has further improved the mrr@10 to 30.9 (+8.08% overall relative increase), resulting in the best non transformer based model. we show that our methods are competitive to bert base despite having very less number of parameters. also, our methods can be easily trained and requires much lesser computational resources. to summarize, the key contributions of this work are as follows: first, we extend the naive coattention encoder to words and phrases making the coattentions to capture local context. we call it as n-gram coattention encoder. second, we further extend the n-gram coattention encoder with query based attention pooling to pick the appropriate clauses which are distributed across the coattention encoding of the passage. we call it as n-gram coattention encoder with attention pooling. we show that this can further improve the model in deciding the relevance. third, we apply our methods on ms marco passage re-ranking task and show that our methods have outperformed all the baselines including the previous best non bert model and are competitive to bert base. last, we use examples to compare and discuss our methods with naive coattention encoder. in section 2, we discuss related work. then in section 3, we describe our two methods of improving naive coattention encoder. in section 4, we describe the dataset, baselines and the settings we used in all our experiments. next, we analyze and discuss the results in section 5. finally, we conclude our work with future plans in section 6. ## related work deep learning methods have been successfully applied to a variety of language and information retrieval tasks. by exploiting deep architectures, deep learning techniques are able to discover from training data the hidden structures and features at dif-ferent levels of abstractions useful for the tasks. therefore a new direction of neural ir is proposed to resort to deep learning for tackling the feature engineering problem of learning to rank, by directly using only automatically learned features from raw text of query and passage. the first successful model of this type is deep structured semantic model (dssm) @xcite introduced in 2013, which is a neural ranking model that directly tackles the adhoc retrieval task. in the same year lu and li proposed deep-match @xcite which is a deep matching method applied to the community question answering and micro-blog matching tasks. later from 2014 and 2015, there is a rapid increase in neural ranking models, such as new variants of dssm @xcite , arc i and arc ii @xcite , matchpyramid @xcite , etc., with the introduction of large scale datasets such as ms marco @xcite , we have seen a tremendous improvements in neural ranking models. well known architectures include duet @xcite , duet v2 (mitra and craswell, 2019), knrm @xcite , conv-knrm @xcite , coattention encoder (alaparthi, 2019) including the transformer based architectures such as bert @xcite , duobert @xcite , repbert @xcite . ## methodology in this section, we first describe with notations. next in section 3.2, we briefly describe the naive coattention encoder first proposed in @xcite . in section 3.3 and 3.4, we describe our two methods to improve the naive coattention encoder. ## experimental setup this section describes our datasets, how training and testing were performed and our implementation details. ## results and discussion in this section, we present the evaluation results of our models and compare our models with various baselines. ## conclusion and future work in this paper, we proposed two simple extensions to naive coattention encoder, namely, n-gram coattention encoder which attends the words and word n-grams to better capture the interactions between query and passage. later, we proposed simple attention pooling to pick the appropriate clauses which are distributed across the coattention encoding of the passage. our experiments on ms marco passage re-ranking task shows that our models outperformed all the baselines including the naive coattention encoder. we also compare our methods with bert and show that our methods are competitive to bert base despite having very less number of parameters, thus our models are very easy to train and are computationally efficient. we have also compared the performance of coattention encoders with respect to the query types and also qualitatively analyzed the performances by taking few examples. we show that n-gram coattention encoders now capture the local context very well and also show the delimitation of coattention mechanism. in the future, we would like to perform more deeper analysis on delimitations of coattention mechanism. apart from this, our future line of research would be as follows: incorporating the handcrafted features such as bm25 and study the performance. it would be interesting to see how the performance of the models will change with respect to the context based embeddings such as elmo, @xcite