BERT meets Cranfield: Uncovering the Properties of Full Ranking on Fully Labeled Data (2021)

Abstract: Recently, various information retrieval models have been proposed based on pre-trained BERT models, achieving outstanding performance. The majority of such models have been tested on data collections with partial relevance labels, where various potentially relevant documents have not been exposed to the annotators. Therefore, evaluating BERT-based rankers may lead to biased and unfair evaluation results, simply because a relevant document has not been exposed to the annotators while creating the collection. In our work, we aim to better understand a BERT-based ranker’s strengths compared to a BERT-based re-ranker and the initial ranker. To this aim, we investigate BERT-based rankers performance on the Cranfield collection, which comes with full relevance judgment on all documents in the collection. Our results demonstrate the BERT-based full ranker’s effectiveness, as opposed to the BERT-based re-ranker and BM25. Also, analysis shows that there are documents that the BERT-based full-ranker finds that were not found by the initial ranker.

## introduction transformers-based pre-trained language representations, such as elmo @xcite , bert @xcite , and t5 @xcite have been counted as a promising approaches to various information retrieval tasks, such as document ranking @xcite and question answering @xcite . prior work argued that utilizing bert for ranking can achieve state-of-the-art results on popular ad-hoc retrieval collections such as robust04 @xcite @xcite , clueweb09-b @xcite , and ms marco @xcite @xcite . however, two major limitations make these collections unfair to bert. one of the limitations of these collections is that they have not been created to reflect bert's superiority to its best @xcite . as bertbased models did not contribute to their assessment pool, testing a bert-based ranking system with these collections can lead to unfair and biased results @xcite . there may be relevant documents that traditional methods could not find, hence assumed irrelevant in the pooling process. therefore, new collections are needed or collections with full relevant judgments. the second limitation is that although reusable test collections play an important role in ir, conducting a lot of research on a single collection can direct the results to an outstanding value by chance @xcite . to avoid these two limitations, we use a previously unused collection with characteristics such as containing a relatively large number of queries in the form of short, full questions. also, to address the first mentioned limitation, the collection contains a full set of judgments for each query. the cranfield foot_0 collection has all the mentioned features. unlike other collections, cranfield's main feature is a complete judgment, so documents uniquely found by a bert-based ranker can be fairly assessed. this collection is built using abstracts of aerospace-related documents such as papers, research reports and articles from the collection of the college of aeronautics, cranfield, england. @xcite . the documents' authors were asked to provide a set of related terms for their documents which were turned into natural language queries @xcite . the collection contains 225 queries and 1400 documents. the collection has not been used for document ranking with bert before and can also address the second limitation. furthermore, queries and most of the documents are short, which seems to be a good fit for bert due to the bert's token limitation. using the cranfield collection, we address the following research questions: our work to answer these research questions includes two steps: first, we provide experimental results on the cranfield collection for document re-ranking with bert, following the bertbased re-ranker of @xcite using bm25 @xcite as the initial ranker. second, we study bert's behavior as a full-ranker. in the paper, we refer to a ranker without any initial filtering method as a full-ranker. the code to reproduce our results is available at https://gitlab.science.ru.nl/ nghasemi/bert-meets-cranfield . the contributions of this work are as follows: the rest of the paper is structured as follows: section 2 reviews the related works on bert and its usage in ranking problems. in section 3, we describe the model used for document ranking with bert in more detail. experimental results and analysis are presented in section 4. the final section 5 concludes the paper and discusses potential research questions for future work. bert is not limited to the discussed pre-trained tasks. the self-attention mechanism in the transformers block makes bert capable of modeling many tasks as long as the task inputs are appropriately processed with bert's desired setups, namely using proper special tokens ([cls], @xcite ) and bert's specialized tokenizer. finetuning the bert pre-trained model helps to gain better performance on a specific task. adding one additional output layer to a bert pre-trained model is suggested in the fine-tuning phase to minimize the number of learning parameters. we refer readers to @xcite for more details. ## method although bert pre-trained models were trained on large corpora, fine-tuning is necessary for us-ing bert effectively. inspired by the work proposed by @xcite , we fine-tuned the bert model and used it to find the representation of the query and document pairs. due to limited resources, we use the bert-base, uncased, pre-trained model. this model produces 768-dimensional representation vectors. our input vector to bert is similar to the next sentence prediction task's input vector used in pre-training bert. following the bert's standard input format @xcite , we converted each query q and document d to a pair [cls] + q + [sep ] + d + @xcite . the query always remains unchanged, but as bert has a limitation on the number of tokens, we truncate documents that were longer than the model's maximum length. we use a pre-trained bert model with an added single linear classification layer on top of the [cls] output vector to suit the ad-hoc retrieval task. in this case, each input's [cls] output vector would be fed to the classification layer, and both the pre-trained bert model and the additional untrained classification layer is tuned on training samples of queries and documents from the cranfield collection. as mentioned in section 1, the cranfield collection contains 225 queries and 1400 documents. all queries in the cranfield collection have full, graded relevance judgments. there are five different relevancy grades. grades one to four are known to be relevant, and five shows irrelevant documents. we test a bert-based re-ranker and a bertbased full-ranker, using the same fine-tuning method and hyperparameters. after tuning the bert model, we use the top-100 bm25 results to do re-ranking, but we use all documents for fullranking. we evaluate the results using map@100, and ndcg@20 @xcite for both models. ## conclusion this paper investigates bert for document ranking. in our experiments, we explore the cranfield collection, which has not been used on bert-based ranking approaches and gives new insights because of its characteristics, such as full relevance judgments and a large number of queries. in addition to the document re-ranking with bert, we considered using a full-ranker under the same experimental settings. the results show that the re-ranker and the full-ranker improve a bm25 baseline significantly. furthermore, the bert-based full-ranker outperforms the bert-based re-ranker. based on our studies, the bert-based full-ranker is a different model than the bm25 ranker as it retrieves a notable number of new documents that were not found by bm25. this is especially true for highly relevant documents.