  BERT-based Neural Ranking Models (NRMs) can be classified according to how
the query and document are encoded through BERT's self-attention layers -
bi-encoder versus cross-encoder. Bi-encoder models are highly efficient because
all the documents can be pre-processed before the query time, but their
performance is inferior compared to cross-encoder models. Both models utilize a
ranker that receives BERT representations as the input and generates a
relevance score as the output. In this work, we propose a method where
multi-teacher distillation is applied to a cross-encoder NRM and a bi-encoder
NRM to produce a bi-encoder NRM with two rankers. The resulting student
bi-encoder achieves an improved performance by simultaneously learning from a
cross-encoder teacher and a bi-encoder teacher and also by combining relevance
scores from the two rankers. We call this method TRMD (Two Rankers and
Multi-teacher Distillation). In the experiments, TwinBERT and ColBERT are
considered as baseline bi-encoders. When monoBERT is used as the cross-encoder
teacher, together with either TwinBERT or ColBERT as the bi-encoder teacher,
TRMD produces a student bi-encoder that performs better than the corresponding
baseline bi-encoder. For P@20, the maximum improvement was 11.4%, and the
average improvement was 6.8%. As an additional experiment, we considered
producing cross-encoder students with TRMD, and found that it could also
improve the cross-encoders.
