Benchmarking a transformer- FREE model for ad-hoc retrieval (2021)

Abstract: Transformer-based “behemoths” have grown in popularity, as well as structurally, shattering multiple NLP benchmarks along the way. However, their real-world usability remains a question. In this work, we empirically assess the feasibility of applying transformer-based models in real-world ad-hoc retrieval applications by comparison to a “greener and more sustainable” alternative, comprising only 620 trainable parameters. We present an analysis of their efficacy and efficiency and show that considering limited computational resources, the lighter model running on the CPU achieves a 3 to 20 times speedup in training and 7 to 47 times in inference while maintaining a comparable retrieval performance. Code to reproduce the efficiency experiments is available on “ https://github.com/bioinformatics-ua/EACL2021-reproducibility/ “.

## introduction the natural language processing (nlp) field has been revolutionised by the simple, yet extremely successful idea of transfer learning allied to the transformer neural architecture capability of exploring long sequences, while being computationally more efficient than the recurrent counterparts. more precisely, @xcite and later @xcite introduced the transfer learning approach to the nlp field by pre-training a language model and then fine-tuning it to multiple nlp tasks. however, it was @xcite who applied the same technique to a large transformer encoder, shattering the sota in 9 of 12 nlp tasks. since then, the nlp field became dominated by works that use, explore or improve, in some way, the transformer-based architecture. although their performance is undeniable, these are large models that comprise millions of parameters and require large computational resources to use and maintain, making them inoperable for the majority of smaller institutions. furthermore, it also gives the impression that the nlp field was reduced to throwing a lot of computation power (money) to continually achieve sota results in multiple benchmarks, leaving behind the careful process of designing a neural solution that follows the human intuition to solve some specific task. moreover, given their running costs and the consequent impact on the environment, their real-world virtues remain questionable, i.e., "is it really feasible to use these models in real-world applications, such as search engines or question answering systems to aid a broader range of people in their day-to-day tasks?". as an example, biomedical experts need to routinely search an unprecedented amount of scientific literature to keep updated with their research, which could be facilitated by an intelligent system. in contradiction with this trend, this work examines a lightweight interaction-based model, with only 620 trainable parameters, carefully designed by considering years of research in ad-hoc retrieval systems and interaction-based architectures. this system was evaluated in two ad-hoc retrieval competitions where it was able to compete with the state-of-the-art transformer-based models. in this paper, we demonstrate a "greener and more sustainable" alternative to the transformer-based architecture, by thoroughly testing such a system against the most popular transformer behemoths in a series of performance experiments. following this section, we frame our work in the current literature, we then present the lightweight model, and finally detail and discuss the evaluation performed. ## related work the information retrieval (ir) field is nowadays considered as being divided into traditional ir @xcite and neural ir @xcite . the former explores exact match signals, i.e., the co-occurrence of query terms in the document, to derive handcrafted rules and formulas to directly compute the query-document importance, with the bm25 @xcite ranking function being the most popular example. on the other end, neural ir explores the increasing success of neural networks to approximate a (sub)optimal ranking function by exploiting labelled examples. in the literature, the most successful neural ir architectures are interaction based, which measure multiple matching signals to create a joint representation of the query and the documents. moreover, considering that the transformer architecture is now widely adopted in interaction-based architectures, we further propose to subdivide the works into shallow interaction-based and transformer interactionbased models. ## proposed neural interaction based model our neural interaction-based model is an enhancement of almeida and matos (2020a) and was already used in the two international competitions, namely bioasq @xcite and trec-covid @xcite . however, in order to keep this paper self-contained, we will now introduce its insight and architecture. from a general perspective, our model follows the successful ideas presented over years of research in the neural ir field, taking into its implementation key concepts of previous works. more precisely, it follows the deeprank @xcite intuition to mimic the human judgment process with neural networks; adopts the extraction capabilities, based on 2d convolution, of the pacrr @xcite and matchpyramid @xcite models; explores the same assumption of independently scoring each sentence as addressed by @xcite ; and finally, uses the drmm @xcite query term gating mechanism to consider the importance of different query terms. overall, the model was designed with the intuition of weighting the importance of the document sentences in terms of the query by considering the context where the exact match occurs, i.e., this model produces a more refined judgment of the initial exact match signal evaluated by the bm25. regarding the architecture, the neural model is divided into three major blocks that can be observed in figure 1 . to clearly describe the model, let us first define a query as a sequence of tokens @xmath0, u 1 , ..., u q }, where u i is the i-th token of the query and q the size of the query; a document passage as @xmath1, v 1 , ..., v s }, where v j is the j-th token of the passage and s the size of the passage; and a document as a sequence of passages @xmath2, p 1 , ..., p n }, where n is the total number of passages in the document. in the current implementation, a passage corresponds to a document sentence, as described below. the first block aims to rearrange the input, query and document, to a suitable format for the downstream interaction model. more precisely, it splits the document into passages using the punkt algorithm @xcite and rearranges the passages by query-term, so that each query-term is associated with a set of passages where that term occurs. at this point, the document is represented by d(u i ) = {p i0 , p i1 , ..., p ip }, where p ik corresponds to the k-th passage with respect to the query term u i , and p is the maximum number of passages that can be associated with each query-term. note that it is possible to have repeated passages for different query-terms at this point since one passage can contain more than one query-term. the second block, named interaction network, has the objective of scoring each passage according to its relevance for a given query. to accomplish this, we employed a shallow interaction-based model to capture the most relevant matching signals that are present in each query-passage combination. in more detail, for each query-term-passage pair, we construct an interaction matrix with dimension q×s, also designated similarity tensor, by computing the cosine similarity between the query and the passage embedding. then 3 by 3 convolutions are applied to learn n-gram relevance signals that are further extracted by multiple pooling layers (max, average, and k-max-average) applied to the filter dimension, creating a feature vector r per each query-term-passage pair. consequently, d(u i ) at this point is given by d(u i ) = {r i0 , r i1 , ..., r ip }, where r ik corresponds to the k-th passage feature vector with respect to the query term u i . finally, each resulting feature vector is linearly combined with a trainable vector, and a sigmoid function, σ, is applied to produce a sentence level relevance signal (1=relevant, 0=irrelevant). the third block, the aggregation network, is responsible for producing the final document relevance score for the given query. this is accomplished by weighting the importance of each document passage score with respect to the importance of each query term. more precisely, this importance is given by a probabilistic distribution computed (softmax) over a linear combination of each query term embedding and a trainable vector and is defined as @xmath3, a u 1 , ..., a u q }, where a u i represents the probabilistic importance of the query-term u i . each passage score is then weighted by the corresponding query-term importance, a u i , and the resulting set of weighted scores is fed to a multi-layer perceptron to produce the final document score. our model is trained in a pairwise fashion and uses word-level pre-trained embeddings, specifically word2vec embeddings @xcite in these experiments. more details on hyperparameters, configuration, and training can be found in previous work @xcite . ## evaluation in this section, we aim to empirically demonstrate the efficacy of the proposed model and compare its efficiency against the state-of-the-art transformerbased models. ## discussion in this section, we take into consideration the presented results, both in terms of efficacy and performance, and present an overall overview of the comparison of our lightweight model and the transformer-based approaches. first of all, we acknowledge some limitations of the proposed model, which are all related to the current implementation that processes a large portion of padding data, increasing the number of tokens that the model sees and thus injuring the performance of the current solution. however, this is something that can be addressed in future work by rearranging the model data flow and better understanding the required maximum input size of the query and document tokens. moreover, to get a sense of speed up that we could achieve, we made a comparison with a transformer-based model that is able to encode up to 4096 tokens, which is close to the 4500 tokens of the proposed model. in this case, the lightweight model ends up being 1379 times faster, which gives an idea that the expected speedup may be in the order of a thousand. recalling the three proposed dimensions of comparison, the proposed model excels in all of them. first of all, it is a model with 10 thousand times fewer trainable parameters when compared to dis-tilbert. in the efficacy evaluation, the results show that the proposed model could achieve close to state of the art performance in both biomedical ad-hoc retrieval tasks. finally, according to the presented inference times, more precisely, with the times presented in figure 2 , it is observable that the proposed model was the only one to perform inference under one second even with a batch size of 256, which is more than enough to adopt in real-world ad-hoc retrieval applications. furthermore, this observation enables us to conclude that the transformer-based model solution is currently not suitable for a realworld ad-hoc retrieval application unless large scale computational resources are available. moreover, our results also show that the proposed model was always the fastest model by a significant margin, even though using only the cpu. this, in our view, ends up to be a preferable characteristic, since it facilitates the deployability and scalability of systems that implement this model. ## conclusion in this paper, we discuss the feasibility of using transformer-based models in real-world adhoc retrieval applications and show an extreme lightweight solution, with only 620 trainable parameters. we evaluate the solution against the transformerbased models, in terms of efficacy and performance, and show that this model is capable of matching the efficacy offered by the transformer-based models in two biomedical ad-hoc retrieval challenges while being considerably faster by just using the cpu compared to the best gpu runs of the transformerbased models.