The Inefficiency of Language Models in Scholarly Retrieval: An Experimental Walk-through (2022)

Abstract: Language models are increasingly becoming popular in AI-powered scientific IR systems. This paper evaluates popular scientific language models in handling (i) short-query texts and (ii) textual neighbors. Our experiments showcase the inability to retrieve relevant documents for a short-query text even under the most relaxed conditions. Additionally, we leverage textual neighbors, generated by small perturbations to the original text, to demonstrate that not all perturbations lead to close neighbors in the embedding space. Further, an exhaustive categorization yields several classes of orthographically and semantically related, partially related and completely unrelated neighbors. Retrieval performance turns out to be more influenced by the surface form rather than the semantics of the text.

## introduction representation learning methods have drastically evolved large scientific volume exploration strategies. the popular applications include summarization, construction of mentor-mentee network @xcite , recommendation @xcite @xcite , qa over scientific documents @xcite , and verification of scientific claims @xcite . the growing community's interest has led to the development of several scientific document embedding models such as oag-bert @xcite , specter @xcite , scibert @xcite , and biobert @xcite over the past five years. oag-bert has been deployed in the aminer production system. given similar possibilities of future deployments of scientific document embeddings models in the existing scholarly systems, it is crucial to evaluate and identify limitations robustly. however, we do not find any existing work that critically analyzes the scientific language models to the best of our knowledge. to motivate the reader, we present a simple experiment. queries 'document vector' and 'document vectors' fetch no common candidates among first page results on google scholar and semantic scholar (candidates in appendix a). this illustrates the extremely brittle nature of such systems to minor alterations in query text, leading to completely different search outcomes. to motivate further, we experiment with textual queries encoded by the popular scibert model. a perturbed text 'documen vector' (relevant in ai) is closer to the biomedical term 'virus vector' in the embedding space. similar observations were found for many other queries. as scholarly search and recommendation systems are complex systems and their detailed algorithms are not publicly available, we analyze the behavior of scientific language models, which are (or will be) presumably an integral component of each of these systems. motivated by the usage of perturbed inputs to stress test ml systems in interpretability analysis, we propose to use 'textual neighbors' to analyze how they are represented in the embedding space of scientific lms. unlike previous works @xcite , which analyze the effect of perturbations on downstream task-specific models, we focus on analyzing the embeddings which are originally inputs to such downstream models. with the explosion of perturbation techniques for various kinds of robustness and interpretability analysis, it is difficult to generalize the insights gathered from perturbation experiments. we propose a classification schema based on orthography and semantics, to organize the perturbation strategies. the distribution of various types of textual neighbors in a training corpus is non-uniform. specifically, the low frequency of textual neighbors results in non-optimized representations, wherein semantically similar neighbors might end up distant in the space. these non-optimal representations have a cascading effect on the downstream task-specific models. in this work, we analyze whether all textual neighbors of input x are also x's neighbors in the embedding space. further, we also study if their presence in the embedding space can negatively impact downstream applications dependent on similarity-based document retrieval. our main contributions are: ## related works several works utilize textual neighbors to interpret decisions of classifiers @xcite , test linguistic capabilities of nlp models @xcite , measure progress in language generation @xcite , and generate semantically equivalent adversarial examples @xcite . similar to these works, we use textual neighbors of scientific papers to analyze the behavior of scientific lms. @xcite analyze the behavior of neural ir models by proposing test strategies: constructing test samples by controlling specific measures (e.g., term frequency, document length), and by manipulating text (e.g., removing stops, shuffling words). this is closest to our work, as we also employ text manipulation to analyze the behavior of scientific language models using a simple alternative-self retrieval scheme (section 4). however, our focus is not evaluation of retrieval-augmented models and we only use a relaxed document retrieval pipeline in our evaluation to analyze the behavior of scientific lms trained on diverse domains, in encoding scientific documents. we organize the textual neighbors into categories which capture different capabilities of lms. we also show that it is crucial to evaluate models on dissimilar texts rather than just semantically similar textual neighbors. ours is a first work in analyzing the properties of scientific lms for different inputs and can be utilized by future works to design and evaluate future scientific lms. due to space limitations, we present the detailed discussion on scientific lms in appendix c. ## short queries and textual neighbors in this paper, we experiment with short queries to fetch relevant scientific documents. the term 'short' signifies a query length comparable to the length of research titles. the candidates are constructed from either title (t) or title and abstract (t+a) text. we, further, make small alterations to the candidate text to construct 'textual neighbors'. the textual neighbors can be syntactically, semantically, or structurally similar to the candidate text. unlike previous works that explore textual neighbors to analyze and stress test complex models (q&a, sentiment, nli, ner (rychalska et al., 2019)), we experiment directly with representation learning models and analyze the placement of textual neighbors in their embedding space. while semantically similar neighbors are frequently used in previous works (ribeiro et al., 2018); we also explore semantically dissimilar textual neighbors to analyze scientific language models. while an lm is expected to represent semantically similar texts with high similarity, some orthographically similar but semantically dissimilar texts can have highly similar embeddings, which is undesirable behavior. note that we restrict the current query set to titles for two main reasons: (i) most real-world search queries are shorter in length, and (ii) flat keyword-based search lacks intent and can lead to erroneous conclusions. textual neighbors have a similar word form or similar meaning. we observe that textual neighbors possess the following properties based on two aspects: (i) orthography (surface-level information content of text in terms of characters, words, and sentences), and (ii) semantics. these two aspects are integral for a pair of texts to be textual neighbors. the properties of these aspects are: 1. orthography: orthographic-neighbors are generated by making small surface-level transformations to the input. based on the textual content, neighbors can be generated by: (a) lossy perturbation (lo): 'lossy' behavior indicates that the information @xmath0: neighbor code is in format txx_ayy_zz, where xx and yy denote the perturbation to the paper title (t) and the abstract (a) respectively. zz denotes perturbation to both t and a. missing t or a denotes that the corresponding input field is deleted completely. from the original text is lost, either due to addition or deletion of textual content. e.g., random deletion of 2-3 characters from a word. (b) lossless perturbation (ll): 'lossless' behavior indicates that the original information content is unaltered. e.g., shuffling sentences of a paragraph. 2. semantics: textual neighbors can either be semantically similar or not. while semantic categories are subjective, we try to define them objectively: (a) dissimilar (ds): semantically dissimilar neighbors are generated by deletion of nouns (nns) or noun phrases (nps), or by changing the directionality of text such as replacing adjectives (adjs) with antonyms. (b) partially similar (ps): partially similar neighbors are constructed by sentence level modifications such as sentences scrambling (while preserving the word order in each sentence), word deletion of non-nns (or nps), or character deletion in words (including nns) such that the textual neighbor preserves at least 70% of the words in the original text. (c) highly similar (hs): highly similar neighbors are constructed by (i) nonword-semantic changes such as changing the case of words or altering the whitespace characters, (ii) word deletions or additions such that the textual neighbor preserves at least 90% of the words in the original text. note that deletion of nns only from the title while preserving the abstract will qualify to be the hs category. since specter and oag-bert utilize paper titles and abstracts to learn embeddings, we generate textual neighbors by altering texts of these two input fields. we present 32 textual neighbor types in ## experiment design the alternative-self retrieval: we propose an embarrassingly simple binary retrieval scheme dataset domain subdomain #papers acl-anth cs nlp 35,482 iclr cs deep learning 5,032 arxiv-cs-sy cs systems and control 10,000 arxiv-math mt algebraic topology 10,000 arxiv-hep hep hep theory 10,000 arxiv-qbio qb neurons and cognition 6,903 arxiv-econ eco econometrics, general & theoretical economics 4,542 we make available our code and dataset for public access foot_0 . the notations: d is the set of seven datasets described in evaluation metrics: we report performance scores on the following retrieval metrics: mean reciprocal rank (mrr): all our tasks use binary relevance of documents to compute mrr. t100: it represents the percentage of queries which retrieve the one and only relevant document among the top-100 documents. nnk_ret: % of queries in textual neighbor category whose k nearest neighbors (k-nn) retrieve the original document. aop-10: average overlap percentage among 10-nn of x and y, where @xmath5. f is a text manipulation function, represented by the textual neighbor codes presented in ## analysing embeddings for scientific document titles and abstracts in this section, we experiment with the inputs to the scientific language models. due to free availability and ease of parsing paper title and abstract, majority scientific lms learn embeddings from the title and the abstract of the paper. however, multiple downstream applications such as document search involve short queries (often keywords). we present two alternative-self retrieval experiments to compare the embeddings of paper title (t) with the embeddings of paper titles and abstract (t+a). in both experiments, |q| = 1000 queries for each dataset. task ii: introducing all titles in the candidate set to increase the complexity of the previous task, we add all the title embeddings (t) in the candidate set (figure 1(b) ). we test if the t embeddings are more similar to other titles, or to their corresponding t+a embeddings. query: @xmath6: the task and evaluation metrics are same as task i. extremely poor values for scibert ( specter still performs the best, but a significant drop in mrr and t100 suggests that both t and t+a embeddings tightly cluster together in partially overlapping small groups (can be verified from figure 2 ). however, comparable t100 for oag-bert to the previous experiment suggests that the model does not falter when the input text length is small. ideally, we expect t and t+a embeddings to overlap, indicating that the embeddings of the same paper are closer. the pretraining of these models could be the reason for such distribution of t and t+a embeddings. scibert is trained on sentences from full text of research papers, leading to different representations for short (t) and longer (t+a) texts. as specter and oag-bert are trained on title and abstract fields both, such non-overlapping behavior is not observed. ## analysing scientific lms with textual neighbors in the previous section, we experimented with different input fields (t vs t+a). in this section, we experiment with the 32 textual neighbors classes (which alter different input fields: t, a, or t+a). we present our results for the following experiments for the five broad categories: ll-hs, ll-ps, ll-ds, lo-hs, and lo-ds. due to space constraints, we present plots for selective datasets for each of the experiment in the paper. the rest of the plots are presented in appendix e. ## conclusion we propose five categories of textual neighbors to organize the increasing number of textual neighbor types: ll-hs, lo-hs, ll-ps, lo-ps, and lo-ds. we evaluate scibert, specter, and oag-bert models on thirty-two textual neighbor classes organized into the previous five categories. we show that evaluation of language models on 'semantically dissimilar' texts is also important rather than just evaluation on 'semantically similar' texts. we show that the scibert model is highly sensitive to the input length. specter embeddings for all types of textual neighbors are highly similar irrespective of whether the textual neighbor is semantically dissimilar or not. specter embeddings show sensitivity to the presence of specific keywords. lastly, oag-bert embeddings of all categories of textual neighbors are highly dissimilar to the original title and abstract (t+a) embed-dings. we believe that our insights could be used to develop better pretraining strategies for scientific document language models and also to evaluate other language models. one example for mlm (or replaced token identification) could be utilizing a weighted-penalty based loss, i.e. partially similar tokens if predicted should be penalized less in comparison to the prediction of unrelated (or dissimilar) tokens. additionally, these insights could also be utilised by several systems that use these scientific document language models to incorporate informed strategies in downstream systems such as recommendation systems. @xmath7: sparse composite document vectors using soft clustering over distributional representations legal document retrieval using document vector embeddings and deep learning music genre classification with word and document vectors document vector embeddings for bibliographic records indexing constructing document vectors using kernel density estimates document vector extension for documents classification text classification with sparse composite document vectors multi-document summarization by creating synthetic document vector based on language model words are not equal: graded weighting model for building composite document vectors document vector representations for feature extraction in multi-stage document ranking a document descriptor using covariance of word vectors an adaptive topic tracking model based on 3-dimension document vector document embedding with paragraph vectors a support vector machine mixed with tf-idf algorithm to categorize bengali document document classification with distributions of word vectors a framework for a feedback process to analyze and personalize a document vector space in a feature extraction model text document clustering using global term context vectors google scholar google scholar document vector representations for feature extraction in multi-stage document ranking document embedding with paragraph vectors document ranking and the vector-space model scdv: sparse composite document vectors using soft clustering over distributional representations legal document retrieval using document vector embeddings and deep learning text document clustering using global term context vectors improving a tf-idf weighted document vector embedding document classification with distributions of word vectors efficient vector representation for documents through corruption using sparse composite document vectors to classify vba macros enhancing web service clustering using length feature weight method for service description document vector space representation words are not equal: graded weighting model for building composite document vectors document vector compression and its application in document clustering image-based document vectors for text retrieval a vector space model for automatic indexing improving document vectors representation using semantic links and attributes hierarchical document categorization with support vector machines self organization of a massive document collection using an n-gram-based document representation with a vector processing retrieval model music genre classification with word and document vectors