MIR e AD : Simple Method for Learning High-quality Representations from Scientific Documents (2023)

Abstract: Learning semantically meaningful representations from scientific documents can facilitate academic literature search and improve performance of recommendation systems. Pretrained language models have been shown to learn rich textual representations, yet they cannot provide powerful document-level representations for scientific articles. We propose MIReAD, a simple method that learns highquality representations of scientific papers by fine-tuning transformer model to predict the target journal class based on the abstract. We train MIReAD on more than 500,000 PubMed and arXiv abstracts across over 2,000 journal classes. We show that MIReAD produces representations that can be used for similar papers retrieval, topic categorization and literature search. Our proposed approach outperforms six existing models for representation learning on scientific documents across four evaluation standards.

## introduction a significant increase in the volume of scientific publications over the past decades has made the academic literature search a more challenging task. one of the key steps to improve the recommendation systems (rs) for research articles is to obtain high-quality document-level representations. recently, transformer-based models have brought substantial progress to the field of natural language processing (nlp), obtaining state-of-the-art results on a variety of benchmarks @xcite . while transformer models are effective in language modeling and learning abstracts and journal data is available through hug-gingface hub at https://huggingface.co/datasets/brainchalov/ pubmed_arxiv_abstracts_data full code is provided at https://github.com/arazd/miread figure 1 : miread representations allow distinguishing abstracts by scientific domain, without using domain or citation information during finetuning. tsne with abstracts' representations from unseen journals is shown. previous transformer-based methods for representation learning on scientific documents are derived from bert model @xcite . classic examples of such approaches are pubmedbert, biobert and scibert -scientific domain adaptations of bert, which were pre-trained with masked language modeling (mlm) objective on pubmed abstracts, as well as full-text articles from pubmed-central and semantic scholar, respectively @xcite @xcite . while mlm objective allows to efficiently capture the context of the sentence, it cannot achieve accurate paper representations that can be used "off-theshelf" to discover similar articles. to address this problem, recent works explored fine-tuning the pretrained models with supervised objectives based on citation graphs @xcite . despite their efficiency, citationbased objectives have several disadvantages: (1) citations are not distributed uniformly, with novel papers and articles from certain fields being less favoured; (2) citations have a bias related to the increased self-citation and existence of over-cited papers; (3) citation graphs are often large and difficult to preprocess. hence, there is a gap in representation learning for scientific articles, requiring approaches which would derive high-quality document-level representations, without relying on the citation graphs. in this paper, we propose miread, an approach that requires minimal information for representation learning of academic documents. miread combines the scibert architecture with novel training objective -a target journal classification. we show that such a simple training objective leads to high-quality representations of academic papers, suitable for rs usage. figure 1 illustrates how miread representations from unseen abstracts are separated based on scientific domain, even though this information was not accessed during training. we trained miread by predicting one of 2,734 journal classes from the paper's title and abstract for 500,335 articles from pubmed and arxiv. then we measured the quality of paper representations obtained with miread using three evaluation standards -linear evaluation, information retrieval, and clustering purity scores -on three different datasets. miread substantially outperforms 5 previous approaches (bert, pubmedbert, biobert, scibert, citebert) across all evaluation benchmarks and outperforms specter in most cases. ## evaluation of representations we evaluate the information content of the representations of scientific abstracts produced by different approaches. ideally, we are interested in representations that contain information about scientific domain, and allow to distinguish specific subdomains within larger fields. we use three common strategies for representation quality assessment: linear evaluation, clustering purity and information retrieval. ## results we compared miread with the original bert model and 5 other approaches that use bert architecture: pubmedbert, biobert, scibert, cite-bert and specter. we evaluated the quality of representations with the purity metric of k-means clusters. to compute clustering purity, each cluster is assigned to its "true" class (most frequent class in the cluster), then accuracy is measured by counting the number of correctly assigned documents and dividing by the number of samples. clustering purity on mesh (shown in similar results were obtained on information retrieval experiments with arxiv subcategories (average precision is shown in overall, three types of evaluations on various datasets reveal that miread produces powerful representations of academic papers whose information content outperforms or matches the performance of the current state-of-the-art feature extraction models. ## conclusions we present miread, a transformer-based method for representation learning of research articles using minimal information. we fine-tuned miread by predicting the target journal using the paper's title and abstract, and assembled a training dataset spanning over half a million data points from over two thousands journals. we show that this simple training objective results in high-quality documentlevel representations, which can be used for various applications and are suitable for recommendation systems. earlier we have seen this effect on biological data -where the prediction of subcellular localization (dozens of classes) @xcite or protein (thousands of classes) @xcite from the fluorescent microscopy images allows to obtain high-quality features. these resulting features had higher information content and could be applied for solving various downstream analysis tasks. similarly to our findings, more classification labels improved feature quality, which was reflected in downstream task performance. we found that journal title is a high-quality label for scientific manuscripts, which can be explained by several reasons. firstly, scientific journals are often highly specialized and focused on a single topic. therefore, the journal name can serve as a precise topic label. additionally, journals with different impact factors may accept slightly different types of research works, making journal name a valuable human-annotated label. in our study, the number of journals was determined by available datasets. in a preliminary experiment, we found that increasing the number of labels resulted in better specificity of the representations (data not shown). for example, an increase from 100 to 1000 labels helps the model to learn better separations between sub-fields (e.g.medical sub-domains). we found that lower-level labels encourage the model to learn more fine-grained features to distinguish between journal classes, while high-level labels encourage model to focus on few important features, which may lead to oversimplification of representations content. our experimental results show that miread substantially outperforms 6 previous approaches (bert, pubmedbert, biobert, scibert, cite-bert, sentencebert) across three evaluation benchmarks, and outperforms specter, the current sota approach for representation learning on scientific articles, in most cases. the major advantage of miread compared to specter is that miread uses solely paper's abstract and metadata, but does not require the additional information, such as the reference graph. hence, miread can be trained on novel papers that have not obtained citations or papers that have no open access.