Pre-training Multi-task Contrastive Learning Models for Scientific Literature Understanding (2023)

Abstract: Scientific literature understanding tasks have gained significant attention due to their potential to accelerate scientific discovery. Pre-trained language models (LMs) have shown effectiveness in these tasks, especially when tuned via contrastive learning. However, jointly utilizing pre-training data across multiple heterogeneous tasks (e.g., extreme multi-label paper classification, citation prediction, and literature search) remains largely unexplored. To bridge this gap, we propose a multi-task contrastive learning framework, SciMult, with a focus on facilitating common knowledge sharing across different scientific literature understanding tasks while preventing task-specific skills from interfering with each other. To be specific, we explore two techniques – task-aware specialization and instruction tuning. The former adopts a Mixture-of-Experts Transformer architecture with task-aware sub-layers; the latter prepends task-specific instructions to the input text so as to produce task-aware outputs. Extensive experiments on a comprehensive collection of benchmark datasets verify the effectiveness of our task-aware specialization strategy, where we outperform state-of-the-art scientific pre-trained LMs. Code, datasets, and pre-trained models can be found at https://scimult.github.io/.

## introduction scientific literature understanding tasks, such as paper classification @xcite , citation prediction @xcite , scientific literature search @xcite , and recommendation @xcite , have received increasing attention because they can be broadly applied to academic service platforms @xcite @xcite , and more importantly, uncover knowledge structures to accelerate scientific discovery @xcite . recent studies have demonstrated the effectiveness of pre-trained language models (lms) @xcite @xcite in these tasks as they generate highquality scientific text representations, especially when the lms are further tuned via contrastive learning. for example, micol @xcite proposes a metadata-induced contrastive learning that can perform extreme multi-label paper classification with more than 10,000 classes; specter @xcite and scincl @xcite leverage citation information to create training pairs and achieve remarkable performance in predicting various types of links between papers. nevertheless, jointly using data across different scientific literature understanding tasks for lm pretraining remains largely unexplored. intuitively, there are some common knowledge and skills that can be shared across related tasks. for example, accurately identifying fine-grained topic classes of a paper not only helps classification but also provides hints to link prediction and search. therefore, a multi-task learning framework is expected to be beneficial with improved parameter efficiency. however, if all parameters of the backbone lm are shared across tasks @xcite , the model is observed to suffer from the undesirable task interference @xcite , that is, the model sacrifices the performance on some tasks to boost the others when jointly trained to a certain extent. this is because specialized skills are still required in different tasks competing for the limited shared parameter space. for instance, the encoder for extreme multi-label paper classification should focus more on fine-grained fields-of-study entities in each paper, while the encoder for citation prediction should put more effort into understanding citation intents. mixing these two skills may result in a negative transfer across the two tasks. in this paper, to mitigate task interference in multi-task scientific literature understanding, we propose to consider two techniques: task-aware specialization and instruction tuning. task-aware specialization, inspired by the mixture-of-experts (moe) transformer architecture @xcite @xcite , modifies the transformer block in the lm to have multiple parallel sub-layers, each of which is dedicated for one task. when performing different tasks, the input will be routed to different sublayers based on task types. in this way, the encoder contains both shared parameters and task-specific parameters, making it capable of producing taskaware outputs when tapping into shared knowledge. in contrast, instruction tuning adopts one encoder for all tasks, but it prepends task-specific instructions @xcite @xcite @xcite to the input text during training so that the encoder can learn to produce task-aware representations. these two ideas are different from the techniques (e.g., adapters @xcite and control codes @xcite ) explored in previous studies @xcite for multitask scientific text representation learning. indeed, as far as we know, this is a pioneering study that explores the effect of the moe transformer architecture and instruction tuning in scientific nlp. to validate the efficacy of our proposed techniques, we conduct a comprehensive empirical study using datasets from multiple sources @xcite @xcite @xcite for evaluating various scientific literature understanding tasks. for each task, models will be tested on not only in-domain but also cross-domain evaluation datasets. specifically, for extreme multi-label text classification, models trained on computer science and biomedicine papers will be tested in the geography and psychology fields @xcite ; for link prediction, models trained on citation signals @xcite need to be evaluated on patient summary retrieval @xcite and paper recommendation @xcite ; for search, models will be tested on datasets specific to covid-19 @xcite or related to claim verification @xcite which are not seen during pre-training. experimental results show that scimult-mhaexpert outperforms competitive scientific pre-trained lms @xcite @xcite on most datasets and achieves the new state-of-the-art performance on the leaderboard of pmc-patients @xcite . ablation studies further prove that task-aware specialization can effectively mitigate task interference, while the improvement brought by instruction tuning is not consistent across all tasks. ## background we consider three widely studied tasks in scientific literature understanding: classification, link prediction, and search. (extreme multi-label) classification. classifying academic papers to their relevant label(s) is a fundamental task in scientific text mining. it can help organize papers according to their fields/themes and benefit downstream applications such as trend analysis of scientific topics @xcite . the label space l can be either coarse-grained @xcite ; e.g., predicting whether a paper belongs to "computer science" or "biology") or finegrained @xcite @xcite @xcite ; e.g., predicting whether a paper is relevant to "alphacoronavirus" or "betacoronavirus" or both). when l is large and fine-grained (e.g., with more than 10,000 labels), it is natural to assume that each paper p can be relevant to more than one label. this task is called extreme multi-label classification @xcite @xcite , which aims to rank all labels @xmath0. link prediction. link prediction aims to predict if a certain type of link exists between a query paper p q and a candidate paper p c . narrowly, "links" refer to citation links @xcite @xcite i.e., p q cites p c ). broadly, "links" can be defined as relations that p q and p c are co-viewed frequently by users, co-cited frequently by other papers, and so on @xcite ). an accurate link prediction model can benefit tasks like paper recommendation @xcite and help identify the potential use of scientific literature @xcite . search. scientific literature search helps researchers track their interested fields-of-study and prevents them from drowning in the whole literature. given a search query q and a pool of papers p, the task is to find papers @xmath1. search also serves as the initial step of more complex scientific text mining tasks, such as claim verification @xcite and open-domain question answering @xcite . ## conclusions in this work, we propose to pre-train scientific lms via multi-task contrastive learning. to mitigate task interference, we adopt two strategies: task-aware specialization considers a mixture-of-experts transformer architecture so that each task has its unique components, while instruction tuning relies on task-specific instructions to produce task-aware text representations. extensive experiments on a comprehensive collection of benchmark datasets demonstrate the advantages of our models against competitive scientific lms in extreme multi-label classification, link prediction, and search. in particular, we achieve the new state-of-the-art performance on the pmc-patients leaderboard. we also show that task-specific mha sub-layers are beneficial to the model performance across all examined tasks, whereas the benefit of other proposed techniques is not consistent. further analysis validates some of our design choices, such as hard negative mining in extreme classification.