Fine-Tuning Language Models on Multiple Datasets for Citation Intention Classification (2024)

Abstract: Citation intention Classification (CIC) tools classify citations by their intention (e.g., background, motivation) and assist readers in evaluating the contribution of scientific literature. Prior research has shown that pretrained language models (PLMs) such as SciBERT can achieve state-of-the-art performance on CIC benchmarks. PLMs are trained via self-supervision tasks on a large corpus of general text and can quickly adapt to CIC tasks via moderate fine-tuning on the corresponding dataset. Despite their advantages, PLMs can easily overfit small datasets during fine-tuning. In this paper, we propose a multi-task learning (MTL) framework that jointly fine-tunes PLMs on a dataset of primary interest together with multiple auxiliary CIC datasets to take advantage of additional supervision signals. We develop a data-driven task relation learning (TRL) method that controls the contribution of auxiliary datasets to avoid negative transfer and expensive hyper-parameter tuning. We conduct experiments on three CIC datasets and show that fine-tuning with additional datasets can improve the PLMs’ generalization performance on the primary dataset. PLMs fine-tuned with our proposed framework outperform the current state-of-the-art models by 7% to 11% on small datasets while aligning with the best-performing model on a large dataset.

## introduction citation count is a crucial bibliometric for assessing the impact of scientific papers @xcite . highly cited papers are often regarded as seminal works in their respective fields. scientists cite other papers for various reasons, each contributing differently to the impact of the cited papers. for example, they may cite a paper because its the bedrock of their work or because it provides background knowledge. recently, it is also found that citations can be purchased and manipulated @xcite . this highlights the need for tools that can identify the intention behind citations to create more nuanced bibliometrics. citation intention classification (cic) tools classify citations based on their underlying intentions. prior research formulates cic as a text classification problem and solves it using machine learning methods @xcite @xcite . they extract the citation context from the citing papers (i.e., a span of text around the citation), and use it as the input to classifiers. among these methods, pretrained language models (plms) @xcite achieve the current state-of-the-art performance on cic benchmarks @xcite . researchers apply plms to the cic problem by fine-tuning them on citations with intention labels. however, obtaining labeled citations is challenging, as labeling citations in a scientific domain requires experts with in-depth domain knowledge. over the years, different cic datasets have been curated that assign the citations to different intention categories @xcite . they share an input space which is a set of citation contexts extracted from scientific papers. the set of citation labels of these datasets may contain semantically identical or similar intention categories (e.g., "background" and "unused"). accordingly, fine-tuning a plm on one such dataset may benefit its generalization performance on others. in this paper, we aim to improve plms' generalization performance on cic datasets by leveraging supervision signals from additional cic datasets. we propose a multi-task learning (mtl) framework that jointly fine-tunes plms on auxiliary cic datasets to improve the plms' generalizability on a primary cic dataset of interest. to prevent negative transfer @xcite (wherein sharing information with unrelated tasks harms the per-formance of the primary task) and to reduce the burden of hyper-parameter search, we propose a task relation learning (trl) method to control the contribution of the auxiliary datasets to the training. the trl method measures the relevance of an auxiliary dataset to the primary dataset by evaluating the information gain @xcite of a model trained on the auxiliary dataset on the primary dataset. we also find that the position of a citation within its context provides useful information for cic tasks and a position-aware readout function, i.e., a function that aggregates the plm output token embeddings to a fixed length citation embedding, can improve plms' performance. our contributions are summarized as follows: we carefully design and conduct experiments on three benchmark datasets which show that jointly fine-tuning plms on multiple datasets with the proposed mtl framework improves the plms' performance on the primary dataset. plms fine-tuned with our framework outperform the current stateof-the-art models by 7% to 11% on small datasets while align with the best-performing model on a large dataset. we release the code and datasets used in our experiments at https://github.com/ shuix007/deep-citation.git . ## conclusion we propose a multi-task learning (mtl) framework to fine-tune pretrained language models (plms) for citation intention classification (cic) tasks. our framework treats additional cic datasets as auxiliary tasks to be jointly trained with a primary cic dataset. we develop an efficient, data-driven task relation learning (trl) method that controls the contribution of auxiliary datasets to avoid negative transfer. the proposed trl method effectively identifies a set of coefficients that is critical to the performance of the mtl framework with magnitudes lower computational cost compared to grid search. we introduce a position-aware readout function and demonstrate that a citation's position within the context is informative for predicting its intention. experimental results suggest that jointly fine-tuning plms on primary and auxiliary datasets with our proposed mtl framework effectively improves their performance on the primary datasets.