On the Use of Context for Predicting Citation Worthiness of Sentences in Scholarly Articles (2021)

Abstract: In this paper, we study the importance of context in predicting the citation worthiness of sentences in scholarly articles. We formulate this problem as a sequence labeling task solved using a hierarchical BiLSTM model. We contribute a new benchmark dataset containing over two million sentences and their corresponding labels. We preserve the sentence order in this dataset and perform document-level train/test splits, which importantly allows incorporating contextual information in the modeling process. We evaluate the proposed approach on three benchmark datasets. Our results quantify the benefits of using context and contextual embeddings for citation worthiness. Lastly, through error analysis, we provide insights into cases where context plays an essential role in predicting citation worthiness.

## introduction and background citation worthiness is an emerging research topic in the natural language processing (nlp) domain, where the goal is to determine if a sentence in a scientific article requires a citation 1 . this research has potential applications in citation recommendation systems @xcite @xcite , and is also useful for scientific publishers to regularize the citation process. providing appropriate citations is critical to scientific writing because it helps readers understand how the current work relates to existing research. citation worthiness was first introduced by @xcite , where the authors formulated as a sentence-level binary classification task solved using classical machine learning techniques like support vector machines (svms). subsequent works from @xcite use similar approach but employ deep learning models like convolutional neural networks (cnns) and recurrent neural networks (rnns). more recently, zeng et al. @xcite proposed a bidirectional long shortterm memory (bilstm) based architecture and demonstrated that context, specifically the two adjacent sentences, can help improve the prediction of citation worthiness. citation worthiness is closely related to citation recommendation (suggest a reference for a sentence in a scientific article), which is often approached as a ranking problem solved using models that combine textual, contextual, and documentlevel features @xcite . more recent works employ deep learning models @xcite and personalization @xcite @xcite . citation analysis @xcite and citation function @xcite @xcite , other closely related domains, aim to predict the sentiment and motivation of a citation respectively. researchers have used many supervised approaches like sequence labeling @xcite , structure-based prediction @xcite , and multi-task learning @xcite to address these problems. in this paper, we want to investigate two research question about citation worthiness. first, we posit that citation worthiness is not purely a sentencelevel classification task because the surrounding context could influence if a sentence requires a citation. this context could include not only adjacent sentences but also information about the section titles, paragraphs, and other included citations. previous work @xcite has explored using the adjacent two sentences; we predict that citation worthiness models would improve with access to more contextual information. to pursue this hypothesis, we propose two new formulations: (a) sentence pair classification and (b) sentence se-the baseline tts systems in the project utilize the hts toolkit which is built on top of the htk framework @xcite . the hmm-based tts systems have been developed for finnish, english, mandarin and japanese [no cite]. the systems include an average voice model for each language trained over hundreds of speakers taken from standard asr corpora, such as speecon @xcite . using speaker adaptation transforms, thousands of new voices have been created and new voices can be added using a small number of either supervised or unsupervised speech samples @xcite . the lexrank method addressed was very successful in generic multi-document summarization @xcite . a topic-sensitive lexrank is proposed @xcite . as in lexrank, the set of sentences in a document cluster is represented as a graph, where nodes are sentences and links between the nodes are induced by a similarity relation between the sentences [no cite]. the second research objective is to understand if contextual embedding models would help citation worthiness. recent developments in language modeling, specifically contextual embedding models @xcite , have already demonstrated significant improvements in various nlp research tasks. we expect to observe similar gains in citation worthiness. following is a summary of the main contributions of this work: ## conclusions in this paper, we study the impact of context and contextual models on citation worthiness. we propose two new formulations for this problem: sentence-pair classification and sentence sequence modeling. we contribute a new benchmark dataset with document-level train/dev/test splits, which enables to incorporate contextual information better. we propose a hierarchical bilstm approach for sequence modeling, but we could also consider a transformer-based approach and further improve with a crf layer. likewise, we also want to consider some of the newer language models @xcite ) that handle longer sentences. we expect citation worthiness would be an important part of developing writing assistants for scientific documents. we studied the citation worthiness of sentences in scholarly articles in this paper, but we believe these findings are relevant to other domains like news, wikipedia, and legal documents.