Dynamic Context Extraction for Citation Classification (2022)

Abstract: We investigate the effect of varying citation context window sizes on model performance in citation intent classification. Prior studies have been limited to the application of fixed-size contiguous citation contexts or the use of manually curated citation contexts. We introduce a new automated unsupervised approach for the selection of a dynamic-size and potentially non-contiguous citation context, which utilises the transformer-based document representations and embedding similarities. Our experiments show that the addition of non-contiguous citing sentences improves performance beyond previous results. Evalu- ating on the (1) domain-specific (ACL-ARC) and (2) the multi-disciplinary (SDP-ACT) dataset demonstrates that the inclusion of additional context beyond the citing sentence significantly improves the citation classifi- cation model’s performance, irrespective of the dataset’s domain. We release the datasets and the source code used for the experiments at: https://github.com/oacore/dynamic_citation_context

## introduction understanding citation types has served a wide range of applications, including research evaluation @xcite , article summary generation @xcite and information retrieval @xcite to name a few. classifying citation types according to their purpose or intent can make use of a variety of features, the most essential of which is the contextual textual fragment (context window) surrounding the citation marker within the citing article @xcite . this information, also known as citation context, articulates how a cited work is presented in a research paper. several citation type taxonomies of widely varying granularity have been used for citation type classification in the past @xcite . the taxonomy originally introduced by jurgens et al. has been used across the two largest annotated datasets for citation typing, act @xcite and acl-arc @xcite and is shown in appendix a. although evidence indicates that the size of the citation context window matters, there is not yet a consensus about its optimal size. while some researchers argue that multi-sentence context windows only add noise, thus confining their focus to the citing sentence alone @xcite , others emphasise the need to incorporate longer citation context to avoid information loss @xcite @xcite . most citation intent classification methods rely on a fixed-size contiguous citation context window (most typically one sentence) @xcite @xcite , or a defined number of characters @xcite . significant variation in contextual lengths however for each citation makes considering fixed context window size less desirable @xcite . the use of a fixed citation context comes also with the risk of either the addition of noise (when the surrounding sentences have one or more citations) or loss of information (when the implicit citation context is beyond the static window size). additionally, previous research shows that the document structure can influence the citation context window size, where it is more likely that context size is smaller for citations in the introduction section than in other sections, thus questioning the reliability of fixed citation contexts @xcite . the use of adaptive longer than one sentence context methods for determining the optimal context span was also investigated by the earlier works @xcite . these methods involving supervised sentence classification require manual annotations for identifying the citation context boundary. additionally, prior work on citation context extraction is mostly domain-centric, with many previous studies explicitly focusing on articles from computational linguistics. it was shown however in @xcite that citation behaviour of researchers differs across disciplines. the goal of this study is to answer the following research questions: rq1: to what extent does the performance of citation classification models vary depending on the size of the applied context window? previous studies have not provided a definitive answer to this question. this is largely due to the results from previous studies not being comparable, as they use different datasets, type classifications and methodologies. our work tests the effect of changing the citation context window size under the same experimental conditions, i.e. using identical state-of-the-art models; across two benchmark datasets, one multidisciplinary and one domainspecific. accurately measuring this effect then enables us to measure the extent to which the citation intent classification performance varies depending on the context window size. should we find that such difference is significant, this would motivate us to answer: rq2: how can we create a dynamic-size context extraction model that adaptively identifies sentences in the vicinity of the citation marker that should be semantically part of a given citation context window? such models would constitute a component that dynamically, i.e. adaptively for each citation marker, identifies the boundaries for a semantically coherent and complete citation context. the output of this component could be fed to the input of a citation intent classification model to increase its performance. rotondi et al. (2018) categorise citation context determination strategies depending on the size of the context used as follows: (1) fixed number of characters, (2) citing sentence, (3) fixed extended context and (4) adaptive extended context. for automatic classification of citation functions, jurgens et al. (2018) utilised fixed context size of 200 characters from either side of the citation, which was extracted using parscit (councill et al., 2008), an open-source scientific document parser. the developers of the scicite dataset (cohan et al., 2019) on the other hand, noted that the addition of more context besides citing sentences resulted in the introduction of noise. using sequence classification approach, abu-jbara et al. (2013) experimented with different citation context window sizes for citation purpose and polarity classification. the authors concluded that the best context span constituted the previous, citing and two following sentences. sequence classification approaches for context window detection use nlp-based features for identifying dynamic citation contexts. kaplan et al. (2016) did extensive analysis on citation context teams method used context used macro f-score irel scibert citing sentence 0.2670 duke data bilstm attention prev sent,citing 0.2590 science + elmo sent, next sent ## methodology our experiments for rq1 are designed to systematically test the performance of citation typing classification models on different fixed-size context windows. for this purpose we utilise a state-of-theart model based on scibert @xcite , which is the highest performing system from the previous two 3c shared @xcite . additionally, to understand the extent to which performance is impacted by the size of the citation context window, we evaluate a non-deterministic oracle approach. this approach assigns the correct label if at least one of the fixed window models make the right prediction. we extract several fixedsize contexts ( in rq2, we address the limitations of the existing fixed-size context approach by exploring a new adaptive unsupervised approach for dynamically extracting citation context. as illustrated in figure 1 , there are two types of the dynamic-size context: (1) contiguous and (2) non-contiguous. our extraction method utilises transformer-based scientific document embedding methods, specter @xcite and scincl @xcite and features from the citing and cited article, in addition to the citing sentence. finally, we evaluate the extracted dynamic context on citation function classification task using a sample of the multi-disciplinary act dataset @xcite and domain-specific acl-arc dataset @xcite . paragraph containing citing sentence fc11 ## results tables 4, 5 and 6 show the results we obtained for the domain-specific acl-arc and the multidisciplinary sdp-act datasets for the fixed-size, dynamic-size contiguous and dynamic-size noncontiguous contexts. it also contains the theoretical performance boundary of the oracle. from by adding the previous sentence to the citing sentence. however, on the multi-disciplinary sdp-act dataset, models perform well when using the immediate sentences following the citing sentence. in both cases, we can see that the theoretical performance boundary, represented by the oracle approach, performs substantially better. this empirically shows high dependence of classification performance on the context window size, indicating a strong potential for improvement with the dynamic-size context approaches. the results for the three context window approaches are as follows: fixed-size context -the highest macro and micro f-score for the acl-arc dataset is obtained by adding up to one or two previous sentences from the citing sentence. however, surprisingly, the performance drops when the subsequent sentences from the citing sentence are added to the citation context. this contrasts with the findings of @xcite who previously reported that "...the related context almost always falls within a window of four sentences. the window includes the citing sentence, one sentence before the citing sentence, and two sentences after the citing sentence.." @xcite , where the authors performed experiments using papers from computational linguistics, similar to the acl-arc dataset. in the case of multi-disciplinary sdp-act corpus, the sentences from the next context proved to be more valuable for citation classification. the highest performance was reported when up to three sentences following the citing sentence were added to the fixed citation context. the experimental results across both datasets ( dynamic non-contiguous context -the performance of the non-contiguous context on the acl-arc citation classifier falls by 3.4% when compared to its contiguous counterpart ( ## discussion citation type classification based on purpose reflects the author's citing intention and is therefore important for a wide range of applications, including research evaluation and scholarly document retrieval. prior citation classification research has primarily been restricted to specific domains, notably computer science, computational linguistics and bio-medicine. this has severe drawbacks as methods developed for a singular discipline cannot capture the varying differences in citation practices across disciplines. this is why we conducted all our experiments on a domain-specific as well as on a multi-disciplinary corpora. the outcome that adding further contexts beyond one sentence significantly improve results is impor-tant for further practice. as the optimal size of the citation context window for a given dataset is not known in advance, as can be seen from our experiments on the sdp-act and acl-arc dataset, there are two options: 1) to manually annotate the citation boundaries (which may be tedious) or 2) to apply a dynamic-size context extraction approach prior to feeding data into the citation type classifier. we argue that option 2 is well suited in situations where manual annotation of the boundaries is not available, which is the case on all current citation type datasets, except multicite @xcite , and whenever one needs to apply the model in practice across large volumes of citations. one potential limitation of this work is the usage of a restricted set of contextual features for dynamic boundary detection. as a direction for future work, we would be interested in applying additional scientific features (both contextual and non-contextual) to further improve the dynamic non-contiguous method and verify the performance against the existing manually annotated multicite corpus @xcite . also, the challenges involved in extracting features resulted in a considerable number of missing values for the cited abstract, which is another limitation of this paper. we believe employing additional sources for meta-data extraction might reduce the missing feature values in the future. the acl-arc and sdp-act datasets used in these experiments were chosen for comparison due to their similarities, notably the usage of the six-way classification system. the most significant difference however is the range of domains from which the citation contexts are drawn. the acl-arc dataset uses data from just one domain, computational linguistics, whereas the sdp-act dataset is compiled from citations across 36 domains. the significant differences in the evaluation scores for the acl-arc and sdp-act datasets suggest that citation classification models trained on a specific domains are less effective when used to classify a multi-disciplinary dataset. this is an important direction for future work. ## conclusion this work provides the first comprehensive study of the effect of different citation context window sizes on citation type classification performance. our results on fixed-size contexts conclusively shows that using only the citing sentence, as it is com-mon in previous work @xcite , leads to lower performance than what can be achieved with longer citation contexts. furthermore, our analysis of fixed-size context reveals that the optimal citation context size is domain-dependent. this emphasises the need for determining context dynamically. we therefore present the first unsupervised adaptive dynamic-size context extraction method for contiguous and non-contiguous context extraction. this method significantly improves performance of citation classification models compared to using the citing sentence only. the results from our performance boundary test using the oracle system suggest a large scope for further improvement which can be achieved in the future with the use of dynamic-size context extraction methods.