Classification of URL Citations in Scholarly Papers for Promoting Utilization of Research Artifacts (2022)

Abstract: Utilizing citations for research artifacts (e.g., dataset, software) in scholarly papers contributes to efficient expansion of research artifact repositories and various applications e.g., the search, recommendation, and evaluation of such artifacts. This study focuses on citations using URLs (URL citations) and aims to identify and analyze research artifact citations automatically. This paper addresses the classification task for each URL citation to identify (1) the role that the referenced resources play in research activities, (2) the type of referenced resources, and (3) the reason why the author cited the resources. This paper proposes the classification method using section titles and footnote texts as new input features. We extracted URL citations from international conference papers as experimental data. We performed 5-fold cross-validation using the data and computed the classification performance of our method. The results demonstrate that our method is effective in all tasks. An additional experiment demonstrates that using cited URLs as input features is also effective.

## introduction open science is an activity for promoting sharing and utilizing research artifacts 1 . one strategy to promote these activities is to provide repositories for research artifacts, and such repositories have been developed recently, e.g., zenodo 2 and mendeley data 3 . in addition, national infrastructures for sharing research artifacts have been de-veloped 4 . to develop a research artifact repository, it is required to register research artifacts and create their metadata 5 . automating these processes can improve the efficiency of developing repositories and increase the number of research artifacts registered in the repositories. to this end, research artifact citations in scholarly papers can be utilized because scholarly papers citing research artifacts generally describe the name or usage of the artifacts. in addition, information about research artifacts not in existing metadata may be described in the scholarly papers @xcite . unlike citations for literature (paper citations), there are various ways to cite research artifacts. therefore, automating the identification of the research artifact citations is not trivial task. this study focuses on citations using urls in scholarly papers (url citations) and aims to identify and analyze research artifact citations. figure 1 shows examples of url citations. url citations can refer to not only scholarly papers but also various resources, e.g., datasets, software, homepages, and articles. therefore, an analysis of url citations leads to the identification of research artifact citations. in addition, it can clarify the reality of url citations performed informally. this paper proposes a method to classify url citations in scholarly papers according to the following viewpoints: ## task definition this study addresses three classification tasks determining the followings for each url citation. ## experiment 5.1 dataset there was no dataset for the classification of url citations with the corresponding section titles, footnotes, and this paper's classification labels. therefore, we created an experimental dataset. we collected the scholarly papers as the source of url citations from the acl anthology 7 . the papers were collected from the proceedings of acl/emnlp/naacl 2000-2021. we collected a total of 15,761 papers. the pdf of each paper was converted to text by pdfnlt-1.0 8 (abekawa and aizawa, 2016). the urls 9 , footnote numbers in the body text that refer to the footnotes, and the citation anchors referring to bibliographic information in the reference section were detected for each paper. the citation anchors were detected by regular expressions 10 based on those described by @xcite . they are compatible with both the harvard and vancouver referencing style. using the detected results, paragraphs where the url citations appeared were extracted as the citation contexts of the citations. we evaluated the performance of identifying the location of url citations using 65 randomly selected papers. as a result, precision was 0.995 (199/200), and recall was 0.948 (199/210). the extracted url citations were annotated by an expert in the natural language processing field. before the annotation, a part of url removed mechanically, such as url citations whose citation context had only a few words and the urls attached as an auxiliary to the bibliographic information in the paper citation. the annotator was instructed to refer to the label definitions and examples of annotated url citations before the work and could refer to them anytime during the work. the created dataset contained 2,037 url citations from 652 papers. figure 3 shows the distribution of labels. although the distribution of labels is skewed, there is a certain balance in the ratio 7 https://aclanthology.org/ 8 https://github.com/kmcs-nii/pdfnlt-1. 0 9 strings beginning with either "http://," "https://," or "ftp://" were identified as urls. 10 details are described in the appendix. of corresponding resource types for each resource role. for example, "dataset," "knowledge," and "datasource" defined by this paper correspond to "material," and there is not much difference in their ratios. in the dataset, the rate of url citations using footnotes is 0.725, the rate of url citations using the reference sections is 0.170, and the rate of url citations in the body texts is 0.105. figure 4 shows the examples of dataset 11 . another researcher in the natural language processing field annotated 100 citations in the dataset as with the original annotator. as a result, the cohen's kappa of the resource roles, resource types and citation functions were 0.644, 0.456, and 0.615, respectively. ## limitation in this paper, experimental data was constructed from one domain. since paper styles, including structure of sections, how to use footnotes, and which type of url citation the authors prefer may differ according to the domain, constructing experimental data from other domains and verifying our method on the data remain as the future works. this paper defined the "mixed" label for multiple resource citations. if citations are classified to the "mixed" label, the resource roles and types can not be identified. therefore, in practice, additional classification is required. otherwise, it can be considered to employ the multi-label classification as with @xcite 's study which applied the multi-label formulation to the classification of citation function. in that case, it is necessary to discuss how citations using ambiguous terms as referenced resources should be regarded (e.g., "all code and resources are available at @xcite ."). this paper addressed the automatic classification of url citation to generate metadata of research artifacts. it contributes to the efficient expansion of research artifact repository, enrichment of the existing repositories, and automatic analysis of research artifact citations. however, resources cited by urls tend to become unreachable within some years @xcite . to promote utilization of research artifacts cited by urls, establishing systems and platforms to preserve the artifacts and maintaining them are also required. as for the maintaining, the automatic predicting the longevity of research artifacts cited by urls (acuna et al., 2022) might be useful. ## conclusion this paper addressed the classification task of identifying the resource role, resource type, and citation function, for each url citation in scholarly papers. this paper proposed the classification method using not only citation contexts but also section titles and footnote texts as input fea16 if there was a significant difference compared to our method by the paired t-test, daggers ( †) are assigned. the significance level was 0.05. tures. our method was evaluated experimentally and the results demonstrated the effectiveness of our method on all tasks. however, the effective features differ depending on the task and how the url is cited. when classifying resource types, an approach that obtains and uses an embedding for the url string used for the citation was effective.