A Fine-grained citation graph for biomedical academic papers: the finding-citation graph (2024)

Abstract: Citations typically mention findings as well as papers. To model this richer notion of citation, we introduce a richer form of citation graph with nodes for both academic papers and their findings: the finding-citation graph (FCG). We also present a new pipeline to construct such a graph, which includes a finding identification module and a citation sentence extraction module. From each paper, it extracts rich basic information, abstract, and structured full text first. The abstract and vital sections, such as the results and discussion, are input into the finding identification module. This module identifies multiple findings from a paper, achieving an 80% accuracy in multiple findings evaluation. The full text is input into the citation sentence extraction module to identify inline citation sentences and citation markers, achieving 97.7% accuracy. Then, the graph is constructed using the outputs from the two modules mentioned above. We used the Europe PMC to build such a graph using the pipeline, resulting in a graph with 14.25 million nodes and 76 million edges.

## introduction in recent years, the volume of biomedical literature has been constantly growing. more than 3000 articles are published every day on average and pubmed alone has a total of 29m articles as of january 2019 @xcite . this makes it difficult for experts to understand and assess the publications within a short amount of time. citations play a crucial role in academic papers, linking the new work to related research @xcite . they can assist in evaluating research outputs @xcite , and tracking the progression of research while predicting future directions @xcite . the citation network, a graph that records the citation relationship between papers, is commonly used in such studies @xcite @xcite . { pmid: 38399565, cited_pmid: 35198509, article_title: "a prospective analysis of the effects of a powder-type hemostatic agent on the short-term outcomes after liver resection", cited_article_title: "in vivo study for the hemostatic efficacy and foreign body reaction of a new powder-type polysaccharide hemostatic agent", cited_article_finding: "oozfix caused a minimal fbr that disappeared within 2 weeks in vivo, and its hemostatic performance was comparable with that of an existing agent, arista ah. further clinical studies are required in the future.", citation_sentence: "in a study comparing oozfix (theracion biomedical, seongnam, republic of korea), a new polysaccharide hemostatic agent, with arista ah, both products showed comparable hemostatic performance in animal models, with both agents demonstrating minimal foreign body reactions that resolved within two weeks @xcite ." } figure 1 : an example of the relation between paper and cited paper's finding through the citation sentence. in recent years, many academic databases, which also can be regarded as academic citation networks, have been developed to facilitate detailed citation studies on biomedical publications. they provide basic information for hundreds of millions of academic papers and the citations between these documents. some of these databases are commercial, like clarivate's web of science (wos) and elsevier's scopus. others are open-source, in line with current trends. these include microsoft academic graph (mag) @xcite , opencitations index of crossref open doi-doi citations (coci) @xcite , dimensions @xcite , national institutes of health's open citation collection (nih-occ) @xcite , semantic scholar's open research corpus (s2orc) @xcite . some statistics about these databases are shown in from in this paper, we define the finding-citation graph first in section 3. then, we outline the process of constructing the finding-citation graph using the european pmc dataset in section 4. we also evaluate the construction to ensure quality. the summary statistics of the graph and some interesting observations are presented in section 5. ## related work constructing a fine-grained citation graph directly relates to cite-worthiness detection, and finding identification. we will briefly introduce these aspects in the following sub-sections. since the biomedical large language model (llm) has recently gained popularity and may be used in our project, it will also be introduced in the subsequent sub-sections. cite-worthiness detection cite-worthiness detection involves identifying citation sentences in an academic paper. these sentences contain ref-erences to external sources cited within the paper. there are many different forms of citation, but the most common are: however, the datasets from these studies are confined to one or a limited number of domains and exhibit a high class imbalance. as per @xcite , only a tenth of all sentences hold at least one citation marker, leaving the remaining 90% without any. furthermore, these studies lack an indepth discussion on dataset creation and qualitative analysis. in response to these issues, wright and augenstein (2021) introduced a dataset for spotting citation-worthy sources across six domains. they detailed the process for creating the dataset and provided a qualitative analysis. however, their approach to dataset creation was limited to using regular expressions to identify the first and second citation forms mentioned above. besides, the authors trained a set of baseline models on their dataset to evaluate performance and understand the complexity of the problem. the results of these models are displayed in fig 2 . finding identification the process of pinpointing and extracting results or conclusions from an academic paper is known as finding identification. prabhakaran, hamilton, mcfarland, and jurafsky (2016) designed a conditional random field (crf) model that manages sentence-level sequence labeling, designating each sentence in the abstract a rhetorical role, including result and conclusion. dernoncourt and lee (2017 then, they fine-tuned a roberta model @xcite on this dataset, classifying each sentence in the abstract into categories such as result, conclusion, method, background, and others. the model achieved an f1 score of 92%, and when applied to the full text of papers, it performed well. previous studies have overlooked the importance of certain sections of papers, such as the results and conclusion sections. these sections often contain important findings. past studies mainly focus on finding extraction, neglecting the potential for finding generation. however, the development of large generative language models, like llama @xcite , now provides the opportunity for effective finding generation. in the finding identification task, there is a subtask known as claim identification or argumentation mining exists. according to the definition from @xcite , a claim is (1) a statement declaring something as superior, (2) a statement proposing something new, or (3) a statement describing a new discovery or a new causeeffect relationship. the definition of a claim differs from that of a finding, being stricter and more precise. nonetheless, some ideas from this research could be useful. @xcite developed a tool for annotating claims and collected 1500 labeled abstracts (scice) from pubmed articles published from 2008 to 2018. these abstracts incorporate 11,702 sentences in total, with each sentence labeled as a claim or non-claim. this tool effectively tackles the issue of data scarcity in the task. they also constructed a new model incorporating transfer learning, which improved the f1 score by 14 percentage points compared to the baseline model without transfer learning. in 2023, wei et al. undertook the same task, achieving a new state-of-the-art (sota) performance on the dataset using supervised contrastive learning and transfer learning, with an 87.45% f1 score. as observed, all these models operate on the abstract rather than the full-text article, and the shift to the full-text article still poses a challenge due to the writing structure of the complete publication. biomedical llms back in 2018, elmo @xcite pioneered the use of a context-sensitive language model pre-trained on a huge data corpus. this sparked a wave of llms such as gpt @xcite , bert @xcite , ernie @xcite , gpt-2 @xcite , gpt3 (brown et al., 2020), among others. these llms are incredibly useful for a variety of natural language processing (nlp) tasks. however, general-purpose llms, which are trained on resources like english wikipedia and bookcorpus, often struggle with biomedical nlp tasks due to the numerous domain-specific terms and proper nouns. to counter this, many llms have been pre-trained on biomedical corpus like pubmed abstracts and pubmed central fulltext articles, enhancing their performance in the biomedical field. there are two main approaches to domainspecific neural language model paradigms: mixdomain pre-training and domain-specific pretraining from scratch. mix-domain pre-training, such as biobert @xcite and bluebert @xcite , begins with parameters from a general-purpose language model and adopts its vocabulary. on the other hand, domain-specific pre-training from scratch, like pubmedbert @xcite , biolinkbert @xcite , biomedlm @xcite , and bioformer-8l @xcite , generates vocabulary and conducts pre-training using only the in-domain corpus. models like pubmedbert and biomedlm have shown that domain-specific pre-training from scratch can outperform mix-domain pre-training. ## the finding-citation graph: definition building on the work of @xcite , we define a finding as a statement that describes a specific research outcome from a scientific study. we also describe a citation sentence as a sentence that references knowledge from other papers. subsequently, we define a finding-citation graph (fcg) as @xmath0, f, c, b), where p , f , c, and b represent sets of papers, findings, citations, and basic information respectively. a paper in this graph is an academic paper. a finding in this graph is a statement same as the above definition. a citation within the graph refers to instances where the citation sentence includes the findings of the cited paper, which we will now refer to as a useful citation. the basic information includes the author, journal, publication year, etc. of the paper containing the finding. the defined finding-citation graph is a heterogeneous graph and can be perceived as a variant of the citation graph, where the node is the paper and the relation is the citation. ## constructing the fcg we now introduce our pipeline to construct the finding-citation graph (figure 3 ), which allows us to analyze findings from the perspective of the citation network. the pipeline takes a europe pubmed article in xml format as input and produces three types of information for each paper: basic information, all citation sentences, and all findings. this information is then used to construct the graph. the pipeline comprises four main modules, as follows: the above four modules will be introduced in the following sections excluding the xml parser. our parser was primarily based on an open-source pubmed parser @xcite , with minor changes made to increase speed. ## experiments we utilized europe pmc articles in xml format for our experiment. europe pmc is an open-source, global biomedical literature repository that houses life science articles, preprints, micropublications, books, patents, and clinical guidelines from around the world. our main comparison is s2orc, which is the most similar database to use and includes citation contexts, so we evaluated s2orc on the latest version again. it is not only for the comparison but also for the confirmation of our comparison. the our_o and our_d are the original and filtered pmid-pmid relations respectively. when we did the evaluation on the filtered pmid-pmid relations, we did the same filter to the gold dataset. type of article count(%) europe pmc xml 6m (100%) successfully parsed 5.75m (95.8%) with pmcid 5.75m (95.8%) with pmid 5.75m (95.8%) with abstract 4.83m (80%) with paragraph 5.75m (95.8%) with references 5.21m (86.8%) next, the parsed text is processed by the finding identification module and the citation sentence extraction module. in terms of finding identification results, approximately 4.25 million articles have at least one finding. on average, we obtained 2.4 findings for each article that had findings, totaling around 10 million findings. for citation sentence extraction results, roughly 3.69 million articles have at least one citation sentence. we obtained 56 citation sentences on average for each article with citation sentences, amounting to approximately 200 million citation sentences in total. after dropping the citations not in europe pmc, roughly 3.28 million articles have at least one citation sentence, with 20 citation sentences on average for each article and 67 million citation sentences in total. finally, the similarity scores were calculated based on these findings and citation sentences. these findings, citation sentences, and similarity scores are then utilized to create the finding-citation graph. we obtained 14.25 million nodes in total, consisting of 4.25 million article nodes and 10 million finding nodes. we got 77 million edges in total, of which there are 67 million edges representing citation relationships. ## conclusion we introduce a new fine-grained citation graph, the finding-citation graph. unlike the traditional citation graph which only contains papers as nodes, the finding-citation graph also includes the findings, representing the results of the academic papers. this graph facilitates more detailed studies at the finding level, such as evaluating findings and tracking the progression of research. we also present a new pipeline for constructing this graph. this pipeline mainly consists of three modules: finding identification, citation sentence extraction, and graph construction. as there is no such pipeline to build the finding-citation graph, we design an evaluation to confirm the graph's quality. the finding identification module achieved 91% accuracy for finding sentence identification and 80% accuracy for multiple findings. the citation sentence extraction module got a 90% f1-score on the paper-level evaluation and 97.7% accuracy on the tuple-level evaluation. the outputs of the two modules are used to construct the graph and confirm its quality. finally, we built a finding-citation graph using europe pmc. our graph comprises 14.25 million nodes, with 4.25 million being academic papers and the rest being findings from those papers. it also includes 76 million edges, with 66 million representing citation relations. the definition and creation of the fcg is an essential step for our future research. we plan to use it to assess research findings from a citation perspective and pinpoint future research directions at the finding level.