Acknowledgement Entity Recognition in CORD -19 Papers (2020)

Abstract: Acknowledgements are ubiquitous in scholarly papers. Existing acknowledgement entity recognition methods assume all named entities are acknowledged. Here, we examine the nuances between acknowledged and named entities by analyzing sentence structure. We develop an acknowledgement extraction system, AckExtract based on open-source text mining software and evaluate our method using manually labeled data. AckExtract uses the PDF of a scholarly paper as input and outputs acknowledgement entities. Results show an overall performance of F_1=0.92. We built a supplementary database by linking CORD-19 papers with acknowledgement entities extracted by AckExtract including persons and organizations and find that only up to 50–60% of named entities are actually acknowledged. We further analyze chronological trends of acknowledgement entities in CORD-19 papers. All codes and labeled data are publicly available at https://github.com/lamps-lab/ackextract .

## introduction acknowledgements have been an institutionalized part of research publications for some time @xcite . acknowledgement statements show the authors' public gratitude and recognition to individuals, organizations, and grants for various contributions. acknowledged individuals and organizations have been under-presented in author ranking and citation impact analysis mostly due to their presumed sub-authorship contribution. a recent survey found that discipline, academic rank, and gender have a significant effect on coauthorship disagreement rate @xcite , leading to non-author collaborators receiving less attention. recently, the presence of non-author collaborators in the biomedical and social sciences @xcite showed that non-author collaborators are not rare and their presence varies significantly by disciplines. acknowledgements can be classified depending on the nature of the contribution. @xcite classified sentences in acknowledgement sections into 6 categories: declaration, financial, peer interactive communication and technical support, presentation, general acknowledgement, and general statement. they can also be classified based on the type of entities such as individual, organization, and grant. since 2008, funding acknowledgements have been indexed by web of science. however, there is still no dedicated software to accurately recognize acknowledged people and organizations and generate a centralized acknowledgement database. early works on acknowledgements were based on datasets manually extracted from specific journals, which was not scalable. building such a large database can support further study of acknowledgements at a larger scale. there are several scenarios that make the acknowledgement entity recognition (aer) task challenging. the upper panel of figure 1 shows examples of sentences appearing in an isolated acknowledgement section. the "utrecht university" is mentioned but should not be counted as an acknowledgement entity because it is just the affiliation of "arno van vliet" who is acknowledged. acknowledgement statements can also appear at footnotes (figure 1 bottom), mixed with other footnote and/or body text. author names may also appear in the statements, such as "andreoni" in this example, and should be excluded. existing works on aer leverage off-the-shelf named entity recognition (ner) packages, such figure 1 : upper: acknowledgement statements appear in an isolated section @xcite . bottom: acknowledgement statements appear in a footnote (andreoni and gee, 2015). as the natural language toolkits (nltk) @xcite e.g., @xcite ; paul-hus et al. (2020), followed by simple semi-manual data cleansing, resulting in a fraction of entities that are mentioned but not actually acknowledged. in this paper, we design an automatic aer system called ackextract that further classifies extraction results from open source ner packages that recognize people and organizations and distinguish entities that are actually acknowledged. the extractor finds acknowledgement statements from isolated sections and other locations such as footnotes, which is common for papers in social and behavioral sciences. our contributions are: 1. develop ackextract as open-source software to automatically extract acknowledgement entities from research papers. 2. apply ackextract on the cord-19 dataset and supplement the dataset with a corpus of classified acknowledgement entities for further studies. 3. use the cord-19 dataset as a case study to demonstrate that acknowledgement studies without classifying named entities, can significantly overestimate the number of entities that are actually acknowledged because many people and organizations are mentioned but not explicitly acknowledged. early work on acknowledgement extraction was manually applied, which was labor-intensive. @xcite extracted a total of 9561 peer interactive communication (pic) names from a total of 4200 research sociology articles, most were persons' names. they also defined the following six categories of acknowledgement: moral support, financial support, editorial support, presentational support, instrumental/technical support, and conceptual support, or pic @xcite . @xcite used a hybrid method for automatic aer from research papers and automatically created an acknowledgement index @xcite . the algorithm first used a heuristic method for identifying acknowledgement passages. it then uses an svm model for identifying lines containing acknowledgement sentences outside labeled acknowledgement sections. a regular expression was used to extract entity names from acknowledging text. this method achieved an overall precision of about 0.785 and a recall of 0.896 on citeseer papers @xcite . the algorithm does not distinguish entity types. @xcite leveraged opencalais foot_0 and alchemyapi 2 , free services at that time, to extract named entities from acknowledgement sections and built ackseer, a search engine for acknowledgement entities. they merged outputs of both ner apis and generated a list and disambiguated entity mentions using the longest common subsequence (lcs) algorithm. the ground truth contains 200 top-cited citeseerx papers in which 130 had acknowledgement sections. they achieved 92.3% and 91.6% precision and recall for acknowledgement section extraction but did not evaluate entity extraction. recent studies of acknowledgements tend to use results from off-the-shelf ner packages with simple filters, assuming that named entities were acknowledged entities. for example, paul-hus et al. (2020) uses the stanford ner module in nltk to extract persons. @xcite also directly use people and organizations recognized by the stanford corenlp @xcite . these works achieved a high recall by recognizing most name entities in the acknowledgements but ignored their relations to the papers where they appear, resulting in a fraction of entities that are mentioned but not actually acknowledged. @xcite consider grammar structure such as verb tense and voice and sentence patterns when labeling sentences to their six categories. for example, "was funded" is followed by an "organization". however, they only label sentences and do not annotate them down to the entity level. our system examines the relationship between entities and the current work, with the purpose of discriminating acknowledgement entities from named entities. in this system, we focus on people and organizations. recently, @xcite proposed grantextractor, a pipeline system to extract grant support information from scientific papers. a model combining bilstm-crf and pattern matching was used to extract entities of grant numbers and agencies from funding sentences, which are identified using heuristic methods. the system achieves a micro-f 1 up to 0.90 in extracting grant pairs (agency, number). @xcite proposed an ensemble approach called fundingfinder for extracting funding information from text. the authors construct feature vectors for candidate entities using whether the entities are recognized by four ner implementation: stanford (conditional random field model), lingpipe (hidden markov model), opennlp (maximum entropy model), and elsevier's fingerprint engine. the f 1 -measure for funding body is only 0.68 ± 0.3. our method is different from existing methods in threefold. (1) it is built on top of state-of-the-art neural ner methods, which results in a relatively high recall. (2) it uses a heuristic method to filter out entities that are just mentioned but not acknowledged. (3) it extracts both organizations and people. ## dataset on march 16th, 2020, allen institute of artificial intelligence, released the first version of the covid-19 open research dataset (cord-19) @xcite , in collaboration with several other institutions. the dataset contains metadata and segmented full text of research articles selected by searching a list of keywords about coronavirus, sars-cov, mers, and other related terms from four digital libraries including who, pubmed central, biorxiv, and medrxiv. the initial dataset contained about 28k papers and was updated weekly with papers from new sources and the latest publication. we used the dataset released on april 10, 2020 containing over 59,312 metadata records, among which 54,756 have the full text in json format. the cord-19 papers were generated by processing pdfs using the s2ord pipeline @xcite , in which grobid @xcite was employed for document segmentation and metadata extraction. the full text in json files is directly used for sentence segmentation. however, we observe that grobid extraction results are not perfect. in particular, we estimate the fraction of acknowledgement sections omitted. we also estimate the number of acknowledgement entities omitted in the data release due to the extraction error of grobid. to do this, we downloaded 45,916 full-text pdf papers collected by the internet archive (ia) because the cord-19 dataset does not include pdf files foot_2 . we found 13,103 cord-19 papers in the ia dataset. ## data analysis the cord-19 dataset contains pmc and pdf folders. we found that almost all papers in the pmc folders are included in the pdf folders. for consistency, we only work on papers in the pdf folders, containing 37,612 unique pdf papers 7 . using ackextract, we extracted a total of 102,965 named entities from 21,469 cord-19 papers. the fraction of papers with named entities (21469/37612≈57%) is roughly consistent with the fraction obtained from the 200 samples (120/200≈60% in section 4.2). among all named entities, 58,742 are acknowledgement entities. these numbers suggest that using our model, only about 50% of named entities are acknowledged. using only named entities, acknowledgement studies could significantly overestimate the 7 we excluded 2003 duplicate papers with exactly the same sha1 values. algorithm 1: relation-based classifier figure 6 shows the numbers of cord-19 papers with vs. without acknowledgement (person or organization) recognized from 1970 to 2020. the huge leap around 2002 was due to the wave of coronavirus studies during the sars period. the small drop-down in 2020 was due to data incompleteness. the plot indicates that the fraction of papers with acknowledgement has been increasing gradually over the last 20 years. figure 7 shows the numbers of the top 10 acknowledged organizations from 1983 to 2020. the figure indicates that the number of acknowledgements to nih has been gradually decreasing over the past 10 years while the acknowledgements to nsf is roughly constant. in contrast, the number has been gradually increasing from nsfc (a chinese funding agency). note that the distribution of acknowledged organizations has a long tail and organizations behind the top 10 actually dominate the total number. however, the top 10 organizations are the biggest research agencies and the trend to some extent reflects strategic shifts of funding support. ## conclusion and future work here, we extended the work of @xcite and built an acknowledgement extraction framework denoted as ackextract for research articles. ackextract is based on heuristic methods and state-of-the-art text mining libraries (e.g., grobid and stanza) but features a classifier that discriminates acknowledgement entities from named entities by analyzing the multiple subjectpredicate-object relations in a sentence. our approach successfully recognizes acknowledgement entities that cannot be recognized by oie packages such as reverb, ollie, and the allennlp srl library. this method is applied to the cord-19 dataset released on april 10, 2020, processing one pdf document in 5 seconds on average. our results indicate that only 50-60% named entities are acknowledged. the rest are mentioned to provide additional information (e.g., affiliation or location) about acknowledgement entities. working on clean data, our method achieves an overall f 1 = 0.92 for person and organization entities. the trend analysis of the cord-19 papers verifies that more and more papers include acknowledgement entities since 2002, when the sars outbreak happened. the trend also reveals that the overall number of acknowledgements to nih is gradually decreasing over the past 10 years, while more papers acknowledge nsfc, a chinese funding agency. one caveat of our method is that organizations in different countries are not distinguished. for example, many countries have agencies called "ministry of health". in the future, we plan to build learning-based models for sentence classification and entity classification. the code and data of this project have been released on github at: https://github.com/lamps-lab/ackextract .