Synthetic vs. Real Reference Strings for Citation Parsing, and the Importance of Re-training and Out-Of-Sample Data for Meaningful Evaluations: Experiments with GROBID , GIANT and CORA (2020)

Abstract: Citation parsing, particularly with deep neural networks, suffers from a lack of training data as available datasets typically contain only a few thousand training instances. Manually labelling citation strings is very time-consuming, hence synthetically created training data could be a solution. However, as of now, it is unknown if synthetically created reference-strings are suitable to train machine learning algorithms for citation parsing. To find out, we train Grobid, which uses Conditional Random Fields, with a) human-labelled reference strings from ‘real’ bibliographies and b) synthetically created reference strings from the GIANT dataset. We find that both synthetic and organic reference strings are equally suited for training Grobid (F1 = 0.74). We additionally find that retraining Grobid has a notable impact on its performance, for both synthetic and real data (+30% in F1). Having as many types of labelled fields as possible during training also improves effectiveness, even if these fields are not available in the evaluation data (+13.5% F1). We conclude that synthetic data is suitable for training (deep) citation parsing models. We further suggest that in future evaluations of reference parsers both evaluation data similar and dissimilar to the training data should be used for more meaningful evaluations.

## introduction accurate citation data is needed by publishers, academic search engines, citation & research-paper recommender systems and others to calculate impact metrics @xcite , rank search results @xcite , generate recommendations @xcite @xcite @xcite and other applications e.g. in the field of bibliometric-enhanced information retrieval @xcite . citation data is often parsed from unstructured bibliographies found in pdf files on the web (figure 1 ). to facilitate the parsing process, a dozen @xcite open source tools were developed including parscit @xcite , grobid @xcite , and cermine @xcite . grobid is typically considered the most effective one @xcite . there is ongoing research that continuously leads to novel citation-parsing algorithms including deep learning algorithms @xcite @xcite @xcite @xcite and meta-learned ensembles @xcite . most parsing tools apply supervised machine learning @xcite and require labelled training data. however, training data is rare compared to other disciplines where datasets may have millions of instances. to the best of our knowledge, existing citation-parsing datasets typically contain a few thousand instances and are domain specific (figure 2 ). this may be sufficient for traditional machine learning algorithms but not for deep learning, which shows a lot of potential for citation parsing @xcite @xcite @xcite . even for traditional machine learning, existing datasets may not be ideal as they often lack diversity in terms of citation styles. recently, we published giant, a synthetic dataset with nearly 1 billion annotated reference strings @xcite . more precisely, the dataset contains 677,000 unique reference strings, each in around 1,500 citation styles (e.g. apa, harvard, acm). the dataset was synthetically created. this means, the reference strings are not 'real' reference strings extracted from 'real' bibliographies. instead, we downloaded 677,000 references in xml format from crossref, and used citeproc-js (frank g. bennett, 2011) with 1,500 citation styles to convert the 677,000 references into a total of 1 billion annotated citation strings (1,500 * 677,000). 2 we wonder how suitable a synthetic dataset like giant is to train machine learning models for citation parsing. therefore, we pursue the following research question: potentially, synthetic data could lead to higher citation parsing performance, as synthetic datasets may contain more data and more diverse data (more citation styles). synthetic datasets like giant could potentially also advance (deep) citation parsing, which currently suffers from a lack of 'real' annotated bibliographies at large scale. in addition to the above research question, we aimed to answer the following questions: is it important to have many different fields (author, year, . . . ) for training, even if the fields are not available in the final data? ## related work we are aware of eleven datasets (figure 2 ) with annotated reference strings. the most popular ones are probably cora and citeseer. researchers also often use variations of pubmed. several datasets are from the same authors, and many datasets include data from other datasets. for instance, the grobid dataset is based on some data from cora, pubmed, and others @xcite . new data is continuously added to grobid's dataset. as such, there is not "the one" grobid dataset. giant @xcite is the largest and most diverse dataset in terms of citation styles, but giant is, as mentioned, synthetically created. cora is one of the most widely used datasets but has potential shortcomings @xcite @xcite . cora is homogeneous with citation strings only from computer science. it is relatively small and only has labels for "coarse-grained fields" @xcite . for example, the author field does not label each author separately. prasad et al. conclude that a "shortcoming of [citation parsing research] is that the evaluations have been largely limited to the cora dataset, which is [...] unrepresentative of the multilingual, multidisciplinary scholastic reality" @xcite . ## methodology to compare the effectiveness of synthetic vs. real bibliographies, we used grobid. grobid is the most effective citation parsing tool @xcite and the most easy to use tool based on our experience. grobid uses conditional random fields (crf) as machine learning algorithm. of course, in the long-run, it would be good to conduct our experiments with different machine learning algorithms, particularly deep learning algorithms, but for now we concentrate on one tool and algorithm. given that all major citation-parsing tools -including grobid, cermine and parscit -use crf we consider this sufficient for an initial experiment. also, we attempted to re-train neural parscit @xcite but failed doing so, which indicates that the ease-of-use of the rather new deep-learning methods is not yet as advanced as the established citation parsing tools like grobid. we trained grobid, the crf respectively, on two datasets. train grobid denotes a model trained on 70% (5,460 instances) of the dataset that grobid uses to train its out-of-the box version. we slightly modified the dataset, i.e. we removed labels for 'pubplace', 'note' and 'institution' as this information is not contained in giant, and hence a model trained on giant could not identify these labels foot_3 . train giant denotes the model trained on a random sample (5,460 instances) of giant's 991,411,100 labeled reference strings. our expectation was that both models would perform similar, or, ideally, train giant would even outperform train grobid . to analyze how the amount of training data affects performance, we additionally trained train giant , on 1k, 3k, 5k, 10k, 20k, and 40k instances of giant. we evaluated all models on four datasets. eval grobid comprises of the remaining 30% of grobid's dataset (2,340 reference strings). eval cora denotes the cora dataset, which comprises, after some cleaning, of 1,148 labelled reference strings from the computer science domain. eval giant comprises of 5,000 random reference strings from giant. these three evaluation datasets are potentially not ideal as evaluations are likely biased towards one of the two trained models. evaluating the models on eval giant likely favors train giant since the data for both train giant and eval giant is highly similar, i.e. it originates from the same dataset. similarly, evaluating the models on eval grobid likely favors train grobid as train grobid was trained on 70% of the original grobid dataset and this 70% of the data is highly similar to the remaining 30% that we used for the evaluation. also, the cora dataset is somewhat biased, because grobid's dataset contains parts of cora. we therefore created another evaluation dataset. eval webpdf is our 'unbiased' dataset with 300 manually annotated citation strings from pdfs that we collected from the web. to create eval webpdf , we chose twenty different words from the homepages of some universities 4 . then, we used each of the twenty words as a search term in google scholar. from each of these searches, we downloaded the first four available pdfs. of each pdf, we randomly chose four citation strings. this gave approximately sixteen citation strings for each of the twenty keywords. in total, we obtained 300 citation strings. we consider this dataset to be a realistic, though relatively small, dataset for citation parsing in the context of a web-based academic search engine or recommender system. we measure performance of all models with precision, recall, f1 (micro average) and f1 (macro average) on both field level and token level. we only report 'f1 macro average on field level' as all metrics led to similar results. all source code, data (including the webpdf dataset), images, and an excel sheet with all results (including precision and recall and token level results) is available on github 4 the words were: bone, recommender systems, running, war, crop, monetary, migration, imprisonment, hubble, obstetrics, photonics, carbon, cellulose, evolutionary, revolutionary, paleobiology, penal, leadership, soil, musicology. figure 3: f1 of the two models (train grobid and train giant ) on the four evaluation datasets. https://github.com/beelgroup/giant-the-1billion-annotated-synthetic-bibliographic-reference-string-dataset/. ## results the models trained on grobid (train grobid ) and giant (train giant ) perform as expected when evaluated on the three 'biased' datasets eval grobid , eval cora and eval giant (figure 3 ). when evaluated on eval grobid , train grobid outperforms train giant by 35% with an f1 of 0.93 vs. 0.69. when evaluated on eval giant , results are almost exactly the opposite: this time, train giant outperforms train grobid by 32% with an f1 of 0.91 vs. 0.69. on eval cora , the difference is less strong but still notable. train grobid outperforms train giant by 19% with an f1 of 0.74 vs. 0.62. this is not surprising as grobid's training data includes some cora data. while these results generally might not be surprising, they imply that both synthetic and real data lead to very similar results and 'behave' similarly when used to train models that are evaluated on data being (not) similar to the training data. also interesting is the evaluation on the webpdf dataset. the model trained on synthetic data (train giant ) and the model trained on real data (train grobid ) perform alike with an f1 of 0.74 each (figure 3 ) 5 . in other words, synthetic and humanlabelled data perform equally well for training our machine learning models. looking at the data in more detail reveals that some fields are easier to parse than others (figure 4 ). for instance, the 'date' field (i.e. year of publication) has a constantly high f1 across all models and evaluation datasets (@xmath0.86; @xmath1.0). the 'author' field also has a high f1 throughout all experiments (@xmath2.75; @xmath3.99). in contrast, parsing 'booktitle' and 'publisher' seems to strongly benefit from training based on samples similar to the evaluation data. when evaluation and training data is highly similar (e.g. train giant -eval giant or train grobid -eval grobid ), f1 is relatively high (typically above 0.7). if the evaluation data is different (e.g. train giant -eval grobid ), f1 is low (0.15 and 0.16 for train grobid and train giant respectively on eval webpdf ). the difference in f1 for parsing the book-title is around factor 6.5, with an f1 of 0.97 (train grobid ) and 0.15 respectively (train giant ) when evaluated on eval grobid . similarly, f1 for parsing the book-title on eval giant differs by around factor 3 with an f1 of 0.75 (train giant ) and 0.27 (train grobid ) respectively. while it is well known, and quite intuitive, that different fields are differently difficult to parse, we are first to show that field accuracy varies for different fields differently depending on whether or not the model was trained on data (not) being similar to the evaluation data. in a side experiment, we trained a new model train grobid + with additional labels for institution, note and pubplace (those we removed for the other experiments). train grobid + outperformed train grobid notably with an f1 of 0.84 vs. 0.74 (+13.5%) when evaluated on eval webpdf . this indicates that the more fields are available for training, the better the parsing of all fields becomes even if the additional fields are not in the evaluation data. this finding seems plausible to us and confirms statements by anzaroot and mccallum but, to the best of our knowledge, we are first to quantify the benefit. it is worth noting that citation parsers do not always use the same fields (figure 6 ). for instance, cermine extracts relatively few fields, but is one of few tools extracting the doi field. our assumption that more training data would generally lead to better parsing performance -and hence giant could be useful for training standard machine learning algorithms -was not confirmed. increasing training data from 1,000 to 10,000 instances improved f1 by 6% on average over the four evaluation datasets (figure 5 ). more precisely, increasing data from 1,000 to 3,000 instances improved f1, on average, by 2.4%; increasing from 3,000 to 5,000 instances improved f1 by another 2%; increasing further to 10,000 instances improved f1 by another 1.6%. however, increasing to 20,000 or 40,000 instances leads to no notable improvement, and in some cases even to a decline in f1 (figure 5 ). ## summary and discussion in summary, both models -one trained on synthetic data (giant) and one trained on 'real' humanannotated reference strings (grobid) -performed very similar. on the main evaluation dataset (webpdf) both models achieved an f1 of 0.74. similarly, if a model was evaluated on data different from its training data, f1 was between 0.6 and 0.7. if a model was evaluated on data similar to the training data, f1 was above 0.9 (+30%). f1 only increased up to a training size of around 10,000 instances (+6% compared to 1,000 instances). additional fields (e.g. pubplace) in the training data increased f1 notably (+13.5%), even if these additional fields were not in the evaluation data. these results lead us to the following conclusions. first, there seems to be little benefit in using synthetic data (e.g. giant @xcite ) for training traditional machine learning models (i.e. conditional random fields). the existing datasets with a few thousand training instances seem sufficient. second, citation parsers should, if possible, be (re)trained on data that is similar to the data that should actually be parsed. such a re-training increased performance by around 30% in our experiments. this finding may also explain why researchers often report excellent performance of their tools and approaches with e.g. f1's of over 0.9. these researchers typically evaluate their models on data highly similar to the training data. this might be considered a realistic scenario for those cases when re-training is possible. however, reporting such results creates unrealistic expectations for scenarios without the option to re-train, i.e. for users who just want to use a citation parser like grobid out-of-the-box. therefore, we propose that future evaluations of citation parsing algorithms should be conducted on at least two datasets: one dataset that is similar to the training dataset, and one out-of-sample dataset that differs from the training data. third, citation parsers should be trained with as many labelled field types as possible, even if these fields will not be in the data that should be parsed. such a fine-grained training improved f1 by 13.5% in our experiments. fourth, having ten times as much training data (10,000 vs. 1,000) improved the parsing performance by 6%, without notable improvements beyond 10,000 instances. annotating a few thousand instances should be feasible for many scenarios. hence, businesses and organizations who want the maximum accuracy should annotate their own data for training as this likely will lead to large increases in accuracy (+30%, see conclusion 3). fifth, given how similar synthetic and traditionally annotated data perform, synthetic data likely is suitable to train deep neural networks for citation parsing. this, of course, has yet to be empirically to be shown. however, if our assumption holds true, deep citation parsers could greatly benefit from synthetic data like giant. for the future, we see the need to extend our experiments to different machine learning algorithms and datasets (e.g. unarxive @xcite or core @xcite ). it would also be interesting to analyze if and to what extend synthetic data could improve related disciplines. this may include citation-string matching, i.e. analyzing whether two different reference strings refer to the same document @xcite , or the extraction of mathematical formulae @xcite or titles @xcite from scientific articles.