Bootstrapping Multilingual Metadata Extraction: A Showcase in C yrillic (2021)

Abstract: Applications based on scholarly data are of ever increasing importance. This results in disadvantages for areas where high-quality data and compatible systems are not available, such as non-English publications. To advance the mitigation of this imbalance, we use Cyrillic script publications from the CORE collection to create a high-quality data set for metadata extraction. We utilize our data for training and evaluating sequence labeling models to extract title and author information. Retraining GROBID on our data, we observe significant improvements in terms of precision and recall and achieve even better results with a self developed model. We make our data set covering over 15,000 publications as well as our source code freely available.

## introduction the use of scholarly data becomes more and more important as the rate of academic publications keeps increasing and automated processing gains relevance, such as scientometric analysis and scholarly recommendation @xcite . consequentially, limitations of scholarly data and approaches based thereon directly translate into disadvantages for the affected publications, in terms of, for example, discoverability and impact. one particular limitation of scholarly data nowadays is an underrepresentation of non-english content @xcite . while supporting multiple languages poses challenges, such as language-specific preprocessing requirements @xcite , disregarding non-english work is problematic @xcite . to further the availability of high-quality scholarly data beyond the anglophone publication record, we showcase the creation and application of a data set for training and evaluating sequence labeling tasks on cyrillic publications. recent years have seen an increased focus on multilinguality in natural language processing approaches, such as language models @xcite and data sets @xcite . furthermore, there are efforts to specifically support languages that use non-latin scripts @xcite . with regards to cyrillic script languages, approaches concerned with named entity linking in web documents @xcite , as well as approaches to extracting keywords from scientific texts @xcite exist. model training for these types of information extraction tasks is increasingly done using automatically generated high-quality training data. this has, for example, been done for tasks such as text extraction from scholarly pdf files @xcite , identification of publication components such as figures and tables in scanned documents @xcite , and the parsing of bibliographic references @xcite . we extend this approach to non-english scholarly data. to this end, we use cyrillic script documents from the core data set @xcite to train and evaluate sequence labeling mod-els for identifying publications' metadata (title and authors) in unlabeled text, as illustrated in figure 1 . overall, the contributions we make with this paper are as follows. ## application to assess the utility of our data set, we use it to retrain grobid @xcite , a widely used metadata extraction tool @xcite , as well as a standalone sequence labeling model, and evaluate their performance against an off-theshelf version of grobid. ## evaluation to assess the performance of both the off-theshelf and retrained grobid as well as the standalone bilstm model, we perform five-fold crossvalidations and measure the overall precision, recall, and f1 score. 12 because grobid retraining is only possible on roughly two thirds of our data (see section 3.1) we evaluate the off-the-shelf ("vanilla") grobid model on the same subset in order to maximize comparability of the evaluation results. regarding the comparability to our standalone bilstm model, a key difference lies in the fact that we use four labels (author, b-title, i-title and misc) instead of grobid's two (author and title). to adjust for this difference, we decide to disregard the misc label and combine the two types of *-title label by a weighted average. the overall evaluation scores resulting from this are shown in for further assessment of the bilstm model's performance, we evaluate its predictions per language as shown in ## conclusion inspired by recent approaches creating high-quality data for training and evaluating information extraction tasks involving scholarly publications, we utilize this approach to tackle the problem of underrepresented non-english scholarly (training) data. to this end, we use cyrillic script documents found in the core data set to train sequence labeling models for identifying publications' metadata. we create a data set of 15,553 papers spanning 27 years and three languages. using this data set, we retrain grobid and thereby greatly improve its performance. furthermore, we train and evaluate a separate sequence labeling model that is less constrained by pdf parsing restrictions (see section 3.1), showing even better overall performance results than the retrained grobid model. by showcasing the use of freely available non-english publications to improve the availability of high-quality data and models covering areas beyond the anglophone publication record, we hope to inspire similar efforts for other languages. for our own approach, we plan to extend it to the extraction of bibliographic references in the future.