S ci WING – A Software Toolkit for Scientific Document Processing (2020)

Abstract: We introduce SciWING, an open-source soft-ware toolkit which provides access to state-of-the-art pre-trained models for scientific document processing (SDP) tasks, such as citation string parsing, logical structure recovery and citation intent classification. Compared to other toolkits, SciWING follows a full neural pipeline and provides a Python inter-face for SDP. When needed, SciWING provides fine-grained control for rapid experimentation with different models by swapping and stacking different modules. Transfer learning from general and scientific documents specific pre-trained transformers (i.e., BERT, SciBERT, etc.) can be performed. SciWING incorporates ready-to-use web and terminal-based applications and demonstrations to aid adoption and development. The toolkit is available from http://sciwing.io and the demos are available at http://rebrand.ly/sciwing-demo .

## introduction automated scientific document processing (sdp) deploys natural language processing (nlp) on scholarly articles. as scholarly articles are long-form, complex documents with conventional structure and cross-reference to external resources, they require specialized treatment and have specialized tasks. representative sdp tasks include parsing embedded reference strings @xcite ; identifying the importance, sentiment and provenance for citations @xcite ; identifying logical sections and markup @xcite ; parsing of equations, figures and tables @xcite ; and article summarization (qazvinian and radev, 2008; qazvinian 1 watch our demo video at https://rebrand.ly/ sciwing-video figure 1 : sciwing components: text classification and sequence labelling datasets, models composed from low-level modules, and engine to train and record experiment parameters. infer middleware does the inference and high-level functionality (e.g., developing apis; low-level and web applications). @xcite @xcite . sdp tasks, in turn, help downstream systems and assist scholars in finding relevant documents and manage their knowledge discovery and utilization workflows. next-generation introspective digital libraries such as semantic scholar @xcite , google scholar and microsoft academic have begun to incorporate such services. while nlp, in general, has seen tremendous progress with the introduction of neural network architectures and general toolkits and datasets to leverage them, their deployment for sdp is still limited. over the past few years, many opensource software packages have accelerated the development of state-of-the-art (sota) nlp models. however, these frameworks have a few limitations concerning sdp. first, most are general purpose frameworks aimed at producing sota models for natural language understanding tasks or only for specific domains such as biomedicine. second, they do not provide deployable, sota models for sdp. most provide limited or no means for researchers to train models on their datasets, or experiment with model architectures. a key barrier to entry is accessibility: a nontrivial level of expertise in nlp and machine learning is a prerequisite. practitioners who wish to deploy sdp on their field's literature may lack knowledge and motivation to learn it for the sake of deployment. thus, there is a clear need for a toolkit that unifies different efforts and provides access to pre-trained, sota models for sdp, while also allowing researchers to experiment with models rapidly to create deployable applications. we introduce sciwing to address this need with respect to other frameworks (cf. sciwing is mit licensed and comes with downloadabled pre-trained models and freelyavailable datasets. the package runs on python 3.7 and can be easily installed from python packaging index (pypi) using pip install sciwing. for researchers aiming to further develop sciwing, we provide installation tools that set up the system, alongside documentation. ## system overview our view is that sdp-specific considerations are best embodied as an abstraction layer over exist-ing nlp frameworks. sciwing incorporates al-lennlp, the generic nlp pipeline @xcite , developing models on top of it when necessary, while using the transformers package @xcite to enable transfer learning via its pretrained general-purpose representations such as bert @xcite and sdp-specific ones like scibert @xcite . fig. 1 shows sciwing's dataset, model and engine components facilitating flexible re-configuration. we now describe these components. datasets: there are many challenges for the researcher-practitioner to experiment with different sdp tasks. first, researchers must handle various data formats and datasets: for reference string parsing, the conll format is most common; for text classification, csv is most common. sciwing enables reading of dataset files in different formats and also facilitates the download of open datasets using command-line interfaces. for example, sciwing download data --task scienceie downloads the official, openly-available dataset for the scienceie task. current methods for pre-processing are cumbersome and error-prone. processing can become complex when different models require different tokenisation and numericalisation methods. sciwing unifies these various input formats through a pipeline of pre-processing, tokenisation and numericalisation, via tokenisers and numericalisers. sciwing also handles batching and padding of examples. models: paired embedder-encoder subcomponents combine to form a neural network model, themselves pytorch classes. embedders: modern nlp models represent nat-ural language tokens as continuous vectors -embeddings. sciwing abstracts this concept via embedders. generic (non-sdp specific) embeddings such as glove @xcite are natively provided. tokens in scientific documents can benefit from special attention, as most are missing from pre-trained embeddings. sciwing includes task-specific trained embeddings for reference strings @xcite . sciwing also supports contextual word embeddings: elmo @xcite , bert @xcite , scibert @xcite , etc. sota embedding models are built by concatenating multiple representations, via sciwing's concatembedders module. as an example, word and character embeddings are combined in ner models @xcite , and multiple contextual word embeddings are combined in various clinical and bionlp tasks @xcite . neural network encoders: sciwing ports commonly-used neural network components that can be composed to form neural architectures for different tasks. for example in text classification, encoding input sentence as a vector using an lstm is a common task (sciwing's seq2vecencoder). another common operation is obtaining a sequence of hidden states for a set of tokens, often used in sequence labelling tasks and sciwing's lstm2seq achieves this. further, it also includes attention based modules. sciwing builds in generic linear classification and sequence labelling with crf heads that can be attached to the encoders to build production models. it provides pretrained sota models for particular sdp tasks that work out-of-the-box or further fine-tuned. engine: sciwing handles all the boilerplate code to train the model, monitor the loss and metrics, check-pointing parameters at different stages of training, validation and testing. it helps researchers adopt best practices, such as clipping gradient-norms, as well as saving and deploying best performing models. users can customize the following: optimisers: sciwing supports all the optimisers supported by pytorch, and various learning rate schedulers that dynamically manage learning rates based on validation performance. experiment logging: sciwing adopts current best practices in leveraging logging tools to monitor and manage experiments. sciwing writes logs for every experiment run and facilitates cloud-based experiment logging and corresponding charting of relevant metrics via the third-party api service of weights and biases 2 , with the integration of alternative logging services on the way. metrics: different sdp tasks require their respective metrics. sciwing abstracts a separate metrics module to select appropriate metrics for each task. ## use cases sciwing caters to both use cases of practitioners looking to deploy pre-trained models as well as researchers looking to refine model architectures and perform fine-tuning domain adaptation on top of state-of-the-art contextual word embedding models. we now examine both use cases. ## related work grobid @xcite is the closest to a general workbench for scientific document processing. similarly to sciwing, grobid also performs document structure classification, reference string parsing, among other tasks. but grobid is architected in the traditional, manual feature engineering approach, leading to performance losses for many sdp tasks, and difficulties in retrofitting neural models into its framework. scispacy @xcite focuses on biomedical related tasks such as pos-tagging, syntactic parsing and biomedical span extraction. however, scispacy primarily caters for practitioners; it does not easily allow for the development and testing of new models and architectures. task-and domain-agnostic frameworks also exist. ncrf++ @xcite is a tool for performing sequence tagging using neural networks and conditional random fields and flair @xcite is a framework for generalpurpose nlp and mainly provide access to different embeddings and ways to combine them. ## conclusion and future work we introduce sciwing, an open-source scholarly document processing (sdp) toolkit, targeted at practitioners and researchers interested in rapid experimentation. it provisions pre-trained models for key sdp tasks that achieve state-of-the-art performance and aids practitioners to deploy models directly on their community's literature. sciwing's modular design also greatly facilitates sdp researchers in model architecture development, speeding train/test cycles for architecture search, and supporting transfer learning for use cases with limited annotated data. sciwing allows declaration of models, datasets and experiment parameters in a single configuration file. sciwing is actively being developed. we consider the following improvements in our roadmap: