Pimlico: A toolkit for corpus-processing pipelines and reproducible experiments (2020)

Abstract: We present Pimlico, an open source toolkit for building pipelines for processing large corpora. It is especially focused on processing linguistic corpora and provides wrappers around existing, widely used NLP tools. A particular goal is to ease distribution of reproducible and extensible experiments by making it easy to document and re-run all steps involved, including data loading, pre-processing, model training and evaluation. Once a pipeline is released, it is easy to adapt, for example, to run on a new dataset, or to re-run an experiment with different parameters. The toolkit takes care of many common challenges in writing and distributing corpus-processing code, such as managing data between the steps of a pipeline, installing required software and combining existing toolkits with new, task-specific code.

## introduction it is becoming more and more common for conferences and journals in nlp and other computational areas to encourage, or even require, authors to make publicly available the code and data required to reproduce their reported results. it is now widely acknowledged that such practices lie at the center of open science and are essential to ensuring that research contributions are verifiable, extensible and useable in applications. however, this requires extensive additional work. and, even when researchers do this, it is all too common for others to have to spend large amounts of time and effort preparing data, downloading and installing tools, configuring execution environments and picking through instructions and scripts before they can reproduce the original results, never mind apply the code to new datasets or build upon it in novel research. whilst sometimes it may be sufficient to release a script that performs all of the data processing, model training and experimental evalua-tion steps, often this is not a practical approach to multi-stage processing of large corpora. we present a toolkit, pimlico (pipelined modular linguistic corpus processing), that addresses these problems. it allows users to write and run potentially complex processing pipelines, with the key goals of making it easy to: the toolkit is written in python and released under the open source lgplv3 license foot_0 . it comes with pre-defined modules to wrap a number of existing nlp toolkits (including non-python code) and carry out many other common pre-processing or data manipulation tasks. comprehensive documentation is maintained online foot_1 . in this paper, we describe the core concepts that pimlico is built around and some of its key features. we also describe a number of the core modules that come built into the toolkit and we present an example pipeline. finally, we explain how the toolkit addresses the stated goals and outline plans for future development. ## building pipelines pimlico addresses the task of building of pipelines to process large datasets. it allows you to run one or several steps of processing at a time, with highlevel control over how each step is run, manages the data produced by each step, and lets you observe these intermediate outputs. pimlico provides simple, powerful tools to give this kind of control, without needing to write any code. developing a pipeline with pimlico involves defining the structure of the pipeline itself in terms of modules to be executed and connections between their inputs and outputs describing the flow of data. modules correspond to some data-processing code, with some parameters. they may be of a standard type, so-called core modules, for which code is provided as part of pimlico. a pipeline may also incorporate custom module types, for which metadata and data-processing code must be provided by the author. ## code distribution one of the key problems that pimlico sets out to solve is the difficulty of distributing code in a way that makes it easy for others to reproduce and extend the processing. it achieves this by making the full processing pipeline explicit in the pipeline conf file. it is therefore crucial that (a) it is easy to distribute all the necessary files to re-run a pipeline; and (b) it is easy for someone else, given these files to get the pipeline running. ## core module types pimlico comes with a large number of core module types, for which a pipeline author needs to write no code, but simply define the module configuration in their config file. this set is being constantly expanded. the following list gives some examples of core module types provided with pimlico. the full list is available in the documentation. ## a worked example an example of a full pipeline config file is shown in fig. 2 . this simple pipeline loads a corpus from a directory containing text files, each representing a single document. it applies tokenization to each document using the core document-map module that wraps spacy's tokenizer. then it applies some custom processing to the tokens of each document, using a module type defined specifically for this pipeline and found in the accompanying source directory foot_6 . the resulting corpus is finally passed through the core vocabulary builder, which builds a vocabulary from all the words used in the corpus. ## software licenses pimlico itself is released under the gnu lgplv3 license. however, it provides access to a large number of software packages, with a wide range of different licenses. software dependencies are installed only when required, so use of pimlico does not fall under the terms of all of these -only those required by the modules of the user's pipeline. it can be important to know what licenses apply to all the code used by a pipeline. the pimlico codebase keeps track of the licenses that apply to software that may be installed to support the use of the core module types. the command licenses produces a list of the licenses of all of the software used by a given pipeline, or alternatively just particular modules. ## related toolkits some proprietary tools exist for similar purposes to pimlico foot_7 . however, the use of a proprietary tool to build a pipeline in itself precludes easy replication and extension by other authors, so we focus here on open source tools. two recently released examples of toolkits for building nlp pipelines are forte and psi. forte @xcite is constructed around similar concepts to pimlico and it too provides wrappers around other nlp toolkits. psi @xcite , platform for situated intelligence) is similar in its goals and design to forte. pimlico's focus is on control of the execution of static pipelines to process large datasets and the management of the data as it passes through the pipeline. for these purposes, it provides a powerful set of tools not built into other toolkits. it does not provide facilities to run pipelines in a way that can be dynamically integrated into other systems. we see this as a distinct use case with different design requirements, one that is well catered for by toolkits like forte and psi. many other toolkits focus specifically on nlp tools, allowing models to be trained and applied for standard nlp tasks. some provide their own structures for defining pipelines that chain multiple tasks (e.g., @xcite @xcite . pimlico provides a general framework for processing of large datasets, incorporating nlp tasks by providing wrappers around toolkits such as these. unlike with these toolkits, data loading, pre-and post-processing can be handled in a single pipeline definition, requiring minimal (or no) code to be written. other general toolkits exist for building and running data-processing pipelines, such as bonobo foot_8 . an alternative approach to developing pimlico would have been to define a library of modules for nlp-specific tasks that could be used from such a toolkit. we chose instead to develop an infrastructure tuned to the type of corpus processing and data management that is typical in nlp experiments and tasks. the toolkit effectively addresses common problems encountered in using nlp tools to process large datasets, releasing code for experiments or other corpus processing for others to use, and running someone else's released code in a new environment or on new data. as such, we present it as a key contribution to free distribution of code to accompany nlp research and replicability of experiments.