Label Sleuth: From Unlabeled Text to a Classifier in a Few Hours (2022)

Abstract: Label Sleuth is an open source platform for building text classifiers which does not require coding skills nor machine learning knowledge.- Project website: [ https://www.label-sleuth.org/ ]( https://www.label-sleuth.org/ )- Link to screencast video: [ https://vimeo.com/735675461 ]( https://vimeo.com/735675461 )### AbstractText classification can be useful in many real-world scenarios, saving a lot of time for end users. However, building a classifier generally requires coding skills and ML knowledge, which poses a significant barrier for many potential users. To lift this barrier we introduce *Label Sleuth*, a free open source system for labeling and creating text classifiers. This system is unique for: - being a no-code system, making NLP accessible for non-experts. - guiding its users throughout the entire labeling process until they obtain their desired classifier, making the process efficient - from cold start to a classifier in a few hours. - being open for configuration and extension by developers. By open sourcing Label Sleuth we hope to build a community of users and developers that will widen the utilization of NLP models.

## introduction text classification is an nlp task with great practical importance. practitioners working with large amounts of textual data often need to categorize snippets of text. for instance, a lawyer reviewing contracts may need to find clauses specifying the terms under which a contract can be terminated. or, a psychologist analyzing treatment notes may be interested in finding all sentences that indicate that a patient is suffering from depression. often, the text snippets of interest are rare and scattered throughout the dataset. manually reviewing the entire dataset is inefficient or impractical, thus raising the need for an automated solution in the form of a custom text classification model. practitioners, or domain experts (a.k.a subject matter experts) who need such models, typically lack the skills to build them, and thus must rely on machine learning (ml) experts. this, in turn, creates a gap between modern text classification * these authors contributed equally to this work. † work done while author was working at ibm research. we present label sleuth foot_0 -an open source foot_1 system designed to enable domain experts to create a text classifier by themselves, with no dependency on ml experts. label sleuth is both a labeling platform and a machine learning platform, and is thus used to collect labeled data as well as to build text classifiers. it enables a domain expert to create a good quality custom classifier from a cold start (no labels) in a few hours, in several short rounds of labeling -enhanced via active learning @xcite -that provide feedback to models being trained in the background. label sleuth was designed to be intuitive and easy to use by domain experts. rather than trying to cover many different nlp tasks and increasing the system complexity, it focuses on a single broadly applicable use case of binary text classification, and provides a fully automated flow for building such classifiers. to the best of our knowledge, label sleuth is the first text classification platform intentionally designed for a broad audience -domain experts that typically lack coding skills or an understanding of ml concepts. by open sourcing label sleuth, we hope the community will join this effort, to further expand and improve its existing capabilities for the benefit of a potentially wide community of users. ## system comparison text labeling (or annotation) tools have proliferated in recent years. @xcite surveyed 78 tools. they can be classified into two categories: basic labeling tools simply allow users to assign a label(s) to data elements. examples include early tools, such as callisto @xcite , brat @xcite , and webanno @xcite , and more recent ones, such as doccano @xcite . labeling tools with ml support are more similar to label sleuth, since in addition to collecting labels, they train a classifier with these labels, or accelerate the labeling process by integrating ml. in our comparison, we focus on representative systems that are most similar to label sleuth and have gained users popularity. all these systems offer some form of ml labeling support, however, they are designed with technical users in mind, such as data scientists and developers; they often require complex actions to get started, which assume ml knowledge. they do not offer ml integration outof-the-box, relying instead on the user to configure the system (e.g., by connecting it to external models). furthermore, ml support is typically limited to active learning and lacks advanced features that could help domain experts, such as identifying and guiding the user in resolving potential labeling issues. we next provide a brief overview of each of the reviewed systems. prodigy @xcite is a paid, closed source labeling tool by the makers of spacy. while it offers an intuitive frontend, it targets mainly data scientists, as most tasks (except for basic labeling) -including dataset upload -require using the command-line. moreover, it does not show examples in context and thus the user must label them in isolation from their source document, and according to a predefined order. label studio @xcite is offered in a free open source community edition and a paid enterprise edition. while the latter offers ml and active learning support, setting up the process requires invoking external models (which in their simplest form are pre-built container images). inception @xcite ) -an open source labeling tool from tu darmstadt -is arguably the most configurable tool in the list. it enables finegrained control of several aspects of the labeling process, including the label granularity and when model predictions are shown. however, this customizability further increases the barrier to entry compared to other tools. even setting up a classification task requires creating complex annotation layers, while integrating a model, in many cases, requires the use of external libraries. in contrast to label sleuth, these systems support nlp tasks other than text classification, such as ner and question answering, or even nontextual tasks, such as audio and image classification. thus, label sleuth and these systems correspond to different points on the trade-off between ease of use and task support. existing systems support a wide variety of tasks but assume a technical user, while label sleuth focuses on text classification but creates an end-to-end model building experience tailored specifically to non-technical users. we believe that it is important to have tools that strike different balances in this trade-off. ## architecture label sleuth is composed of backend and frontend layers. the backend is written in python and uses the flask framework for exposing a web service; the frontend is a react application which uses the mui design library. for additional details see our architecture webpage. while label sleuth is well-suited to users with no ml background, it also offers configurability and extensibility options for advanced users. users can choose from the available models and active learning strategies, and can also contribute new ones by implementing one or two straightforward functions. in addition, it is possible to configure the system to dynamically switch between models and/or strategies as the labeling progresses. large models that require a gpu are also supported. another extensible component is training set selection. while a basic approach would be to train classifiers using the set of examples labeled by the user, more advanced methods can provide added benefits. the default setting leverages the fact that the negative prior is high (since positive examples are relatively rare), and randomly selects elements from the unlabeled set to be added as weak negative examples for training. the various system configurations (e.g., classification model, active learning strategy, criterion to trigger the training of a new model) together constitute a policy that shapes the flow and experience of building a classifier. the default policy (see app. a) can be extended and modified to further improve efficiency or to support different scenarios. the data access layer is responsible for saving and exposing the dataset and user labels. the current implementation relies on a combination of in-memory for performance, and local disk storage for persistency. finally, while english is used as the default language, label sleuth provides an infrastructure to easily support other languages. ## open source and research opportunities label sleuth is the product of a collaboration between industry and academia, and aims to continue evolving by leveraging insights from multiple stakeholders and perspectives. we welcome further contributions and feedback from domain experts and the open source community, as well as researchers in related fields, including natural language processing and human-computer interaction. to facilitate this, label sleuth was released in july 2022 as open source under the apache 2.0 license. following the example of other successful projects, in addition to the source code of the system, the open source release includes material aimed to facilitate the use of the system and contributions to its development. the material on the project's website includes an overview of the system (including a short video), quick installation instructions, and a walk-through tutorial tailored to domain experts (building upon the animal habitat scenario and dataset of § 2). there is also detailed documentation of the system's internals for open source contributors and/or researchers who want to understand the underlying techniques and extend the system for their own needs. as detailed in § 4, the system is highly extensible, allowing researchers to further improve the system by incorporating novel techniques. a research aspect that we believe will be of particular interest to the nlp community is the unique requirements that arise from the interactive nature with the non-technical target audience of label sleuth. we next outline a few examples of such requirements, which we hope the nlp community will contribute solutions to. ## conclusions label sleuth is a production-ready freely available open-source system that seeks to lower the accessibility barrier for domain experts to label and build text classifiers. it provides unique opportunities for a more productive and efficient classifier building process -one where the system guides the labeling process, and both the domain expert and the ml components can provide timely feedback to each other. we encourage domain experts, the open source community, and researchers to use, extend, and contribute back to the label sleuth project.