MEGA nno: Exploratory Labeling for NLP in Computational Notebooks (2022)

Abstract: We present MEGAnno, a novel exploratory annotation framework designed for NLP researchers and practitioners. Unlike existing labeling tools that focus on data labeling only, our framework aims to support a broader, iterative ML workflow including data exploration and model development. With MEGAnno’s API, users can programmatically explore the data through sophisticated search and automated suggestion functions and incrementally update task schema as their project evolve. Combined with our widget, the users can interactively sort, filter, and assign labels to multiple items simultaneously in the same notebook where the rest of the NLP project resides. We demonstrate MEGAnno’s flexible, exploratory, efficient, and seamless labeling experience through a sentiment analysis use case.

## introduction data labeling is an important step in the machine learning life cycle since the quality and quantity of training data directly affect the model performance @xcite . unfortunately, existing annotation tools tend to consider the data labeling step in isolation from the broader ml life cycle, ignoring the iterative workflow of researchers and practitioners. however, activities such as data selection, exploratory data analysis, data labeling, model training, and evaluation do not happen sequentially @xcite . instead, continuous iterations are required to improve data, annotation, and models @xcite . to further investigate this gap, we examine the data annotation practices within the ml model development life cycle. based on a formative study with six researchers from our organization, we characterize their annotation practices as a "dual-loop" model shown in fig. 1 . after data preprocessing (fig. 1 1 ## use case: sentiment analysis we present a use case to illustrate how meganno can support data annotation in nlp researchers and practitioners' workflow. meggie is a data scientist who wants to train a sentiment-related model for her project. she obtains a twitter dataset foot_0 about us airlines and decides to label it using meganno. import data in an empty notebook, meggie starts by initializing the project named "tweet sentiment" using a meganno python api. she gets a copy of the data from the product manager who often uses google spreadsheet and imports the data from its published link. explore and annotate she starts by exploring the first 300 data points in the widget's table view. using the search box, she filters the subset with the keyword "amazing". as expected, most of the data records reflect a positive sentiment, so she assigns a positive label to multiple data items using the "bulk label" button. next, she wants to examine tweets with hashtags related to failing, so she tries regular expression search using #fai[^] * (fig. 7 ). with better understanding of the dataset, meggie chooses to perform more advanced exploration using part-of-speech metadata. she imports a pos tagger from spacy @xcite and retrieves tweets that match interesting patterns such as @xmath0. with such an exploratory approach and the batch labeling feature, she can annotate much faster and in a more controlled way. candidate suggestion after a few more heuristics, she runs out of ideas, so she takes advantage of the suggestion feature. she selects the sentence bert @xcite encoder as the meta-data generation function. she first issues a query with strategy similarity to collect data points similar to the ones retrieved from the previous pos search. finally, she wants to see samples from less covered areas in the embedding space to improve the diversity of the training data and issues a coverage query. update the schema meggie is now happy with the collected labels, but she aims to go one step further to understand which words or phrases lead to the sentiment judgment. so she updates her schema by adding a new span-level label called sentiment_span with label options being "happy" or "sad". each data record can have arbitrary numbers of such span-level labels. after updating the schema, she fetches all records with neutral labels in a widget. to highlight and annotate spans, she goes into the single view (as shown in fig. 4 ). at any step of the iteration, she could refer to the dashboard widget to monitor the project progress. after several rounds of similar iterations, she feels good and concludes her exploration. finally, meggie can export the annotated data in json or csv formats for training or plug in the model directly. in conclusion, with meganno, meggie can explore her dataset using various heuristic-based or automated search functions and better understand the data corpus as she labels. she has the flexibility to iteratively update her schema as the project evolves. using the widget, meggie can finish the entire ml life cyle in the same jupyter notebook. ## related work there exist numerous annotation tools that can support nlp tasks, which are extensively surveyed by @xcite . in this section, we focus on works that are closer to our flexible, exploratory, efficient, and seamless framework. flexible schema unfortunately, most of existing tools are not designed for iterative schema development, and thus they are not flexible enough for evolving projects. @xcite and @xcite allow users to progressively define document classes by inspecting documents that are assigned to each class. but these works are more similar to interactive topic modeling or interactive classification, where users assign documents to classes, than document labeling, where users assign labels to documents. our tool goes beyond document label refinement and supports a broader task of progressively defining annotation schema (e.g., additionally collecting span-level labels). ## conclusion in this paper, we present meganno, an annotation framework designed for nlp researchers and practitioners. through meganno's programmatic interfaces and interactive widget, users can iteratively explore and search for interesting data subsets, annotate data, train models, evaluate and debug models within a jupyter notebook without the overhead of context switch or data migration.