iobes: Library for Span Level Processing (2020)

Abstract: Many tasks in natural language processing, such as named entity recognition and slot-filling, involve identifying and labeling specific spans of text. In order to leverage common models, these tasks are often recast as sequence labeling tasks. Each token is given a label and these labels are prefixed with special tokens such as B- or I-. After a model assigns labels to each token, these prefixes are used to group the tokens into spans. Properly parsing these annotations is critical for producing fair and comparable metrics; however, despite its importance, there is not an easy-to-use, standardized, programmatically integratable library to help work with span labeling. To remedy this, we introduce our open-source library, iobes. iobes is used for parsing, converting, and processing spans represented as token level decisions.

## introduction tasks like named entity recognition, finding mentions for real world things in text, and slot-filling, finding mentions of relevant objects, often in a dialogue, require identifying contiguous sections of the input text and classifying them into one of several pre-defined classes. while some work solves span labeling by scoring all possible spans, followed by filtering with a threshold @xcite , most work recasts span identification as a token labeling task @xcite @xcite @xcite . special prefixes like b-are used in conjunction with token level type labels, like "per" for person or "org" for organization, to signal where different spans begin and end. once a standard sequence labeling model is used to produce tags for each token, we use these prefixes to convert the token level annotations into spans. for example, the token b-loc means that we need to start a location span here. the tokens labeled with i-loc are continuations of this span. finally, a token of a different type, or the special o label, representing a token that is "outside" of a span, will signal the end of our span. once we have decoded our spans, we often use a metric like exact match f1, where both the span type and the span boundaries have to match, to compare our predicted spans to the reference spans. unfortunately, there are many subtle places where implementations of this approach to span labeling can diverge. the semantics of these special prefixes can change. there are multiple encoding schemes that are all equally expressive but have differences in how easy it is for a model to learn them @xcite . many common datasets are old and therefore distributed in older formats. researchers often convert these datasets to newer formats, but differences or bugs in this process can introduce discrepancies in the data used to train models. policy decisions on handling token level annotations that do not conform to the rules of the encoding scheme also introduce differences that render different models incomparable. the problems mentioned above all stem from the lack of a community wide standardization on how to handle these edge cases. the lack of common tooling for working with these spans, represented as token level annotations, has led to many rolling their own, slightly different implementations. we introduce the iobes library. a python @xcite library that encapsulates all the rough edges of processing and evaluating spans. we aim to provide the community with a single, easy-to-use toolkit whose adoption will ensure the comparability of span labeling metrics reported by different researchers. ## formats there are several forms that span encoding via token labeling can take. later we will see how these multiple formats-and the conversion between them-are the cause of many problems, but for now we will summarize them here. ## discrepancies differences in how processing of these spans is done can cause discrepancies in both the datasets and the evaluation metrics used by researchers. while this section draws examples of errors from specific pieces of work, we would like to emphasize these are not failures on the part of the authors, but rather a failing of the community for not providing tested, reusable tooling. ## our library to help avoid these kind of preventable mistakes, and to create a standardized policy on the creation of entities from illegal tag sequences, we introduce our library iobes. ## conclusion many span level tasks in natural language processing are recast as token-level labeling. there are many encoding schemes used to convert these tokens into spans and these schemes dictate which tokens can follow other ones. unfortunately, processing these tokens is a common place where errors and mistakes manifest. we have shown how mistakes in conversion code, gold annotations, and entity resolution, in the presence of malformed tag sequences, render results incomparable. to remedy these problems, we introduce iobes, a small, well-tested python library that helps standardize the processing of spans. our library helps with parsing token labels into a list of spans, identifying locations of errors in token sequences, converting between span encoding schemes, and enumerating which transitions are allowed and which are not. our library will help avoid these errors and will ensure that results created by different researchers are comparable.