Masked Conditional Random Fields for Sequence Labeling (2021)

Abstract: Conditional Random Field (CRF) based neural models are among the most performant methods for solving sequence labeling problems. Despite its great success, CRF has the shortcoming of occasionally generating illegal sequences of tags, e.g. sequences containing an “I-” tag immediately after an “O” tag, which is forbidden by the underlying BIO tagging scheme. In this work, we propose Masked Conditional Random Field (MCRF), an easy to implement variant of CRF that impose restrictions on candidate paths during both training and decoding phases. We show that the proposed method thoroughly resolves this issue and brings significant improvement over existing CRF-based models with near zero additional cost.

## introduction sequence labeling problems such as named entity recognition (ner), part of speech (pos) tagging and chunking have long been considered as fundamental nlp tasks and drawn researcher's attention for many years. traditional work is based on statistical approaches such as hidden markov models @xcite and conditional random fields @xcite , where handcrafted features and task-specific resources are used. with advances in deep learning, neural network based models have achieved dominance in sequence labeling tasks in an end-to-end manner. those models typically consist of a neural encoder that maps the input tokens to embeddings capturing global sequence information, and a crf layer that models dependencies between neighboring labels. popular choices of neural encoder have been convolutional neural network @xcite , and bidirectional lstm @xcite . recently, pretrained language models such as elmo @xcite or bert * corresponding author. @xcite have been proven far superior as a sequence encoder, achieving state-of-the-art results on a broad range of sequence labeling tasks. most sequence labeling models adopt a bio or bioes tag encoding scheme @xcite , which forbids certain tag transitions by design. occasionally, a model may yield sequence of predicted tags that violates the rules of the scheme. such predictions, subsequently referred to as illegal paths, are erroneous and must be dealt with. existing methods rely on hand-crafted post-processing procedure to resolve this problem, typically by retaining the illegal segments and re-tagging them. but as we shall show in this work, such treatment is arbitrary and leads to suboptimal performance. the main contribution of this paper is to give a principled solution to the illegal path problem. more precisely: the remainder of the paper is organized as follows. section 2 describes the illegal path problem and existing strategies that resolve it. in section 3 we propose mcrf, its motivation and an approximate implementation. section 4 is devoted to numerical experiments. we conclude the current work in section 5. ## approach in this section we introduce the motivation and implementation of mcrf. we first go over the conventional neural-based crf models in section 3.1. we then introduce mcrf in section 3.2. its implementation will be given in section 3.3. ## experiments in this section, we run a series of experiments 2 to evaluate the performance of mcrf. the datasets used in our experiments are listed as follows: algorithm 1 (mcrf) 1: input: library for computing the gradients of conventional crf loss (2), training dataset s, stopping criterion c, set of illegal transitions ω, masking constant c 0. 2: initialize: model weight w and tag transition matrix @xmath0. 3: while c is not met do for (i, j) ∈ ω do 7: a ij ← c maintain the mask 8: for chinese ner tasks, we use the publicavailable foot_1 bert base as the pretrained model. for english ner and chunking tasks, we use the cased version of bert base model. we use uncased bert base for english slot filling tasks. the standard fine-tuning as recommended by @xcite . in discriminative fine-tuning, one uses different learning rates for each layer. let r l be the learning rate for the last (l-th) layer and η be the decay factor. then the learning rate for the (l -n)-th layer is given by r l-@xmath1. in our experiments, we use r @xmath2, 5e -5} and η ∈ {1/2, 2/3} depending on the dataset. the standard adam optimizer is used throughout, and the mini-batch size is fixed to be 32. we always finetune for 5 epochs or 10000 iterations, whichever is longer. ## conclusion our major contribution is the proposal of mcrf, a constrained variant of crf that masks illegal transitions during crf training, eliminating illegal outcomes in a principled way. we have justified mcrf from a theoretical perspective, and shown empirically in a number of datasets that mcrf consistently outperforms the conventional crf. as mcrf is easy to implement and incurs zero additional overhead, we advocate always using mcrf instead of crf when applicable.