Training for G ibbs Sampling on Conditional Random Fields with Neural Scoring Factors (2020)

Abstract: Most recent improvements in NLP come from changes to the neural network architectures modeling the text input. Yet, state-of-the-art models often rely on simple approaches to model the label space, e.g. bigram Conditional Random Fields (CRFs) in sequence tagging. More expressive graphical models are rarely used due to their prohibitive computational cost. In this work, we present an approach for efficiently training and decoding hybrids of graphical models and neural networks based on Gibbs sampling. Our approach is the natural adaptation of SampleRank (Wick et al., 2011) to neural models, and is widely applicable to tasks beyond sequence tagging. We apply our approach to named entity recognition and present a neural skip-chain CRF model, for which exact inference is impractical. The skip-chain model improves over a strong baseline on three languages from CoNLL-02/03. We obtain new state-of-the-art results on Dutch.

## introduction complex probabilistic graphical models were widely adopted for nlp tasks before the prevalence of deep learning (e.g. the skip-chain crf of @xcite and @xcite for ner). although modern neural architectures are able to learn much better feature representations (e.g. the contextualized word representations of @xcite , @xcite , and @xcite ) than the hand-crafted features used classically in graphical model's log-linear potentials, these advances in feature learning do not negate the need for modeling the output label space. consider two contrasting approaches to structured prediction: transition-based models and graphical models. transition-based models (e.g. the sequence-to-sequence models of @xcite ) have enjoyed recent success thanks to their ability to have unbounded memory of past transitions when predicting subsequent ones; yet because no conditional independence assumptions are made, inference is typically restricted to (heuristic) greedy search and its variants. by contrast, graphical models make strong conditional independence assumptions, but enjoy a wealth of inference algorithms, both exact and approximate, as a result. moreover, graphical models readily admit the incorporation of domain knowledge about interactions between the output variables. in this paper, we focus on this latter approach to modeling. specifically, we explore conditional random fields (crfs) @xcite with neural potential functions. prior state-of-the-art approaches utilizing such models (e.g. crf-lstms) for sequence tagging tasks like named entity recognition (ner) have focused on simple linear-chain crfs, which only model bi-gram dependencies of adjacent labels @xcite , and the exact inference can be done in polynomial time with dynamic programming. by contrast, we are motivated by crfs that do not admit exact inference. we propose neural samplerank, a novel algorithm that is computationally efficient for approximate inference and training of complex crfs (where exact inference is impractical) with neural factors. the main inspiration of our work is @xcite , a training algorithm for complex graphical models based on gibbs sampling, that has been shown to work well with linear factors. we extend samplerank to work with neural scoring factors. neural samplerank enables us to use crfs that are far more expressive than the linear-chain structures seen in ner models. the loss does not require full inference to compute the gradient in training, which makes it more computationally efficient. comparing with message-passing based algorithms like loopy belief propagation (lbp), neural samplerank is conceptually simpler and easier to implement with modern deep learning tools like pytorch @xcite . we empirically evaluate neural samplerank on the conll-02 and conll-03 ner task on english, german and dutch @xcite . we show that for linear-chain bilstm-crf model ( lample et al., 2016 ), training with neural samplerank achieves competitive f1 score compared with mle and exact inference. with a new neural skip-chain crf model trained with neural samplerank, we achieve higher f1 on english and german than all existing models that do not use contextualized word embeddings or external labeled data. with contextualized word embeddings, our skip-chain model obtains new state-ofthe-art results on dutch. ## related work various approaches have been taken in nlp to combine graphical models and neural architectures. for sequence tagging tasks like ner, it is common to use a linear-chain crf model @xcite , for which exact inference can be done in polynomial time with forward-backward. @xcite adopt a factorized crf to model the output space of morphological tagging, and the exact inference is tractable with belief propagation. @xcite propose a fully connected binary crf to model mention sequence for entity linking task, and they use loopy belief propagation for approximate inference. other approaches have been proposed to adopt expressive graphical models while keeping the inference computationally feasible, but have not been applied to deep neural networks. @xcite propose to select non-local contexts while keeping the model feasible for exact inference. @xcite use gibbs sampling with simulated annealing for fast approximate inference for models with non-local factors. @xcite propose a skip-chain crf for ner learned with loopy belief propagation. @xcite propose a new training objective targeted for samplingbased inference which is efficient both in terms of computation cost and task performance. in prior work, gibbs sampling has been used with deep neural networks for bayesian posterior inference @xcite , and sampling from conditional sequence models @xcite . gibbs sampling was only widely applied to discriminative models before the prevalence of deep learning, and restricted to generative models when used with neural models @xcite @xcite . to the best of knowledge, we are the first to use gibbs sampling to obtain point estimation for neural network graphical model hybrids, for the task of structured prediction. state-of-the-art approaches for ner all use a simple linear-chain crf model to model the label space. neural architectures to learn a better representation of the text input include bi-directional lstm @xcite , gru @xcite and character cnn @xcite . a major recent step in the field is contextualized word embedding like elmo @xcite , bert @xcite and the character-based flair @xcite . however, none of these approaches model longer range context dependencies in the document, limited by the linear-chain structure of the crf. ## conclusion in this work, we have proposed neural samplerank (nsr), an efficient algorithm for approximate inference and training for crf models with neural network factors. with a novel skip-chain crf model that models long range context dependencies, nsr can significantly improve ner performance over the linear-chain crf on multiple datasets. nsr is computationally efficient for arbitrarily complex graphical models, thus applicable to a wide range of structured prediction tasks. graphical models with task specific inductive bias have been successful for tasks like ner, coreference resolution, relation extraction, and parsing. our proposed method paves the way for new neural graphical models to be designed for these tasks.