  Linear chain conditional random fields (CRFs) combined with contextual word
embeddings have achieved state of the art performance on sequence labeling
tasks. In many of these tasks, the identity of the neighboring words is often
the most useful contextual information when predicting the label of a given
word. However, contextual embeddings are usually trained in a task-agnostic
manner. This means that although they may encode information about the
neighboring words, it is not guaranteed. It can therefore be beneficial to
design the sequence labeling architecture to directly extract this information
from the embeddings. We propose locally-contextual nonlinear CRFs for sequence
labeling. Our approach directly incorporates information from the neighboring
embeddings when predicting the label for a given word, and parametrizes the
potential functions using deep neural networks. Our model serves as a drop-in
replacement for the linear chain CRF, consistently outperforming it in our
ablation study. On a variety of tasks, our results are competitive with those
of the best published methods. In particular, we outperform the previous state
of the art on chunking on CoNLL 2000 and named entity recognition on OntoNotes
5.0 English.
