An Investigation of Potential Function Designs for Neural CRF (2020)

Abstract: The neural linear-chain CRF model is one of the most widely-used approach to sequence labeling. In this paper, we investigate a series of increasingly expressive potential functions for neural CRF models, which not only integrate the emission and transition functions, but also explicitly take the representations of the contextual words as input. Our extensive experiments show that the decomposed quadrilinear potential function based on the vector representations of two neighboring labels and two neighboring words consistently achieves the best performance.

## introduction sequence labeling is the task of labeling each token of a sequence. it is an important task in natural language processing and has a lot of applications such as part-of-speech tagging (pos) @xcite @xcite , named entity recognition (ner) @xcite , chunking @xcite . the neural crf model is one of the most widelyused approaches to sequence labeling and can achieve superior performance on many tasks @xcite @xcite @xcite . it often employs an encoder such as a bilstm to compute the contextual vector representation of each word in the input sequence. the potential function at each position of the input sequence in a neural crf is typically decomposed into an emission function (of the current label and the vector representation of the current word) and a transition function (of the previous and current labels) @xcite . ## models our overall neural network architecture for sequence labeling is shown in figure 1 . it contains three parts: a word representation layer, a bi-directional lstm (bilstm) encoder, and an inference layer. the bilstm encoder produces a sequence of output vectors h 1 , h 2 , ...h @xmath0, which are utilized by the inference layer to predict the label sequence. the inference layer typically defines a potential function s(x, y, i) for each position i of the input sequence x and label sequence y and computes the conditional probability of the label sequence given the input sequence as follows: vanilla crf in most previous work of neural crfs, the potential function is decomposed to an emission function and a transition function (figure 2(b) ), and the transition function is represented by a table φ maintaining the transition scores between labels. twobilinear instead of one-hot vectors, we may use dense vectors to represent labels, which has the benefit of encoding similarities between labels. accordingly, the emission and transition functions are modeled by two bilinear functions. threebilinear figure 2 (c) depicts the structure of threebilinear. compared with twobilinear, threebilinear has an extra emission function between the current word representation and previous label. trilinear instead of three bilinear functions, we may use a trilinear function to model the correlation between h i , t y i and t y i-1 . it has strictly more representational power than the sum of three bilinear functions. d-trilinear despite the increased representational power of trilinear, its space and time complexity becomes cubic. to reduce the computational complexity without too much compromise of the representational power, we assume that u has rank d r and can be decomposed into the product of three matrices u t 1 , u t 2 ∈ r dt×dr and u @xmath1. then the trilinear function can be rewritten as, d-quadrilinear we may take the representation of the previous word as an additional input and use a quadrilinear function in the potential function. we call the resulting model d-quadrilinear and its factor graph is shown in figure 2 (e). d-pentalinear following the same idea, we extend d-quadrilinear to d-pentalinear by taking the representation of the next word as an additional input. figure 2 (f) shows the structure of d-pentalinear. ## conclusion in this paper, we investigate several potential functions for neural crf models. the proposed potential functions not only integrate the emission and transition functions, but also take into consideration representations of additional neighboring words. our experiments show that d-quadrilinear achieves the best overall performance. our proposed approaches are simple and effective and could facilitate future research in neural sequence labeling. manning. 2014. glove: global vectors for word representation. in proceedings of emnlp. thai-hoang pham, xuan-khoai pham, tuan-anh nguyen, and phuong le-hong. 2017. nnvlp: a neural network-based vietnamese language processing toolkit. in proceedings of the ijcnlp 2017, system demonstrations, pages 37-40, tapei, taiwan. association for computational linguistics. lev ratinov and dan roth. 2009. design challenges and misconceptions in named entity recognition. in proceedings of the thirteenth conference on computational natural language learning (conll-2009), pages 147-155, boulder, colorado. association for computational linguistics. nils reimers and iryna gurevych. 2017. reporting score distributions makes a difference: performance study of lstm-networks for sequence tagging. in proceedings of the 2017 conference on empirical methods in natural language processing, pages 338-348, copenhagen, denmark. association for computational linguistics. alan ritter, sam clark, mausam, and oren etzioni. 2011. named entity recognition in tweets: an experimental study. in proceedings of the 2011 conference on empirical methods in natural language processing, pages 1524-1534, edinburgh, scotland, uk. association for computational linguistics. jun suzuki, erik mcdermott, and hideki isozaki. 2006. training conditional random fields with multivariate evaluation measures. in proceedings of the 21st international conference on computational linguistics and 44th annual meeting of the association for computational linguistics, pages 217-224, sydney, australia. association for computational linguistics. erik f. tjong kim sang. 2002. introduction to the conll-2002 shared task: language-independent named entity recognition. in coling-02: the 6th conference on natural language learning 2002 (conll-2002). erik f. tjong kim sang and sabine buchholz. 2000. introduction to the conll-2000 shared task chunking. in fourth conference on computational natural language learning and the second learning language in logic workshop. erik f. tjong kim sang and fien de meulder. 2003. introduction to the conll-2003 shared task: language-independent named entity recognition. in proceedings of the seventh conference on natural language learning at hlt-naacl 2003, pages 142-147. kristina toutanova, dan klein, christopher d. manning, and yoram singer. 2003. feature-rich part-ofspeech tagging with a cyclic dependency network. in proceedings of the 2003 human language technology conference of the north american chapter of the association for computational linguistics, pages 252-259. yingwei xin, ethan hart, vibhuti mahajan, and jean-david ruvini. 2018. learning better internal structure of words for sequence labeling. in proceedings of the 2018 conference on empirical methods in natural language processing, pages 2584-2593, brussels, belgium. association for computational linguistics. jie yang, shuailong liang, and yue zhang. 2018. design challenges and misconceptions in neural sequence labeling. in proceedings of the 27th international conference on computational linguistics, pages 3879-3889, santa fe, new mexico, usa. association for computational linguistics.