More Embeddings, Better Sequence Labelers? (2020)

Abstract: Recent work proposes a family of contextual embeddings that significantly improves the accuracy of sequence labelers over non-contextual embeddings. However, there is no definite conclusion on whether we can build better sequence labelers by combining different kinds of embeddings in various settings. In this paper, we conduct extensive experiments on 3 tasks over 18 datasets and 8 languages to study the accuracy of sequence labeling with various embedding concatenations and make three observations: (1) concatenating more embedding variants leads to better accuracy in rich-resource and cross-domain settings and some conditions of low-resource settings; (2) concatenating contextual sub-word embeddings with contextual character embeddings hurts the accuracy in extremely low-resource settings; (3) based on the conclusion of (1), concatenating additional similar contextual embeddings cannot lead to further improvements. We hope these conclusions can help people build stronger sequence labelers in various settings.

## introduction in recent years, sequence labelers equipped with contextual embeddings have achieved significant accuracy improvement @xcite @xcite over approaches that use static non-contextual word embeddings @xcite and character embeddings @xcite . different types of embeddings have different inductive biases to guide the learning process. however, little work has been done to study how to concatenate these contextual embeddings and non-contextual embeddings to build better sequence labelers in * yong jiang and kewei tu are the corresponding authors. ‡ : this work was conducted when xinyu wang was interning at alibaba damo academy. multilingual, low-resource, or cross-domain settings over various sequence labeling tasks. in this paper, we empirically investigate the effectiveness of concatenating various kinds of embeddings for multilingual sequence labeling and try to answer the following questions: ## experiments and results for simplicity, we use m to represent m-bert embeddings, f to represent flair embeddings, w to represent fasttext embeddings, c to represent non-contextual character embeddings, all to represents the concatenation of all types of embeddings and the operator "+" to represent the concatenation operation. we use the maxent approach for all experiments foot_2 . due to the space limit, some detailed experiment settings, extra experiments and discussions are included in the appendix. ## conclusion in this paper, we analyze how to get a better sequence labeler by concatenating various kinds of embeddings. we make several empirical observations that we hope can guide future work to build better sequence labelers: (1) in most settings, concatenating more embedding variants leads to better results, while in extremely low-resource settings, only using cses and nwes performs better; (2) nces become less effective when concatenated with contextual embeddings, while nwes are still beneficial; (3) neural models can automatically learn which embeddings are beneficial to the task; (4) additionally concatenating similar contextual embeddings with the best concatenations from ( 1 ) cannot further improve the accuracy in most cases. a. beygelzimer, f. d alché-buc, e. fox, and r. garnett, editors, advances in neural information processing systems 32, pages 8024-8035. curran associates, inc. jeffrey pennington, richard socher, and christopher manning. 2014. glove: global vectors for word representation. in proceedings of the 2014 conference on empirical methods in natural language processing (emnlp), pages 1532-1543. matthew peters, mark neumann, mohit iyyer, matt gardner, christopher clark, kenton lee, and luke zettlemoyer. 2018. deep contextualized word representations. in proceedings of the 2018 conference of the north american chapter of the association for computational linguistics: human language technologies, volume 1 (long papers), pages 2227-2237, new orleans, louisiana. association for computational linguistics. telmo pires, eva schlinger, and dan garrette. 2019. how multilingual is multilingual bert? in proceedings of the 57th annual meeting of the association for computational linguistics, pages 4996-5001, florence, italy. association for computational linguistics. alec radford, karthik narasimhan, tim salimans, and ilya sutskever. improving language understanding by generative pre-training. cícero dos santos and victor guimarães. 2015. boosting named entity recognition with neural character embeddings. in proceedings of the fifth named entity workshop, pages 25-33, beijing, china. association for computational linguistics. cicero d santos and bianca zadrozny. 2014. learning character-level representations for part-of-speech tagging. in proceedings of the 31st international conference on machine learning (icml-14), pages 1818-1826. rico sennrich, barry haddow, and alexandra birch. 2016. neural machine translation of rare words with subword units. in proceedings of the 54th annual meeting of the association for computational linguistics (volume 1: long papers), pages 1715-1725, berlin, germany. association for computational linguistics. erik f. tjong kim sang. 2002. introduction to the conll-2002 shared task: language-independent named entity recognition. in coling-02: the 6th conference on natural language learning 2002 (conll-2002). erik f. tjong kim sang and fien de meulder. 2003. introduction to the conll-2003 shared task: language-independent named entity recognition. in proceedings of the seventh conference on natural language learning at hlt-naacl 2003, pages 142-147. ashish vaswani, noam shazeer, niki parmar, jakob uszkoreit, llion jones, aidan n gomez, łukasz kaiser, and illia polosukhin. 2017. attention is all you need. in advances in neural information processing systems, pages 5998-6008. yonghui wu, mike schuster, zhifeng chen, quoc v le, mohammad norouzi, wolfgang macherey, maxim krikun, yuan cao, qin gao, klaus macherey, et al. 2016. google's neural machine translation system: bridging the gap between human and machine translation. arxiv preprint arxiv:1609.08144. jie yang, shuailong liang, and yue zhang. 2018. design challenges and misconceptions in neural sequence labeling. in proceedings of the 27th international conference on computational linguistics, pages 3879-3889, santa fe, new mexico, usa. association for computational linguistics.