Bridging Pre-trained Language Models and Hand-crafted Features for Unsupervised POS Tagging (2022)

Abstract: In recent years, large-scale pre-trained language models (PLMs) have made extraordinary progress in most NLP tasks. But, in the unsupervised POS tagging task, works utilizing PLMs are few and fail to achieve state-of-the-art (SOTA) performance. The recent SOTA performance is yielded by a Guassian HMM variant proposed by He et al. (2018). However, as a generative model, HMM makes very strong independence assumptions, making it very challenging to incorporate contexualized word representations from PLMs. In this work, we for the first time propose a neural conditional random field autoencoder (CRF-AE) model for unsupervised POS tagging. The discriminative encoder of CRF-AE can straightforwardly incorporate ELMo word representations. Moreover, inspired by feature-rich HMM, we reintroduce hand-crafted features into the decoder of CRF-AE. Finally, experiments clearly show that our model outperforms previous state-of-the-art models by a large margin on Penn Treebank and multilingual Universal Dependencies treebank v2.0.

## introduction unsupervised learning has been an important yet challenging research direction in nlp @xcite @xcite . training models directly from unlabeled data can relieve painful data annotation and is thus especially attractive for low-resource languages @xcite . as three typical tasks related to syntactic analysis, unsupervised part-of-speech (pos) tagging (or induction), dependency parsing, and constituency parsing have attracted intensive interest during the past three decades @xcite . compared with tree-structure dependency and constituency parsing, pos tagging corresponds i looked at my watch . ## vanilla crf-ae in this work, we adopt the crf-ae approach as our basic model for unsupervised pos tagging. the non-neural crf-ae model is first proposed by @xcite for unsupervised sequence labeling tasks, inspired by neural network autoencoders. @xcite also extend the idea to non-neural unsupervised dependency parsing. the basic idea is first producing latent structures, i.e., pos tag sequences, with a discriminative crf over the observed sentence, and then reconstructing the original sentence given each latent structure. the two steps correspond to the encoder and the decoder respectively. training loss. we denote a sentence as @xmath0, x 2 , ⋯, x i , ⋯, x n , and a pos tag sequence as @xmath1, y 2 , ⋯, y i , ⋯, y n . given an unlabeled dataset d which does not contain any pos tag sequences, the training loss is: inference. during evaluation, we follow ammar et al. (2014) and use both the crf and the reconstruction probabilities to obtain the optimal tag sequence: ammar et al. (2014) use hand-crafted discrete features to obtain bigram scores. s (x, y i-1 , y i ; φ) = φ ⊺ g(x, y i-1 , y i , i). (5) decoder: p(x y; θ). the decoder computes the reconstruction probability of x given a pos tag sequence y, which is factorized into position-wise generation probabilities based on a strong independence assumption. p(x y; θ) = n @xmath2; θ). (6) ammar et al. (2014) use a categorical distribution matrix θ, which is updated via em training, to maintain all generation probabilities p(x i y i ), i.e., a word x i generated by a tag y i . ## proposed approach in this work, we for the first time propose a neural crf-ae and leverage plm representations and hand-crafted features for unsupervised pos tagging. ## conclusions this work bridges plms and hand-crafted features for unsupervised pos tagging. based on the crf-ae framework, we employ powerful contextualized representations from plms in the crf encoder, and incorporate rich morphological features for better reconstruction. our proposed approach achieves new sota on 45-tag english ptb and 12tag multilingual ud datasets, outperforming previous results by large margin. experiments and analysis show that rich features and plm representations are critical for the superior performance of our model. meanwhile, simple adjustments of hand-crafted features are key for the success of our model on languages other than english.