Masked Part-Of-Speech Model: Does Modeling Long Context Help Unsupervised POS -tagging? (2022)

Abstract: Previous Part-Of-Speech (POS) induction models usually assume certain independence assumptions (e.g., Markov, unidirectional, local dependency) that do not hold in real languages. For example, the subject-verb agreement can be both long-term and bidirectional. To facilitate flexible dependency modeling, we propose a Masked Part-of-Speech Model (MPoSM), inspired by the recent success of Masked Language Models (MLM). MPoSM can model arbitrary tag dependency and perform POS induction through the objective of masked POS reconstruction. We achieve competitive results on both the English Penn WSJ dataset as well as the universal treebank containing 10 diverse languages. Though modeling the long-term dependency should ideally help this task, our ablation study shows mixed trends in different languages. To better understand this phenomenon, we design a novel synthetic experiment that can specifically diagnose the model’s ability to learn tag agreement. Surprisingly, we find that even strong baselines fail to solve this problem consistently in a very simplified setting: the agreement between adjacent words. Nonetheless, MPoSM achieves overall better performance. Lastly, we conduct a detailed error analysis to shed light on other remaining challenges.

## introduction unsupervised part-of-speech (pos) tagging is the task of discovering pos tags from text without any supervision. these unsupervised syntax induction approaches can reduce the effort needed for collecting expensive syntactic annotation, and can bring us insights about what inductive bias leads to the emergence of syntax. recent pos induction models have made great progress using different frameworks @xcite @xcite ; stratos dt nn in dt nn vbz rb jj . some areas of the factory are particularly dusty . dt nns in dt nn vbp rb jj . some news about the factory is particularly cheerful . et @xcite . however, most of them assume certain independence assumptions among pos tags, e.g., markov @xcite @xcite , unidirectional @xcite , local dependency @xcite , etc. on the contrary, complex and long-term dependency appear in many real languages and plays an important role in defining the pos tags. for example, in figure 1 , the vbp tag of are and the nns tag of areas depend on each other, and so do the vbz tag of is and the nn tag of news. foot_1 so in this case, models only conditioning the immediate preceding tag (markov) or 1-2 neighboring words (local) cannot explain the distinction between nns and nn, or between vbz and vbp. while unidirectional (e.g., using a unidirectional lstm @xcite ) models are in theory capable of modeling long-term dependency through optimizing the joint probability of tags, bidirectional architectures still show clear advantage in language modeling literature @xcite @xcite . in this work, we present a novel framework for pos induction that is capable of modeling arbitrary long-term bidirectional dependencies: masked part-of-speech model (mposm), inspired by recent success of masked language models (mlm) @xcite . specifically, mposm consists of two modules (see figure 2 ): a local pos prediction module that maps each word to its pos tag and a masked pos reconstruction module that masks a certain portion of tags produced from the previous step, and then learns to first predict the masked tags as latent variables and then reconstruct the corresponding words. we use a bidirectional lstm (bi-lstm) to predict the mask tags conditioned on the remaining tags, which grants our model the ability to model complex long-term and bidirectional dependencies among tags. through the training signal back-propagated from this module, the tags predicted from the local pos prediction module will also be encouraged to have global inter-dependency, which leads to better tags. since we do not have gold pos tags, at the masked positions, we marginalize over all possible tags and optimize word reconstruction probabilities. intuitively, the correct induction of pos tags is beneficial for the prediction of the correct masked words. for example, in figure 1 , if we mask the second positions of the two sentences (corresponding to areas and news), inducing two different tags (i.e., nns for areas and nn for news) correctly will make the word prediction easier than inducing the same tag. from a probabilistic view, our model is conceptually similar to approximately modeling the probability of generating the whole sentence from latent tags using masked loss as a surrogate. mposm achieves competitive performance on both the 45-tag english penn wsj dataset @xcite and the 12-tag universal treebank @xcite containing 10 diverse different languages. it achieves comparable oracle m-1 compared to the sota (state-of-theart) models @xcite on penn wsj dataset and achieves higher performance than @xcite on 4 out of 10 languages on the universal treebank. we also show that substantial improvements can be made with the help of contextualized representations in mbert, similar to @xcite . we conduct an ablation study on multiple languages by replacing the bi-lstm architecture with a window-based mlp that models the local dependency of tags. surprisingly, while modeling the full-sentence context can improve the performance of english and german, modeling local context is better for indonesian and korean. our mutual information analysis indicates that this difference may be resulted from the different degrees of gold-tag dependency of different languages. since real-life datasets can contain many con-founding factors, we next design a suite of wellcontrolled synthetic experiments from the angle of agreement-learning to examine how well the model can take advantage of the long-term dependency. our synthetic datasets can guarantee enough training signals for the model to capture the agreements. however, we show that all current models fail to consistently solve the agreement learning problems, even with the most basic agreement happening between adjacent words. nonetheless, our model shows the best performance with the highest percentage of solving these problems in multiple runs. we conjecture that this is relate to the general optimization problems of latent variable inference problems @xcite (see more discussions in section 7). such obstacles prevent models from gaining additional benefits from modeling longterm dependency. finally, we did error analysis on the predicted clusters for english and portuguese and identify remaining challenges both from imperfect modeling and lack of data diversity. in summary, our main contributions are: (1) a novel pos induction architecture with mlminspired loss that allows learning arbitrary tag dependencies and reaches close-to-sota performance; and (2) examining the effectiveness of using long-term context and providing a suite of synthetic datasets to expose the challenges in agreement learning and pointing out future challenges. ## masked part-of-speech model inspired by the recent success in masked language modeling (mlm) @xcite , we present masked part-of-speech model (mposm). next, we will first describe the model architecture and then introduce several useful additional techniques. ## connections to related works the hmm-based pos induction model @xcite has many extensions, including using handengineered linguistic features @xcite , pretrained embeddings @xcite , task-specific modifications @xcite , flow-based transformations @xcite , external resources @xcite @xcite , etc. they all optimize the probability of the sequence, p (x). however, it requires certain dependency assumptions to be tractable. our model instead optimizes the sum of conditional word probabilities given the remaining context i log p (x i |x 1..i-1 , x i+1..l ), i.e., mlm loss @xcite . while being different from p (x), this objective is an effective surrogate and makes modeling complex dependencies possible. there also exist some earlier methods that do not require the markov assumption. for example, @xcite design a method to directly cluster the embeddings containing distributional and morphological information of the word, and then identify prototype words to facilitate the final pos induction. @xcite propose a neural hmm model. similar to our model, it can also model long-term dependency (due to the use of lstm), however, they still constrain the dependence to be uni-directional (due to the hmm nature). our model does not have such constraints and empirically works better. architecture-wise, our model is conceptually similar to a denoising auto-encoder @xcite , where the masking step can be viewed as adding noises to the pos tags. the idea of using auto-encoder models for unsupervised learning has been explored with crfs in @xcite . however, they still require markov independence assumption to make inference on crf tractable, while our model has the ability to model complex long-term dependencies. plus, we use an mlminspired loss instead of reconstructing brown clusters @xcite as @xcite . our model also provide additional insight on the relation between mlm and syntax emergence. such connection has also been explored in previous works. pretrained transformers using mlm @xcite @xcite have shown strong syntactic abilities @xcite @xcite . cbow and skip-gram embeddings @xcite can be viewed as an mlm with a limited context window (i.e., local context), and have been shown to be useful for syntax induction, especially with small window sizes @xcite @xcite . some recent pos induction works explore cbow-style objectives @xcite . however, using the sentence-level mlm objective for syntax induction is under-explored. the only exception is the recent work of @xcite , which focuses on a different task: unsupervised parsing. the different tasks lead to substantially different focuses and designs in the architecture. they use mlm with a dependency-constrained self-attention mechanism to extract parses, while we extend mlm to the pos-tag level (mposm) and explicitly discretizes the latent variables to extract tags. ## conclusion we propose mposm, a pos induction model inspired by mlm and can model complex long-term dependencies between pos tags. our model shows competitive performance on both english and multilingual datasets. we analyze the effectiveness of using long context compared to local context. finally, we use synthetic datasets and analyses to point out remaining challenges.