Unsupervised Chunking as Syntactic Structure Induction with a Knowledge-Transfer Approach (2021)

Abstract: In this paper, we address unsupervised chunking as a new task of syntactic structure induction, which is helpful for understanding the linguistic structures of human languages as well as processing low-resource languages. We propose a knowledge-transfer approach that heuristically induces chunk labels from state-of-the-art unsupervised parsing models; a hierarchical recurrent neural network (HRNN) learns from such induced chunk labels to smooth out the noise of the heuristics. Experiments show that our approach largely bridges the gap between supervised and unsupervised chunking.

## introduction understanding the linguistic structure of language (e.g., parsing and chunking) is an important research topic in nlp. most previous work employs supervised machine learning methods to predict linguistic structures. while these methods achieve high performance, they need massive data labeled with linguistic structures, such as treebanks @xcite . existing resources are mainly constructed for widely used languages (e.g., english); further constructing new treebanks for lowresource languages is cumbersome and expensive. unsupervised syntactic structure induction has been attracting increasing interest in recent years @xcite . this task concerns discovering linguistic structures of text without using labeled data. it is important to nlp research because it can be potentially used for low-resource languages and also be a first pass in annotating large treebanks for them. moreover, grammar learned by these unsupervised methods shed light on linguistic theories. previous unsupervised syntactic structure mainly focuses on the task of constituency parsing which organize words in a hierarchical manner @xcite . recently, @xcite propose to jointly induce constituency and dependency structures from text. in this work, we address unsupervised chunking, another meaningful task of linguistic structure discovery. the chunking task aims to group the words of a sentence into chunks (roughly speaking, phrases) in a non-hierarchical fashion @xcite , and our setting is to detect chunks without the supervision of annotated linguistic structures. in fact, unsupervised chunking has real-world applications, as understanding text fundamentally requires finding spans like noun phrases and verb phrases. it would benefit various downstream tasks, such as keywords extraction @xcite , named entity recognition @xcite , and open information extraction @xcite . in our paper, we propose a knowledge-transfer approach to unsupervised chunking by hierarchical recurrent neural networks (hrnn). we utilize the recent advances of unsupervised parsers, and propose a maximal left-branching heuristic to induce chunk labels from unsupervised parsing. without any supervision of annotated grammars, such heuristic leads to reasonable (albeit noisy and imperfect) chunks. we further design an hrnn model that learns from the heuristic chunk labels. our hrnn involves a trainable chunking gate that switches between a lower word-level rnn and a upper phrase-level rnn. this explicitly models the composition of words into chunks and chunks into the sentence. results on three datasets show that our hrnn can indeed smooth out the noise of heuristically induced chunk labels, with a considerable improvement in terms of the phrase-f1 score; such observations are consistent in different domains and languages. related work. unsupervised syntactic structure detection has attracted much attention in early nlp research because of its use in low-resource scenarios @xcite . @xcite propose to model constituency and context for each spans with an expectation-maximization (em) algorithm. early work also focuses on unsupervised dependency parsing for syntactic structure induction @xcite . @xcite combine constituency and dependency models via co-training to further boost their performance. to learn the syntactic structures, @xcite propose a probabilistic context-free grammar (pcfg), augmented with manually designed features. @xcite perform clustering by syntactic features to obtain labeled parse trees. @xcite clusters sequences of tags based on their local mutual information to build parse trees. such early studies typically used heuristics, linguistic knowledge, and manually designed features for unsupervised syntactic structure induction @xcite @xcite . in the deep learning era, unsupervised parsing has revived the interest. @xcite propose the recursive autoencoder, where a binary tree is built by greedily minimizing the reconstruction loss. such recursive tree structures can also be learned in an unsupervised way by cyk-style marginalization @xcite and gumbelsoftmax @xcite . @xcite learn a shift-reduce parser by reinforcement learning towards a downstream task. however, evidence shows the above approaches do not yield linguistically plausible trees @xcite . @xcite propose to model the syntactic distance (2018a) or syntactic ordering (2018b) to build parse trees. @xcite propose a compound pcfg for unsupervised parsing. the trees given by these approaches are more correlated with constituency trees. @xcite propose to transfer knowledge among several unsupervised parsers and obtain better performance. our work is inspired by such knowledge transfer, but we propose insightful heuristics that induces chunk labels from unsupervised parsers. we also design hierarchical rnn to learn from induced chunk labels. previous studies address unsupervised chunking as an important task in speech processing; they use acoustic information to determine the chunks @xcite . our work only considers textual information, and views unsupervised chunking as a new task of syntactic structure induction. ## model in this section, we will first induce chunking labels from state-of-the-art unsupervised parsing. then, we will train a hierarchical rnn to learn from induced labels to smooth out the noise. ## experiments setup. we used the @xcite , @xcite , and english web treebank @xcite for evaluation. we compare the model output with groundtruth chunks in terms of phrase f1 and tag accuracy. dataset details and our experimental settings are presented in appendix b. ## conclusion in this paper, we address a new task of syntactic structure discovery, namely, unsupervised chunking. we propose a hierarchical rnn with soft gates to learn from the chunk labels inducted by a stateof-the-art unsupervised parser, compound pcfg. results show that our approach largely bridges the gap between supervised and unsupervised chunking. we also show rigorous analysis on our chunk heuristics and the student model's architecture.