C o-training an U nsupervised C onstituency P arser with W eak S upervision (2022)

Abstract: We introduce a method for unsupervised parsing that relies on bootstrapping classifiers to identify if a node dominates a specific span in a sentence. There are two types of classifiers, an inside classifier that acts on a span, and an outside classifier that acts on everything outside of a given span. Through self-training and co-training with the two classifiers, we show that the interplay between them helps improve the accuracy of both, and as a result, effectively parse. A seed bootstrapping technique prepares the data to train these classifiers. Our analyses further validate that such an approach in conjunction with weak supervision using prior branching knowledge of a known language (left/right-branching) and minimal heuristics injects strong inductive bias into the parser, achieving 63.1 F _1 on the English (PTB) test set. In addition, we show the effectiveness of our architecture by evaluating on treebanks for Chinese (CTB) and Japanese (KTB) and achieve new state-of-the-art results.

## introduction pre-trained language models (plms) have become a standard tool in the natural language processing (nlp) toolkit, offering the benefits of learning from large amounts of unlabeled data while providing modular function in many nlp tasks that require supervision. recent work has shown that plms capture different types of linguistic regularities and information, for instance, the lower layers capture phrase-level information which becomes less prominent in the upper layers @xcite , span representations constructed from these models can encode rich syntactic phenomena, like the ability to track subject-verb agreement @xcite , dependency trees can be embedded within the geometry of bert's hidden states @xcite , and most relevantly to this paper, syntactic information via self-attention mechanisms @xcite . we offer another perspective on the way plms represent syntactic information. we demonstrate the usability of plms to capture syntactic information by developing an unsupervised parsing model that makes heavy use of plms. the learning algorithm is light in the injection of hard bias to parse text, emphasizing the role of plms in capturing syntactic information. our approach to unsupervised parsing is inspired by recent work in the area of spectral learning for parsing @xcite and unsupervised estimation of probabilistic context-free grammars (pcfgs; @xcite . at its core, our learning algorithm views the presence or absence of a node dominating a substring in the final parse tree as a latent variable, where patterns of co-occurrence of the string that the node dominates (the "inside" string) and the rest of the sentence (the "outside" string) dictate whether the node is present or not. with spectral learning for latent-variable pcfgs (l-pcfgs; @xcite the notion of inside trees versus out-side trees is important, but in our case, given that the trees are not present during learning, we have to further specialize it to extract information only from the strings. consider the diagram of a syntax tree in figure 1 , decomposed into two parts. following the main notion in spectral learning, each of these parts (the orange part and the blue part) is a "view" of the whole tree that provides information on the identity of the node that spans the words x i • • • x j . in the case of the tree being unobserved during training, we have to rely only on the substrings that are spanned by the blue part or the orange part, to hypothesize whether indeed a node exists there. to represent the inside and outside views, we make use of plms. we encode these substrings, and then bootstrap a classifier that determines whether a given span is a constituent or not. the bootstrapping process alternates between the two views, and at each point adds predictions on the training set that it is confident about to train a new classifier. this can be thought of as a form of co-training @xcite , a training technique that relies on multiple views of training instances. we formulate the task of identifying constituents and distituents (referring to spans that are not constituents) in a sentence as a binary classification task by devising a strategy to convert the unlabeled data into a classification task. firstly, we build a sequence classification model by fine-tuning a transformer-based plm on the unlabeled training sentences to distinguish between the true and false inside strings of constituents. secondly, we use the highly-confident inside strings to produce the outside strings. additionally, through the use of semi-supervised learning techniques, we jointly use both the inside and outside passes to enrich the model's ability to determine the breakpoints in a sentence. our final model achieves 63.1 sentence f 1 averaged over multiple runs with random seed on the penn treebank test set. we also report strong results for the japanese and chinese treebanks. ## problem formulation and inference we give a treatment to the problem of unsupervised constituency parsing. in that setup, the training algorithm is given an unlabeled corpus (set of sentences) and its goal is to learn a function mapping a sentence x to an unlabeled phrase-structure tree y that indicates the constituents in x. in previous work with models such as the constituent-context model (ccm; @xcite , the dependency model with valence (dmv; @xcite , and unsupervised maximum likelihood estimator for data-oriented parsing (uml-dop; bod 2006), the parts of speech (pos) of the words in x are also given as input both during inference and during training, but we do not make use of such pos tags. inference while our learning algorithm is grammarless, for inference we make use of a dynamic programming algorithm, akin to cyk, to predict the parse tree. inference assumes that each possible span in the tree was scored with a score function s(i, j) where i and j are endpoints in the sentence. the score function is learned through our algorithm. we then proceed by finding the tree t * such that: when s(i, j) is the probability of a span (i, j) being in the correct tree, this formulation gives the tree with the highest expected number of correct constituents @xcite . this formulation has been used recently by several unsupervised constituency parsing algorithms @xcite @xcite . ## training algorithm at the core of our approach lies the notion of inside and outside strings. for a given sentence @xmath0, j), the inside string of span (i, j) is the sequence we denote by h in (i, j) representations for inside strings and h out (i, j) representations for outside strings. both are vectors derived from a plm (roberta; liu et al. 2019, as we see later). these two types of strings provide two views of a given possible splitting point in the syntax tree. we offer three ways, with increasing complexity, to bootstrap a score function that helps identify whether a node should dominate a given span. the main idea behind this bootstrapping is to start with a small seed set of training examples (x, i, j, b) where (i, j) is a span in a sentence x and @xmath1, 1}, depending on whether the span (i, j) is dominated by a node in the syntactic tree or not. bootstrapping the seed set is dependent only on either the inside string or the outside string, and the corresponding classifier built from this bootstrapped seed set returns a probability p(b | x, i, j). once a classifier is learned using the bootstrapping seed set, the classifier is applied on the training set, and the seed set is added to more examples where the classifier is confident of the label b. this is also known as self-training @xcite . in the next three sections, we present three learning algorithms of increasing complexity in their use of inside and outside strings. ## experimental setup in this section, we describe our experimental setup: the data we use, the exact details of the experimental use of our approach to unsupervised parsing, and our evaluation methodology. ## results and discussion ## related work recently, neural network-based approaches have shown promising results on inducing parse trees directly from words. our weakly-supervised parser is comparable in behavior to a fully unsupervised parser as it does not rely on syntactic annotations. we highlight some themes most relevant to our method. learning from distant supervision. a related work to ours @xcite uses answer fragments and webpage hyperlinks to mine syntactic constituents for parsing. many previous studies depend on punctuation as a strong signal to detect constituent boundaries @xcite . ## conclusion we propose a simple yet effective method of inducing constituency trees which is the first of its kind in achieving performance comparable to the supervised binary tree rnng model and setting a new state-of-the-art result for unsupervised parsing using weak supervision. our model generalizes to multiple languages of known treebanks. we have done comprehensive linguistic error analysis showing a step-by-step breakdown of the f 1 performance for the inside model, inside model with self-training, and the inside-outside model with a co-training-based approach. the effectiveness of our multi-view learning strategy is evident in our experiments. future work could aim to augment the parser's capabilities to investigate cross-domain generalization and efficient cross-lingual transfer.