Improved Latent Tree Induction with Distant Supervision via Span Constraints (2021)

Abstract: For over thirty years, researchers have developed and analyzed methods for latent tree induction as an approach for unsupervised syntactic parsing. Nonetheless, modern systems still do not perform well enough compared to their supervised counterparts to have any practical use as structural annotation of text. In this work, we present a technique that uses distant supervision in the form of span constraints (i.e. phrase bracketing) to improve performance in unsupervised constituency parsing. Using a relatively small number of span constraints we can substantially improve the output from DIORA, an already competitive unsupervised parsing system. Compared with full parse tree annotation, span constraints can be acquired with minimal effort, such as with a lexicon derived from Wikipedia, to find exact text matches. Our experiments show span constraints based on entities improves constituency parsing on English WSJ Penn Treebank by more than 5 F1. Furthermore, our method extends to any domain where span constraints are easily attainable, and as a case study we demonstrate its effectiveness by parsing biomedical text from the CRAFT dataset.

## introduction syntactic parse trees are helpful for various downstream tasks such as speech recognition @xcite , machine translation @xcite , paraphrase generation @xcite , semantic parsing @xcite , and information extraction @xcite . while supervised syntactic parsers are state-of-the-art models for creating these parse trees, their performance does not transfer well across domains. moreover, new syntactic annotations are prohibitively expensive; the original penn treebank required eight years of annotation @xcite , and expanding ptb annotation to a new domain can be ⇤ equal contribution. the driver heading to evermore falls span constraint the driver heading to evermore falls ✓ distant supervision ❌ unsupervised learning figure 1 : an example sentence and parsing to illustrate distant supervision via span constraints. top: the unsupervised parser predicts a parse tree, but due to natural ambiguity in the text the prediction crosses with a known constraint. bottom: by incorporating the span constraint, the prediction improves and, as a result, recovers the ground truth parse tree. in our experiments, we both inject span constraints directly into parse tree decoding and separately use the constraints only for distant supervision at training time. we find the latter approach is typically more effective. a large endeavor. for example, the 20k sentences of biomedical treebanking in the craft corpus required 80 annotator hours per week for 2.5 years, include 6 months for annotator training @xcite . however, although many domains and many languages lack full treebanks, they do often have access to other annotated resources such as ner, whose spans might provide some partial syntactic supervision. we explore whether unsupervised parsing methods can be enhanced with distant supervision from such spans to enable the types of benefits afforded by supervised syntactic parsers without the need for expensive syntactic annotations. we aim to "bridge the gap" between supervised and unsupervised parsing with distant supervision through span constraints. these span constraints in-dicate that a certain sequence of words in a sentence form a constituent span in its parse tree, and we obtain these partial ground-truths without explicit user annotation. we take inspiration from previous work incorporating distant supervision into parsing @xcite @xcite , and design a novel fully neural system that improves a competitive neural unsupervised parser (diora; @xcite ) using span constraints defined on a portion of the training data. in the large majority of cases, the number of spans constraints per sentence is much lower than that specified by a full parse tree. we find that entity spans are effective as constraints, and can readily be acquired from existing data or derived from a gazetteer. in our experiments, we use diora as our baseline and improve upon it by injecting these span constraints as a source of distant supervision. we introduce a new method for training diora that leverages the structured svm loss often used in supervised constituency parsing @xcite , but only depends on partial structure. we refer to this method as partially structured svm (ps-svm). our experiments indicate ps-svm improves upon unsupervised parsing performance as the model adjusts its prediction to incorporate span constraints (depicted in figure 1 ). using ground-truth entities from ontonotes @xcite as constraints, we achieve more than 5 f1 improvement over diora when parsing english wsj penn treebank @xcite . using automatically extracted span constraints from an entity-based lexicon (i.e. gazetteer) is an easy alternative to ground truth annotation and gives 2 f1 improvement over diora. importantly, training diora with ps-svm is more effective than simply injecting available constraints into parse tree decoding at test time. we also conduct experiments with different types of span constraints. our detailed analysis shows that entity-based constraints are similarly useful as the same number of ground truth np constituent constraints. finally, we show that diora and ps-svm are helpful for parsing biomedical text, a domain where full parse tree annotation is particularly expensive. ## background: diora the deep inside-outside recursive autoencoder (diora; @xcite is an extension of tree recursive neural networks (treernn) that does not require pre-defined tree structure. it depends on the two primitives compose : r 2d ! r d and score : r 2d ! r 1 . diora is bi-directionalthe inside pass builds phrase vectors and the outside pass builds context vectors. diora is trained by predicting words from their context vectors, and has been effective as an unsupervised parser by extracting parse trees from the values computed during the inside pass. inside-outside typically, a treernn would follow a parse tree to continually compose words or phrases until the entire sentence is represented as a vector, but this requires knowing the tree structure or using some trivial structure such as a balanced binary tree. instead of using a single structure, diora encodes all possible binary trees using a soft-weighting determined by the output of the score function. there are a combinatorial number of valid parse trees for a given sentence -it would be infeasible to encode each of them separately. instead, diora decomposes the problem of representing all valid parse trees by encoding all subtrees over a span into a single phrase vector. for example, each phrase vector is computed in the inside pass according to the following equations: the outside pass is computed in a similar way: training diora is trained end-to-end directly from raw text and without any parse tree supervision. in our work, we use the same reconstruction objective as in @xcite . for a sentence x, we optimize the probability of the i-th word x i using its context (x i ): parsing diora has primarily been used as an unsupervised parser. this requires defining a new primitive treescore : s(y) = p i,j,k2y s in i,j,k . a tree y can be extracted from diora by solving the search problem that can be done efficiently with the cky algorithm @xcite : ## experimental setup in this section, we provide details on data preprocessing, running experiments, and evaluating model predictions. in addition, code to reproduce our experiments and the model checkpoints are available on github. foot_1 ## results and discussion in our experiments and analysis we aim to address several research questions about incorporating span constraints for the task of unsupervised parsing. ## conclusion in this work, we present a method for enhancing diora with distant supervision from span constraints. we call this approach partially structured svm (ps-svm). we find that span constraints based on entities are effective at improving parsing performance of diora on english newswire data (+5.1 f1 using ground truth entities, or +2 f1 using a gazetteer). furthermore, we show ps-svm is also effective in the domain of biomedical text (+1 f1 using ground truth entities). our detailed analysis shows that entities are effective as span constraints, giving equivalent benefit as a similar amount of np-based constraints. we hope our findings will help "bridge the gap" between supervised and unsupervised parsing.