Learning Syntax from Naturally-Occurring Bracketings (2021)

Abstract: Naturally-occurring bracketings, such as answer fragments to natural language questions and hyperlinks on webpages, can reflect human syntactic intuition regarding phrasal boundaries. Their availability and approximate correspondence to syntax make them appealing as distant information sources to incorporate into unsupervised constituency parsing. But they are noisy and incomplete; to address this challenge, we develop a partial-brackets-aware structured ramp loss in learning. Experiments demonstrate that our distantly-supervised models trained on naturally-occurring bracketing data are more accurate in inducing syntactic structures than competing unsupervised systems. On the English WSJ corpus, our models achieve an unlabeled F1 score of 68.9 for constituency parsing.

## introduction constituency is a foundational building block for phrase-structure grammars. it captures the notion of what tokens can group together and act as a single unit. the motivating insight behind this paper is that constituency may be reflected in markups of bracketings that people provide in doing natural tasks. we term these segments naturallyoccurring bracketings for their lack of intended syntactic annotation. these include, for example, the segments people pick out from sentences to refer to other wikipedia pages or to answer semanticallyoriented questions; see figure 1 for an illustration. gathering such data requires low annotation expertise and effort. on the other hand, these data are not necessarily suitable for training parsers, as they often contain incomplete, incorrect and sometimes conflicting bracketing information. it is thus an empirical question whether and how much we science fiction (sometimes shortened to sci-fi or sf) is a genre of speculative fiction that typically deals with imaginative and futuristic concepts such as advanced science and technology, space exploration, time travel, parallel universes, and extraterrestrial life. republicans have been imploring the white house to compromise on the wage issue. ## naturally-occurring bracketings constituents are naturally reflected in various human cognitive processes, including speech production and perception @xcite , reading behaviors @xcite , punctuation marks @xcite , and keystroke dynamics @xcite . conversely, these externalized signals help us gain insight into constituency representations. we consider two such data sources: a) answer fragments when questions are answered with fragments instead of full sentences, those fragments tend to form constituents. this phenomenon corresponds to a well-established constituency test in the linguistics literature @xcite , inter alia). 2 we worked with articles in english. b) webpage hyperlinks since a hyperlink is a pointer to another location or action (e.g., mailto: links), anchor text often represents a conceptual unit related to the link destination. indeed, spitkovsky et al. ( 2010 ) first give empirical evidence that around half of the anchor text instances in their data respects constituent boundaries and @xcite demonstrates that hyperlink data can help boost chunking accuracy in a multi-task learning setup. both types of data have been considered in previous work on dependency-grammar induction @xcite , and in this work, we explore their efficacy for learning constituency structures. for answer fragments, we use he et al.'s (2015) question-answering-driven semantic role labeling (qa-srl) dataset, where annotators answer whquestions regarding predicates in sentences drawn from the wall street jounal (wsj) section of the penn treebank (ptb; @xcite . for hyperlinks, we used a 1% sample of 2020-05-01 english wikipedia, retaining only within-wikipedia links. 3 we compare our extracted naturally-occurring bracketings with the reference phrase-structure annotations: foot_2 ## parsing model preliminaries the inputs to our learning algorithm are tuples (w, b), where @xmath0, . . . , w n is a length-n sentence and @xmath1, e k )} is a set of naturally-occurring bracketings, denoted by the beginning and ending indices b k and e k into the sentence w. as a first step, we extract bert-based contextualized word representations @xcite to associate each token w i with a vector x i . foot_3 see appendix b for details. ## experiments and results data and implementation we evaluate on the ptb @xcite with the standard splits model plm mean max random trees 19.2 19.5 left branching 8.7 right branching 39.5 upper bound 84.3 urnng (kim et al., 2019b) -45.4 prpn @xcite 47.3 47.9 on @xcite 48.1 50.0 diora @xcite • -58.9 cpcfg @xcite 55.2 60.1 s-diora @xcite • 57.6 64.0 constituency tests @xcite • 62.8 65.9 +urnng @xcite • 67.9 71. evaluation we follow the evaluation setting of @xcite . more specifically, we discard punctuation and trivial spans (single-word and full-sentence spans) during evaluation and report sentence-level f1 scores as our main metrics. results learning from partial annotations full syntactic annotations are costly to obtain, so the alternative solution of training parsers from partiallyannotated data has attracted considerable research attention, especially within the context of active learning for dependency parsing @xcite @xcite @xcite @xcite and grammar induction for constituency parsing @xcite @xcite . these works typically require expert annotators to generate gold-standard, though partial, annotations. in contrast, our work considers the setting and the challenge of learning from noisy bracketing data, which is more comparable to @xcite and @xcite on transfer learning for dependency parsing. ## conclusion and future work we argue that naturally-occurring bracketings are a rich resource for inducing syntactic structures. they reflect human judgment of what constitutes a phrase and what does not. more importantly, they require low annotation expertise and effort; for example, webpage hyperlinks can be extracted essentially for free. empirically, our models trained on qa-srl and wikipedia bracketings achieve competitive results with the state of the art on unsupervised constituency parsing. structural probes have been successful in extracting syntactic knowledge from frozen-weight pre-trained language models (e.g., hewitt and manning, 2019), but they still require direct syntactic supervision. our work shows that it is also feasible to retrieve constituency trees from bert-based models using distant supervision data. our models are limited to the unlabeled setting, and we leave it to future work to automatically cluster the naturally-occurring bracketings and to induce phrase labels. our work also points to potential applications in (semi-)supervised settings including active learning and domain adaptation @xcite . future work can also consider other naturally-occurring bracketings induced from sources such as speech production, reading behavior, etc. span scoring mlp left and mlp right are singlelayer mlps: they both consist of a linear layer projecting bert representations to 256-dimensional vectors, followed by a leaky relu activation function @xcite . the constituent scoring component has parameter @xmath2. all the parameters are randomly initialized @xcite .