On the Role of Supervision in Unsupervised Constituency Parsing (2020)

Abstract: We analyze several recent unsupervised constituency parsing models, which are tuned with respect to the parsing F1 score on the Wall Street Journal (WSJ) development set (1,700 sentences). We introduce strong baselines for them, by training an existing supervised parsing model (Kitaev and Klein, 2018) on the same labeled examples they access. When training on the 1,700 examples, or even when using only 50 examples for training and 5 for development, such a few-shot parsing approach can outperform all the unsupervised parsing methods by a significant margin. Few-shot parsing can be further improved by a simple data augmentation method and self-training. This suggests that, in order to arrive at fair conclusions, we should carefully consider the amount of labeled data used for model development. We propose two protocols for future work on unsupervised parsing: (i) use fully unsupervised criteria for hyperparameter tuning and model selection; (ii) use as few labeled examples as possible for model development, and compare to few-shot parsing trained on the same labeled examples.

## introduction recent work has considered neural unsupervised constituency parsing @xcite @xcite , showing that it can achieve much better performance than trivial baselines. however, many of these approaches use the gold parse trees of all sentences in a development set for either early stopping @xcite @xcite or hyperparameter tuning @xcite . in contrast, models trained and tuned without any labeled data @xcite are much less competitive. are the labeled examples important in order to obtain decent unsupervised parsing performance? how well can we do if we train on these labeled examples rather than merely using them for tuning? in this work, we consider training a supervised constituency parsing model @xcite with very few examples as a strong baseline for unsupervised parsing tuned on labeled examples. we empirically characterize unsupervised and few-shot parsing across the spectrum of labeled data availability, finding that (i) tuning based on a few (as few as 15) labeled examples is sufficient to improve unsupervised parsers over fully unsupervised criteria by a significant margin; (ii) unsupervised parsing with supervised tuning does outperform few-shot parsing with fewer than 15 labeled examples, but few-shot parsing quickly dominates once there are more than 55 examples; and (iii) when few-shot parsing is combined with a simple data augmentation method and self-training @xcite @xcite , only 15 examples are needed for few-shot parsing to begin to dominate. based on these results, we propose the following two protocols for future work on unsupervised parsing: we suggest future work to tune and compare models under each protocol separately. in addition, we present two side findings on unsupervised parsing: (i) the vocabulary size in unsupervised parsing, which has not been widely considered as a hyperparameter and varies across prior work, greatly affects the performance of all unsupervised parsing models tested; and (ii) selftraining can help improve all investigated unsupervised parsing @xcite @xcite and few-shot parsing models, and thus can be considered as a post-processing step in future work. ## few-shot constituency parsing we apply benepar ( §3.1; @xcite as the base model for few-shot parsing. we present a simple data augmentation method ( §3.2) and an iterative self-training strategy ( §3.3) to further improve the performance. we suggest that such an approach should serve as a strong baseline for unsupervised parsing with supervised tuning. ## discussion while many state-of-the-art unsupervised parsing models are tuned on all labeled examples in a development set @xcite @xcite , inter alia), we have demonstrated that, given the same data, few-shot parsing with simple data augmentation and self-training can consistently outperform all of these models by a large margin. we suggest that one possibility for future work is to focus on fully unsupervised criteria, such as language model perplexity @xcite @xcite @xcite and model stability across different random seeds @xcite , for model selection, as discussed in unsupervised learning work @xcite @xcite ). an alternative is to use as few labeled examples in the development set as possible, and compare to few-shot parsing trained on the used examples as a strong baseline. in addition, we find that self-training is a useful post-processing step for unsupervised parsing. our work does not necessarily imply that unsupervised parsers produce poor parses; they may be producing good parses that clash with the conventions of treebanks @xcite . if this is the case, then extrinsic evaluation of parsers in downstream tasks @xcite , e.g., machine translation @xcite @xcite , may better show the potential of unsupervised methods. we follow @xcite to use a 3-layer on-lstm, and used the master gates in the second layer to compute the syntactic distance @xcite . diora @xcite . 13