An Empirical Comparison of Unsupervised Constituency Parsing Methods (2020)

Abstract: Unsupervised constituency parsing aims to learn a constituency parser from a training corpus without parse tree annotations. While many methods have been proposed to tackle the problem, including statistical and neural methods, their experimental results are often not directly comparable due to discrepancies in datasets, data preprocessing, lexicalization, and evaluation metrics. In this paper, we first examine experimental settings used in previous work and propose to standardize the settings for better comparability between methods. We then empirically compare several existing methods, including decade-old and newly proposed ones, under the standardized settings on English and Japanese, two languages with different branching tendencies. We find that recent models do not show a clear advantage over decade-old models in our experiments. We hope our work can provide new insights into existing methods and facilitate future empirical evaluation of unsupervised constituency parsing.

## introduction unsupervised constituency parsing, a task in the area of grammar induction, aims to learn a constituency parser from a training corpus without parse tree annotations. while research on unsupervised constituency parsing has a long history @xcite @xcite , recently there is a resurgence of interest in this task and several approaches based on neural networks have been proposed that achieve impressive performance @xcite @xcite @xcite . with the recent development in research of unsupervised constituency parsing, however, the problem of lacking a unified experimental setting begins to emerge, which makes empirical comparison between different approaches difficult. first of all, although almost all previous approaches are evaluated on the penn treebank @xcite , they differ in how they preprocess the training data, with respect to the sentence length limit, punctuation removal, vocabulary pruning, and so on. for example, non-neural methods such as constituent context model (ccm) @xcite are trained on short sentences, while modern neural based methods such as parsing-reading-predict network (prpn) @xcite do not impose any limit on sentence length. furthermore, existing approaches also differ in their evaluation metrics, with respect to the methods of computing averages, counting trivial spans, and so on. the evaluation results of the same approach using different metrics can differ significantly in some cases. unfortunately, we have seen more than one paper that directly compares approaches evaluated with different metrics. in this paper, we propose three standardized experimental settings with respect to data preprocessing, post-processing, evaluation metrics, and tuning. we then empirically compare five existing methods under the standardized settings, including two decade-old methods and three recently proposed neural methods. we run our experiments on english and japanese, two languages with different branching tendencies. interestingly, the overall experimental results show that the recent methods do not show a clear advantage over the decade-old methods. we hope our empirical comparison could provide new insights into the relative strength and weakness of existing methods and our standard-ized experimental settings could facilitate future evaluation of unsupervised constituency parsing. our pre/post-processing and evaluation source code can be found at https://github.com/i-lijun/ unsupconstparseeval . ## experimental results we list the experimental results of all the models and the left/right-branching baselines for ptb and ktb in also note that for the left and right branching baselines and the binary upper bound, the scores for "length 10 no punct" and "length 40 no punct" are the same, because these baselines do not require training and are evaluated on the same test sets. overall comparison there is no universal winner for all the settings but there is clear winners for specific settings. on ptb, it is surprising to see that each model is the winner of at least one setting. right-branching is a very strong base-line and with post-processing it outperforms all the models in some settings of "ptb len40 punct". on ktb, diora is the winner in most of the settings, while ccm has a strong performance on "ktb len10 nopunct". left-branching is a strong baseline especially when evaluated on sentences with @xmath1. although ccm and diora achieve the best overall performance, we note that they both utilize additional resources. ccm uses gold pos tags and diora uses pretrained word embedding. our preliminary experiments on ptb show a significant drop in performance when we run ccm using words without gold pos tags, with the evalb f1 score dropping from 70.14 to 57.29 when evaluated on @xmath2. diora also performs worse when pretrained word embedding is replaced by randomly initialized embedding, with the average evalb f1 score dropping from 49.39 to 42.63 when evaluated on all sentences under the "ptb len40 nopunct" setting. overall, we do not see a clear advantage of more recent neural models over traditional models. there are two factors that should be taken into account though. first, neural models are significantly slower and therefore may not have been sufficiently tuned because of the fixed tuning time budget. second, the training data may still be too small from the perspective of neural models. finally, we also note that our post-processing method for adding back punctuation almost always improves the score in ptb, sometimes by a large margin (e.g., for ccm and rbranch). on ktb, however, it sometimes decreases the score. this may be caused by different annotation standards for punctuation in the two treebanks. ## discussion we make the following recommendations for future experiments on unsupervised constituency parsing. for the sentence length limit, we think one can set any limit on the training data, but should report evaluation results on both @xmath3. for the evaluation metrics, since small details in implementing micro and macro average will lead to nontrivial differences, we suggest using parseval which has publicly available implementation. for models sensitive to random seeds, we recommend reporting means and standard deviations from multiple runs. we also recommend evaluation on treebanks of both leftbranching and right-branching languages, such as ptb and ktb.