Paper: Ensemble Distillation for Unsupervised Constituency Parsing (ICLR-2024)

Reviewer comment (Novelty):
Somewhat narrow scope (ensembling constituency trees)

Author response:
Unsupervised constituency parsing, in fact, fits ICLR themes well, as it concerns **learning the representations** of human language. In the literature, a number of impactful papers were published in previous editions of ICLR, such as Shen et al., 2018, 2019.  

In addition, our study has general machine learning implications, especially for multi-teacher distillation. Our study shows that a straightforward union distillation from multiple teachers (as was done in previous work) may not yield improved performance, whereas our ensemble-then-distill approach is able to alleviate the over-smoothing issue in traditional methods. We’re happy to explore multi-teacher distillation for different data types (such as sequences and graphs), mentioned in Future Work.

Reviewer comment (Reproducibility):
Some details missing (see the questions)

Author response:
Thanks for the detailed question. By “Best/Worst teacher across runs”, we mean that we must still choose exactly one run for a model, but which run may not correspond among models. A simple example:

Model1: Run1=85, Run2=90, Run3=95

Model2: Run1=65, Run2=60, Run3=55

Then, the best teachers across runs will be Model1-Run3 and Model2-Run1. The worst teachers across runs will be Model1-Run1 and Model2-Run3. 

We’ve clarified this in the revision (third paragraph in Sec 3.3).

Reviewer comment (Evaluation):
Would "Our ensemble (X teacher across runs)" in Table 2 use candidate trees from different models, or do they all end up from the same model (e.g., ConTest for X="best")? It's good to know the answer to this question because one of the main claims of the paper is that it's important to exploit the large qualitative differences between different methods (Table 1). The fact that we get the best result by just using outputs from the same model seems to refute that claim (i.e., it's all variance reduction).

Author response:
Thanks for the detailed question. By “Best/Worst teacher across runs”, we mean that we must still choose exactly one run for a model, but which run may not correspond among models. A simple example:

Model1: Run1=85, Run2=90, Run3=95

Model2: Run1=65, Run2=60, Run3=55

Then, the best teachers across runs will be Model1-Run3 and Model2-Run1. The worst teachers across runs will be Model1-Run1 and Model2-Run3. 

We’ve clarified this in the revision (third paragraph in Sec 3.3).

Reviewer comment (Novelty):
MBR-style or consistency-based methods have also been applied to parsing for decoding or model selection, but the authors failed to recognize and discuss them. To name a few, [Smith and Smith, 2007](https://aclanthology.org/D07-1014.pdf) and [Zhang et al., 2020](https://aclanthology.org/2020.acl-main.302.pdf) used MBR-decoding to improve dependency parsing; [Shi et al., 2019](https://aclanthology.org/P19-1180.pdf) adapted an agreement-based model selection process for distantly supervised constituency parsing.

Author response:
Thanks for sharing the literature. We have included the suggested references, as well as some of their references, in the revision.

Reviewer comment (Novelty):
The motivation is not completely convincing. Recent trends in NLP demonstrate that explicit parses might not be crucial or even necessary to many user-facing applications (e.g., GPT models do not really use explicit language structures), which contradicts the first sentence in this paper (*Constituency parsing is a core task in natural language processing*). Traditionally, such structures served as a backbone for many NLP models, and the prediction of them was therefore referred to as *core NLP*. I am not sure if the parsing is as important as what I receive from this paper. Please consider revising or including more justification.

Author response:
We’d clarify that, by saying “a core task,” we mean that parsing has been traditionally referred to as a core NLP task. In the revision, we call parsing “a well-established task.”

Nevertheless, we would like to point out the significance of unsupervised parsing as a curious task of language structure discovery. It verifies linguistic theory, showing that linguistically defined constituents can naturally emerge in an unsupervised way. In our related work, we also discussed how unsupervised parsing may inspire the structure discovery of motion-sensor data (Peng et al., 2011).

Reviewer comment (Novelty):
The references in this paper, especially to conventional linguistic literature, need some work:  
- Section 1: Carnie (2007) and Fromkin et al. (2003) are introductory books. Both should be changed to Chomsky (1957). *Syntactic Structures*.  
- [Spitkovsky et al. (2013)](https://aclanthology.org/D13-1204.pdf) is worth a mention in the related work section.

Author response:
We’re grateful for the suggestions and have included them in the revision.

Reviewer comment (Evaluation):
Page 1: low correlation among different unsupervised parsers: [Williams et al. (2018)](https://direct.mit.edu/tacl/article/doi/10.1162/tacl_a_00019/43445/Do-latent-tree-learning-models-identify-meaningful) discovered a similar issue within the same model architectures (Table 1, right settings). This is worth a discussion.

Author response:
Thanks for the suggestion. The mentioned work is discussed and cited in the revision. Williams et al., 2018 showed low correlations among early latent-tree models that were claimed to induce task-specific tree structures, whereas our paper shows low correlation among unsupervised parsing models that are aimed at discovering linguistically plausible tree structures. Further, our paper shows that such low correlation suggests different expertise of unsupervised parsers, which can be utilized by model ensembles for performance improvement.

Reviewer comment (Evaluation):
Since the topic of this paper is highly linguistic, I am willing to see a detailed analysis of what patterns are fixed. For example, do NPs/PPs with rare words receive more fixes than those with frequent words, or the opposite, or not significant? Do VPs with transitive verb heads receive more fixes than those with intransitive verb heads, or the opposite, or insignificant? Does the student extract any constituent that does not receive any vote from teacher models, due to fixes on shorter spans and CYK?

Author response:
In the performance-by-type analysis, we’ve shown that our approach achieves consistently high performance across all constituency types. We’re further presenting a case study in the appendix, suggesting that our ensemble indeed performs voting for local structures and fixes teachers’ predictions.

Reviewer comment (Experiments):
Which split of PTB did you use to generate the statistics in Table 1?

Author response:
We used the standard split, namely, Section 23 of PTB for tests.

Reviewer comment (Evaluation):
Table 2: why are the +RNNG/+URNNG oracle performance different from the basic one (83.3)?

Author response:
In Table 2, the +RNNG column for Oracle row (Row 15) is, in fact, a supervised RNNG trained on the binarized PTB-training set (binarized ground-truth). +URNNG is further tuning the RNNG in an unsupervised manner. The performance is lower than Oracle, which is the performance upper bound.

Reviewer comment (Evaluation):
The fact that the F1 gains from distillation do not carry over to the out of domain setting is a drawback and somewhat underexplored.

Author response:
Thanks for the insight! We acknowledge that distillation does not help our ensemble, or even the supervised model, in the SUSANNE experiment. To be honest, this is also a little bit unexpected to us as well, but we have honestly stated that in the paper (left to the caption of Table 3). We’re happy to explore domain adaptation of unsupervised parsing more in future work.

Reviewer comment (Evaluation):
There is a lack of qualitative analysis of the types of behaviors that different model types exhibit, and how ensembling actually combines those. Some of this is done in the Appendix but it would be nice to see specific examples in the main paper, especially since that analysis is wrt constituency labels which the model isn’t actually being evaluated on.

Author response:
Thanks for the suggestion. We have included a case study in the revision (which inevitably overflows to the appendix due to the volume and substance of our paper).

Reviewer comment (Experiments):
Regarding the experiment in Figure 1, do you see similar results if you measure the gains from the *distilled* ensemble? That would be useful to see alongside.

Author response:
Thanks for the suggestion. Distillation analysis is expensive, as it requires performing inference for all teachers on the training data, training RNNG, and then refining with URNNG. For Figure 1, it requires 21 teachers' inference over the training set and training 14 RNNG/URNNG models, which may be unaffordable to us. 

Nevertheless, we expect the phenomenon to hold for RNNG/URNNG, because during our development, we observed similar patterns among RNNG, URNNG, and their ensemble teacher. The F1 scores are also similar, with RNNG slightly lower and URNNG slightly higher, consistent with all PTB experiments.

Reviewer comment (Experiments):
The averaging tree is derived with the highest total F1 score compared with different teachers. Have the authors tried other methods of calculating the similarities between trees? Perhaps a fair comparison is needed to further indicate the effectiveness of the proposed tree averaging method.

Author response:
* (F1 score) We chose the F1 score because F1 is the standard measure for constituency parsing, explicitly mentioned in the second paragraph of Sec 2.2. Therefore, our similarity score is well motivated. We would be grateful if the reviewer can recommend alternatives that are more justified.

Reviewer comment (Experiments):
The authors state in Introduction that combining the different parsers may leverage their different expertise. The authors attempt to verify this statement in Section 3.4 by comparing two settings: the ensemble of three runs of the same model and that of three heterogeneous models. I wonder if it is more appropriate to choose the model with highest performance in the former setting so that it can be further validated there exists additional boost due to different expertise (e.g., Neural PCFG for Group 2).

Author response:
In Figure 1, four of the seven groups (namely, Groups 1, 4, 6, and 7) meet the criterion that the reviewer mentioned. Hence, it's possible to draw the conclusion that the reviewer is seeking from a subset of our experimentation. 
Moreover, we do not believe it is essential to use the best-performing model of the latter setting for multiple runs in the former, control setting. As we can see from the figure, the aforementioned four groups are not different from the rest.

Reviewer comment (Writing):
Minor comment on writing - Paragraph #1 in Introduction: ...to explore unsupervised methods as it eliminates... -> ...to explore unsupervised methods as they eliminate...

Author response:
> Weakness 3 (two typos in a paper)
 
Thanks for catching two typos in our paper! We’ve fixed them in the revision.

Reviewer comment (Writing):
Minor comment on writing - Paragraph #1 in Section 3.1: ... on the widely used the Penn Treebank -> ... on the widely-used Penn Treebank

Author response:
> Weakness 3 (two typos in a paper)
 
Thanks for catching two typos in our paper! We’ve fixed them in the revision.

Reviewer comment (Evaluation):
The ensemble method demonstrates its effectiveness on PTB. However, in 2020, the F1 score of CRF parser on PTB variants had already been above 90 (Zhang et al., 2020). There is indeed a performance boost in comparison to oracle score (the highest possible F1 score of binarize groundtruth trees). But it is not appropriate to claim that “largly bridging the gap between supervised and unsupervised constituency parsing” on Page 6.

Author response:
Thanks for pointing this out. We have revised our claim as “largely bridging the gap between supervised and unsupervised *binary* constituency parsing”.

Reviewer comment (Experiments):
In Results on SUSANNE, the authors claim that “This is a realistic experiment to examine the models’ performance in an unseen low-resource domain.” However, SUSANNE is an English dataset. In CoNLL Shared task, there are tree-banks on low-resource languages (Zeman et al., 2017). It’s better that the authors can demonstrate the effectiveness of the approach on some of these low-resource language datasets.

Author response:
Here, our claim about the SUSANNE experiment is domain shift, instead of low-resource language. As shown in Table 3, models trained on PTB underperform in other domains in the same language, showing that unsupervised parsing remains challenging in low-resource domains, even for English.

As mentioned in the response to Reviewer yYrJ, a key difficulty of a non-English ensemble is to find teacher unsupervised parsers, which are less addressed in the literature. We’re happy to address this direction as future work.

Reviewer comment (Experiments):
It is comparing only for English, and it would be better to compare the model with other languages, e.g., Chinese, for further strengthening this submission.

Author response:
Thanks for pointing out that our approach currently works for the English language. A key difficulty of non-English ensemble is to find teacher unsupervised parsers, which are less addressed in the literature. Nevertheless, our ensemble approach brings new opportunities for unsupervised multilingual and non-English parsing (such as transferring and building ensembles with the structural knowledge of different languages). We’re happy to explore this direction in the future, as has been mentioned in our future work section.

Reviewer comment (Evaluation):
I'd like to know the impact of length of inputs, e.g., whether the proposed method is better in lengthy input or not.

Author response:
Thanks for the suggestion! We actually had the length analysis in our development but didn’t include it in the submission because our paper had already overflowed to the appendix. The finding is similar to the analysis of constituency labels: our ensemble maintains high performance across different lengths, compared with teacher models. 

We now show the results in Appendix B.2.