Deep Inside-outside Recursive Autoencoder with All-span Objective (2020)

Abstract: Deep inside-outside recursive autoencoder (DIORA) is a neural-based model designed for unsupervised constituency parsing. During its forward computation, it provides phrase and contextual representations for all spans in the input sentence. By utilizing the contextual representation of each leaf-level span, the span of length 1, to reconstruct the word inside the span, the model is trained without labeled data. In this work, we extend the training objective of DIORA by making use of all spans instead of only leaf-level spans. We test our new training objective on datasets of two languages: English and Japanese, and empirically show that our method achieves improvement in parsing accuracy over the original DIORA.

## introduction constituency parsing produces constituent parse trees that can be useful for downstream tasks, such as text classification @xcite , machine translation @xcite and semantic role labelling @xcite . although supervised neural methods have very high accuracy, they require annotated data which can be very limited for low-resource languages and domains. unsupervised constituency parsing is a task aiming to learn a constituency parser from text without annotated parse trees. there is a recent trend of solving this task with neural approaches @xcite @xcite @xcite . diora @xcite is one of these models. it mimics the inside-outside algorithm @xcite to build an inside chart and an outside chart for each input sentence. each cell in these charts represents a representation of the inside text or the outside context of a span of the sentence. diora is trained by using the contextual representation of every single word to predict the word itself, which can be seen as a reconstruction procedure of the input sentence. the original training objective of diora only takes the n leaf-level spans into account for an n-word sentence. since such a sentence has (n 2 + n)/2 spans in total, we suppose that utilizing the rest of the spans during training can help the model learn better. thus we propose a new training objective for diora that takes into account the reconstruction of all spans. we design weighting strategies to balance the influence from spans and sentences of different lengths in the new objective function. in the experiments, we train and test diora with our new objective function on datasets of two languages, english and japanese. for both languages, we show empirically that the new training objective improves the performance of unsupervised constituency parsing. ## conclusion deep inside-outside recursive autoencoder (diora) is a neural-based model for unsupervised constituency parsing. in this paper, we propose a new training objective function for diora that considers all-level spans instead of leaf-spans only. with the new objective, we show that the model can achieve better overall performance for sentences on english and japanese.