Structural Optimization Ambiguity and Simplicity Bias in Unsupervised Neural Grammar Induction (2024)

Abstract: Neural parameterization has significantly advanced unsupervised grammar induction. However, training these models with a traditional likelihood loss for all possible parses exacerbates two issues: 1) *structural optimization ambiguity* that arbitrarily selects one among structurally ambiguous optimal grammars despite the specific preference of gold parses, and 2) *structural simplicity bias* that leads a model to underutilize rules to compose parse trees. These challenges subject unsupervised neural grammar induction (UNGI) to inevitable prediction errors, high variance, and the necessity for extensive grammars to achieve accurate predictions. This paper tackles these issues, offering a comprehensive analysis of their origins. As a solution, we introduce *sentence-wise parse-focusing* to reduce the parse pool per sentence for loss evaluation, using the structural bias from pre-trained parsers on the same dataset.In unsupervised parsing benchmark tests, our method significantly improves performance while effectively reducing variance and bias toward overly simplistic parses. Our research promotes learning more compact, accurate, and consistent explicit grammars, facilitating better interpretability.

## introduction grammar induction (gi) has been intensively studied due to its critical role in understanding, controlling, and utilizing the structural information of internal symbols called constituents in linguistic analysis. unsupervised approaches to gi are continuously studied for their benefits of avoiding structured data as parses, which requires enormous human efforts and expertise to generate supervised data. in the literature of this field, neural parameterization for representing grammars @xcite has shown significant improvement over traditional probabilistic models @xcite @xcite . furthermore, advanced models that incorporate conventional approaches, such as lexicalization @xcite and symbol expansion @xcite have been proposed. however, these unsupervised neural grammar inductions (ungis) lack analyses of the traditional challenges, particularly regarding the training loss. some studies, such as @xcite , have highlighted the difficulty of optimization presented by the complex loss landscape in ugis @xcite @xcite . subsequent efforts @xcite have alleviated these issues by only enhancing expressiveness through overparameterization rather than the cause analysis. in this paper, we aim to reveal two unhandled problems in ungis: 1) structural optimization ambiguity, which increases the number of optima of the same quality that prefer different parse structures, leading to prediction errors by random selection for branching and high variance, and 2) structural simplicity bias, which induces the grammar to use only a few rules in parse trees and resulting in simplistic parse trees even with a large capacity. both problems are caused by training a neural grammar with sentence probability. to address these issues, we propose a simple but effective method to inject focusing-bias that accounts for only a few selected parses for training loss in the inside algorithm. this approach reduces the ambiguity by selecting more distinct parses across sentences and restricts the simplicity bias by excluding of unnecessary simplistic parse trees by sentences. we also introduce a method of generating structural bias by parse selection from single or multiple parsers pre-trained on the same training data. our method outperforms state-of-the-art implicit and explicit grammar induction models in english parsing tasks on the penn treebank (ptb) @xcite , as well as in ten other languages, including the chinese penn treebank (ctb) @xcite and the spmrl datasets @xcite . in-depth analysis shows a reduction in high variance and overly simplistic parses. we provide the investigation results of various structural biases, recommending heterogeneous multi-parsers. our contributions are summarized as follows: ## limits on unsupervised learning of neural grammar induction as explained earlier, ungi loss has the advantage of inducing grammar without the information of gold parse trees. however, ungis still inherit the limitations of probabilistic ugis. in this section, we reveal two issues arising from these limitations in ungis:structural optimization ambiguity and structural simplicity bias, and discuss derived negative effects and their causes. for precise analysis, we focus on inducing explicit neural grammars based on pcfgs, mainly neural pcfgs (n-pcfgs). ## conclusion in this paper, we tackle the challenges of high variance and overly simplistic parse in unsupervised neural grammar induction (ungi). we identify structural optimization-ambiguity and structural simplicity bias as primary causes, arising from ugi loss on all possible parses of all sentence parses. we introduce a parse-focusing approach that narrows down the parse set per sentence and incorporates biases from parses generated by pre-trained parsers on the same dataset. this method has been shown to effectively mitigate these issues, significantly surpassing state-of-the-art methods in both english and multilingual parsing benchmarks. our contributions facilitate the consistent learning of more compact, explicit, accurate, and therefore interpretable grammar models in unsupervised learning environment.