Re-evaluating the Need for Visual Signals in Unsupervised Grammar Induction (2024)

Abstract: Are multimodal inputs necessary for grammar induction? Recent work has shown that multimodal training inputs can improve grammar induction. However, these improvements are based on comparisons to weak text-only baselines that were trained on relatively little textual data. To determine whether multimodal inputs are needed in regimes with large amounts of textual training data, we design a stronger text-only baseline, which we refer to as LC-PCFG. LC-PCFG is a C-PFCG that incorporates embeddings from text-only large language models (LLMs). We use a fixed grammar family to directly compare LC-PCFG to various multimodal grammar induction methods. We compare performance on four benchmark datasets. LC-PCFG provides an up to 17% relative improvement in Corpus-F1 compared to state-of-the-art multimodal grammar induction methods. LC-PCFG is also more computationally efficient, providing an up to 85% reduction in parameter count and 8.8\times reduction in training time compared to multimodal approaches. These results suggest that multimodal inputs may not be necessary for grammar induction, and emphasize the importance of strong vision-free baselines for evaluating the benefit of multimodal approaches.

## introduction prior studies have shown that multimodal inputs can facilitate grammar induction. these studies paired text with inputs from images and videos, and found that models trained with paired multimodal inputs outperform text-only models on grammar induction @xcite @xcite . these results suggest that multimodal inputs improve grammar induction by grounding textual inputs to the visual world. indeed, a long line of work in human language learning suggests that paired multimodal inputs are crucial for language acquisition in humans @xcite . while multimodal inputs can undoubtedly help with grammar induction, especially in regimes with low textual data, are multimodal inputs necessary to learn a grammar? to investigate this question, we test whether the benefits of multimodal inputs for grammar induction can be achieved by more textual data. prior studies of multimodal grammar induction compared multimodal methods to weak text-only baselines which were trained with relatively little data @xcite @xcite . however, recent grammar induction approaches that incorporate representations from large language models (llms) produced large improvements in text-only grammar induction performance (e.g., @xcite @xcite . the performance of these llm-based grammar induction methods suggest that exposure to larger quantities of textual training data can substantially improve grammar induction. however, prior studies used different settings to evaluate multimodal and textonly methods for grammar induction. thus, it is unclear whether the performance of llm-based grammar induction approaches can match the performance of multimodal approaches. here we compare multimodal methods for grammar induction to a strong text-only baseline. our text-only baseline, which we refer to as lc-pcfg, is a c-pcfg that incorporates embeddings from text-only llms. we use and use a fixed grammar family (c-pcfgs) to directly compare lc-pcfg to multimodal methods, and perform comparisons with four multimodal grammar induction datasets. we find that compared to previous state-of-theart multimodal methods, lc-pcfg achieves up to 17% relative improvement in corpus-f1 score while requiring 8.8× less time to train. moreover, the benefits of incorporating llm embeddings does not straightforwardly stack with the benefits of multimodal training inputs: adding mul- text-only parser prior works ours + pixels + llm embeddings 59.3 67.2 53.6 figure 1 : comparison with prior multimodal methods on image-assisted grammar induction. prior works showed that paired images can improve grammar induction (53.6 → 59.3 corpus-level f1). we show that a strong text-only baselined line that incorporates embeddings from large language models (llm) can match (and surpass) multimodal methods, suggesting that multimodal inputs may not be necessary for grammar induction (53.6 → 67.2). ## related work grammar induction grammar induction, the task of inducing syntactic structure without explicit supervision, has been extensively studied over the past few decades (e.g., @xcite @xcite @xcite . many methods for grammar induction train on data from text alone (e.g., @xcite @xcite @xcite . however, based on the intuition that multimodal inputs capture information that is missing in text, recent studies have devised methods for grammar induction that incorporate information from images and videos @xcite @xcite . these multimodal methods have been shown to outperform some text-only methods @xcite @xcite . ## lc-pcfg: grammar induction with large language models the goal of grammar induction is to learn syntactic structure without explicit supervision. methods for grammar induction assume a grammar formalism and then optimize grammar parameters to fit the data. we use compound probabilistic context-free grammars (c-pcfgs) @xcite as a grammar formalism. we construct a c-pcfg that incorporates llm representations. we refer to this method as lc-pcfg. we compare lc-pcfg to prior methods that incorporate multimodal data @xcite @xcite . figure 2 provides an overview of our experiments. background. c-pcfgs extend the probabilistic context free grammar (pcfg) formalism, and are defined by a 5-tuple @xmath0, n , p, σ, r), consisting of a start symbol s, a set of non-terminals n , a set of pre-terminals p, a set of terminals σ, and a set of derivation rules r: pcfgs define a probability distribution over transformation rules π = {π r } @xmath1. then the inside algorithm @xcite can be used to efficiently perform inference over this probability distribution. in neural pcfgs, this distribution may be formullm sentence c-pcfg constituency parse tree sentence embedding a squirrel jumps on a stump fc latent figure 3: lc-pcfg workflow. a sentence x is fed to an llm to obtain a sentence embedding e(x). e(x) is passed through a fully-connected layer (fc), producing the latent z. z is fed to the c-pcfg to obtain a constituency parse tree. note that unlike prior work, our approach does not require multimodal data. because pcfgs contain a strong context-free assumption, pcfgs cannot leverage global information that is useful for computing production probabilities during inference. c-pcfgs @xcite extend pcfgs to incorporate global information. c-pcfgs formulate rule probabilities as a compound probability distribution @xcite : where z is a latent variable generated by a prior distribution (generally assumed to be spherical gaussian) and e @xmath2. rule probabilities π z are additionally conditioned on this latent variable: importantly, the latent variable z allows global information to be shared across production decisions during, while maintaining the context-free assumption needed for efficient inference when z is fixed. becase the introduction of z makes inference intractable, variational methods are used to optimize c-pcfgs (kingma and welling, 2013) at inference time, given a sentence x, the variational inference network q ϕ is used to produce the latent here, g is a sentence encoder used to generate a vector representation given token embeddings e(x). for a more thorough treatment of c-pcfgs, please see @xcite . ## conclusion and future work we propose lc-pcfg, a strong text-only baseline for grammar induction. lc-pcfg is a c-pcfg model that incorporates representations from llms trained on text alone. on four benchmarks for multimodal grammar induction, lc-pcfg outperforms several prior state-of-the-art multimodal approaches. furthermore, adding visual inputs to lc-pcfg does not improve grammar induction. these experiments show that for grammar induction, the benefits of multimodal inputs can be achieved by more textual data. our results challenge the notion that multimodal inputs are necessary for grammar induction. based on the result that lc-pcfg performs as well as methods trained on multimodal inputs, we speculate that representations from llms provide information that is redundant with information provided by multimodal inputs. indeed, some work has shown that multimodal inputs improve grammar induction by providing signals of noun concreteness @xcite . other work has shown that llms acquire some knowledge of word concreteness @xcite . thus, large amounts of textual training data may provide signals of word concreteness that obviate multimodal inputs for grammar induction.