Vision-aided Unsupervised Constituency Parsing with Multi- MLLM Debating (2025)

Abstract: This paper presents a novel framework for vision-aided unsupervised constituency parsing (VUCP), leveraging multimodal large language models (MLLMs) pre-trained on diverse image-text or video-text data. Unlike previous methods requiring explicit cross-modal alignment, our approach eliminates this need by using pre-trained models like Qwen-VL and VideoLLaVA, which seamlessly handle multimodal inputs. We introduce two multi-agent debating mechanisms—consensus-driven (CD) and round-driven (RD)—to enable cooperation between models with complementary strengths. Extensive experiments demonstrate that our approach achieves state-of-the-art performance on both image-text and video-text datasets for VUCP, improving robustness and accuracy.

## introduction unsupervised constituency parsing (ucp), which extracts syntactic structures from unannotated text, has long relied on textual context and semantic cues for sequence labeling @xcite . by assigning syntactic tags to tokens, these models aim to uncover hierarchical structures within sentences @xcite . however, recent advances in natural language processing (nlp) have revealed the limitations of a purely textual approach, especially in domains like social media or instructional content, where visual information such as images or videos often appears. studies have explored how incorporating visual information can improve constituency parsing accuracy, highlighting that visual context provides crucial cues absent in text alone, thereby aiding the resolution of syntactic ambiguities. this has led to the development of vision-aided unsupervised constituency parsing (vucp) @xcite . however, leveraging multimodal data for unsupervised parsing presents several challenges. two ## methodology figure 2 illustrates the simplified framework of our approach. task definition: in the context of large language models (llms), constituency parsing is redefined as sequence generation tasks. given an input sentence @xmath0, x 2 , . . . , x n } nd a pre-trained model m, the sentence structure is analyzed and predicted as follows: ## experimentation for a thorough evaluation of our mmd, we conduct extensive experiments and detailed analysis. ## conclusion we present mmd, a novel framework advancing vision-aided unsupervised constituency parsing through pre-trained mllms and multi-agent debates. by eliminating cross-modal alignment costs and synergizing model strengths via rd/cd protocols, mmd achieves sota performance (about 10% f1 gains vs. single mllm) on image/videotext benchmarks. the debate mechanisms effectively resolve syntactic ambiguities by integrating complementary predictions, while pre-trained mllms ensure scalable multimodal processing. this work demonstrates the viability of resourceefficient, collaboration-driven parsing in complex multi-modal contexts, offering new directions for unsupervised syntactic analysis.