On the Transferability of Visually Grounded PCFG s (2023)

Abstract: There has been a significant surge of interest in visually grounded grammar induction in recent times. While a variety of models have been developed for the task and have demonstrated impressive performance, they have not been evaluated on text domains that are different from the training domain, so it is unclear if the improvements brought by visual groundings are transferable. Our study aims to fill this gap and assess the degree of transferability. We start by extending VC-PCFG (short for Visually-grounded Compound PCFG [[Zhao and Titov, 2020](https://aclanthology.org/2020.emnlp-main.354/)]) in such a way that it can transfer across text domains. We consider a zero-shot transfer learning setting where a model is trained on the source domain and is directly applied to target domains, without any further training. Our experimental results suggest that: the benefits from using visual groundings transfer to text in a domain similar to the training domain but fail to transfer to remote domains. Further, we conduct data and result analysis; we find that the lexicon overlap between the source domain and the target domain is the most important factor in the transferability of VC-PCFG.

## introduction research in unsupervised grammar induction has long focused on learning grammar models from pure text @xcite @xcite . observing that human language learning is largely grounded in perceptual experiences, a new trend in the area has been inducing grammars of language with visual groundings @xcite @xcite . these grounded models are typically trained on parallel images and text such as image captioning data. once trained, they can be directly used to parse text without access to the aligned images. though these studies have concluded that visual grounding is beneficial, they evaluated their models only on in-domain text; it is unclear if the improvements transfer across domains or, equivalently, if the models acquire a general grammar or only a grammar suitable for a certain domain. in this work, we bridge the gap by studying the transferability of vc-pcfg, a performant visually grounded pcfg model @xcite . to enable vc-pcfg to transfer across different domains, we extend it by using pre-trained word embeddings and obtain transferrable vc-pcfg (model dubbed tvc-pcfg). this modification allows for directly applying vc-pcfg to a target domain, without requiring training on any data from the target domain. we learn tvc-pcfg on the mscoco imagecaption pairs @xcite and consider two evaluation setups: (1) proximate-domain transfer evaluates the model on the captions from another image-captioning data set; and (2) remote-domain transfer evaluates the model on sentences from 14 non-caption text domains, ranging from news, biography, and fiction to informal web text. our experiments show that vc-pcfg can transfer to similar text domains but struggle on text domains that are different from the training domain. to investigate the factors that influence the transferability of tvc-pcfg, we further conduct an analysis of the source-and target-domain data and the parsing results. we propose to measure the distance between the source domain and the target domain via the overlaps of phrasal labels, grammar rules, and words. our findings suggest that lexicon overlap plays an important role in the transferability of tvc-pcfg. for example, when tested on wall street journal data @xcite , we observe that the fewer words from a test sentence are present in the mscoco data, the lower the performance tends to be. ## conclusion and future work we have presented a simple approach that enables vc-pcfg to transfer to text domains beyond the training domain. our approach relies on pre-trained word embeddings and does not require training on the target domain. we empirically find that our tvc-pcfg is able to transfer to similar text domains but struggles to transfer to remote-domain text. through data and result analysis, we find that the lexicon overlap between the source domain and the target domain has a great effect on the transferability of tvc-pcfg. in the future, we would like to explore alternative transfer learning settings. for example, when training sentences from the target domain are available, we can additionally optimize the lm objective on them, apart from optimizing the full objective function on source-domain image-text pairs. moreover, we may consider a few-shot transfer learning setting and learn vc-pcfg only on a few target-domain sentences.