Lexicon-Level Contrastive Visual-Grounding Improves Language Modeling (2024)

Abstract: Today’s most accurate language models are trained on orders of magnitude more language data than human language learners receive— but with no supervision from other sensory modalities that play a crucial role in human learning. Can we make LMs’ representations and predictions more accurate (and more human-like) with more ecologically plausible supervision? This paper describes LexiContrastive Grounding (LCG), a grounded language learning procedure that leverages visual supervision to improve textual representations. LexiContrastive Grounding combines a next-token prediction strategy with a contrastive visual grounding objective, focusing on early-layerrepresentations that encode lexical information. Across multiple word-learning and sentence-understanding benchmarks, LexiContrastiveGrounding not only outperforms standard language-only models in terms of learning efficiency in small and developmentally plausible data regimes, but also improves upon vision-and-language learning procedures including CLIP, GIT, Flamingo, and Vokenization.Moreover, LexiContrastive Grounding improves perplexity by around 5% on multiple language modeling tasks compared to other models trained on the same amount of text data. This work underscores the potential of incorporating visual grounding into language models, aligning more closely with the multimodal nature of human language acquisition.

## introductions neural language models (lms; @xcite @xcite have shown utility in modeling aspects of human language processing, including neuronal responses to linguistic stimuli @xcite @xcite and data about human language production @xcite and comprehension @xcite . nevertheless, these models currently lack plausibility as models of human cognitive development. this discrepancy primarily stems from the immense volume of training data necessitated for effective lm performance, surpassing-by orders of magnitude-the linguistic input received during human language acquisition @xcite . specifically, children may be exposed to at most sixty million words in their first five years @xcite , whereas training modern lms requires hundreds of billions of words. can insights from human language acquisition guide the training of new lms that are both better cognitive models and more sample-efficient in an absolute sense? a key contrast in language learning between humans and lms is that humans ground language learning in perceptual signals across various modalities, encompassing hearing, touch, and vision @xcite @xcite . multimodal training has also been studied in natural language processing as a potential avenue towards more human-like language learning @xcite . encouragingly, recent years have witnessed a surge in the development of multi-modal models and learning algorithms, primarily tailored for tasks requiring simultaneous reasoning across both modalities @xcite @xcite @xcite . however, none of these multi-modal models have been shown to learn language more efficiently than language-only models. in fact, @xcite show that several different visual-language models (clip, git, and flamingo) learn word meanings less efficiently than models trained on language alone. this finding suggests that these existing visual-language learning algorithms cannot model how humans leverage vision to help learn language. however, @xcite also find that vision-language 116 102 perplexity 0.82 0.67 0.52 similarity to humans 0.20 0.17 0.14 0.45 0.40 0.35 0.86 0.83 0.80 accuracy c. mixed ungrouned -grounded learning vokenization git flamingo language-only lexi-voken grouding double same half double same half double same half double same half double same half number of tokens ratio from image-caption to language-only datasets lexicontrastive grounding forest beckons winter forest beckons with its the winter forest beckons with its mysterious purity. image feature visual encoder image feature the winter forest beckons with similar disimilar narrow attention with figure 1: lexicontrastive grounding models leverage visual information to facilitate word learning when they are trained on image-caption datasets. a. pretraining schema for the lexicontrastive grounding models. the images are sent to a frozen visual encoder pretrained using unsupervised learning algorithms to generate image features. these image features and the hidden representations of the first layer after the token-embedding layer are used to compute a vision-language contrastive loss. this loss is added to the next-token prediction loss to form the final loss. b. results from the grounded-only learning scenario on word-learning benchmarks for lexicontrastive grounding (•), language-only (■), clip (◆), git (▼), and flamingo (✖). the x-axis is plotted in the log scale. each point represents the average performance from four models initialized from different random seeds, and the line width represents the s.e.m. from these four models. c. results from the mixed-learning scenario on the language modeling and the word learning benchmarks. we also add lexivoken grounding (✚) and vokenization (♦) models. the ungrounded dataset is smashwords-5m. different dots of the same color represent models with different random initialization seeds. unlike the methods studied by @xcite , we find that lexicontrastive grounding yields improved learning efficiency compared to language-only models in small and developmentally plausible data regimes. we evaluate it, and other visual-language learning procedures, on two learning scenarios: grounded-only and mixed ungrounded-grounded language learning. in both scenarios, we systematically vary the size of the training datasets. additionally, we vary the source of the ungrounded corpus in the mixed-learning scenario with a focus on the amount of data children can receive. after training the models on the same amount of text data, we evaluate them on tasks that assess different aspects of word and language learning, including semantic similarity judgment, lexical relation and semantic norm prediction, and language modeling. lexicontrastive grounding outperforms existing methods on most of the evaluated benchmarks, demonstrating a consistent and significant benefit of visual grounding on language modeling compared to the languageonly models when trained on the same text data. to the best of our knowledge, lexicontrastive grounding is the first multi-modality learning algorithm to transfer benefits from visual grounding to (unconditional) language modeling when learning from the same amount of text data. these results show how natural language semantics can be better acquired by grounded learning, and suggest steps toward human-level efficiency in language learning with lms. ## background grounded language learning algorithms in ai. in recent years, there have been notable advancements in multi-modality learning. for instance, the vokenization model @xcite ) finds contextually relevant images (called "vokens") for language tokens and uses the vokens to additionally supervise the lms. although the vokenization model is shown to outperform language-only baselines on language-understanding benchmarks like mnli @xcite , the model is first finetuned on the corresponding training sets of these benchmarks. this finetuning process makes it unclear whether the learned representation is directly better than the representations of the languageonly models. the clip model produces transferable visual representations and word representations that demonstrate strong performance on certain tests of word similarity @xcite , after being trained contrastively on a massive amount of image-caption pairs. in contrast, git is a generative model that achieves state-of-the-art performance on various visual-language tasks, such as image captioning and visual question answering, through utilizing visual inputs to condition next-word predictions @xcite . the related flamingo model also yields strong results on these tasks by employing visual representations to modulate attention in a transformer language model @xcite . models of human language acquisition using ungrounded and grounded learning algorithms. al. (2021) and warstadt et al. (2023) target grammar learning in models trained on small ungrounded datasets. @xcite analyze word-acquisition trajectories in languageonly models, but their focus on model surprisal changes during training makes their findings less relevant to learning word meanings. for grounded models, @xcite and @xcite propose multi-modality algorithms for understanding word acquisition, with specific emphasis on word categories and function words, respectively. vong et al. (2024) apply a clip-like learning algorithm on first-person videos collected from children and report that the trained visual-language model yields good performance in visual-referent mapping tasks on the same child's experience and modest generalization ability to out-of-domain datasets. @xcite evaluate multiple existing visual-language models on word-learning benchmarks and show that visual-grounding yields limited and conditioned help in low-data regimes. our work introduces a stronger visual-language learning algorithm that outperforms these existing algorithms in different learning scenarios. ## discussion in this paper, we introduce lexicontrastive grounding, a visually grounded language learning objective motivated by models of grounded language acquisition in humans. lcg combines next-token prediction with a word-level contrastive visual grounding objective applied to early-layer representations. lcg not only outperforms other visual-language learning algorithms across various benchmarks, including evaluations of similarity, lexical relations, and semantic norms, but also surpasses traditional language-only models in learning efficiency on language modeling. this result underscores the potentially significant benefits of visual grounding in language modeling, offering insights into the role of multimodal learning in human-like language acquisition and suggesting a pathway toward more efficient and cognitively aligned language learning technologies. our analysis shows that the word meanings acquired by lexicontrastive grounding are more human-like when the words are concrete. this concreteness-based bias should not exist for a perfectly human-aligned word-meaning encoding. therefore, this shows that the representations learned by lexicontrastive grounding still differ from those in human adults. one possible explanation for this difference is that an additional learning mechanism is needed to augment the lex-figure 2 : lexicontrastive grounding models better learn concrete words than language-only models. a. scatter plot for an analysis on the word-relatedness benchmark for the lcg model trained with 2.1m image-caption pairs. each point on the plot corresponds to a pair of words, with its y-value indicating the relative rank obtained by sorting the word pairs based on the difference between human and model judgments. a greater y-value signifies a closer resemblance to human judgment. additionally, linear regression lines are depicted on the graph along with their respective 95% confidence intervals. b. the results of simverb-3500, a word-relatedness benchmark evaluating models only on verb words. the marker-to-model map is the same as that in fig. 1 . c. distributions of the per-word prediction performance difference between lexicontrastive grounding and language-only models grouped by concreteness of words. the prediction performance is the negative loglikelihood of the corresponding word averaged across all appearances in the test dataset. the lexicontrastive grounding and language-only models are taken from the "same" condition in fig. 1c . a positive difference means that the lexicontrastive grounding model is better than the language-only model. icontrastive grounding algorithm to better learn abstract words. another possible explanation is that our training corpus needs to be closer to what human adults perceive in both its quantity and distribution. since we only have at most 50m tokens in the training set, our learned representations may better capture what children have compared to what adults have. more experiments are needed to test these explanations.