VLG - BERT : Towards Better Interpretability in LLM s through Visual and Linguistic Grounding (2025)

Abstract: We present VLG-BERT, a novel LLM model conceived to improve the language meaning encoding. VLG-BERT provides a deeper insights about meaning encoding in Large Language Models (LLMs) by focusing on linguistic and real-world semantics. It uses syntactic dependencies as a form of a ground truth to supervise the learning process of the words representation. VLG-BERT incorporates visual latent representations from pre-trained vision models and their corresponding labels. A vocabulary of 10k tokens corresponding to so-called concrete words is built by extending the set of ImageNet labels. The extension is based on synonyms, hyponyms and hypernyms from WordNet. Thus, a lookup table for this vocabulary is used to initialize the embedding matrix during training, rather than random initialization. This multimodal grounding provides a stronger semantic foundation for encoding the meaning of words. Its architecture aligns seamlessly with foundational theories from across the cognitive sciences. The integration of visual and linguistic grounding makes VLG-BERT consistent with many cognitive theories. Our approach contributes to the ongoing effort to create models that bridge the gap between language and vision, making them more aligned with how humans understand and interpret the world. Experiments on text classification have shown an excellent results compared to BERT Base.

## introduction the growing need for interpretability and grounding in large language models (llms) is driven by their increasing use in critical and diverse applications, as well as ethical, practical, and technical challenges. llms assist in diagnosing diseases and generating treatment plans. they are also used for contract analysis and legal reasoning. they personalize the learning experience for students. despite their outstanding performance in many downstream tasks, llms often produce plausible but factually incorrect outputs, referred to as hallucination. this behavior results from their reliance on patterns in training data rather than true semantic understanding. llms must provide explainable insights about their black-boxes. their decisions must meet legal and ethical standards. therefore, interpretability allows users to trace the reasoning or data sources behind a model's outputs, providing accountability. the integration of visual real-world data and domain knowledge into llms, could be good lead to anchor their responses to verifiable facts. textbased llms have made significant advancements in natural language processing. llms two fundamental learning policies are next-word generation and bidirectional representation. the first approach is used for text generation, by predicting the next word based on prior context. the second approach focuses on understanding text by predicting masked words using both left and right context. however, these models have notable limitations when it comes to representing meaning, particularly in relation to real-world semantics. while llms excel at capturing contextual relationships between words, they do not inherently ground meaning in the real-world, unlike humans who learn language through sensory and perceptual experiences. in this paper, we introduce vlg-bert, a multimodal model which combines syntactic knowledge and visual grounding to improve word representation learning. it extends our recent modal capabilities to incorporate real-world semantics. unlike traditional models that learn embeddings solely from textual space, vlg-bert uses latent representations of real-world concepts to learn embeddings. latent representations are extracted from the vision transformer (vit) trained on the imagenet dataset. vlg-bert aims to go beyond the purely textual space as the only source of words representation learning, by involving the real-world semantics in the learning process. this grounding bridges the gap between vision and language, allowing the model to process and encode richer semantic information. it is also particularly useful for multimodal downstream tasks. vlg-bert is also designed to inject syntactic knowledge into the attention mechanism using augmented lagrange multipliers. the model employs syntactic dependencies as a form of ground truth to supervise the learning process of word representation, thereby ensuring that syntactic structure exerts an influence on the model's word representations. the application of augmented lagrangian optimization imposes constraints on the attention mechanism. it makes the learning of syntactic relationships easier. this approach involves the customization of the prediction layer of the standard bert architecture. the objective is to predict an adjacency matrix that encodes words' syntactic relationships rather than masked tokens. vlg-bert merges a bottom-up, data-driven approach with a top-down, rule-driven approach. furthermore, vlg-bert brings clear insights about the interpretability of transformer-based models 2 related work transformer models like bert and its variants have paved the way for great advancements in nlp. these models are primarily geared towards modeling the semantics of language. they've resulted in tremendous performance in many different fields @xcite @xcite @xcite . the scientific community developed new versions of bert as a consequence of the inaccurate results in some downstream tasks and appraisal of the linguistic properties of the natural language @xcite @xcite . some of the proposed models aim to inject linguistic knowledge into transformer models, while others try to ground language via visual data. syntactic connections between words are not just what lends language its richness, but are also what make meaning beyond mere word correlations @xcite . one way of adding syntactic knowledge to transformer models is syntax-bert. it is an extension of the original bert that introduces explicit syntactic information through syntax trees and instructs the self-attentional system in relation to linguistic dependencies such as parent, child, and sibling. this strategy preserves bert's pre-trained expertise and combines it with structure and efficiency to help it better excel in nlp scenarios when syn-tactic clarity is required or data is finite. syntax-bert is a system that allows syntax trees to be included during fine-tuning without the need to train from scratch @xcite . the syntactic knowledge via graph attention with bert is another proposed model which adopts syntactic knowledge injection into transformer models. sgb is a machine translation dedicated model. it explicitly uses the syntactic dependency knowledge via graph attention networks (gat) and bert-based encoders. the gat treats syntactic structures as graphs, enhancing token representations with dependency relations. it also combines them with bert outputs through two methods. the first one is called sgbc. it concatenates bert and gat outputs for encoder-decoder attention. the second one is sgbd (decoderguided syntax). this approach leaverage a translation fluency @xcite . in addition to the syntax-aware model in transformer models, visionoriented models have emerged. one of these models has been developed with the objective of grounding natural language in visual data is visualbert. it is based on the architecture of bert. visual-bert uses image-text alignment to ground language in visual contexts. it employs cross-attention layers to establish a connection between the visual and textual modalities. visual information is conveyed through a convolutional neural network (cnn) to extract visual embeddings, which are subsequently integrated with the textual embeddings. the cross-modal attention layers grant bidirectional influence between text and image representations during the encoding process. visual-bert employs a fusion strategy that unites textual tokens and visual features within a unified transformer @xcite . lxmert, which stands for learning cross-modality encoder representations from transformers is a multimodal model. it processes both visual and textual data. it uses a cross-attention mechanism to merge the image and text features. lxmert architecture is based on two-stream transformer. the first stream processes the visual features. it consists of image regions such as objects and objects parts encoded by a pretrained faster r-cnn model. the encoded visual features are then fed into lxmert to learn contextual relationships between image regions. the second stream processes textual features. it comprises bert's word embeddings. both streams interact with each other through cross-attention encoder. this interaction enables the model to learn relationships between the image and its corresponding textual description @xcite . the list of multimodal models is too long to fit within the limited number of pages of this paper. without dissecting technical details, we mention among others, uniter, imagebert, and multimodal-bert, which are transformer-based models. they are conceived to connect visual and textual data in order to improve the performance in multimodal tasks @xcite @xcite . uniter, universal image-text representation learns joint embeddings by pretraining on diverse image-text datasets, enabling tasks like image-text retrieval and visual question answering @xcite . similarly, image-bert depends on a shared embedding space and cross-modal interaction to align text and images @xcite . in turn, multimodal-bert customize bert's architecture to handle multimodal inputs. it is particularly dedicated to applications like medical image and text classification @xcite . the research community is moving toward the integration of visual and textual data to encode the meaning of language. these models offer an excellent way of grounding the language by aligning visual information, such as images, with textual context. in the next sections, we present vlg-bert, a multimodal model which combines syntactic knowledge and visual grounding to improve word representation learning. ## two categories of words the present work assumes two categories of words. the first is called concrete words, while the second is called abstract words. the former refers to all the words that have a physical referent in the real world. the latter refers to all words that do not have a physical referent in the real world. from a cognitive sciences point of view, the term realworld here differs from lakoff's definition @xcite . it is more in line with the definitions of materialism and empirical realism. ## visual grounding most llms use a random initialization to learn word embeddings. we propose a human-like model by initializing the embeddings matrix of words with their corresponding latent representation from the real world. in other words, the visual grounding in vlg-bert consists of using the latent representations extracted from the vision transformer vit. the latent representations are learned by vit based on the imagenet dataset, which contains 1000 labels or classes corresponding to real objects @xcite . we extend the vocabulary by building a lookup table that corresponds to our embeddings matrix, using wordnet. the vocabulary extension uses synonymy, hyponymy and hypernymy relations @xcite . semantically similar words are extended using wordnet semantic relations. hyponyms are more specific terms, while hypernyms are general terms or categories. the semantic similarity of hyponyms should be more similar to each other than to their hypernyms. this can be done by incorporating hierarchical wordnet semantic relations. in other words, several path-based similarity measures can be used to compute the shortest path between two words in the hypernym-hyponym tree. the shorter the path between the two words, the more semantically related they are. finally, the lookup table is implemented using json, where keys are the token ids and values are the latent representations before and after regularization. the second category of words which have no referent in the real world, are randomly initialized as in traditional llms. the metric that measures the relationship between a word w and its hyponym w hypo , and its hypernym w hyper is given by : r(w, w hyp , w hyper ) = λ • max 0, pathdist(w, w hyper )-pathdist(w, w hypo ) + δ . the intuition behind this regularization is to penalize the model when the path distance between a word w and its hypernym w hyper is smaller than the path distance between the word and its hyponym w hypo . using the above metric, we compute hyponyms and hypernyms latent representations. thus, we built a vocabulary of 10 000 concrete words. it takes the form of a lookup table. it is used to initialize the embeddings. if the word is concrete and does not exist in the lookup table, we initialize it randomly. ## linguistic grounding vlg-bert is a syntax-aware model. it is designed to inject syntactic knowledge into the attention mechanism. it uses augmented lagrange multipliers as a constraint based convex optimization method. vlg-bert deploys syntactic dependencies as a ground truth to supervise the learning process. the syntactic relations between the sentence words are encoded in an adjacency matrix. vlg-bert is forced to predict a matrix that approximates the adjacency matrix that encodes the syntactic relations between words. the use of the augmented lagrangian optimization method is an innovative way of integrating constraints into attention mechanisms. the prediction layer of the standard bert architecture is customized to predict the syntactic matrix. ## conceptual model the model is based on transformer architectures and incorporates syntactic dependencies through the use of an adjacency matrix, m . m is used to encode the syntactic dependencies. during the training phase, it is employed as the ground truth to converge toward. the positional encoding is kept as in bert base, while the next sentence prediction is not integrated. ## vlg-bert under the spotlight of cognitive sciences llms learn the probability distribution of sequences of words in natural language. they are designed based on the idea of maximizing the probability of certain words under certain conditions. this can be the next word in a sequence, or a masked word. in an auto-regressive model, given a sequence of words w 1 , w 2 , . . . , w n-1 , the model learns to predict the probability distribution for the next word w n . unlike the auto-regressive model, bidirectional models learn to predict a word by conditioning on both the preceding and succeeding words in the sequence. given a sequence of words w 1 , w 2 , . . . , w n , the model predicts a representation for each word by conditioning on both the left and right context. the llms community considers next word prediction models to be text generation models, while they consider bidirectional encoding models to be text understanding models. the integration of different sensory modalities is necessary to humans to perceive and understand the world. the architecture of vlg-bert can be seen as a computational model that mimics humans by combining textual and visual data for a better and deeper encoding of the language meaning. vlg-bert aligns with many theories like symbol grounding. symbol grounding refers to the association of the abstract symbols like words with real-world objects. in cognitive science, grounding is fundamental to how humans link linguistic symbols to sensory experiences like seeing an apple. in embodied cognition theory, the mind is considered to be rooted in the body's interactions with the world. this implies that understanding comes from both perceiving and acting in the world. vlg-bert aligns with the idea of embodied cognition by grounding language in visual data. the representations in vlg-bert approximate rosch prototypes theory @xcite by clustering features from both latent visual features and linguistic domains, improving generalization for concept categories. vlg-bert aligns with dual coding theory @xcite ) that combines verbal and imaginal codes that reinforce the comprehension and the retrieval of concrete concepts. by combining visual signs and linguistic signs, vlg-bert aligns with peirce's triadic model of signification, offering a robust semiotic framework for word meaning. the visual and linguistic signs can be considered as iconic and symbolic representa-mens while the learned embeddings of words like interpretants @xcite . ## architecture the proposed architecture consists of two interconnected components: the bert base and a customized prediction layer. the former is bert base follows the standard transformer architecture, which operates without any constraints and leverages gradient descent optimization and the latter is the modified prediction layer that introduces a novel constraint-based optimization mechanism using augmented lagrangian optimization. at the input layer, lookup table is used to map visual latent representation to corresponding tokens of the sentence to initialize the embedding matrix. ## experiments in order to evaluate and test vlg-bert, the same datasets already used by bert were employed: the english wikipedia dump and bookcorpus. the wikipedia dump yielded 16 gb of plain text. in turn, bookcorpus provides access to a substantial corpus of over 11,000 free, unpublished books sourced from the internet. to ensure a meaningful comparison with bert and its derived models, we used a high performance hardware configuration. the training was conducted on a commercial cloud platform utilizing 8 gpus, 128 gb of ram and 32 vcpus cores. for model evaluation, we concentrated on a text classification task. to evaluate the generated embedding from vlg-bert, the ag news dataset is used to focus on categorizing news articles into predefined categories. hyperparameters are defined as follos λ for equation 1 is 0.01,µ for equation 4 is 0.001, learning rate: 2×10 -5 , train batch size: 16, evaluation batch size: 8, seed: 42, optimizer: adam with β 1 = 0.9, β 2 = 0.999, and ϵ = 1 × 10 -8 , number of epochs: 30. while bert-base took around 96 hours to train on 16 tpus, we notice that vlg-bert, on the other hand, took a longer training time of 122 hours. this is expected because the hardware configuration in that case was less powerful than that of bert-base. this highlights the efficiency of the learned embeddings with vlg-bert. it confirms that the model converged effectively, demonstrating the benefits of visual grounding and the use of constraint-based optimization with an augmented lagrangian to reduce training time. the comparison of the two models on the ag-news dataset shows that vlg-bert outperforms bert base in all metrics. vlg-bert scored the highest accuracy (97.56%) and f1-scores for all classes. it demonstrates notable improvements in precision, recall, and f1-scores. compared to scabert, which benefits from only syntactic grounding. ## conclusion vlg-bert has valuable contributions from both computer science and cognitive science standpoints. computer science, with regard to the advance of multimodal learning, it efficiently combines visual and linguistic data that could lead to richer, more robust representations of words. the integration of visual grounding with textual information enables this model to handle complex, real-world tasks more efficiently. such a setup from a cognitive science viewpoint is in consonance with vlg-bert, as it grounds the words in the physical world, incorporating syntactic structures to mirror computationally human-like understanding of concepts. the model supports the perceptual gap between language and vision, representing and leveraging visual and linguistic inputs cohesively to interpret the world, much like humans. this will be further demonstrated by future comparisons with models like visualbert, lxmert, and clip, especially on multimodal tasks such as image captioning and visual question answering. these will serve to underline its ability to integrate visual, syntactic, and semantic knowledge to provide a deeper understanding of multimodal interactions.