Contrastive Language-Image Pre-Training with Knowledge Graphs
Anonymous Author(s)
Abstract
Recent years have witnessed the vast development of large-scale pre-training frameworks that can extract multi-modal representations in a unified form and achievepromising performances when transferred to downstream tasks.
Nevertheless, existing approaches mainly focus on pre-training with simple image-text pairs, whileneglecting the semantic connections between concepts from different modalities.
In this paper, we propose a knowledge-based pre-training framework, dubbed
Knowledge-CLIP , that injects semantic information into the widely used CLIPmodel [41].
Through introducing knowledge-based objectives in the pre-trainingprocess and utilizing different types of knowledge graphs as training data, ourmodel can semantically align the representations in vision and language, and alsoenhance the reasoning ability across scenarios and modalities.
Extensive experiments on various vision-language downstream tasks demonstrate the effectivenessof Knowledge-CLIP comparing with the original CLIP and competitive baselines.
Introduction
Large-scale vision-language pre
-training has attracted wide research interests in recent years [12,30, 41, 76].
Different from training different models for each specific task, pre-trained models takethe analogy of human biological intelligence system, trying to perceive the world from variousdata modalities and handle comprehensive tasks.
Specifically, it aims to provide a unified inferenceparadigm that simultaneously learns representations for multi-modal data and can easily transfer to avariety of downstream tasks.
Benefiting from the accessibility of massive image-text pairs from theweb, the pre-training scheme can leverage a broader source of supervision, and effectively improvesthe model’s generalization power.
Early attempts on vision-language pre-training mainly focus on detecting objects in the images andaligning the corresponding word tokens with object regions [12, 32, 54].
Though effective, theentanglement with the concept of objects, and the additional resources for pre-trained object detectorsimpose restrictions on real-world applications.
One of the pioneer works, CLIP [41], extends thescale of the pre-training dataset to 400 million image-text pairs, and learns representations by directlymatching raw text with the corresponding image.
Through a contrastive-based training scheme, CLIPlearns visual concepts under a large vocabulary which greatly improves the model performances onvarious downstream tasks.
Taking inspiration from CLIP, the following researches further extend thework from several perspectives, including data modality [76], downstream tasks [62], and trainingdata efficiency [24, 47].
Although showing promising results, the current pre-training frameworks also suffer from limitations.
Specifically, the data pairs for pre-training are organized in the simplest manner, where only thedescriptions of matched and unmatched are used to represent the relation between a given imageand text pair.
This usually leads to a degenerated scenario, where the model tends to rely on the correct label: airliner template: a photo of {} airliner military aircraft albatross taxicab high-speed train 0.
template: not a photo of {} airliner airplane wing military aircraft oxygen mask space shuttle 0.
correct label: sports car template: a photo of yellow {} sports car race car car wheel motorboat amphibious vehicle 0.
template: a photo of green {} sports car race car car wheel spaghetti squash convertible 0.
(a) Results on templates with opposite semantic description (b) Results on templates with wrong semantic description co-occurrence of inputs instead of their semantic meanings.
We give a toy example in Fig.
1 byevaluating the zero-shot transfer performance of CLIP on the ImageNet dataset [13] with the templates’a photo of a {}’ and ’not a photo of a {}’.
It is shown that the distributions of CLIP outputs undertwo templates are quite similar, suggesting that the current model fails to understand the semanticmeaning of word tokens.
As a result, the transferability of the model is restricted, and tends to showworse performances on tasks that require reasoning ability, e.g., visual question answering.
To address the limitation of pre-trained models on semantic perceiving, we resort to the technique ofknowledge graph, which has been widely studied in the field of natural language processing [10, 63].
Knowledge graph (KG) is a large-scale semantic network that comprises entities as nodes andsemantic relations as edges.
Through organizing data in a graph structure, knowledge graphs providerich information on describing the relations between entities and enable a reasoning process throughthe whole graph.
These advantages over regular-structured data are favorable on various tasksincluding question-answering [23, 74], relation prediction [33, 46] and knowledge reasoning [9, 64].
In recent years, knowledge graph has also been investigated in the field of computer vision, e.g. ,scene graph [69], and the integration of both language and image [2].
This bridges the gap betweendifferent modalities in the knowledge graph, which inspires us to explore a new knowledge-basedpre-training framework, and inject semantic information into simple image-text pairs.
In this paper, we propose a novel vision-language pre-training approach, dubbed Knowledge-CLIP , byconstructing a knowledge-enhanced pre-training framework based on the widely used CLIP models.
As illustrated in Fig.
2, we follow the structure of CLIP, and use two Transformer-based models asimage and text encoders respectively.
These two encoders take entities and relations in the knowledgegraph as input and extract raw features for both entities and relations.
Notably, entities can be inthe form of image/text, while the relations are constantly described by language tokens.
Then, amulti-modal Transformer encoder is adopted to fuse the entity features conditioned on their relations.
In this way, the pre-trained model is pushed to concentrate on understanding semantic relationsbetween visual and word concepts, thereby establishing strong semantic connections between visionand language modalities.
To additionally improve the training efficiency and avoid the massive computation cost in the pretraining procedure, we adopt a simple continuous learning strategy by training our model basedon the pre-trained weights of CLIP.
This provides a possibility of efficiently promoting the modelperformance of CLIP with low training resources.
We practically train our model on three knowledge graph datasets, namely Visual-Genome [29](scene graph), ConceptNet [49] (language-based graph), and VisualSem [2] (multi-modal graph), andalso adopt part of datasets from CLIP to avoid the model forgetting problem.
With the knowledgeenhanced pre-training, Knowledge-CLIP achieves consistent improvements over the original CLIPmodels on various vision and language downstream tasks.
Our model can also transfer to severalgraph-based tasks, including link prediction and entity classification, and achieve competitive results.
Related works
Large-scale pre
-training.
Large-scale pre-training framework has received wide concerns in recentyears and shown promising results in the field of computer vision and natural language processing.
GPT [42] is one of the pioneer works for language pre-training which optimizes the probability ofoutput based on previous words in the sequence.
BERT [15] adopts the masked language modelingtechnique and predicts the masked tokens conditioned on the unmasked ones.
XLNet [70] takes theadvantages of BERT and GPT, and combines the ability to model bidirectional contexts in BERT, andthe auto-regressive formulation in GPT which additionally improves the generalization performance.
Similarly, computer vision society also witnesses the development of pre-training models thanks tothe emergence of large-scale image datasets.
IGPT [7] proposes a generative pre-training techniqueand shows promising results on classification task.
MAE [21] adopts a similar pre-training scheme as
BERT and predicts the masked regions of an image with unmasked ones.
Another line of researches isbased on contrastive learning and uses siamese architectures for self supervision [5, 8, 22].
With theadvent of Transformer-based models in computer vision, large-scale datasets have gradually becomea common practice in the training process [16, 58].
Multi-modal pre-training bears differences from the aforementioned frameworks and requires thealignment between various data modalities.
Using enormous image-text pairs collected from Internet,vision-language models show significant improvements on various downstream tasks.
Amongthese approaches, various pre-training scheme is adopted, including contrastive learning [1, 31, 35],masked language modeling [50, 55], and masked region modeling [12].
Several approaches also usea pre-trained object detector to align the object with text concepts [12, 32, 54].
Comparing to previous approaches, we are the first to incorporate multi-modal knowledge graphsinto the pre-training process, and effectively enhance the model perception on semantic relationsbetween visual and language concepts.
Knowledge Graph.
Knowledge graph is first introduced in the field of natural language processing,and the knowledge graph embedding approaches have been successful on capturing the semanticsof symbols (entities and relations) and achieving impressive results on a wide range of real-worldapplications including text understanding [17, 71], recommendation system [20, 61] and naturallanguage question answering [23, 74].
On the other hand, scene graphs represent a type of graphstructured data in computer vision, where the visual concepts in the image are connected withsemantic relations.
Scene graphs emphasize the fine-grained semantic features for images and arewidely adopted in various downstream tasks, including scene graph generation [69], and Scene
Graph Parsing [73].
Besides scene graph, knowledge graph is also adopted in other computer visiontasks, including image classification [28], panoptic segmentation [67], and image captioning [75].
On this basis, multi-modal knowledge graph earns wide concerns in recent years.
Considering thenatural alignment between different data modalities, multi-modal knowledge graphs have been widelyadopted in various graph-based tasks including link prediction [3, 34], entity classification [66], whilealso showing great potential on out of graph applications like visual question answering [25, 44] andrecommendation systems [52, 56].
Contrastive Language-Image Pre-training (CLIP)
We first provide a brief review of model architectures and training settings in CLIP.
CLIP uses two separate models for image encoder and text encoder respectively.
For text inputs, a12-layer Transformer is adopted with 512 width and 8 attention heads.
Raw texts are first convertedusing byte pair encoding [43] technique under a vocabulary size of 49,152.
The text sequence length iscapped at 76 and added by a positional encoding before being sent into the text encoder.
On the otherhand, CLIP has different versions of image encoder with ResNet-based and Vision Transformer-basedarchitectures.
As the following researches have demonstrated the better performances of Vision
Transformer models, we only consider Transformer-based image encoders in this paper.
Similar tothe text input, images are first converted to patches, and added by a positional encoding.
At the laststage of both encoders, a global pooling function is adopted to compress the feature map into a singlefeature, which serves as the representation of the whole image/text sequence.
The cosine distance ofthe image and text features is computed as the similarity of the data pair.
For training supervision, acontrastive loss is adopted to maximize the similarity of matched pairs while minimizing the similarity
Knowledge-CLIP Knowledge-CLIP
Contrastive Loss
Knowledge-CLIP
MLP
Classification Loss
Knowledge-CLIP
GNN
Graph Loss
Contrastive j k l Image Encoder (CLIP Initialization) ...
... + + + + +
Text Encoder (CLIP Initialization) <
Has>
<A> <Castle> <in> <
Europe> + + + + + ...
Pooling + + + + + + + + + <head>
Multi-modal Encoder (Random Initialization) image pos embed image patches text pos embed text tokens
Example Input Triplet (h,r,t): has a Castle in Europe triplet sequence triplet embed of unmatched pairs.
Given a batch of N data pairs { I i , T i } Ni =1 , where I i and T represents the i thimage and text respectively, the loss function can be parameterized as:
[Equation]
where f I and f T correspond to image and text encoders, cos( · ) denotes the cosine similarity betweenthe inputs, and τ is a learnable temperature initialized at 0 .
While effective, this simple training framework actually brings several concerns that need to beaddressed.
First, the pre-training framework fails to model the semantic information of inputs dueto the simplicity of the data structure.
This results in inferior performances on tasks that requirereasoning ability, e.g. , visual question answering and visual commonsense reasoning.
Second, theimage and text features reside in separate spaces, which makes it difficult to model the interactionsbetween different modalities.
Third, the massive time and resource consumption in the trainingprocedure set restrictions on performing a full pre-training schedule from scratch.
Knowledge-CLIP
As we have summarized above, there are several concerns that hinder the transferability of CLIPand potential improvements on model performances.
In this paper, we propose a novel pre-trainingframework based on knowledge graphs, that addresses the limitation of the original CLIP modelfrom several perspectives: (1) We introduce knowledge graphs into the training dataset where thegraph-structured data and semantic relations between concepts enable the model to extract semanticfeatures and establish semantic connection across inputs; (2) A multi-modal encoder is added on topof the current image and text encoders to fuse the features from different modalities, and model thejoint distribution between inputs;
(3) A continuous learning strategy based on the pre-trained modelof CLIP is adopted which greatly avoids the massive computation cost in the pre-training procedure,and enhance the generalization power of the model efficiently.
We introduce our framework in detailin the following sections, and show the overview in Fig.
4.1 Data Preparation
Different from raw image-text pairs adopted in the original CLIP, our model takes knowledge graphsas input.
A knowledge graph can be defined as a directed graph G = { ξ, R , T R } , where ξ , Rcorrespond to sets of entities and relations, and T R represent the set of relation triplets.
A triplet( h, r, t ) ∈
T R denotes that entity h ∈ ξ has relation r ∈ R with entity t ∈ ξ .
As illustrated in Fig.
3,we pre-train our model on three types of knowledge graphs, including multi-modal knowledge graph,scene graph, and language-based knowledge graph.
Among these, relations are constantly describedin language tokens, where the entities are from different modalities in different forms.
Pollution
Water conservation
[Figure]
For multi-modal knowledge graph, the entities contain both illustrative images and language descriptions.
Through representing the same entity under various modalities and connecting entities withrelations, it helps to build semantic connections between vision and language concepts.
In practice,language and vision descriptions are randomly chosen for each entity.
In this way, the triplet set T Rcontains different forms including (Img, Rel, Img), (Img, Rel, Text), and (Text, Rel, Text), providingrich information across modalities while also enhancing perceptions within modalities.
Different from multi-modal knowledge graph, scene graph extracts visual concepts (mainly objects)for each image, and connects them with predefined semantic relations describing relative locations,actions, etc.
Therefore, the entities in the scene graph correspond to a certain region in an image, withthe triplet form of (Img, Rel, Img).
We practically use the selected regions as the input and discardthe irrelevant parts.
As two entities in the same triplet denote different regions in the same image, itforces the model to extract more fine-grained features.
Lastly, language-based knowledge graph connects words and phrases of natural language with labelededges.
It is built on only language modality with the triplet form of (Text, Rel, Text), while helping tobuild semantic alignment within word tokens.
4.2 Model Architecture
The model architecture and the training framework are illustrated in Fig.
2(A).
Specifically, we firstprocess the inputs into token sequences with modality-specific tokenizers.
The BPE tokenzier [43] isadopted for language inputs, while image inputs are sliced into non-overlapped patches and convertedinto a sequence of patches following ViT [16].
For convenient processing, we set the length of theimage sequence and text sequence as l I and l T respectively for all inputs.
To preserve the relativeposition information in the input, learnable positional encodings are added to the correspondingsequences before being sent to the model.
Two separate image encoder f I ( · ) and text encoder f T ( · ) are then adopted to extract features fromraw inputs.
For a given triplet ( h, r, t ) , the entities h and t are sent to the encoders with respect totheir modalities (image or text).
The relation r , which is represented by language tokens, is sent totext encoder similar to text entity.
Comparing to the model structure in CLIP, we introduce a modification to better adapt our framework.
Specifically, vanilla CLIP models use a pooling function at the last layer of two encoders to compressthe feature map into a global representation.
Namely, for an input u ∈ R L × d i , where L and d i denotethe sequence length and feature dimension, the output of the encoder can be formulated as:
[Equation]
where f represents the feature extraction module, Pool( · ) denotes the pooling function, and d o is theoutput dimension.
Though efficient, it also leads to inevitable information loss in the local region,especially for the image inputs.
Therefore, we remove the pooling functions for image and textentities to preserve the local information, and use x u ∈ R L × d o as the extracted feature.
The relation,on the other hand, is normally under a limited sequence length, e.g. , one or two word tokens, wherethe information density is smaller than entities.
Therefore, we retain the pooling function for relationinput and use ¯ x u ∈ R d o as the extracted features.
In this way, we have extracted the features defined as ( x h , ¯ x r , x t ) , which correspond to the elementsin the input triplet ( h, r, t ) .
To model the joint distribution of different elements in the triplet,we consider a multi-modal encoder TransEncoder( · ) to fuse the features from different sources.
Specifically, we first concatenate all the features in the triplet into a single sequence and use a headtoken < head > at the beginning of the sequence.
To emphasize the status of the tokens in thesequence, we consider additional learnable encodings for each element h, r, t in the triplet:
[Equation]
After processing by the multi-modal encoder, the feature of the head token < head > finally serves asthe representation of the whole sequence:
[Equation]
Also, representation for relation is extracted from the corresponding token:
[Equation]
4.3 Training Targets
Considering the unique data structure of knowledge graphs, we mainly adopt two types of trainingtargets in our framework, including triplet-based loss and graph-based loss as illustrated in Fig.
2(B).
Besides, a knowledge distillation loss is also considered due to the continuous learning strategyadopted in our framework.
Triplet-based loss considers a batch of triplets as the input and supervises the training of our model byestimating the joint distribution of elements in the triplets.
Inspired by the mask prediction techniquethat models the distribution of masked tokens conditioned on the unmasked regions, we similarlymask the elements in the triplets and predict the distribution with the help of a multi-modal encoder.
Specifically, for incomplete triplets where certain elements are missing in the input, the concatenatedsequence can be similarly derived as in Eq. 3 by masking the corresponding feature.
For example, theconcatenated sequence for an input ( h, r, - ) can be represented as:
[Equation]
On this basis, given a set of input D =
{ ( h i , r i , t i ) }
Ni =1 , we first model the distribution when one ofthe entities, i.e. , t i , is masked, and derive the Entity-Entity (E2E) Loss by minimizing the negativelog-likelihood:
[Equation]
We practically approximate the distribution P ( x t | x h , ¯ x r ) as the cosine similarity of P ( x t ) and
P ( x h , ¯ x r ) , and defined the loss function as:
[Equation]
We also model the distribution when the relation in the triplet is masked, and similarly derive the
Entity-Relation (E2R) Loss:
[Equation]
Different from E2E loss, the relations in the triplets are defined in a limited set of relation groups.
Therefore, we instead extract the representation of relation through an auxiliary two-layer MLPnetwork, and model the objective as a classification problem from a predefined set of relation labels
R .
In this way, the loss function can be defined as:
[Equation]
i ) log( y (¯ x r i )) , where y (¯ x r i ) =
MLP( R ( h i , - , t i )) , (10) is extracted from an MLP model followed by the output of multi-modal encoder defined in Eq.
Graph-based loss.
We also take advantage of the graph structure in knowledge graph datasets, andadopt a graph neural network to extract deeper structural information among entities.
We propagateinformation through connected edges in the graph, and update entity representations with aggregated feature.
Specifically, for a graph neural network with L layers, the update function for the l th layercan be formulated as:
[Equation]
where g ( l ) ( R ( h i , - , t
[Equation]
= W ( l ) R
[Equation]
h
[Equation]
12) calculates the aggregation weights by relation representation R ( h i , - , t ) with a learnable matrix W ( l ) .
Finally, we define the Graph-Entity(G2E) Loss by computing the cosine similarity of entity featuresbefore and after the propagation procedure in the graph:
[Equation]
Continuous Learning.
Large-scale pre-training usually requires massive computation resourceswhich makes it highly inefficient when training from scratch.
Therefore, to inject the semanticinformation in an efficient manner, we consider training our model based on the pre-trained weightsfrom the original CLIP.
This powerful initialization promotes the convergence of our model andgreatly enhances the training efficiency.
However, naively extending the training process with newdata leads to severe forgetting problem that hampers the performance of the original models.
To address this limitation, we adopt simple solutions to maintain CLIP performances while improvingits ability to extract semantic features from knowledge graphs.
(1) Besides the knowledge graphdatasets, we also train our model on several widely adopted image-text datasets that share a similardata distribution with the training data in CLIP.
To better fit our pre-training framework, we convertthe original image-text pair into the form of triplets, with specifically designed relations ’image of’and ’caption of’.
(2) We also use the original CLIP model as the teacher, and use an auxiliary loss
L KD to measure the KL distance between the output of CLIP and our model.
Overall, the final pre-training objective of Knowledge-CLIP is formulated as:
[Equation]
Experiments5.1 Implementation Details
Experimental Setup.
In all the experiments, we use the same model structure as CLIP [41].
A12-layer Transformer model with 512 width is adopted for text encoder, and ViT-L/14 is adoptedfor image encoder.
For text and image encoder, we use the pre-trained weights in the original CLIPas the initialization.
For the multi-modal encoder, we consider a 4 layer Transformer model with1024 width.
The rate for drop path is set as 0.1 during training.
As the added multi-modal encoder istrained from random initialization, we decrease the learning rate for the pre-trained weights from
CLIP to achieve a more balanced step in the optimization.
We train Knowledge-CLIP with an initiallearning rate of 1e-5 for image and text encoders, and 1e-3 for the multi-modal encoder.
Cosinelearning rate with linear warmup is used in the training schedule.
Weight decay and gradient clip arealso adopted.
See more details in the supplemental material.
Pre-train Dataset.
Three knowledge graph datasets are adopted in the pre-training process.
VisualSem [2] is a high-quality multi-modal knowledge graph dataset for vision and language concepts,including entities with multilingual glosses, multiple illustrative images, and visually relevant relations, covering a total number of 90k nodes, 1.3M glosses and 938k images.
13 semantic relationsare used to connect different entities in the graph, while the entities in VisualSem are linked to
Wikipedia articles, WordNet [38], and high-quality images from ImageNet [13].
Visual Genome [29]is a knowledge-based scene graph dataset that connects structured image concepts with semanticrelations.
Visual Genome serves as the benchmark for various vision tasks, e.g. , visual grounding,and scene graph generation.
ConceptNet [49] is a knowledge graph that connects words and phrasesof natural language with labeled edges.
Its knowledge is collected from many sources includingexpert-created resources and crowd-sourcing built on only language modality.
Besides the three knowledge graph datasets, we also train our model on two widely adopted imagetext datasets that share the similar data distribution with the training data in CLIP.
We practically add
[Table]
COCO Caption [11] and CC3M [45] to the training set, while large-scale datasets like CC12M [6] or
YFCC [27] are not considered to maintain training efficiency.
Downstream Task.
To validate the effectiveness of our framework, we conduct experiments onvarious downstream tasks, including multi-modal tasks like text and image retrieval, visual questionanswering [Table] and uni-modal tasks like image classification and natural language understanding.
Wealso show the performances of our models on several knowledge-based tasks including link predictionand triple classification, where our model can benefit from the graph-based training schedule.
5.2 Multi-modal Tasks
Image and text retrieval.
We first conduct experiments on Flickr30k [40] and COCO Caption [11]dataset to show the performances of our model on image-text retrieval tasks.
Given input sets Xand Y of images and texts, we use Knowledge-CLIP to extract features for each input, and modelthe joint probability with the cosine similarity between image and text pairs.
We summarize thecomparison results of Knowledge-CLIP with competitive baselines in Tab. 1.
It is shown that ourmodel consistently achieves better results over the original CLIP on both datasets, while comparablewith competitive baselines like OSCAR.
[Table]
Visual question answering / Visual Entailment.
We also validate the effectivenessof Knowledge-CLIP on other vision-languagetasks, including VQA [19] and SNLI-VE [68].
We show the comparison results in Tab. 2.
Comparing to competitive baselines including
VILLA [18] and ALBEF [30], KnowledgeCLIP with ViT-L/14 shows better performancesunder all settings, while the smaller model alsoachieves competitive results.
Comparing to theoriginal CLIP model, our pre-trained modelpractically improves its transferability on downstream tasks, especially on the datasets like VQA that requires reasoning ability.
5.3 Uni-modal Tasks
[Table]
Image Classification.
To further demonstrate the generalizationpower of Knowledge-CLIP, we compare the performances of pre-trainmodels on the ImageNet classification task [13].
We summarize thecomparison results in Tab. 3, and show that Knowledge-CLIP can alsohandle vision tasks well.
We argue the improvements over baselinesmay attribute to the scene graphs in our pre-training dataset, whichemphasize the visual concepts in the images.
Language Understanding.
We validate the generalization performance of Knowledge-CLIP forlanguage understanding tasks on the widely adopted GLUE dataset [60].
Specifically, we conductexperiments on 7 tasks in GLUE and summarize the comparison results in Tab. 4. It is shown thatour model achieves comparable performances with competitive baseline models. Also, for tasks like
[Table]
[Table]
[Table]
[Table]
[Table]
501 RotatE [53] 0. [Table] [Table] [Table]
[Table] [Table] [Table] InteractE [59]
[Table]
[Table] [Table] [Table] [Table]
Ours [Table] [Table] [Table]
[Table]
QQP and MNLI that require sentence-pair matching, Knowledge-CLIP shows higher performances,due to the existence of language triplets in the pre-training dataset.
5.4 Knowledge-based Tasks
Benefiting from the graph-based learning framework in the pre-training process, our models enjoyadvantages on several knowledge-based downstream tasks.
Therefore, we conduct experiments onlink prediction, entity classification and triple classification tasks.
Link prediction task aim to recover an incomplete triplet when one of the entities is masked, i.e. ,predicting entity h given ( - , r, t ) .
This task shares certain similarities with our pre-training objectives.
We validate the performances of our model on the WN18RR [14] and FB15K-237 [57] datasets, where
MR (MeanRank), MRR(Mean Reciprocal Rank), and Hit@n are adopted as the evaluation metrics.
As shown in Tab. 5, Knowledge-CLIP is able to perform competitive performances comparing toseveral baseline models, and achieves better results on 3 of 5 metrics.
[Table]
Triple classification requires the model to distinguish matched triples from unmatched ones,which can serve as a binary classification task. We validate
our model on YAGO39K 37]dataset, with Accuracy, Precision , Recall, and
F1-Score as the evaluation metric.
It is shown
inTab that our model shows promising resultsover competitive baselines.
Conclusion
In this paper, we propose a novel vision-language pretraining framework that incorporates knowledgeinformation to model the semantic connections between vision and language entities.
We introducethree types of graph-structured datasets into the training process, and adopt a multi-modal encoder tomodel the joint distribution of entities and their semantic relations.
Extensive experiments on variousdownstream tasks including multi-modal, uni-modal, and graph-based tasks validate the transfer andgeneralization ability of our model.
Our approach is now limited in injecting knowledge informationinto the CLIP models.
However, our training objectives and new knowledge graph datasets aretechnically compatible with other large-scale pretraining frameworks.
We will explore the possibilityof further applications in the future.