K-L ITE : Learning Transferable Visual Models with External Knowledge
Abstract
The new generation of state-of-the-art computer vision systems are trained from natural language supervision, ranging from simple object category names to descriptive captions.
This form of supervision ensures high generality and usability of the learned visual models, due to the broad concept coverage achieved via largescale data collection process.
Alternatively, we argue that learning with external knowledge is a promising way which leverages a much more structured source of supervision and offers sample efﬁciency.
We propose K-L ITE 1 , a simple strategy to leverage external knowledge for building transferable visual systems:
In training, it enriches entities in text with WordNet and Wiktionary knowledge, leading to an efﬁcient and scalable approach to learning image representations that uses knowledge about the visual concepts.
In evaluation, the text is also augmented with external knowledge and then used to reference learned visual concepts (or describe new ones) to enable zero-shot and few-shot transfer of the pre-trained models.
We study the performance of K-L ITE on two important computer vision problems, image classiﬁcation and object detection, benchmarking on 20 and 13 different existing datasets, respectively.
The proposed knowledge-augmented models show signiﬁcant improvement in transfer learning performance over existing methods.
Introduction
One of the core aspirations in computer vision (CV) is to develop systems that endow computers with the ability to effectively learn general visual representations, which can be transferred to a variety of downstream recognition datasets with arbitrary visual concepts in the wild.
Though excellent performance has been achieved on standard benchmarks, the traditional supervised approaches are limited to learning a ﬁxed set of concepts, e.g., 22K concepts on ImageNet [15] or 18K concepts on
JFT-300M [82].
This leads to a few issues: ( i )
Annotating each individual vision dataset is not only labor intensive, but also results in a narrow set of visual concepts; ( ii ) Visual models trained on such datasets are good at one task (with the given concept set) and this task only, and show poor transfer learning performance to customized datasets that usually come with a different set of concepts [27].
To tackle this problem, recent large-scale language-augmented visual models, such as CLIP [71],
ALIGN [36] and Florence [101], are trained on a wide variety of images with natural language supervision that is abundantly available on the Internet.
These models demonstrate strong zero-shot transfer capabilities, since they acquire open-set recognition abilities through problem reformulation from classiﬁcation to retrieval.
Moreover, model generalization is improved as natural language supervision typically contains rich semantics.
While these models usually perform well on recognizing common objects, they still struggle on visual concepts that are absent or rare in the pre-training stage.
To ensure good transfer performance, it is required to train such models on huge datasets with sufﬁcient concept coverage( e.g., >400M image-text pairs), which is both labor and compute expensive.
Instead of scaling the number of image-text pairs to increase concept coverage, we propose to leverage structured external knowledge to augment language supervision.
The inspiration comes from how humans generalize to novel concepts: instead of trying to memorize all concepts, humans leverage the structured knowledge such as deﬁnitions and concept hierarchy.
For example, when we visit a
Japanese restaurant for the ﬁrst time, we may struggle to understand the menu by only looking at the dish names ( e.g., Takoyaki, Sashimi ), as it is hard to imagine what they are.
However, it becomes much clearer once a waiter introduces these concepts (Figure 1), leading to success in ordering food ( i.e., matching content to a name).
Similar intuitions have been exploited in computer vision for class-level transfer [94, 12], but not yet for task-level transfer settings (similar to that of CLIP).
[Figure]
To this end, we explore a systematic approach to acquire and learn with external knowledge sources from databases such as WordNet [63] and Wiktionary [62] to train more transferable and sampleefﬁcient visual models.
The concept descriptions and concept hierarchies are purely textual, and the process of collecting external knowledge is fully automatic without extra human annotation.
The acquired knowledge typically provides information that is shared between seen and unseen concepts to facilitate effective transfer.
Speciﬁcally, rare concepts, e.g., Takoyaki, Sashimi in Figure 1, are explained with more common concepts.
Such knowledge sources are generally available for a variety of domains and datasets, making it possible to build a generic approach for task-level transfer
Our main ﬁndings and contributions can be summarized as follows:
• We present the ﬁrst strong evidence that external knowledge can beneﬁt large-scale task-level transfer for two core CV problems, image classiﬁcation (IC) and object detection (OD), by exploring external knowledge sources, including WordNet and Wiktionary.
• A simple and effective strategy K-L ITE (Knowledge-augmented Language Image Training and Evaluation) is proposed:
The acquired knowledge is appended to the original textual concepts as model input during pre-training and evaluation.
It can be viewed as an automatic knowledge-aware language prompting, which makes it easier for the model to access relevant information shared between the training and evaluation data.
A modularized approach is also developed to enable efﬁcient adaptation from vanilla visual models to their knowledge-augmented versions.
• To demonstrate the generality of the K-L ITE , we instantiate it with two recent visual models and develop our knowledge-augmented counterparts: UniCL [95] for IC and GLIP [50] for OD.
Extensive experiments in zero-shot and few-shot learning settings demonstrate that knowledgeaugmented models can signiﬁcantly improve over prior work.
Notably, our model can achieve similar zero-shot performance to previous methods using only half of pre-training image-text pairs in some scenarios, demonstrating sample efﬁciency of the proposed approach.
Related Work
Zero-shot Visual Recognition: Zero-shot learning, i.e., classifying images where there is a lack of labeled training data, has been studied for decades [21], and its popularity has recently increased further [94].
Based on the technique evolution, it can be broadly categorized into two generations: the traditional class-level zero-shot and recently popular task-level zero-shot setting.
Class-level Transfer.
Class-level zero-shot learning aims to recognize object classes whose instances have not been observed during training.
The goal for the zero-shot learning methods is to associate observed and non-observed classes through some form of auxiliary information, which can be either implicit such as pre-trained semantic embeddings [92, 78, 10], or explicit such as attributes [21, 43, 35], text [18, 19, 72, 70], knowledge graphs [91, 74], or rules and ontologies [23].
Please, refer to the recent survey on knowledge-aware zero-shot learning for a more detailed review [12].
Recently, it has been suggested to move away from the restricted nature of standard zero-shot evaluation and make the task more practical by including training classes at test time, i.e., generalized zero-shot learning setting [94].
Despite the progress in this area, the traditional setting is typically limited to studying zero-shot transfer across classes in a single domain with manually deﬁned splits, such as
Animal with Attributes (AwA) [44], Birds-200 [89], SUN attributes [67], and ZS-ImageNet [73, 26].
Concurrently, [85] explore leveraging external knowledge to improve long-tailed visual recognition within individual domains, which falls into the category of class-level transfer.
Task-level Transfer.
Another line of work focuses on task-level zero-shot transfer [52, 46, 64, 71, 36, 97].
They pre-train visual models on hundreds of millions of web-crawled image-caption or image-tags pairs, and evaluate their transfer ability by directly performing inference in a wide range of downstream datasets, without tuning the model weights.
We argue that the task-level transfer is more practical and attractive than class-level transfer, as it is more relevant to real-world scenarios, where we may want to develop models that can serve many visual recognition applications.
Our work bridges the gap between the two lines of works above: it borrows the spirit of exploring knowledge in class-level transfer, and generalizes it for task-level transfer, leveraging the best of both worlds.
To summarize, our work is different in two major aspects: ( i ) Settings.
We focus on the task-level transfer learning across domains, i.e., from large publicly available datasets to a diverse set of downstream datasets in different domains, and demonstrate that external knowledge beneﬁts task-level transfer.
( ii ) Modeling.
Existing class-level transfer works are built upon pre-trained visual features/backbones and shallow word embeddings or tf-idf scores, and only train the classiﬁers.
One representative example is DeViSE [25], where a skip-gram word embedding model and an image classiﬁer are ﬁne-tuned jointly.
In contrast, we are training large Transformer-based models in an end-to-end manner from scratch as in CLIP/ALIGN, providing the ﬁrst empirical evidence that external knowledge can help train a general visual backbone.
Knowledge-Intensive Models:
In natural language processing (NLP), with the increase of model capacity via pre-trained language models [17], there emerges the need for more knowledgeable models [55] with advanced functionalities such as making use of encyclopedic [88, 4, 6] and commonsense knowledge [79, 102].
To address this, a large number of language models augmented with external knowledge sources have been proposed [68, 29, 45, 56, 100, 7], achieving strong performance on a variety of NLP tasks [69, 42].
Please refer to a recent survey [98] for a comprehensive review.
In vision-and-language (V+L) domain, researchers have also started exploring knowledge-intensive tasks, e.g., OK-VQA [61] and WebQA [9].
They often require additional information sources ( e.g., factual and commonsense knowledge) beyond the QA pairs, compared to the established tasks such as VQA [3, 34] and image captioning [54, 2].
Hence, existing pre-trained models [49, 83, 58, 51, 81, 103, 38, 48, 30, 99, 77] would perform poorly on these knowledge-intensive V+L tasks [9].
To address the problem, acquiring external knowledge becomes an essential component for success [93, 60, 96].
The success of knowledge in NLP and V+L tasks inspires us to ask a natural question: Can we learn a transferable visual backbone model with external knowledge?
Thus, we dissect and borrow the vital elements such as knowledge sources [63, 62] and modeling techniques [90, 100], and carry out studies for core computer vision tasks.
Knowledge-Augmented Visual Models
Problem setup.
Computer vision systems have achieved strong transfer performance, when learning with large-scale image-label data [39] and image-caption data [71].
Recently, it has been demonstrated in [95, 101] that the uniﬁcation of image-label and image-text formats into image-text-label achieves superior performance over either of them.
We follow the setting in [95], and deﬁne a uniﬁed triplet-wise data format D
= { ( x n , t n , y n ) } Nn =1 , where x 2 X is an image, t 2 T is its language description, and y 2 Y is a label indicating the index of the unique language description in the dataset.
In a general form, the language description is a text sequence t =
[ t 1 , · · · , t L ] .
It ranges from simple category names representing visual concepts when L is small, to more free-form and semantic-rich sentences such as captions when L is relatively large.
In this paper, we assume there exists an external knowledge source S , where one may use the language description t as a query to seek additional knowledge description s 2 S for t .
Given these triplet data instances D and an external knowledge source S , our goal is to learn generic visual-
[Table]
+gcnjU1nGqCG2RmMeqG2JNOZO0ZZjhtJsoikXIaSec3M79zhNVmsXywUwTGgg8kixiBBsrPbpZP4yQnqFBperVvBxolfgFqUKB5qDy1R/GJBVUGsKx1j3fS0yQYWUY4XTm9lNNE0wmeER7lkosqA6y/OAZOrPKEEWxsiUNytXfExkWWk9FaDsFNmO97M3F/7xeaqLrIGMySQ2VZLEoSjkyMZp/j4ZMUWL41BJMFLO3IjLGChNjM3JtCP7yy6ukfVHzL2v1+3q1cVPEUYYTOIVz8OEKGnAHTWgBAQHP8ApvjnJenHfnY9FacoqZY/gD5/MHtuiPtg==</latexit> s
Knowledge
[Figure]
Language
Query
Knowledge
[Figure]
Knowledge
[Figure]
semantic representations, which are readily transferable to a wide range of downstream datasets, whose category names are not necessarily observed during training.
In Figure 2, we visually illustrate the proposed knowledge-augmentation process and two considered application scenarios.
3.1 External Knowledge
Query Construction.
For a text sequence associated with an image, different tokens may play different roles in contributing to describing the main semantics of the image.
Humans leverage this prior inherently in parsing the sentences to understand the image.
Further, it is infeasible to employ the entire sequence as a query, as this may lead to the lack of coverage in the knowledge bases.
Instead, we propose to construct a query q 2 Q as a compact form of original language description t , represented with the words that convey the main concepts of the image.
Speciﬁcally, we consider a divide-and-conquer approach.
For short text sequences t such as category names in image-tag/label datasets ( e.g., ImageNet [15]), we directly use the category name as the query.
For long text sequences t such as captions ( e.g., YFCC [84]), we ﬁrst parse the sentence to extract the noun-phrases, among which the most rare noun-phrase over the corpus is used as a query for this sentence.
The intuition is to convert the rare concepts into “explanations” represented in common words using external knowledge.
Noun phrases are useful for summarizing the sentence and thus inferring what is being talked about in the image.
For example, in “ professional boxer is introduced to the crowd ”, the noun-phrases are “ boxer ”, “ professional boxer ”, and “ the crowd ”.
We summarize the query construction process g query below for clarity:
[Equation]
t is a category, class or tag name
[Equation]
The most rare noun-phrase ( t ) [Equation] a caption .
[Equation]
Knowledge Acquisition from External Sources.
We consider three knowledge sources S to enrich the language descriptions t .
They are constructed based on the two knowledge bases: WordNet [63] and Wiktionary [62].
To measure the breadth of a knowledge base, we deﬁne the concept coverage as the percentage of non-empty knowledge items retrieved for a given set of issued queries.
( i )
WordNet [63] is a lexical database which links words into semantic relations including synonyms, hyponyms, and meronyms.
Both nouns and verbs are organized into hierarchies, deﬁned by hypernym relationships.
The synonyms in WordNet are grouped into synsets , expressing the same distinct concept.
The synsets serve as a natural link between language and vision domains.
For example,
ImageNet is an image database organized according to the WordNet hierarchy, where each node is depicted by hundreds/thousands of images.
( ii ) Wiktionary [62] is a web-based content dictionary of terms (including words, phrases, proverbs, linguistic reconstructions).
These entries may contain deﬁnitions, illustrations, usage examples etc. .
Next, we provide our knowledge retrieval process s = g retrieve ( q ) for each source S , followed by an example result for the query “ boxer ”:
• WordNet Hierarchy S wn_path .
A WordNet node of the query is located, then we repeatedly search its parent node.
The words along the traversal path are recorded as the knowledge.
- [boxer, combatant, person, causal_agent, physical_entity, entity] • WordNet Deﬁnition S wn_def .
The deﬁnition from the synsets is used to explain the query.
- someone who fights with his fists for sport • Wiktionary Deﬁnition S wiki_defh .
The query is used for dictionary look-up in Wiktionary, and the corresponding deﬁnition is used.
- a fighter in a boxing match
When multiple meanings (senses) exist for a given query, we simply consider the ﬁrst one for simplicity, and leave more sophisticated designs as future work.
After querying each knowledge source, we represent the external knowledge for each language description t in the form of concatenation of its query and the corresponding retrieved result: [ q , s ] .
This external knowledge introduces additional supervision signals to guide visual models to learn better aligned visual-semantic representations, as we explain later.
We next describe how to encode knowledge in multimodal models to improve transfer in the image-level task of image classiﬁcation and the region-level task of object detection.
3.2 Image Classiﬁcation
Recent works that learn visual models with language supervision [71] often employ a dual-encoder architecture.
For each image x , an image encoder model f ✓ parameterized by ✓ ﬁrst represents x as a visual feature vector ˜ v 2 R P ⇥ 1 : ˜ v = f ✓ ( x ) .
For each language description t 2 T , we encode it with a text encoder f
( t ) parameterized by  , and get the [EOS] feature as the vector representation of the sentence ˜
u 2 R P ⇥ 1 : ˜ u
= f  ( t ) .
In this paper, we further leverage this text encoder to encode the external knowledge.
First, the query q is represented in natural language p =
g prompt ( q ) using the language prompt as in [71].
Depending on the language input t and our augmentation scheme, the knowledge-augmented text sequence t k 2 T k ( k stands for k nowledge) is represented as:
[Equation]
t is a category, class or tag name ,
[Equation]
when t is a
caption , and a concat scheme is used ,
[Equation]
when t is a caption, and a combine scheme is used .
[Equation]
For example, for category name t = boxer or for caption t = professional boxer is introduced to the crowd , we have q = boxer , and the corresponding language description are:
• t ke = a photo of a cool boxer ; boxer , a fighter in a boxing match • t kc = professional boxer is introduced to the crowd ; boxer , a fighter in a boxing match
We re-use the same text encoder f  to encode t k as the original t as supervision for image x .
Training.
For i -th image x i and j -th language description t j in a batch B , we normalize their feature vectors in a hyper-sphere using u i =
f ✓ ( x i )
[Equation]
and v [Equation] , and their similarity is calculated as u > i v j .
A bidirectional supervised contrastive objective is considered to train the model: min
[Equation]
with
[Equation]
where P ( i ) =
{ k | k 2 B , y k
= y i } , Q ( j ) =
{ k | k 2 B , y k
= y j } , and ⌧ is a temperature
hyper- parameter controlling the strength of penalties on hard negative samples.
Note (3) is a general form; it reduces to the training objective of CLIP [71] or ALIGN [36] when there is a one-to-one mapping between an image and its paired caption in a batch, i.e., P ( i )
=
{ i } and Q ( j ) =
{ j } .
Evaluation.
Given a downstream image classiﬁcation task with a custom set of category names, we represent them with the knowledge-augmented prompt form in (2); they are fed into the pre-trained text encoder f  to obtain the class embedding.
The test image x is encoded with f ✓ ( x ) , and compared to all class embeddings to get its label from the best matching class.
Extensions with Modularized Modeling.
In our study, we found it is key to ensure consistency between training and evaluation stages: if a model is trained with knowledge, it performs well when also evaluated with knowledge.
Similarly, if a model is trained without knowledge ( e.g., CLIP/UniCL), adding knowledge directly in the evaluation stage results in performance drop.
However, due to the limited knowledge coverage in existing knowledge bases, s could be empty for a large number of queries q .
When the low coverage happens for a downstream evaluation dataset, it may result in a training-evaluation inconsistency for our knowledge-augmented models, and thus lower performance.
It is desired to have a modularized model that can switch between “with” and “without” knowledge settings.
Inspired by [90], we propose to employ adapters [33] to build the network branch to encode knowledge-augmented language t k , where serial MLP adapters are inserted after each self-attention and MLP modules for all Transformer layers of the text encoder, and t k is passed through f  and adapters.
Meanwhile, the original f  is reserved as the branch to encode vanilla natural language t .
The proposed adapter-modularized architecture can also be used for efﬁcient stage-wise continual pre-training: one may start with a vanilla language-image model pre-trained on D , and continue pre-train the adapters with knowledge-augmented data ( D , S ) to build its knowledge version.
3.3 Object Detection
Object detection (OD) typically involves two tasks: L OD =
L cls + L loc , where the localization task
L loc aims to locate the presence of objects in an image with a bounding box, and classiﬁcation task
L cls determines what object categories are present in that box.
Similar to the IC task above, we improve the categorization task of individual boxes L cls in OD with external knowledge, and keep
L loc the same.
Speciﬁcally, we leverage GLIP [50] to reformulate OD as a phrase grounding task, by grounding each region proposed by L loc [ 53 ] to phrases in a text sequence.
For language encoding, we ﬁrst augment a category name t into its knowledge-augmented form t k = [ q , s ] ; this is different from
IC in that p is excluded, as no prompt engineering is used in OD, as in [50].
In the original GLIP, a sequential text encoding scheme is used: a concatenated long sequence [ q 1 , · · · , q K ] over category names is considered as the text encoder input.
In our case, simple concatenation [ q 1 , s· · · , q K , s K ] will quickly break the max length requirement of the language encoder.
To resolve the issue, we propose a parallel text encoding scheme: each t k is passed through language encoder independently, we use the top-layer feature of [CLS] token as the contextual vector representation ˜ u 2 R P ⇥ 1 of t k : ˜ u
= f  ( t k ) .
Given K categories, they can be encoded in parallel in a batch; the encoded phrase feature sequence is the concatenation U 2 R P ⇥ K : U =
[˜ u 1 , · · · , ˜ u K ] .
The region encoding is the same as in GLIP.
The feature pyramid is V 2 R M ⇥ P : V = f ✓ ( x ) , where
M is the number of box features.
The alignment scores S ground are computed:
[Equation]
where T 2 { 0 , 1 } M ⇥ K is the target indicating match or no match, and M ( S ; T ) is the focal loss [53].
The grounding model, consisting of the image encoder f ✓ , the language encoder f  and a cross-modal interaction head introduced in [50], is trained end-to-end by minimizing the loss deﬁned in (4).
In the evaluation stage, the external knowledge is also retrieved, and encoded in the same parallel encoding manner to enrich the category names in the downstream OD tasks.
Experimental Results
In this section, we examine our knowledge-augmented approach to answer two research questions.
Q1 :
To what extent external knowledge beneﬁts visual transfer learning, including sample-efﬁciency in pre-training and downstream?
Q2 : Why does external knowledge help zero-shot transfer (illustrated with success and failure case studies)?
4.1 Settings
Evaluation benchmark.
We apply the proposed knowledge-augmented models to two task-level transfer settings deﬁned in E LEVATER benchmark [47], which evaluates the transferability of the learned visual representations in the wild. We study our models based on the datasets described in Table 1.
The license, PII, and consent details of each dataset are in the respective papers.
Due to the limited computational resources, the pre-training datasets are constrained to the large publicly available datasets used in [95, 50].
This setting is also friendly to the academic community to allow reproducibility of the results.
The number of visual concepts is identical to the number of categories for datasets with category names ( e.g., ImageNet and Object-365). For image-text data (bottom [Table] [Table]
rows of the IC block), use Spacy [32] to extract the noun phrases.
We also use a merged version of GCC-3M and GCC-12M denoted as GCC-15M. Given the pool of concepts, we calculate the number of unique words and report it as the vocabulary size.
For Concepts and Vocab Size, we report 2 numbers: ﬁrst for the full set, second for items with frequency larger than 5.
The latter provides a sense of “long-tailness”.
The statistics ( e.g., ratio of #Instance / #Concept) illustrates the varied trade-off over different datasets: image diversity, semantic richness and long-tailness.
For example,
YFCC is the most long-tail dataset in IC, as it has low mean value and the largest standard derivation value in #Instance / #Concept.
The concept overlap is computed as the percentage of concepts in a downstream dataset that are covered by the pre-training dataset.
For 20-datasets and 13-datasets, the averaged overlap across individual datasets is reported.
It measures the gap (or difﬁculty) in concept transfer between the pre-training and the downstream data.
The dataset statistics are detailed in Section B.1 in Appendix.
Zero/Few-shot image classiﬁcation .
Following UniCL [95], this task evaluates to what extent a model understands novel concepts.
We pre-train on ImageNet-21K [15] and GCC [76, 11]/YFCC [84] datasets, and report results on ImageNet-1K [15] and a suite of 20 datasets used in [71].
We use the same text prompts as in [71, 95], and report scores averaged over 20 datasets.
UniCL/Florence [101] show superior performance to CLIP or ALIGN counterparts; UniCL is Florence in a controlled academic setting, trained on the large publicly available datasets [95].
Zero-shot object detection .
Following GLIP [50], we pre-train on Object365 [75], and transfer the learned visual representations for object detection on LVIS [28] and a suite of 13 small OD datasets, to check the generalization ability.
The box mAP is reported.
4.2 Image Classiﬁcation
ImageNet-21K pre-training.
We start by pre-training on ImageNet-21K, where all ImageNet-1K images are excluded.
We still see a small amount of concept overlap in Table 1, and hypothesize that some category names are given based on different level of WordNet hierarchy.
The beneﬁts of this setting are two-fold: it ensures distinctively less concept overlap, and all concepts can ﬁnd their full
WordNet knowledge.
We report the results in Table 2.
For each checkpoint, we report the results without and with knowledge in the evaluation stage.
We conﬁrm two major ﬁndings below.
[Table]
[71]/UniCL [95]. }
It indicates that the Combine scheme is used for the image-caption data, otherwise the default is the Concat scheme decribed in Section 3.2.
The linear probing and ﬁne-tuning results are reported for 5-shot settings over 3 random seeds.
F1 :
All three knowledge sources are beneﬁcial .
In Section 3.1, we have introduced WordNet hierarchy
S wn_path , WordNet deﬁnition S wn_def , and Wiktionary deﬁnition S wiki_def as external knowledge sources.
It is shown that all three of them are effective, improving the zero-shot accuracy by absolute gain 1-2% on ImageNet-1K (from 28.16% to 30.23%) and 2-6% on the dataset suite in average (from 27.15% to 33.44%), respectively.
Among them, Wiktionary deﬁnition S wiki_def turns out to be the most effective, therefore, we use it as the default knowledge source throughout the remaining experiments.
F2 : The modularized approach is effective .
Our results in Table 2 reveal that training/test inconsistency in terms of involving knowledge can dramatically degrade the model performance.
For example in the 1st row, the baseline UniCL is pre-trained without knowledge, its performance decreases from 28.16% to 4.93% when knowledge is added in the test stage.
In contrast, in the 4th row, our K-L ITE is pre-trained with knowledge, its performance decrease from 33.44% to 29.03% if knowledge is excluded in the evaluation stage.
Therefore, we consider a modularized approach with 2-branch in the model.
First, we continue pre-train our modularized model from a 32-epoch knowledge-free checkpoint by only updating the Adapters on knowledge-augmented image-text pairs for 10 epochs.
It already shows a performance gain from 27.61% to 28.40%.
This suggests a more affordable solution to obtain knowledge-augmented models from existing models.
We can further boost the performance to 28.90% if we evaluate with two branches, each of which only passes its corresponding language version.
Finally, we also train the modularized model from scratch, and it demonstrates a signiﬁcant gain (absolute 4%) on ImageNet-1K, and over 5% improvement on the 20 datasets.
For fair comparisons with knowledge-free models, we train our models with one branch in the rest of experiments.
To demonstrate the performance of K-L ITE in the extreme large-scale settings, we leverage the largest checkpoint of Florence [101] trained on 800M image-text pairs, and continue pre-training the model on ImageNet-21K with external knowledge.
It improves the zero-shot ImageNet-1K accuracy of
Florence from 83.74% to 85.80%.
As an ablation baseline, continuing pre-training without knowledge yields 85.35%.
The absolute 0.45% performance gain shows that external knowledge can still beneﬁt transfer learning, though a huge amount of pre-training data is employed.
Pre-training on image-text-label data.
The uniﬁcation of image-label and image-caption as imagetext-label has been demonstrated superior over either one of them [95].
Therefore, we report overall results on the combined data in Table 3.
The few-shot learning results are reported with 5 training examples, using two model adaptation method: linear probing and full model ﬁne-tuning.
The average numbers over 3 random seeds are reported.
K-L ITE improves its knowledge-free counterpart
UniCL in almost all the cases.
Importantly, K-L ITE can outperform UniCL using only half of the pre-training image-text pairs in several cases.
It demonstrate the high sample-efﬁciency of K-L ITE , and that external knowledge is an effective source to consider, when collecting large-scale image-text pairs to develop language-augmented visual models at scale. [Figure] a
Success examples. The two datasets with largest improvement in Fig. 3: Flowers102 and The description of the
[Figure] Failure examples. The two datasets with the largest performance loss in Fig. Left (EuroSat): both class names have the same knowledge.
Middle & Right (VOC2007):
The knowledge contains spurious words that confuse
[Figure]
Breakdown Analysis. Next, we ask why does external knowledge improve the zero-shot task transfer performance on a broad range of datasets?
To answer this question, we compare the breakdown performance on all 20 dataset in Figure 3, for the ImageNet-21K checkpoints trained with and without Wiki knowledge. Out of 20 datasets
, external knowledge shows superior/comparable/inferior performance to the baseline on 16/1/3 datasets, respectively.
One prominent observation is that Wiki knowledge improves concept overlap for train-evaluation from 13.26% to 51.24% by average.
It is easy to understand, concepts are explained in more commonly used words in Wikitionary, providing a bridge for train and evaluation.
This is reﬂected by the increased height of blue bar for most datasets in Figure 3.
Interestingly, for all datasets with increased accuracy scores, there shows an increase of the concept overlap.
In summary, knowledge is an effective approach to improve concept overlap, a prerequisite for good task transfer performance.
In Figure 4 (a), we provide success examples after adding knowledge; more examples are shown in Appendix.
Limitations.
The failure cases where knowledge-augmented approach does not help mainly belong to two scenarios: ( i ) No external knowledge was extracted from the given knowledge base ( i.e.,
Wiktionary in this case), e.g., StanfordCars and FGVC Aircraft.
They often require domain-speciﬁc knowledge explanations to deﬁne a car brand ( e.g., Volvo C30 Hatchback 2012) or an aircraft model type ( e.g., 737-200), while Wiktionary can hardly provide such professional deﬁnitions.
( ii )
While knowledge is available, the quality is too low to provide useful information.
In Figure(b), we provide failure examples from two datasets with the biggest performance loss after adding knowledge.
In Figure 6 in the Appendix, we show more examples where knowledge only yields slight [Table]
improvement. To summarize, a promising future research direction is to improve the knowledge quality to be more related to the given classiﬁcation tasks.
4.3 Object detection
We evaluate the model’s ability to recognize diverse objects on LVIS [28] and 13 downstream datasets used in [50] in a zero-shot setting.
We report on MiniVal containing 5,000 images on LVIS.
Our
K-L ITE GLIP is trained with Wiktionary deﬁnitions.
The results are presented in Table 4.
The 1st row are the original numbers reported in [50], using the sequential text encoding.
The 2nd row is our implementation of GLIP using parallel text encoding, whose effectiveness is validated by the comparable numbers with 1st row.
The 3rd row is our knowledge-augmented GLIP, i.e., K-L ITE .
The beneﬁt of using external knowledge is evident.
On LVIS, the categories are divided into rare, common, frequent groups, based on the number of training images per category.
K-L ITE improves the detection performance for all three groups with an average of 2.8 points on LVIS, and particularly brings a 4.7 points improvement on MiniVal APc over the GLIP-A reported in [50].
We conclude that the enriched semantics of external knowledge signiﬁcantly helps the model recognize concepts with a decent number of instances.
Since LVIS has its own knowledge source S LVIS , mostly built upon
WordNet deﬁnitions [28], we evaluate our model with S LVIS .
We alter the external knowledge source to S wn_path , S wn_def , and S wiki_def in evaluation, which yields mAP 18.7, 21.4, 20.5, respectively.
It veriﬁes that our knowledge source extraction process is reliable.
Similarly, K-L ITE improve thedownstream OD datasets from 28.8 (or 27.5 with its own knowledge-free counterpart) to 31.7.
The success and failure examples of OD are shown in Section B.5 in Appendix.
Conclusions
In this paper, we have presented a knowledge-augmented approach K-L ITE to learn a generic visual model for task-level transfer.
General external knowledge sources including WordNet and Wiktionary are explored to enrich the natural language supervision, which is then used in both the language-image pre-training stage and the prompt-based evaluation stage.
We have demonstrated the generality and effectiveness of K-L ITE in two core computer vision problems: image classiﬁcation and object detection.
Extensive experimental results show that our method can achieve superior performance over existing methods on 20 IC datasets and 13 OD datasets, respectively.
K-L ITE also outperforms its knowledge-free counterpart UniCL using half of the pre-training data in the large-scale academic data setting, demonstrating that leveraging external knowledge is a promising direction in improving pre-training sample-efﬁciency for learning transferable visual models.
Acknowledgments
The authors gratefully acknowledge Chenguang Zhu, Wenhao Yu, Yuwei Fang for the early insightful discussions on the use of dictionary for rare concepts in NLP task, Hao Cheng for the inspirations of external knowledge in open-domain QA.
The project is partly done in the MSR-Berkeley collaboration program 3 .
The work depends on publicly available knowledge databases;
we