K-L ITE : Learning Transferable Visual Models with External Knowledge
Anonymous Author(s)
AfﬁliationAddress email
Abstract
The new generation of state-of-the-art computer vision systems are trained fromnatural language supervision, ranging from simple object category names to descriptive captions.
This form of supervision ensures high generality and usabilityof the learned visual models, due to the broad concept coverage achieved via largescale data collection process.
Alternatively, we argue that learning with externalknowledge is a promising way which leverages a much more structured source ofsupervision and offers sample efﬁciency.
We propose K-L ITE 1 , a simple strategyto leverage external knowledge for building transferable visual systems:
In training, it enriches entities in text with WordNet and Wiktionary knowledge, leadingto an efﬁcient and scalable approach to learning image representations that usesknowledge about the visual concepts.
In evaluation, the text is also augmented withexternal knowledge and then used to reference learned visual concepts (or describenew ones) to enable zero-shot and few-shot transfer of the pre-trained models.
Westudy the performance of K-L ITE on two important computer vision problems,image classiﬁcation and object detection, benchmarking on 20 and 13 differentexisting datasets, respectively.
The proposed knowledge-augmented models showsigniﬁcant improvement in transfer learning performance over existing methods.
Introduction
One of the core aspirations in computer vision (CV) is to develop systems that endow computers withthe ability to effectively learn general visual representations, which can be transferred to a varietyof downstream recognition datasets with arbitrary visual concepts in the wild.
Though excellentperformance has been achieved on standard benchmarks, the traditional supervised approaches arelimited to learning a ﬁxed set of concepts, e g., 22K concepts on ImageNet [15] or 18K concepts on
JFT-300M [80].
This leads to a few issues: ( i )
Annotating each individual vision dataset is not onlylabor intensive, but also results in a narrow set of visual concepts; ( ii ) Visual models trained on suchdatasets are good at one task (with the given concept set) and this task only, and show poor transferlearning performance to customized datasets that usually come with a different set of concepts [27].
To tackle this problem, recent large-scale language-augmented visual models, such as CLIP [70],
ALIGN [36] and Florence [98], are trained on a wide variety of images with natural language supervision that is abundantly available on the Internet.
These models demonstrate strong zero-shot transfercapabilities, since they acquire open-set recognition abilities through problem reformulation fromclassiﬁcation to retrieval.
Moreover, model generalization is improved as natural language supervisiontypically contains rich semantics.
While these models usually perform well on recognizing commonobjects, they still struggle on visual concepts that are absent or rare in the pre-training stage.
To ensure good transfer performance, it is required to train such models on huge datasets with sufﬁcientconcept coverage( e.g., >400M image-text pairs), which is both labor and compute expensive.
Instead of scaling the number of image-text pairs to increase concept coverage, we propose to leveragestructured external knowledge to augment language supervision.
The inspiration comes from howhumans generalize to novel concepts: instead of trying to memorize all concepts, humans leveragethe structured knowledge such as deﬁnitions and concept hierarchy.
For example, when we visit a
Japanese restaurant for the ﬁrst time, we may struggle to understand the menu by only looking at thedish names ( e.g., Takoyaki, Sashimi ), as it is hard to imagine what they are.
However, it becomesmuch clearer once a waiter introduces these concepts (Figure 1), leading to success in ordering food( i.e., matching content to a name).
Similar intuitions have been exploited in computer vision forclass-level transfer [91, 12], but not yet for task-level transfer settings (similar to that of CLIP).
A ball-shaped Japanese dumpling made of batter, filled with diced octopus, tempura scraps , pickled ginger, and green onion .
Takoyaki
A dish consisting of thin slices or pieces of raw fish or meat .
Sashimi the content of the rare dish concepts
To this end, we explore a systematic approach to acquire and learn with external knowledge sourcesfrom databases such as WordNet [62] and Wiktionary [61] to train more transferable and sampleefﬁcient visual models.
The concept descriptionsand concept hierarchies are purely textual, and theprocess of collecting external knowledge is fullyautomatic without extra human annotation.
The acquired knowledge typically provides informationthat is shared between seen and unseen concepts tofacilitate effective transfer.
Speciﬁcally, rare concepts, e.g., Takoyaki, Sashimi in Figure 1, areexplained with more common concepts.
Such knowledge sources are generally available for a variety ofdomains and datasets, making it possible to build ageneric approach for task-level transfer.
Our main ﬁndings and contributions can be summarized as follows:
• We present the ﬁrst strong evidence that external knowledge can beneﬁt large-scale task-leveltransfer for two core CV problems, image classiﬁcation (IC) and object detection (OD), byexploring external knowledge sources, including WordNet and Wiktionary.•
A simple and effective strategy K-L ITE (Knowledge-augmented Language Image Training andEvaluation) is proposed:
The acquired knowledge is appended to the original textual concepts asmodel input during pre-training and evaluation.
It can be viewed as an automatic knowledge-awarelanguage prompting, which makes it easier for the model to access relevant information sharedbetween the training and evaluation data.
A modularized approach is also developed to enableefﬁcient adaptation from vanilla visual models to their knowledge-augmented versions.
• To demonstrate the generality of the K-L ITE , we instantiate it with two recent visual modelsand develop our knowledge-augmented counterparts: UniCL [92] for IC and GLIP [49] for OD.
Extensive experiments in zero-shot and few-shot learning settings demonstrate that knowledgeaugmented models can signiﬁcantly improve over prior work.
Notably, our model can achievesimilar zero-shot performance to previous methods using only half of pre-training image-text pairsin some scenarios, demonstrating sample efﬁciency of the proposed approach.
Related Work
Zero-shot Visual Recognition: Zero-shot learning, i.e., classifying images where there is a lackof labeled training data, has been studied for decades [21], and its popularity has recently increasedfurther [91].
Based on the technique evolution, it can be broadly categorized into two generations:the traditional class-level zero-shot and recently popular task-level zero-shot setting.
Class-level Transfer.
Class-level zero-shot learning aims to recognize object classes whose instanceshave not been observed during training.
The goal for the zero-shot learning methods is to associateobserved and non-observed classes through some form of auxiliary information, which can be eitherimplicit such as pre-trained semantic embeddings [89, 76, 10], or explicit such as attributes [21, 43,35], text [18, 19, 71, 69], knowledge graphs [88, 73], or rules and ontologies [23].
Please, refer tothe recent survey on knowledge-aware zero-shot learning for a more detailed review [12].
Recently,it has been suggested to move away from the restricted nature of standard zero-shot evaluation and make the task more practical by including training classes at test time, i.e., generalized zero-shotlearning setting [91].
Despite the progress in this area, the traditional setting is typically limited tostudying zero-shot transfer across classes in a single domain with manually deﬁned splits, such as
Animal with Attributes (AwA) [44], Birds-200 [86], SUN attributes [66], and ZS-ImageNet [72, 26].
Concurrently, [82] explores leveraging external knowledge to improve long-tailed visual recognitionwithin individual domains, which falls into the category of class-level transfer.
Task-level Transfer.
Another line of work focuses on task-level zero-shot transfer [51, 46, 63, 70,36, 94].
They pre-train visual models on hundreds of millions of web-crawled image-caption orimage-tags pairs, and evaluate their transfer ability by directly performing inference in a wide rangeof downstream datasets, without tuning the model weights.
We argue that the task-level transfer ismore practical and attractive than class-level transfer, as it is more relevant to real-world scenarios,where we may want to develop models that can serve many visual recognition applications.
Our work bridges the gap between the two lines of works above: it borrows the spirit of exploringknowledge in class-level transfer, and generalizes it for task-level transfer, leveraging the best of bothworlds.
To summarize, our work is different in two major aspects: ( i ) Settings.
We focus on thetask-level transfer learning across domains, i.e., from large publicly available datasets to a diverseset of downstream datasets in different domains, and demonstrate that external knowledge beneﬁtstask-level transfer.
( ii ) Modeling.
Existing class-level transfer works are built upon pre-trained visualfeatures/backbones and shallow word embeddings or tf-idf scores, and only train the classiﬁers.
Onerepresentative example is DeViSE [25], where a skip-gram word embedding model and an imageclassiﬁer are ﬁne-tuned jointly.
In contrast, we are training large Transformer-based models in anend-to-end manner from scratch as in CLIP/ALIGN, providing the ﬁrst empirical evidence thatexternal knowledge can help train a general visual backbone.
Knowledge-Intensive Models:
In natural language processing (NLP), with the increase of modelcapacity via pre-trained language models [17], there emerges the need for more knowledgeablemodels [54] with advanced functionalities such as making use of encyclopedic [85, 4, 6] and commonsense knowledge [77, 99].
To address this, a large number of language models augmented withexternal knowledge sources have been proposed [67, 29, 45, 55, 97, 7], achieving strong performanceon a variety of NLP tasks [68, 42].
Please refer to a recent survey [95] for a comprehensive review.
In vision-and-language (V+L) domain, researchers have also started exploring knowledge-intensivetasks, e.g., OK-VQA [60] and WebQA [9].
They often require additional information sources ( e.g.,factual and commonsense knowledge) beyond the QA pairs, compared to the established tasks suchas VQA [3, 34] and image captioning [53, 2].
Hence, existing pre-trained models [57, 50, 79, 100,38, 48, 30, 96] would perform poorly on these knowledge-intensive V+L tasks [9].
To address theproblem, acquiring external knowledge becomes an essential component for success [90, 59, 93].
The success of knowledge in NLP and V+L tasks inspires us to ask a natural question: Can we learna transferable visual backbone model with external knowledge?
Thus, we dissect and borrow thevital elements such as knowledge sources [62, 61] and modeling techniques [87, 97], and carry outstudies for core computer vision tasks.
Knowledge-Augmented Visual Models
Problem setup.
Computer vision systems have achieved strong transfer performance, when learningwith large-scale image-label data [39] and image-caption data [70].
Recently, it has been demonstratedin [92, 98] that the uniﬁcation of image-label and image-text formats into image-text-label achievessuperior performance over either of them.
We follow the setting in [92], and deﬁne a uniﬁedtriplet-wise data format D
= { ( x n , t n , y n ) } Nn =1 , where x 2 X is an image, t 2 T is its language 134 description, and y 2 Y is a label indicating the index of the unique language description in the dataset.
In a general form, the language description is a text sequence t =
[ t 1 , · · · , t L ] .
It ranges from simplecategory names representing visual concepts when L is small, to more free-form and semantic-richsentences such as captions when L is relatively large.
In this paper, we assume there exists an external knowledge source S , where one may use thelanguage description t as a query to seek additional knowledge description s 2 S for t .
Given thesetriplet data instances D and an external knowledge source S , our goal is to learn generic visualsemantic representations, which are readily transferable to a wide range of downstream datasets,whose category names are not necessarily observed during training.
In Figure 2, we visually illustratethe proposed knowledge-augmentation process and two considered application scenarios.
[Table]
+gcnjU1nGqCG2RmMeqG2JNOZO0ZZjhtJsoikXIaSec3M79zhNVmsXywUwTGgg8kixiBBsrPbpZP4yQnqFBperVvBxolfgFqUKB5qDy1R/GJBVUGsKx1j3fS0yQYWUY4XTm9lNNE0wmeER7lkosqA6y/OAZOrPKEEWxsiUNytXfExkWWk9FaDsFNmO97M3F/7xeaqLrIGMySQ2VZLEoSjkyMZp/j4ZMUWL41BJMFLO3IjLGChNjM3JtCP7yy6ukfVHzL2v1+3q1cVPEUYYTOIVz8OEKGnAHTWgBAQHP8ApvjnJenHfnY9FacoqZY/gD5/MHtuiPtg==</latexit> s
Knowledge [Figure]
Language
Query
Knowledge
[Figure]
3.1 External Knowledge
Query Construction.
For a text sequence associated with an image, different tokens may playdifferent roles in contributing to describing the main semantics of the image.
Humans leverage thisprior inherently in parsing the sentences to understand the image.
Further, it is infeasible to employthe entire sequence as a query, as this may lead to the lack of coverage in the knowledge bases.
Instead, we propose to construct a query q 2 Q as a compact form of original language description t ,represented with the words that convey the main concepts of the image.
Speciﬁcally, we consider a divide-and-conquer approach.
For short text sequences t such as categorynames in image-tag/label datasets ( e.g., ImageNet [15]), we directly use the category name as thequery.
For long text sequences t such as captions ( e.g., YFCC [81]), we ﬁrst parse the sentence toextract the noun-phrases, among which the most rare noun-phrase over the corpus is used as a queryfor this sentence.
The intuition is to convert the rare concepts into “explanations” represented incommon words using external knowledge.
Noun phrases are useful for summarizing the sentenceand thus inferring what is being talked about in the image.
For example, in “ professional boxeris introduced to the crowd ”, the noun-phrases are “ boxer ”, “ professional boxer ”, and“ the crowd ”.
We summarize the query construction process g query below for clarity:
[Equation]
t is a category, [Equation] or tag name ,
The most rare noun-phrase ( t ) , when t is a caption .
[Equation]
Knowledge Acquisition from External Sources.
We consider three knowledge sources S to enrichthe language descriptions t .
They are constructed based on the two knowledge bases: WordNet [62]and Wiktionary [61].
To measure the breadth of a knowledge base, we deﬁne the concept coverageas the percentage of non-empty knowledge items retrieved for a given set of issued queries.
( i )
WordNet [62] is a lexical database which links words into semantic relations including synonyms,hyponyms, and meronyms.
Both nouns and verbs are organized into hierarchies, deﬁned by hypernymrelationships.
The synonyms in WordNet are grouped into synsets , expressing the same distinctconcept.
The synsets serve as a natural link between language and vision domains.
For example,
ImageNet is an image database organized according to the WordNet hierarchy, where each node isdepicted by hundreds/thousands of images.
( ii ) Wiktionary [61] is a web-based content dictionaryof terms (including words, phrases, proverbs, linguistic reconstructions).
These entries may containdeﬁnitions, illustrations, usage examples etc. .
Next, we provide our knowledge retrieval processs = g retrieve ( q ) for each source S , followed by an example result for the query “ boxer ”:
• WordNet Hierarchy S wn_path .
A WordNet node of the query is located, then we repeatedly search its parent node.
The words along the traversal path are recorded as the knowledge.
- [boxer, combatant, person, causal_agent, physical_entity, entity] • WordNet Deﬁnition S wn_def .
The deﬁnition from the synsets is used to explain the query.
- someone who fights with his fists for sport • Wiktionary Deﬁnition S wiki_defh .
The query is used for dictionary look-up in Wiktionary, and the corresponding deﬁnition is used.
- a fighter in a boxing match
When multiple meanings (senses) exist for a given query, we simply consider the ﬁrst one for simplic- ity, and leave more sophisticated designs as future work.
After querying each knowledge source, we represent the external knowledge for each language description t in the form of concatenation of itsquery and the corresponding retrieved result: [ q , s ] .
This external knowledge introduces additionalsupervision signals to guide visual models to learn better aligned visual-semantic representations,as we explain later.
We next describe how to encode knowledge in multimodal models to improvetransfer in the image-level task of image classiﬁcation and the region-level task of object detection.
3.2 Image Classiﬁcation
Recent works that learn visual models with language supervision [70] often employ a dual-encoderarchitecture.
For each image x , an image encoder model f ✓ parameterized by ✓ ﬁrst represents x asa visual feature vector ˜ v 2 R P ⇥ 1 : ˜ v = f ✓ ( x ) .
For each language description t 2 T , we encode itwith a text encoder f
( t ) parameterized by  , and get the [EOS] feature as the vector representationof the sentence ˜ u
2 R P ⇥ 1 : ˜ u
= f  ( t ) .
In this paper, we further leverage this text encoder to encodethe external knowledge.
First, the query q is represented in natural language p =
g prompt ( q ) usingthe language prompt as in [70].
Depending on the language input t and our augmentation scheme,the knowledge-augmented text sequence t k 2 T k ( k stands for k nowledge) is represented as:
[Equation]
when t is a category, class or tag name ,
[Equation] [Equation]
[Equation]
when t is a caption, and a concat scheme is used , { t k c , t k e } , when t is a caption, and a combine scheme is used .
[Equation]
For example, for category name t = boxer or for caption t = professional boxer is introducedto the crowd , we have q = boxer , and the corresponding language description are:
• t ke = a photo of a cool boxer ; boxer , a fighter in a boxing match • t kc = professional boxer is introduced to the crowd ; boxer , a fighter in a boxing match
We re-use the same text encoder f  to encode t k as the original t as supervision for image x .
Training.
For i -th image x i and j -th language description t j in a batch B , we normalize their featurevectors in a hyper-sphere using u i =
f ✓ ( x i )
k f ✓ ( x i ) k
and v j = f  ( t j ) k f  ( t j ) k , and their similarity is calculatedas u > i v j .
A bidirectional supervised contrastive objective is considered to train the model:min
[Equation]
i , with
[Equation]
where P ( i ) =
{ k | k 2 B , y k
= y i } , Q ( j ) =
{ k | k 2 B , y k
= y j } , and ⌧ is a temperature hyper-
199 parameter controlling the strength of penalties on hard negative samples.
Note (3) is a general form;it reduces to the training objective of CLIP [70] or ALIGN [36] when there is a one-to-one mappingbetween an image and its paired caption in a batch, i.e., P ( i )
=
{ i } and Q ( j ) =
{ j } .
Evaluation.
Given a downstream image classiﬁcation task with a custom set of category names, werepresent them with the knowledge-augmented prompt form in (2); they are fed into the pre-trainedtext encoder f  to obtain the class embedding.
The test image x is encoded with f ✓ ( x ) , and comparedto all class embeddings to get its label from the best matching class.
Extensions with Modularized Modeling
In our study, we found it is key to ensure consistencybetween training and evaluation stages: if a model is trained with knowledge, it performs well whenalso evaluated with knowledge.
Similarly, if a model is trained without knowledge ( e.g., CLIP/UniCL),adding knowledge directly in the evaluation stage results in performance drop.
However, due to thelimited knowledge coverage in existing knowledge bases, s could be empty for a large number ofqueries q .
When the low coverage happens for a downstream evaluation dataset, it may result in atraining-evaluation inconsistency for our knowledge-augmented models, and thus lower performance.
It is desired to have a modularized model that can switch between “with” and “without” knowledgesettings.
Inspired by [87], we propose to employ adapters [33] to build the network branch to encodeknowledge-augmented language t k , where serial MLP adapters are inserted after each self-attentionand MLP modules for all Transformer layers of the text encoder, and t k is passed through f  andadapters.
Meanwhile, the original f  is reserved as the branch to encode vanilla natural language t .
The proposed adapter-modularized architecture can also be used for efﬁcient stage-wise continual pre-training: one may start with a vanilla language-image model pre-trained on D , and continuepre-train the adapters with knowledge-augmented data ( D , S ) to build its knowledge version.
3.3 Object Detection
Object detection (OD) typically involves two tasks: L OD =
L cls + L loc , where the localization task
L loc aims to locate the presence of objects in an image with a bounding box, and classiﬁcation task
L cls determines what object categories are present in that box.
Similar to the IC task above, weimprove the categorization task of individual boxes L cls in OD with external knowledge, and keep
L loc the same.
Speciﬁcally, we leverage GLIP [49] to reformulate OD as a phrase grounding task, bygrounding each region proposed by L loc [ 52 ] to phrases in a text sequence.
For language encoding, weﬁrst augment a category name t into its knowledge-augmented form t k = [ q , s ] ; this is different from
IC in that p is excluded, as no prompt engineering is used in OD, as in [49].
In the original GLIP, asequential text encoding scheme is used: a concatenated long sequence [ q 1 , · · · , q K ] over category 231 names is considered as the text encoder input.
In our case, simple concatenation [ q 1 , s· · · , q K , s K ] 232 will quickly break the max length requirement of the language encoder.
To resolve the issue, we propose a parallel text encoding scheme: each t k is passed through languageencoder independently, we use the top-layer feature of [CLS] token as the contextual vector representation ˜ u 2 R P ⇥ 1 of t k : ˜ u
= f  ( t k ) .
Given K categories, they can be encoded in parallel in abatch; the encoded phrase feature sequence is the concatenation U 2 R P ⇥ K : U =
[˜ u 1 , · · · , ˜ u K ] .
The region encoding is the same as in GLIP.
The feature pyramid is V 2 R M ⇥ P : V = f ✓ ( x ) , where
M is the number of box features.
The alignment scores S ground are computed:
[Equation]
where T 2 { 0 , 1 } M ⇥ K is the target indicating match or no match, and M ( S ; T ) is the focal loss [52].
The grounding model, consisting of the image encoder f ✓ , the language encoder f  and a cross-modalinteraction head introduced in [49], is trained end-to-end by minimizing the loss deﬁned in (4).
In theevaluation stage, the external knowledge is also retrieved, and encoded in the same parallel encodingmanner to enrich the category names in the downstream OD tasks.
Experimental Results
In this section, we examine our knowledge-augmented approach to answer two research questions.
Q1 :
To what extent external knowledge beneﬁts visual transfer learning, including sample-efﬁciencyin pre-training and downstream?
Q2 : Why does external knowledge help zero-shot transfer (illustratedwith success and failure case studies)?
4.1 Settings
Evaluation benchmark.
We evaluate the transferability of the learned representations by applyingthe proposed knowledge-augmented models to two task-level transfer settings deﬁned in [47].
Thedataset statistics are detailed in Section B.1 in Appendix.
Zero/Few-shot image classiﬁcation .
Following UniCL [92], this task evaluates to what extent a modelunderstands novel concepts.
We pre-train on ImageNet-21K [15] and GCC [75, 11]/YFCC [81]datasets, and report results on ImageNet-1K [15] and a suite of 20 datasets used in [70].
We use thesame text prompts as in [70, 92], and report scores averaged over 20 datasets.
UniCL/Florence [98]show superior performance to CLIP or ALIGN counterparts; UniCL is Florence in a controlledacademic setting, trained on the large publicly available datasets [92].
Zero-shot object detection .
Following GLIP [49], we pre-train on Object365 [74], and transfer thelearned visual representations for object detection on LVIS [28] and a suite of 13 small OD datasets,to check the generalization ability.
The box mAP is reported.
4.2 Image Classiﬁcation
ImageNet-21K pre-training.
We start by pre-training on ImageNet-21K, where all ImageNet-1Kimages are excluded.
We still see a small amount of concept overlap in Table 4, and hypothesize thatsome category names are given based on different level of WordNet hierarchy.
The beneﬁts of thissetting are two-fold: it ensures distinctively less concept overlap, and all concepts can ﬁnd their full
[Table]
WordNet knowledge.
We report the results in Table 1.
For each checkpoint, we report the resultswithout and with knowledge in the evaluation stage.
We conﬁrm two major ﬁndings below.
F1 :
All three knowledge sources are beneﬁcial .
In Section 3.1, we have introduced WordNet hierarchy
S wn_path , WordNet deﬁnition S wn_def , and Wiktionary deﬁnition S wiki_def as external knowledge sources.
It is shown that all three of them are effective, improving the zero-shot accuracy by absolute gain 1-2%on ImageNet-1K (from 28.16% to 30.23%) and 2-6% on the dataset suite in average (from 27.15% to33.44%), respectively.
Among them, Wiktionary deﬁnition S wiki_def turns out to be the most effective,therefore, we use it as the default knowledge source throughout the remaining experiments.
F2 : The modularized approach is effective .
Our results in Table 1 reveal that training/test inconsistencyin terms of involving knowledge can dramatically degrade the model performance.
For example inthe 1st row, the baseline UniCL is pre-trained without knowledge, its performance decreases from28.16% to 4.93% when knowledge is added in the test stage.
In contrast, in the 4th row, our K-L ITEis pre-trained with knowledge, its performance decrease from 33.44% to 29.03% if knowledge isexcluded in the evaluation stage.
Therefore, we consider a modularized approach with 2-branch inthe model.
First, we continue pre-train our modularized model from a 32-epoch knowledge-freecheckpoint by only updating the Adapters on knowledge-augmented image-text pairs for 10 epochs.
It already shows a performance gain from 27.61% to 28.40%.
This suggests a more affordablesolution to obtain knowledge-augmented models from existing models.
We can further boost theperformance to 28.90% if we evaluate with two branches, each of which only passes its correspondinglanguage version.
Finally, we also train the modularized model from scratch, and it demonstrates asigniﬁcant gain (absolute 4%) on ImageNet-1K, and over 5% improvement on the 20 datasets.
Forfair comparisons with knowledge-free models, we train our models with one branch in the rest ofexperiments.
To demonstrate the performance of K-L ITE in the extreme large-scale settings, we leverage the largestcheckpoint of Florence [98] trained on 800M image-text pairs, and continue pre-training the modelon ImageNet-21K with external knowledge.
It improves the zero-shot ImageNet-1K accuracy of
Florence from 83.74% to 85.80%.
As an ablation baseline, continuing pre-training without knowledgeyields 85.35%.
The absolute 0.45% performance gain shows that external knowledge can still beneﬁttransfer learning, though a huge amount of pre-training data is employed.
Pre-training on image-text-label data.
The uniﬁcation of image-label and image-caption as imagetext-label has been demonstrated superior over either one of them [92].
Therefore, we report overallresults on the combined data in Table 2.
The few-shot learning results are reported with 5 trainingexamples, using two model adaptation method: linear probing and full model ﬁne-tuning.
Theaverage numbers over 3 random seeds are reported.
K-L ITE improves its knowledge-free counterpart
UniCL in almost all the cases.
Importantly, K-L ITE can outperform UniCL using only half of thepre-training image-text pairs in several cases.
It demonstrate the high sample-efﬁciency of K-L ITE ,and that external knowledge is an effective source to consider, when collecting large-scale image-textpairs to develop language-augmented visual models at scale.
Breakdown Analysis.
Next, we ask why does external knowledge improve the zero-shot task transferperformance on a broad range of datasets?
To answer this question, we compare the breakdownperformance on all 20 dataset in Figure 3, for the ImageNet-21K checkpoints trained with andwithout Wiki knowledge.
Out of 20 datasets, external knowledge shows superior/comparable/inferior
[Table]
[70]/UniCL [92].
} It indicates that the Combine scheme is used for the image-caption data, otherwise the default is the Concat scheme decribed in Section 3.2.
The linear probing and ﬁne-tuning results are reported for 5-shot settings over 3 random seeds.
performance to the baseline on 16/1/3 datasets, respectively.
One prominent observation is that Wikiknowledge improves concept overlap for train-evaluation from 13.26% to 51.24% by average.
It iseasy to understand, concepts are explained in more commonly used words in Wikitionary, providing abridge for train and evaluation.
This is reﬂected by the increased height of blue bar for most datasetsin Figure 3.
Interestingly, for all datasets with increased accuracy scores, there shows an increase ofthe concept overlap.
In summary, knowledge is an effective approach to improve concept overlap, aprerequisite for good task transfer performance.
In Figure 4 (a), we provide success examples afteradding knowledge; more examples are shown in Appendix.
Limitations.
The failure cases where knowledge-augmented approach does not help mainly belongto two scenarios: ( i )
No external knowledge was extracted from the given knowledge base ( i.e.,
Wiktionary in this case), e.g., StanfordCars and FGVC Aircraft.
They often require domain-speciﬁcknowledge explanations to deﬁne a car brand ( e.g., Volvo C30 Hatchback 2012) or an aircraftmodel type ( e.g., 737-200), while Wiktionary can hardly provide such professional deﬁnitions.
( ii )
While knowledge is available, the quality is too low to provide useful information.
In Figure 4(b), we provide failure examples from two datasets with the biggest performance loss after addingknowledge.
In Figure 6 in the Appendix, we show more examples where knowledge only yields slightimprovement.
To summarize, a promising future research direction is to improve the knowledgequality to be more related to the given classiﬁcation tasks.
4.3 Object detection
We evaluate the model’s ability to recognize diverse objects on LVIS [28] and 13 downstream datasetsused in [49] in a zero-shot setting.
We report on MiniVal containing 5,000 images on LVIS.
Our
K-L ITE GLIP is trained with Wiktionary deﬁnitions.
The results are presented in Table 3.
The 1strow are the original numbers reported in [49], using the sequential text encoding.
The 2nd row is
[Figure]
marigold :
Any of the Old World plants, of the genus Calendula, with orange, yellow or reddish flowers.
Wallflower :
[Figure]
of
[Figure]
short
[Figure]
the Erysimum genus with bright yellow to red flowers.
Canna lily :
Any of several [Figure] in the genus Lilium of the family Liliaceae, which includes a great many ornamental species.
Bird of paradise : Any of various passerine birds of the family Paradisaeidae native to
Oceania.
In many of the species the breeding plumage of the males is brightly coloured
Food-101 (Wiki knowledge improves performance)
[Figure]
and
[Figure] [Figure]
of
[Figure] [Figure]
in various Asian cuisines
[Figure]
are
[Figure]
bisque
[Figure]
hot
[Figure]
[Figure]
quesadilla :
[Figure]
a tortilla
[Figure]
cheese and
[Figure]
it
[Figure]
is melted
[Figure]
(a) Success examples.
The two datasets with largest improvement in Fig.
3: Flowers102 and Food101.
The description of the parent concept, material, shape, color etc. clariﬁes the concepts, boosting performance for the ﬁne-grained classiﬁcation tasks.
[Figure]
(b) Failure examples.
The two datasets with the largest performance loss in Fig.
Left (
): both class names have the same knowledge.
Middle & Right
): The knowledge contains spurious words that confuse the models.
[Figure]
[Table]
~ GLIP is implemented with parallel text encoding in Section 3.3 without external knowledge.
our implementation of GLIP using parallel text encoding, whose effectiveness is validated by thecomparable numbers with 1st row.
The 3rd row is our knowledge-augmented GLIP, i.e., K-L ITE .
The beneﬁt of using external knowledge is evident.
On LVIS, the categories are divided into rare,common, frequent groups, based on the number of training images per category.
K-L ITE improvesthe detection performance for all three groups with an average of 2.8 points on LVIS, and particularlybrings a 4.7 points improvement on MiniVal APc over the GLIP-A reported in [49].
We conclude thatthe enriched semantics of external knowledge signiﬁcantly helps the model recognize concepts witha decent number of instances.
Since LVIS has its own knowledge source S LVIS , mostly built upon
WordNet deﬁnitions [28], we evaluate our model with S LVIS .
We alter the external knowledge sourceto S wn_path , S wn_def , and S wiki_def in evaluation, which yields mAP 18.7, 21.4, 20.5, respectively.
It veriﬁes that our knowledge source extraction process is reliable.
Similarly, K-L ITE improve the 13downstream OD datasets from 28.8 (or 27.5 with its own knowledge-free counterpart) to 31.7.
Thesuccess and failure examples of OD are shown in Section B.5 in Appendix.
Conclusions
In this paper, we have presented a knowledge-augmented approach K-L ITE to learn a generic visualmodel for task-level transfer.
General external knowledge sources including WordNet and Wiktionaryare explored to enrich the natural language supervision, which is then used in both the language-imagepre-training stage and the prompt-based evaluation stage.
We have demonstrated the generality andeffectiveness of K-L ITE in two core computer vision problems: image classiﬁcation and objectdetection.
Extensive experimental results show that our method can achieve superior performanceover existing methods on 20 IC datasets and 13 OD datasets, respectively.
K-L ITE also outperformsits knowledge-free counterpart UniCL using half of the pre-training data in the large-scale academicdata setting, demonstrating that leveraging external knowledge is a promising direction in improvingpre-training sample-efﬁciency for learning transferable visual models.