ARTEMIS: A TTENTION - BASED
R ETRIEVAL
WITH
T EXT -E XPLICIT M ATCHING & I MPLICIT S IMILARITY
Anonymous authors
Paper under double-blind review
A BSTRACT
An intuitive way to search for images is to use queries composed of an example image and a complementary text.
While the ﬁrst provides rich and implicit context for the search, the latter explicitly calls for new traits, or speciﬁes how some elements of the example image should be changed to retrieve the desired target image.
Current approaches typically combine the features of each of the two elements of the query into a single representation, which can then be compared to the ones of the potential target images.
Our work aims at shedding new light on the task by looking at it through the prism of two familiar and related frameworks: text-to-image and image-to-image retrieval.
Taking inspiration from them, we exploit the speciﬁc relation of each query element with the targeted image and derive light-weight attention mechanisms which enable to mediate between the two complementary modalities.
We validate our approach on several retrieval benchmarks, querying with images and their associated free-form text modiﬁers.
Our method obtains state-of-the-art results without resorting to side information, multi-level features, heavy pre-training nor large architectures as in previous works.
I NTRODUCTION
When using an image search engine, should the user provide a detailed textual description or a visual example?
These two image retrieval tasks, respectively studied as cross-modal search (Wang et al., 2016) and visual search (Philbin et al., 2007) in the computer vision community, have been the topic of extensive studies.
However, limiting search queries to a single modality is restrictive.
On the one hand, text proposes an accurate but only partial depiction of the desired result, as it is impossible to provide an exhaustive textual description.
On the other hand, visual queries are richer but a lot more ambiguous: how should the system guess which characteristics the user wants to maintain for the target image?
The task of image search with free-form text modiﬁers reduces this ambiguity by allowing a dual query composed of an example image (the reference image) and a textual description (the text modiﬁer) which explains how the reference image should be modiﬁed to obtain the desired result (the target image).
This task is illustrated in Figure 1.
From this formulation, a challenging research question arises; How should the two facets of this dual query be leveraged and combined?
First methods to answer this question directly borrowed from the visual question answering literature (Kim et al., 2016;
Santoro et al., 2017).
Now, the standard approach for this task consists in fusing the features of the two components of the query into a single representation, so it can be compared to the representation of any potential target image (Vo et al., 2019; Chen & Bazzani, 2020).
Among the current strategies, some resort to rich external information (Chen et al., 2020;
Liu et al., 2021) while others rely on multi-level visual representations (Chen et al., 2020) or heavy cross-attention architectures (Hosseinzadeh & Wang, 2020; Chawla et al., 2021; Liu et al., 2021).
Departing from these strategies, we draw inspiration from two ﬁelds related to the task, cross-modal and visual search, and we advocate for a combination of the two components of the query, in a way that also takes into account their speciﬁc relationship with the target image.
To this end, we design two independent modules, one for each modality.
Our Explicit matching (EM) module measures the compatibility of potential target images with the properties mentioned in the textual part of the query, as a form of cross-modal retrieval.
Simultaneously, our Implicit similarity (IS) module considers the relevance of the target images with respect to the properties of the reference image implied by the textual
Additionally, we note that the text modiﬁer provides some insights about what should change (as an explicit request from the user), but also about what should be preserved in the target image with respect to the reference image ( i.e. what is not mentioned in the text modiﬁer).
Consequently, each of our modules uses a light-weight attention mechanism, guided by the text modiﬁer which helps selecting the characteristics of the target image to focus on for retrieval.
This results in our proposed model, named A ttention-based R etrieval with T ext-
E xplicit M atching
and I mplicit S imilarity ( ARTEMIS ), which jointly learns these two modules, their respective attention mechanisms, and the way they should be combined, in a uniﬁed manner.
In summary, our contribution is twofold.
First, we suggest a new way to look at the task of image search with free-form text modiﬁers, where we draw inspiration from – and combine – the two ﬁelds of cross-modal and visual retrieval.
Second, we propose a computationally efﬁcient model based on two complementary modules and their respective text-guided light-weight attention layers, where each handles a modality of the query and both can be trained jointly.
We experimentally validate ARTEMIS on several datasets and show that, despite its simplicity in terms of architecture and training strategy, it consistently outperforms the state of the art.
R ELATED WORK
Image retrieval with purely visual queries has been extensively studied and is typically referred to as instance-level image retrieval (Philbin et al., 2007).
Most approaches either learn global image representations for efﬁcient search (Perronnin et al., 2010; Radenovi´c et al., 2016;
Gordo et al., 2016) or extract local descriptors for a costly yet robust matching (Arandjelovi´c & Zisserman, 2012;
Tolias et al., 2016; Noh et al., 2017).
Current state-of-the-art models use both global and local descriptors jointly and perform both strategies sequentially (Cao et al., 2020; Tolias et al., 2020).
The most standard retrieval scenario involving text, cross-modal retrieval , uses a textual description of the image of interest as a query.
A common approach is to learn a joint embedding space and modality-speciﬁc encoders producing global representations in that shared space.
Then, the task boils down to computing distances between those representations (Frome et al., 2013; Wang et al., 2016; Faghri et al., 2018).
The most recent works focus on cross-attention between text segments and image region features (Lee et al., 2018; Li et al., 2019; 2020).
However, these models do not scale, and they require to score each potential text-query target-image pair independently.
Miech et al.
(2021) address this issue by distilling a cross-attention architecture into a dual encoder one.
Despite similarities between these two tasks and the one of image search with text modiﬁers , the ﬁrst baselines for the latter initially adapted visual question answering (VQA) approaches (Kim et al., 2016).
To reason with features from different modalities, Santoro et al.
(2017) use a sequence of MLPs, whereas FiLM (Perez et al., 2018) injects text features into the image encoder at multiple layers, altering its behavior through complex modiﬁcations.
Current approaches address this task by composing the two query elements into a single joint representation that is then compared to any potential target image feature.
As such, in TIRG (Vo et al., 2019), the reference image and the “relative captions” (or text modiﬁers) are fused through a gatingresidual mechanism, and the text feature acts as a bridge between the two images in the visual representation space.
Anwaar et al.
(2021) use an autoencoder-based model to map the reference and the target images into the same complex space and learn the text modiﬁer representation as a transformation in this space.
Lee et al.
(2021) and Chawla et al.
(2021) both propose to disentangle the multi-modal information into content and style.
Chen & Bazzani (2020) resort to image’s descriptive texts as side information to train a joint visual-semantic space, and then train a TIRG model on top.
The alignment of visual and textual features for regularisation is reused in VAL (Chen et al., 2020) which also inserts a composite transformer at many levels of the visual encoder to preserve and transform the visual content depending on the text modiﬁer; the model is then optimized with a hierarchical matching objective.
Hosseinzadeh & Wang (2020) also align image and text through a cross-modal module, but they use region proposals instead of CNN activations.
Liu et al.
(2021) propose a transformer-based model that leverages rich pre-trained vision-and-language knowledge for modifying the visual features of the query conditioned by the text.
We note that these last methods are reminiscent of the heavy cross-attention models for cross-modal retrieval and share the same limitations with respect to scaling.
In contrast to most methods described above, ARTEMIS does not compose modalities into a joint global feature for the query (Vo et al., 2019; Chen & Bazzani, 2020; Lee et al., 2021), it does not compute costly cross-attention involving the target image (Hosseinzadeh & Wang, 2020; Chawla et al., 2021) and does not extract multi-level visual representations (Chen et al., 2020).
Instead it leverages the textual modiﬁer in simple attention mechanisms to weight the dimensions of the visual features, emphasizing the characteristics on which the matching should focus.
This results in a model with a manageable amount of parameters to learn during training, efﬁcient inference on the query side, and leading to state-of-the-art results. About attention.
We note that the term attention has been heavily used in the past, referring to light-weighting mechanisms (Xu et al., 2015; Li et al., 2017; Kang et al., 2018; Wang et al., 2019) similarly to us, or to more complex weighting mechanisms (Lu et al., 2016). This differs from the more recent deﬁnition of attention introduced by Vaswani et al. (2017), which became standard in many approaches tackling the cross-modal retrieval (Lee et al., 2018; Li et al., 2019; 2020)
and VQA (Bai et al., 2018) tasks.  P ROPOSED METHOD This section describes ARTEMIS, our proposed approach for the task of image search with freeform text modiﬁers.
In this setting, queries are bimodal pairs, composed of a reference image I r and a text modiﬁer T m .
Such queries are used to retrieve any relevant image I t from a gallery of images.
Let φ ( . ) and θ ( . ) be the visual and textual encoders respectively.
We denote the resulting L2normalized features of the reference image r , of the text modiﬁer m , and of the target image t , i.e. r = φ
( I r ) ∈ R H
I , m = θ ( T m
) ∈ R H T , and t = φ ( I t ) ∈
R H
I .
Two complementary views of the task.
Both the textual and the visual cues are important to ﬁnd relevant target images.
Consider for example the query in Figure 1, where the dress’ length, among other style components from the visual example, and the colour requested in the text modiﬁer are combined to retrieve relevant results.
The reference image provides strong visual context and displays some semantic details that should be shared with the target image, to the extent that the text does not specify their alteration.
In other words, by the absence of an explicit modiﬁcation request of some properties, the modifying text “validates” the associated visual cues from the reference image: those visual cues can be used as direct query for the target image.
Hence, part of the task boils down to image-to-image retrieval (visual search), implicitly conditioned on the text.
On the other hand, the textual cue explicitly calls for modiﬁcations of the reference image.
It was even observed that, in some cases, users tend to directly describe characteristics of the target im- age, without explicitly stating the discrepancy between the reference and the target image 1 ( c.f.
the analysis presented for FashionIQ in Table 2 of Wu et al.
(2021), for CIRR in Table 1 of Liu et al.
Therefore the text, which describes mandatory details for the target, can be directly used to query for the target image, as is common in text-to-image retrieval.
Yet, due to the speciﬁcities of the task, we go beyond the usual text-to-image retrieval.
Proposed approach.
We propose to revisit text-to-image and image-to-image retrieval to tackle the task at stake, which is at the intersection of both.
Standard strategies, for both, boil down to directly comparing the global features of the query (image or text) and of the potential targets, producing two independent, modality-speciﬁc, and not necessarily immediately compatible scoring strategies.
In particular, directly comparing the representations of the reference and the target images is inaccurate as the two images should differ, based on the modifying text.
Likewise, comparing directly the representations of the modifying text and the target image is undesired since part of the target image should stay similar to the reference image.
Here, we reconcile the two tasks and enhance them by designing a model which is trained to match i) the characteristics of the reference image with the ones of the target image and ii) the requirements provided by the text with properties of the target image.
Based on the intuition above, in both cases we use the text as an attention mechanism to select the visual cues which should be emphasized during matching.
The ﬁrst objective is carried out by the Implicit Similarity module (Section 3.1), which draws from visual search, and the second by the Explicit Matching module (Section 3.2), inherited from crossmodal retrieval.
These two modules respectively output the IS and the EM scores, which are simply summed into one as a form of late fusion.
The modules are jointly trained through this ﬁnal score (Section 3.3).
The full architecture of the model is illustrated in Figure 2.
3. I MPLICIT S IMILARITY
In the ﬁrst module, we use the textual cue m as a guide to compare the target image to the reference image.
An attention mechanism A IS ( . ) :
R H T → [0 , 1] H I , taking m as input, predicts the importance of the visual representation dimensions when comparing the target and the reference images.
The IS score is then computed as:
[Equation]
where (cid:12) is the pointwise product and (cid:104) .
| .
(cid:105) represents cosine similarity (the two terms are L2normalized).
A IS ( . ) is a two-layer MLP separated by a ReLU, and followed by a softmax operation.
3. E XPLICIT M ATCHING
This second module captures how well the target image matches the textual cue.
As the target image t and the textual modiﬁer m belong to different modalities, we ﬁrst project m into the visual space with a learned linear transformation T ( . ) :
R H T → R H
I .
In addition, since the textual modiﬁer should not act on all the characteristics of the image (indeed, the target image should remain partially similar to the reference image), we rely on an attention mechanism, A EM ( . ) :
R H T → [0 , 1]
H I to emphasize the dimensions of the image feature to focus on for matching.
Hence, the EM score is computed as:
[Equation]
T ( . ) consists in a fully connected layer.
Note that A IS ( . ) and A EM ( . ) share the same architecture but not their weights, as the role they give to m is different.
Indeed, intuitively A EM ( . ) focuses on what is mentioned by the textual modiﬁer while A IS ( . ) looks for what is not.
Put differently,
A EM ( . ) “up-weights” the dimensions in the image feature that correspond to the semantic information provided by the textual cue, whereas A IS ( . ) “down-weights” these dimensions to rather focus on the shared characteristics between the images.
3. T RAINING PIPELINE
Given a minibatch of B training samples B = { ( I r i , T m i , I t i ) } Bi =1 where I t i corresponds to an image annotated as correct target for the query ( I r i , T m i ) , we train our model using the batch-based classiﬁcation (BBC) loss:
[Equation]
where γ is a learnable temperature parameter and s ( r i , m j , t k ) is the “compatibility” score of triplet ( i, j, k ) in [1 , B ] 3 .
For ARTEMIS , this compatibility score is the sum of the scores EM and IS respectively obtained by each of the modules presented above:
[Equation]
The BBC loss, proposed as InfoNCE in the self-supervised learning literature (Oord et al., 2018), was introduced for image retrieval with text modiﬁers by Vo et al.
(2019) and
its efﬁcacy for the task was conﬁrmed by Lee et al.
In contrast to the triplet and the soft triplet losses which rely on negative mining strategies to obtain optimal results, the BBC loss considers all the negatives samples in a mini-batch, and therefore simultaneously learns from easy and hard negatives.
Encoders.
Our approach is generic and can accommodate for diverse visual encoders φ and textual encoders θ .
Our results present several combinations, their choice being mostly driven by comparability with prior work.
We typically use GloVe word embeddings (Pennington et al., 2014) followed by either a LSTM (Hochreiter & Schmidhuber, 1997) or a BiGRU (Cho et al., 2014) as text encoder and an ImageNet pretrained ResNet18 or ResNet50 model
(He et al., 2016) as image encoder.
Further implementation details can be found in Appendix A.
E XPERIMENTS 4. E XPERIMENTAL SETTING
Datasets.
We consider two datasets focusing on the fashion domain and one on open-domain images, all three using human-written textual modiﬁers in natural language.
The Fashion IQ dataset (Wu et al., 2021) is composed of 46.6k training images and around 15.5k images for both the validation and test sets.
There are 18k training queries, and 6k queries per evaluation split, covering three fashion categories: women’s tops (toptee), women’s dresses (dress) and men’s shirts (shirt).
The text modiﬁer is composed of two relative captions produced by two different human annotators, [Table] exposed to the same reference-target image pair.
The Shoes dataset (Guo et al., 2018) is extracted from the Attribute Discovery Dataset (Berg et al., 2010).
It consists of 10k training images for 9k training triplets [Table] and 4.7k test images for 1.7k test queries.
The recently released CIRR dataset (Liu et al., 2021) is composed of 36k pairs of open-domain images, arranged in 80-10-10 proportion between the train/validation/test splits.
The annotation process is such that the modifying texts should only be relevant to one image pair, and not be suitable for another similar image pairs containing the same reference image.
Evaluation.
We use the standard evaluation protocol for each dataset.
We report Recall@K ( R @ K ), which is the percentage of queries for which the correct ground truth item is ranked among the top
K retrieved items.
All reported numbers are obtained as an average of 3 runs.
In Section 4.3, we additionally report standard deviations.
For FashionIQ, the standard evaluation metric is the one used for the companion challenge (Gao & Guo, 2020).
It computes the R @10 and R @50 for each of the 3 fashion categories and averages these six results.
For Shoes, we report the R @1 , R @10 and R @50 as well as their average.
For CIRR, following Liu et al.
, we also report Recall subset @
K , in which the set of candidate target images is restricted to images semantically similar to the correct target image.
According to Liu et al.
(2021), this metric is less sensitive to false-negatives.
Cross-validation.
Since the test split annotations of FashionIQ became available only very recently, previous works only report results on the validation set, and resort to ad hoc strategies to set their hyper-parameters and early-stop criteria.
In order to compare our method to previously published approaches, we provide results on both the test and validation sets by performing a bidirectional cross-validation between them: we run our experiments for a ﬁxed number of epochs and, at the end of each epoch, we evaluate our model on both sets.
We select the best performing model on the validation set to report results on the test set and vice-versa.
When no validation split is available, as for Shoes, we report the results on the last checkpoint.
For CIRR, we select the best checkpoint on the validation split to evaluate on their test server.
4. A BLATION STUDY
We ﬁrst conduct an ablative study on the validation set of all datasets, to evaluate the inﬂuence of several design choices in our architecture.
For FashionIQ and Shoes, we additionally report median rank of the correct target.
For CIRR, we report the R @5 and R s @1 in addition of R @50 , since they best capture the model capabilities according to Liu et al.
Results are presented in Table 1.
All experiments in this section use ResNet50 and BiGRU as visual and textual encoders, and the same training pipeline described in the Appendix A. We separate our ablation studies into two parts; ﬁrst we look at several baselines which do not use any attention mechanism, then we challenge the different module choices by training them independently.
Baselines.
Image-only and text-only baselines use only one of the queries elements to retrieve the target.
Their compatibility scores can be written as s img ( r , m , t ) =
(cid:104)
r | t (cid:105) and s txt ( r , m , t )
= (cid:104) m | t (cid:105) , respectively.
We also report the late fusion of the two query embeddings, which can be seen as an attention-free version of ARTEMIS.
In this case, the compatibility score becomes s lf ( r , m , t ) = (cid:104) r + m | t (cid:105) . [Table]
[Table] [Table]
The performance of these baselines changes according to the nature of each dataset.
For FashionIQ, where 68% of the modiﬁers make direct reference to the target image (Wu et al., 2021), the textonly baseline outperforms the image-only one.
For Shoes, based on the image-only results and on visual inspection, we conjecture that the reference-target image pairs are, on average, more similar than for FashionIQ, making the reference image more crucial for a wide part of the queries.
For
CIRR, while querying only with the reference image offers a strong baseline, the text modiﬁer is more discriminative when ranking on a restricted set of very similar images.
This is measured by the R s @1 metric.
Note that the late fusion baseline produces strong results for both FashionIQ and
Shoes, but barely improves upon the image-only baseline for CIRR, which reveals the importance of the attention mechanisms.
Independent modules.
We evaluate the IS and EM independently as separate retrieval models.
Their corresponding compatibility scores are given by equations 1 and 2, respectively.
We observe that IS and EM perform differently for different datasets, and as expected, follow the previous observations made for the image-only and text-only baselines.
We see that both our modules slightly outperform their corresponding baseline.
The full ARTEMIS performs best for all three datasets.
This highlights the synergy between IS and EM, supporting the motivations extensively discussed in
Sections 3.1 and 3.2, but also the fact that the method is robust to different types of datasets / queries: those where the visual part is more important as well as those where the textual part is preponderant.
4. C OMPARISON WITH THE STATE OF THE ART
We now compare ARTEMIS to state-of-the-art approaches on each benchmark.
We also report results for our re-implementation of TIRG (Vo et al., 2019), a well-spread composition network baseline, with the same training pipeline of ARTEMIS, as the original paper does not provide results on these more recent datasets. [Table] [Table]
Results on Fashion IQ are reported in Table 2 and Table 3 for the validation and the test sets, respectively.
Since the test split annotations were not available until very recently (only Wu et al.
(2021) published results on that split), most published approaches report results on the validation set.
In both cases our best ARTEMIS (RN50 + BiGRU) outperforms the state of the art.
As most published works (Chen et al., 2020; Lee et al., 2021) use LSTM as text backbone and
ResNet50 as image backbone, we also report results for ARTEMIS in this setting and show that we remain competitive, outperforming VAL (cid:63) by an average of 1.4% over the challenge metric.
Furthermore, we also report results with ResNet18 to compare to (Anwaar et al., 2021;
Chen &
Bazzani, 2020), who prioritize faster and smaller backbones.
We observe that, for smaller image backbone, LSTM seems to perform better then BiGRU.
This is also true for our re-implementation of TIRG.
Results on Shoes are presented in Table 4.
As the Shoes dataset does not contain a dedicated validation set, we reuse the exact same training setting and hyperparameters as for FashionIQ.
ARTEMIS obtains state-of-the-art results with both LSTM and BiGRU encoders, conﬁrming our approach’s ability to generalize.
CIRR results are reported in Table 5.
As recommended by Liu et al.
(2021), we use pre-extracted
ResNet152 features for the images.
This affects our training pipeline as the image backbone cannot be ﬁnetuned and we apply no image transformation.
CIRPLANT obtains competitive results only when its multi-layer cross-modal transformer is initialized with the weights from the Oscar model (Li et al., 2020) trained on 6.5 million image-caption pairs.
When compared to methods trained on the same data, ARTEMIS outperforms CIRPLANT in all metrics, in spite of a simpler architecture.
In particular, our R s @1 result suggests, according to Liu et al.
(2021), that ARTEMIS has a great ﬁne-grained reasoning ability.
4. Q UALITATIVE RESULTS
In Figure 3 we present a few heatmap examples obtained with the Grad-CAM (Selvaraju et al., 2017) algorithm.
Speciﬁcally, the image activations at the last convolutional layer of the CNN are reweighed with the pooled gradients of the same layer, after back-propagation of the coefﬁcients contributing the most to the EM and IS scores.
It appears clearly that the IS module attends to the [Figure] [Figure] visual cues that are similar between the reference and the target images, such as the shirt’s collar and bottom, the sole shape & the heel, or the color and the shoe opening.
In a complementary way, the
EM module addresses the image parts that are the most related to the caption: the color, the shirt’s ﬁt, the slits on the side or the shoe’s closure.
In Figure 4, we present some query examples from the CIRR dataset along with the Top-6 retrieved images.
In both examples, we see that the reference image contributes a strong semantic context, and that the text modiﬁer provides important information to distinguish the correct target image (framed in green) from other very similar images.
Additional qualitative examples and further analysis are available in the Appendix § D.
C ONCLUSION
We proposed ARTEMIS, a new method for image search with free-form text modiﬁers.
It combines two modules, each focusing on one of the modalities of the query.
The Explicit Matching module retrieves potential targets as described in the textual modiﬁer while the Implicit Similarity module compares potential target images to the reference image, assisted by the text.
Both modules contain a light-weight attention mechanism which leverages the text to guide the retrieval process in a speciﬁc way for each modality.
Extensive experiments on several benchmarks validate our approach and show that we consistently reach state-of-the-art performance.
R EPRODUCIBILITY S TATEMENT
Upon acceptance of this paper we intend to release a precise description of our architecture ( i e. the corresponding scripts) as well as the weights of all our best models for all the datasets we considered in our experimental evaluation.
We will also share the code used for evaluation of all datasets, so fellow researchers can reproduce and validate our results.
A PPENDIX
This document provides additional results that complement the main paper.
Its content includes some implementation details (§ A), our experiments on an additional dataset (§ B), an efﬁciency study of our method and comparison to other approaches (§ C), and some qualitative results for ARTEMIS (§ D).
A I MPLEMENTATION DETAILS
Texts are preprocessed to replace special characters by spaces and to remove all other characters than letters.
We observe that using spelling correction and lemmatization does not improve the results.
The text modiﬁer is ﬁrst tokenized into words, and encoded using 300-dimensional GloVe embeddings (Pennington et al., 2014).
Either a BiGRU (Cho et al., 2014), or a LSTM (Hochreiter &
Schmidhuber, 1997) (followed by an average pooling and a fully-connected layer, as in Chen et al.
(2020)) are learned from scratch on top of the word embeddings to produce a sentence-level feature.
To maintain the aspect ratio and avoid losing information about the top and the bottom of the clothing items, images are ﬁrst padded with white pixels to obtain a square.
They are then resized to 256 ×pixels and augmented using random horizontal ﬂip and a random crop of size of 224 × 224 pixels during training.
For inference, images are resized to 256 × 256 and center cropped to 224 × 224 .
The image features are obtained using a CNN with a GeM pooling (Radenovi ´ c et al., 2018), which is widely used in the instance-level image retrieval community instead of the usual average pooling.
The pooling layer is followed by a newly learned fully-connected layer.
Our experiments employ either a ResNet18 or ResNet50 architecture (He et al., 2016), initialized on ImageNet (Russakovsky et al., 2015).
The only exceptions are the experiments on CIRR, for which not all images are available to download.
Liu et al.
(2021) provide pre-extracted ResNet152 features (outputs of the average pooling layer) that we use as a replacement of the image backbone.
As a consequence, for all experiments on CIRR, the image backbone is not ﬁne-tuned and we do not apply any image transformation.
Following Song & Soleymani (2019), we freeze the base encoders during the ﬁrst 8 epochs to pretrain the sentence encoder, EM and IS modules.
Then, we train our model end-to-end for 50 epochs.
Our training pipeline uses AdamW optimizer (Loshchilov & Hutter, 2017), a batch size of 32 and an initial learning rate of 5 × 10 − 4 with a decay of 0 .
5 every 10 epochs.
The dimension of both the image and the textual embeddings is set to H T
= H
I = 512 .
B E XPERIMENTS ON F ASHION 200K
In the literature (Vo et al., 2019; Chen et al., 2020; Lee et al., 2021), the Fashion200K dataset (Han et al., 2017) is generally used as an example of a dataset for image search with text modiﬁers.
The dataset is composed of 201k in-shop fashion article images, split into 172k train images and 29k test images.
From these test images, Vo et al.
(2019) provides a list of 31k image pairing which is used to produce candidate queries for evaluation.
The text modiﬁers, in contrast to the other presented datasets, are not in natural language: they are automatically generated from the set of image attributes following the “ replace att.
X by att.
Y ” template.
In our work, we focus on freeform human annotations, which is the common point between the three datasets presented in the main paper.
However, we here compare ARTEMIS to the other published works on this dataset.
In order to better address the large train set of Fashion200K, we follow VAL (Chen et al., 2020) and CoSMo (Lee et al., 2021), and we train our model with a larger mini-batch of 128 and forepochs end-to-end.
While Chen et al.
; Chen & Bazzani (2020) use MobileNet-v1 as image backbone, we follow Vo et al.
; Lee et al.
and use ResNet18.
As Lee et al.
(2021) pointed out, though the difference in model architecture may play a role in overall results, both models have almost identical scores on both ImageNet Top1 and Top5 error rates.
Interestingly, while ARTEMIS performed better with a BiGRU as text encoder for all other datasets, our best results for Fashion200K were obtained with a LSTM.
We observe the same for our implementation for TIRG, and more generally for all our experiments that use a ResNet18.
[Table] We observe on Table 6 that VAL (cid:63) obtains the best performance, thanks to the extra captions used to pretrain their auxiliary regularizer.
The same side information was also leveraged by Chen & Bazzani (2020).
Excluding models trained with this extra data, both ARTEMIS and CoSMo outperform
VAL, and our approach obtain the best results in 2 out of 4 evaluation metrics.
C C OMPLEXITY AND EFFICIENCY STUDY
In the main paper, we extensively discuss our method’s performance in terms of accuracy, on several benchmarks.
This section provides a complementary study that compares ARTEMIS’ complexity with the complexity of several approaches.
Since no single measure is enough to assess the complexity of a model (Dehghani et al., 2021), we compare the different models on three metrics:
• Number of parameters.
We measure the number of trainable parameters of the full models.
Many works use this metric as a proxy to model complexity and memory footprint.
• Number of multiply-add operations.
The number of multiply-accumulate operations (MAC) is a theoretical measure that allows to compare the number of operations needed by the model to perform a forward pass, independently of the hardware (Johnson, 2018).
We report the number of GMACs for the forward pass of a triplet ( I r , T m , I t ) , where reference and target images are both of size 224 × 224 , and the modiﬁer text is composed of 20 tokens (in average, CIRR sentences contain 11.3 words and FashionIQ 5.3, according to Tablein Liu et al. (2021)).
• Latency time.
Latency usually refers to the required time to perform an inference forward pass, either for a batch of images or the full dataset.
It is an informative speed metric for real-time systems that require user input, so it is well suited to the retrieval task.
We measure the time each method needs to process all queries, all target images, and all querytarget scores of the FashionIQ validation split.
All latency times are measured on the same GPU NVIDIA T4.
We report results for ARTEMIS, TIRG, VAL and CoSMo.
For a fair comparison, all reported numbers are obtained for ResNet50 and LSTM as image and text encoder respectively.
Since these encoders are an important source of complexity in the models, we also evaluate the encoders alone, to provide a reference point.
For this reference, we estimate the corresponding latency by evaluating the late-fusion baseline described in Section 4.2.
We report our results in Table 7.
We make the following observations:
ARTEMIS and TIRG both add very little complexity compared to the reference.
Similarly to TIRG (Vo et al., 2019), our approach adds only a few operations and parameters to the base
[Table]
encoders used to compute r , m and t .
Both methods’ parameters mostly come from their fullyconnected layers.
TIRG uses more parameters than ARTEMIS because its gating mechanisms takes the concatenated vector [ r , m ] as input instead of just the text embeddings m , doubling the number of required connections with respect to ARTEMIS .
ARTEMIS and TIRG add some reasonable extra-latency compared to the reference.
Both
ARTEMIS and TIRG add little latency time to the encoder reference point.
TIRG is more efﬁcient due to its simple pass on the target representation: pre-computed t can be directly compared to the composite query representation by a simple cosine similarity.
ARTEMIS also leverages precomputed t , but re-weights its dimensions using A IS ( m ) and A EM ( m ) then L2-normalizes the result before comparing to the query features, with cosine similarity as well.
This double use of the target explains ARTEMIS ’ slightly longer latency time.
VAL has a higher computational complexity.
We were unable to reproduce all the same experiments on the publicly available code of VAL to compute the latency.
For the complexity measurements, we can attest to an increase in the number of GMACs and parameters that is more signiﬁcant than for both ARTEMIS and TIRG.
The latency is likely to increase similarly, as VAL inserted composite transformers at different levels of the visual encoder, and compares representations coming from the concatenation of average-pooled features at low-, mid- and high-level ( i.e. features of size 2048 + 1024 + 512 ).
CoSMo adds both extra-complexity and extra-latency.
CoSMo, similarly to TIRG, uses a composition module to fuse r and m , and does a single pass on each target.
However, its composition module, combining a disentangled non-local block for content modulation and text-guided afﬁne transformations for style modulation, is more complex than ARTEMIS .
Furthermore, while
ARTEMIS and TIRG introduce little complexity, CoSMo adds around 25% multiply-accumulation operations, and more than doubles the number of trainable parameters.
D Q UALITATIVE RESULTS
Additional ranking results.
We provide qualitative results for the FashionIQ (Wu et al., 2021), the
Shoes (Guo et al., 2018; Berg et al., 2010) and the CIRR (Liu et al., 2021) datasets in Figures 5, 6, 7, 8 and 9.
They illustrate the model’s ability to properly encode and reason on general concepts.
In the case of FashionIQ, it includes the length of sleeves and dresses, the patterns (ﬂoral, stripes, dots, plaid) or the logo color (different from the background color) and the looseness of clothing items.
As for Shoes it captures shininess, wool/suede material, and the presence of laces, of fur, of
Velcro closures, or of a buckle, for instance.
The CIRR dataset is open-domain, and we notice that the model is able to reason on a wide variety of semantic concepts (from trees to the people’s youth).
The model is also able to deal with more complex information, such as dolphins (Figure 6, second row) or different renderings of the same logo (Figure 6, third row).
Eventually, we observe that the model gathers pieces of information from the reference image when needed (style and cut, length or size, collar form, sleeve length, pattern or logo, sole shape, animal breed, sculpture etc.).
While the model relies on the text modiﬁer to propose coherent images, it also resorts to the reference image to ﬁll in the gaps.
Limitations.
Extensive studies of the qualitative results (Figure 10) reveal that the model has trouble to understand the negative formulations: in the third row of Figure 10, the proposed toptees remain colorful; in the second row, the ranked clothing items are still loose and sheer; the last row shows a mix of shoes with laces and other with straps.
As for FashionIQ, it is likely due to the fact that only 3.5% of the queries use negation, according to a semantic analysis carried out in the companion paper (Wu et al., 2021).
Supposedly, there are not enough examples at training time for the model to learn the associated semantic.
Besides, the captions were written by different human annotators.
They sometimes present a subjective judgment ( e.g. “is cuter”, “more manly”) or lack essential details to discriminate better the target among other images in the gallery.
We give a few illustrative examples of such cases in Figure 10.
For the top query, the speciﬁc required color is not mentioned, hence the various guesses taken by the model.
One limitation inherent to the task is well illustrated by the blue dress example in Figure 5, the boot example in Figure 8 or the bird example in Figure 9: very often, many candidate target images would be acceptable answers to a given query, beyond the one annotated as ground-truth.
Hence we believe
R @
K is a more reliable metric when K is bigger as it’s more lenient to plausible wrong predictions.
Other works (Musgrave et al., 2020; Chun et al., 2021) have pointed out the limitations of relying on
R @ K for retrieval tasks when K is too small, and suggested other metrics.
This is also a limitation that Liu et al.
(2021) tries to address with their R s @
K metric, by annotating images while aware of possible false negatives.
Additional heatmaps on examples extracted from FashionIQ Wu et al.
(2021) are presented in Figure 11.
Similarly, Figure 12 presents results on the Shoes Guo et al.
; Berg et al.
(2010) dataset.
The most important parts of the image with regard to the computation of each score (Explicit Matching (EM) and Implicit Similarity (IS)) are highlighted in red.
The coefﬁcients of EM and IS intermediary results ( i.e. before completing the scalar product with the sum) are referred as “components”.
The heatmaps are obtained by backpropagating to the last convolutional layer of the CNN some of the components that contribute the most to the EM and IS scores.
We show the inﬂuence of a same IS component both on the reference and the target images, since both are involved in the computation of IS.
Conversely, EM is based only on the target image, with regard to the text modiﬁer.
It emerges from the IS visualizations that the model is able to draw the parallel between the two images on several concepts.
For clothing items, it can be the shape of the collar, the sleeve length or the cut of the dress.
For shoe items, it includes the presence of heels or laces, the shoe opening or the sole shape.
In a complementary way, the EM module addresses the image parts that are the most related to the caption: for example, it focuses on the pattern, the color or decoration requirements.
Eventually, these visualizations help to understand what semantic parts of the image are taken into account by the model to evaluate the relevance of a candidate target image for a given query.
[Figure]
[Figure] [Figure] [Figure]
[Figure]
two bottles are identical, and in the middle one pepsi bottle
Dog sits on a rock in the water.
Younger girls and different pajamas
Dog runs through the snow.
Focus on two birds with orange and green bodies perched on a branch.
Take a far away shot showing trees behind scene
Another angle and gorilla's mouth open
More distant pandas and one with his face turned to the left.
[Figure] [Figure]
EM
IS s a m e c oe ff
i
c i en t s a m
e c oe ff i
c i en t
IS
EM
IS s a m e c oe ff
i
c i en t
EM are gray with slits on the sides have laces
[Figure]
not
[Figure]
Velcro closure
[Figure]
Target
Query
Target
Target Query
[Figure]
closed loafers
EM
IS s a m e c oe ff
i
c i en t s a m
e c oe ff i
c i en t
IS
EM
IS s a m e c oe ff
i
c i en t
EM have small silver accents is
[Figure]
folded top
Query
Target
Query
[Figure]
Query
[Figure]
decoration
For each of the six blocks, we provide the heatmaps of an EM component on the target image, and of two relevant IS components, applied to both the reference and the target images.
The EM heatmaps highlight some of the connections made by the model between the text modiﬁer and the target image.
The IS heatmaps display some detected similarities between the two images.
Both EM and IS help to evaluate the relevance of the target image with regard to the query.