V ISUAL E XPLANATION FOR D EEP M ETRIC L EARNING
Anonymous authors
Paper under double-blind review
A BSTRACT
This work explores the visual explanation for deep metric learning and its applications.
As an important problem for learning representation, metric learning has attracted much attention recently, while the interpretation of such model is not as well studied as classiﬁcation.
To this end, we propose an intuitive idea to show where contributes the most to the overall similarity of two input images by decomposing the ﬁnal activation.
Instead of only providing the overall activation map of each image, we propose to generate point-to-point activation intensity between two images so that the relationship between different regions is uncovered.
We show that the proposed framework can be directly deployed to a large range of metric learning applications and provides valuable information for understanding the model.
Furthermore, our experiments show its effectiveness on two potential applications, i.e. cross-view pattern discovery and interactive retrieval.
I NTRODUCTION
Learning the similarity metrics between arbitrary images is a fundamental problem for a variety of tasks, such as image retrieval (Oh Song et al.
, veriﬁcation (Schroff et al.
; Luo et al.
(2019)), localization (Hu et al.
video tracking (Bertinetto et al.
(2016)), etc.
Recently the deep Siamese network (Chopra et al.
based framework has become a standard architecture for metric learning and achieves exciting results on a wide range of applications (Wang et al.
However, there are surprisingly few papers conducting visual analyses to explain why the learned similarity of a given image pair is high or low.
Speciﬁcally, which part contributes the most to the similarity is a straightforward question and the answer can reveal important hidden information about the model as well as the data .
Previous visual explanation works mainly focus on the interpretation of deep neural network for classiﬁcation (Springenberg et al.
; Zhou et al.
; Fong & Vedaldi (2017)).
Guided back propagation (Guided BP) (Springenberg et al.
(2014)) has been used for explanation by generating the gradient from prediction to input, which shows how much the output will change with a little change in each dimension of the input.
Another representative visual explanation, class activation map (CAM) Zhou et al.
(2016), generates the heatmap of discriminative regions corresponding to a speciﬁc class based on the linearity of global average pooling (GAP) and fully connected (FC) layer.
However, the original method only works on this speciﬁc architecture conﬁguration and needs retraining for visualizing other applications.
Based on the gradient of the last convolutional layer instead of the input, Grad-CAM (Selvaraju et al.
(2017)) is proposed to generate activation maps for all convolutional neural network (CNN) architectures.
Besides, other existing methods explore network ablation (Zhou et al.
(2018)), the winner-take-all strategy (Zhang et al.
(2018)), inversion (Mahendran & Vedaldi (2015)), and perturbation (Fong & Vedaldi (2017)) for visual explanation.
Since the veriﬁcation applications like person re-identiﬁcation (re-ID)
(Luo et al.
(2019)) usually train metric learning models along with classiﬁcation, recent work (Yang et al.
(2019)) starts to leverage the classiﬁcation activation map to help improve the overall performance, but the activation map of metric learning is still not well explored.
For two given images, a variant of Grad-CAM has been used for visualization of image retrieval (Gordo & Larlus (2017)) by computing the gradient from the cosine similarity of the embedding features to the last convolutional layers of both images.
However, Grad-CAM only provides the overall highlighted regions of two input images, the relationship between each activated region of two images is yet to be uncovered.
Since the similarity is calculated from two images and possibly based on several similar patterns between them, the relationship between these patterns is critical for understanding the model.
[Figure]
[Figure]
In this paper, we propose an activation decomposition framework for visual explanation of deep metric learning and explore the relationship between each activated region by point-topoint activation response between two images.
As shown in Fig.
1, the overall activation map of the proposed method is generated by decomposing the similarity along each image.
In this example, the query image (A) has a high activation on both the eyes and mouth areas, but the overall map of the retrieved image (B) only highlights the eyes.
It is actually hard to understand how the model works only based on the overall maps.
For image B, the mouth region (green point) has a low activation which means the activation between the mouth and the whole image A is low compared to the overall activation (similarity).
However, by further decomposing this activation (green point) along image A, that is to give the activation between the mouth region of image B and each position in image A , the resulting partial (or point-speciﬁc) activation map (green box) reveals that the mouth region of image B still has a high response on the mouth region of image
A.
This partial activation map can be generated for each pixel, which renders the point-to-point activation intensity representing the relationship between regions in both images, e.g. eye-to-nose or mouth-to-mouth.
The partial activation map provides much more reﬁned information about the model which is crucial for explanation.
The contributions are summarized as follows.
• We propose a novel explanation framework for deep metric learning architectures and it may serve as an analysis tool for a host of applications, e.g. face recognition, person re-ID.
• The proposed method uncovers the point-to-point activation intensity which is not explored by existing methods.
Our experiments further show the importance of the partial activation map on several new applications,
i
e. cross-view pattern discovery and interactive retrieval.
• Our analysis suggests that two widely believed arguments (Section 2.1, 4) about CAM and GradCAM are inaccurate.
R ELATED W ORK 2. I NTERPRETATION FOR C LASSIFICATION
As a default setting, most existing interpretation methods (Zhou et al.
; Springenberg et al.
(2014); Zhang et al.
; Fong & Vedaldi (2017)) are designed in classiﬁcation context where the output score is generated from one input image.
Among the approaches that do not require architecture change, one intuitive idea (Zhou et al.
; Zhang et al.
(2018)) is to look into the model and see where contributes the most to the ﬁnal prediction.
CAM and its variant Grad-CAM are among the most widely used approaches, but recent works simply consider CAM as a heuristic linear combination of convolutional feature maps which is limited to the original architecture.
It is also believed (Zhou et al.
; Selvaraju et al.
that CAM only applies to the speciﬁc architecture conﬁguration of GAP and one FC layer (Argument 1).
On the contrary, we argue that
CAM is only a special case of activation decomposition on the speciﬁc architecture, and the idea applies to much more architectures, even beyond the classiﬁcation problem.
It is also believed that
Grad-CAM is a generalization of CAM for arbitrary CNN architecture (Argument 2), but the original paper only provides the proof on CAM’s architecture.
We show that Grad-CAM is not equivalent to activation decomposition for some architectures (Section 4.3).
Another direction of explanation is to check the black box model by modifying the input and observing the response in the prediction.
A representative method Fong & Vedaldi (2017) aims to optimize the blurred region and see which region has the strongest response on the output signal when it is blurred.
Perturbation optimization is applicable for any black box model, but the optimization can be computational expensive.
2. I NTERPRETATION FOR M ETRIC L EARNING
Guided BP (Springenberg et al.
(2014)) can be easily adopted to metric learning, but recent works (Adebayo et al.
; Fong & Vedaldi (2017)) claim that gradient can be irrelevant to the model and guided-BP fails on the sanity check (Adebayo et al.
Among methods that pass the sanity check, Grad-CAM has been used for visualization of image retrieval (Gordo & Larlus (2017)), but no quantitative result is provided.
Instead of adding more gradient heuristics like Grad-CAM++ (Chattopadhay et al.
, we propose to explain the result of Grad-CAM under the decomposition framework.
As shown in Section 4.3, by removing the heuristic step in Grad-CAM, the meaning of the generated activation map can be explained as the ﬁrst two terms of the proposed framework.
The perturbation optimization needs reformulation for metric learning and can be more computational expensive if the point-to-point activation map is desired.
A CTIVATION D ECOMPOSITION ON A S IMPLE A RCHITECTURE
[Figure]
To better introduce our method, we ﬁrst review the formulation of CAM and illustrate our idea on a simple architecture (CNN+GAP as in Fig. 2) for metric learning.
As shown in Eq.
1, CAM is actually a spatial decomposition of the prediction score of each class, and the original method only applies for CNN with GAP and one FC layer without bias.
The decomposition clearly shows how much each part of the input image contributes to the overall prediction of one class and provides valuable information about how the decision is made inside the classiﬁcation model.
The idea is based on the linearity of GAP:
[Equation]
Here S c denotes the overall score (before softmax) of class c and ω k,c is the FC layer parameter for the k -th channel of class c .
A i,j,k denotes the feature map of the last convolutional layer at position ( i, j ) , and Z is the normalization term of GAP.
In fact, the result (cid:80) k
ω
k,c
A i,j,k can be considered as a decomposition of S c along ( i, j ) .
For a two-stream architecture (see Fig. 2) with GAP and the cosine similarity metric ( S ) for metric learning application (e.g. image retrieval), we propose to do decomposition along ( i, j, x, y ) so that the relationship between different parts of two images are uncovered:
[Equation]
Here the normalization terms of the cosine similarity (L2 norm) and GAP are included in Z .
We use ( i, j ) , ( x, y ) for different streams because cross-view applications (Hu et al.
(2018)) may have different image sizes for two streams.
A denotes the feature map of the last convolutional layer.
The superscripts q and r respectively denote the query and the retrieved image in this paper.
For each query point ( i, j ) in the query image, the corresponding activation map in the retrieved image is given by (cid:80) k
A qi,j,k A rx,y,k which is the contribution of features at ( i, j, x, y ) to the overall cosine similarity.
Like CAM and Grad-CAM, bilinear interpolation is implemented to generate the contribution of each pixel pair and the full resolution map.
The overall activation maps of two images are generated by a simple summation along ( i, j ) or ( x, y ) .
Although we only show the positive activation in the map, the negative value is still available for counterfactual explanation (Selvaraju et al.
Since recent works (Liu et al.
; Wang et al.
(2018)) have highlighted the superiority of L normalization, we use the cosine similarity S as the default metric.
With L2 normalization, the squared Euclidean distance D equals to 2 − 2 S so that S and D are equivalent as metrics.
Although there are still a number of existing works (Luo et al.
(2019); Ristani & Tomasi (2018))
training with the Euclidean distance without L2 normalization, Luo et al.
(2019) shows that the cosine similarity still performs well as the evaluation metric.
We further empirically show that the cosine similarity works well for this case (Section 5.4).
E XTENSION FOR C OMPLEX A RCHITECTURES
Recent metric learning approaches usually leverage more complex architectures to improve performance, e.g. adding several FC layers after the ﬂattened feature or global pooling.
Although different metric learning applications have different head architectures (e.g. GAP+FC) on CNN, the basic components are highly similar.
We introduce a uniﬁed extension to make our method applicable to most existing state-of-the-art architectures for various applications, including image retrieval (Wang et al.
(2019)), face recognition (Deng et al.
re-ID (Luo et al.
and geo-localization (Hu et al.
Note that the extension also applies to classiﬁcation architectures .
In Section 4.1, we address linear components by considering them together as one linear transformation.
Then
Section 4.2 focuses on transforming nonlinear components to linear in the validation phase.
4. L INEAR C OMPONENT
The GAP can be considered as a special case of ﬂattened layer (directly reshape the last convolutional layer
A ∈ R m × n × p as one vector ˆ A ∈ R mnp without pooling) multiplied by a transformation matrix
T GAP ∈ R l × mnp where l denotes the length of the feature embedding vector (see Appendix
A.3 for details).
Without loss of generality, we consider a two-stream framework with ﬂattened layer followed by one FC layer with weights ˆ W ∈ R l × mnp and biases B ∈ R l .
Since the visual explanation is generated at the test phase when typical components such as FC layer and batch normalization (BN) are linear, all the linear components together are formulated as one linear transformation g ( ˆ A ) =
ˆ W ˆ A + B = (cid:80) i,j W i,j A
i,j + B in the FC layer.
Here W i,j ∈ R l × p and A
i,j ∈ R p denote the weights matrix and feature vector corresponding to position ( i, j ) .
Although B is ignored in CAM, we keep it as a residual term in the decomposition.
Then Eq. 2 is re-formulated as:
[Equation]
Here Z denotes the normalization term for cosine similarity (L2 norm), and · is the inner product.
The decomposition of S has four terms, and the last three terms contain the bias B .
The ﬁrst term clearly shows the activation response for location pair ( i, j, x, y ) and is considered as the point-topoint activation in this paper.
The second and third terms correspond to the activation between one image and the bias of the other image.
Although they can be considered as negligible bias term, they actually contribute to the overall activation map.
For the overall activation map of the query image, the second term can be included, because W qi,j A qi,j · B r varies at different ( i, j ) positions while the third and last terms stay unchanged.
Similarly, the ﬁrst and third terms are considered when calculating the overall map for the retrieved image.
We investigate both settings about the bias term (i.e. w or w/o) on the overall activation map and they are referred to as “Decomposition”:
( W qi,j A qi,j ) · ( (cid:80) x,y W rx,y A rx,y ) , and “Decomposition+Bias” : ( W qi,j A qi,j ) · ( (cid:80) x,y W rx,y A rx,y + B r ) in
Section 5.
The last term is pure bias which is the same for every input image.
Therefore, we ignore this term if not mentioned.
4. N ON - LINEAR C OMPONENT
[Figure]
Non-linear transformation increases the difﬁculty of activation decomposition, fortunately, the most widely used non-linear components – global maximum pooling (GMP) and rectiﬁed linear unit (ReLU) – can be transformed as linear operations in the validation phase by multiplying a mask 1 .
We only focus on GMP and ReLU in this paper, and all the mentioned components together have covered all the popular architectures for deep metric learning applications.
In the validation phase, the GMP can be considered as a combination of a maximum mask
M and GAP or ﬂattened layer as shown in Fig.
The result of Hadamard product ( M (cid:12) A ) is considered as the new feature map which can be directly applied to
Eq. 3.
Similarly, by adding a mask for ReLU, the FC layer with ReLU can be included in W and B in the validation phase.
4. R ELATIONSHIP WITH G RAD -CAM
Grad-CAM is another way for generating overall activation map on more complex architecture.
When we calculate the activation map based on Grad-CAM, what do we get?
Selvaraju et al.
(2017) has shown that Grad-CAM is equivalent to CAM on GAP based architecture.
However, for a classiﬁcation architecture with ﬂattened feature and one FC layer, the prediction score for class c may be formulated as S c = (cid:80) i,j,k W i,j,k,
c
A i,j,k where W is the reshaped weights of FC layer following the shape of the last convolutional feature ( A ).
From our activation decomposition perspective, the contribution of each position ( i, j ) is clearly given by (cid:80) k W i,j,k,c
A i,j,k , while the Grad-CAM map for class c at ( i, j ) position is given by:
[Equation]
In this case, the result of Grad-CAM is different from activation decomposition because of the GAP in Eq.
For architectures using GMP, they are also different, because only the maximal value of each channel contributes to the overall prediction score, while Grad-CAM puts the same weight for features at all positions (see Appendix A.4 for details).
Although Selvaraju et al.
(2017) empirically shows that the heuristic GAP step can help improve the performance of localization, this step makes the meaning of the generated map unclear.
By removing this step, which means combining the gradient and feature directly as A i,j,k ∂S c ∂A
i,j,k , the modiﬁed version would generate the same result as activation decomposition for ﬂattened and GMP based architectures.
As for the metric learning architecture discussed in Section 4.1, the Grad-CAM map of query image is written as (the derivation is included in Appendix A.5):
[Equation]
E q denotes the embedding vector of query image before L2 normalization and Z is the normalization term.
The gradient term ∂ ( E q / | E q | ) ∂E q , which comes from the L2 normalization, would put less weights on dominant channels so that the generated activation map becomes more scattering (Appendix A.5).
It can be removed by calculating gradient from E q · E r without L2 normalization.
If we remove the gradient term as well as the GAP term, the result is actually equivalent to the overall map of “Decomposition+Bias” given by the ﬁrst two terms of Eq.
3 as ( W qi,j A qi,j ) · ( (cid:80) x,y W rx,
y A rx,y + B r ) .
E XPERIMENT 5. W EAKLY S UPERVISED L OCALIZATION
Weakly supervised localization has been adopted as evaluation metric in most existing works, we thus conduct localization experiment in the context of metric learning to evaluate the overall ac- tivation map.
We follow the state-of-the-art image retrieval method Wang et al.
(2019) on CUB (Welinder et al.
(2010)), which contains over 10,000 images from 200 bird spices.
CUB is a challenging dataset for metric learning and the bounding box is available for localization evaluation.
We ﬁrst train a model on CUB with the loss function and architecture of Wang et al.
and then conduct the weakly supervised localization to show the differences between the variants of proposed methods.
As CAM and Grad-CAM generate the mask with different threshold settings (0.2 and 0.15), we use multiple thresholds and ﬁnd the result is sensitive to threshold (see Appendix
A.1).
We report the best result of each method in Table1.
The “Grad-CAM (no norm)” means the
Grad-CAM with gradient computed from the product of two embedding vectors without normalization and the “Decomposition+Bias” denotes the ﬁrst two terms of Eq.
The proposed framework outperforms Grad-CAM for metric learning as well as CAM based on classiﬁcation.
As justiﬁed in
Section 4.3, computing the gradients before normalization does improve the performance of GradCAM by a large margin.
Since the architecture of Wang et al.
(2019) is based on GMP, “Grad-CAM (no norm)” is not equivalent to “Decomposition+Bias” as shown in Section 4.3.
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
5. M ODEL D IAGNOSIS
A large number of loss functions have been proposed for metric learning (Schroff et al.
Oh Song et al.
; Wang et al.
(2019)), while only the performance and embedding distribution are evaluated.
Here we show that the overall activation map can also help evaluate the generalization ability of different metric learning methods.
We follow the setting of Wang et al.
and train
[Table] [Table]
[Figure]
two metric learning models with multiple similarity (MS) loss Wang et al.
and triplet loss
Schroff et al.
(2015), respectively.
As shown in Table 2, although they have almost the same accuracy on training set, there is a big gap between their generalization ability on the validation set.
Before checking the result on the validation set, is there any clue in the training set for such gap?
Despite the similar training accuracies, the predictions of two models are actually based on different regions (see
Fig. 5).
The quantitative results in Table 3 also support the fact that “Triplet” model is more likely to focus on the background region rather than the object as we witness the localization accuracy drop for Triplet loss.
Although “Decomposition+Bias” can be more sensitive to different loss functions, implying that the bias term does provide valuable information on overall map, both settings of the proposed method have the same trend on accuracy.
Therefore, our activation decomposition framework can help shed light on the generalization ability of loss functions for metric learning.
5. A PPLICATION I: C ROSS - VIEW P ATTERN D ISCOVERY
[Figure]
When the metric learning is applied to cross-view applications, e.g. image retrieval (Zhai et al.
Hu et al.
; Tian et al.
, the model is capable of learning similar patterns in different views which may provide geometric information of the two views, e.g. the camera pose or orientation information.
We conduct orientation estimation experiment to show the advantage of partial activation map compared with the overall activation map on providing geometric information based on cross-view patterns.
In our experiment, we take street-to-aerial image geo-localization as an example and train a Siamese network following the loss function of Hu et al.
The objective of geo-localization is to ﬁnd the best matched aerial-view image in a reference dataset for a query street-view image.
We conduct the experiment on CVUSA (Zhai et al.
(2017)), which is the most popular benchmark for this problem containing 35,532 training pairs and 8,884 test pairs.
Each pair consists of a query street image and the corresponding orientation-aligned aerial image at the same GPS location.
The orientation example is illustrated as angle degree in Fig.
6 (the yellow lines on both images correspond to 0 ◦ ).
We train the Siamese image matching model with randomly rotated aerial images so that the activation map is approximately rotation invariant for aerial image.
As shown in Fig.
6, the overall activation map may contain multiple highlighted regions contributing to the overall matching/similarity score.
Our method can further provide detailed point-to-point relationship which is critical for orientation estimation.
Here we utilize the maximum activated region for orientation estimation to show the superiority of the partial map compared with the overall map.
We ﬁrst generate the overall activation map as well as the partial map corresponding to the maximum activation position of the query street-view image.
For both activation maps, the activation peaks from different views are selected for orientation estimation as shown in Fig.
[Figure]
In this example, the overall activation maps contain multiple highlighted regions and the relationship between them are unclear.
The activation peaks in the two views (the cyan boxes) actually correspond to different objects which leads to a failure estimation.
However, the partial activation map clearly shows the relationship between these regions and provides more accurate estimation (the red line).
We also present the quantitative results in Fig.
As can be seen from this ﬁgure, the partial activation map signiﬁcantly outperforms the overall map for cross-view orientation estimation.
Partial activation based orientation estimation has over 16% samples with angle error less than ± 3 .
5 ◦ (the red bar at 0 ◦ ), while overall map based method only has less than 12% .
5. A PPLICATION II: I NTERACTIVE R ETRIEVAL
Veriﬁcation applications like face recognition and re-ID usually retrieve images with a complete query image.
However, the complete query image may be not available in some scenarios.
For example, only part of a person is captured in surveillance image/video due to occlusion or viewpoint.
Interactive retrieval, which retrieves images with interactively selected region of the query
[Figure]
image, can provide critical information for such circumstance.
Since our framework provides pointto-point activation map, it can be served as a reasonable tool for measuring partial similarity.
With the partial similarity, we are able to interactively retrieve images having similar parts with the query image instead of the overall most similar image, which may contain irrelevant or noisy information.
As shown in Fig.
8, the partial activation generated by the proposed method works well on retrieving people with similar clothes and faces with similar beard.
We follow the pipeline of recent approaches on face recognition (Deng et al.
and person re-identiﬁcation
(Luo et al.
For re-ID, the model is trained and evaluated on Market- 1501 (Zheng et al.
where some validation images only contain a small part of a person.
Although the model is trained with Euclidean distance without L2 normalization, cosine similarity still works well as the evaluation metric.
[Figure]
For face recognition, we take the trained model on CASIAWebFace (Yi et al.
and evaluate the model on FIW (Robinson et al.
where the face identiﬁcation and kinship relationship are available.
The interactive search is adopted by simply matching the equivalent partial feature in Eq.
3 ( W qi,j A qi,j ) with the reference embedding features.
Fig.
8 shows that different images are retrieved when searching with different partial features on the same query image.
However, we do ﬁnd some failure cases as shown in the last row of Fig.
8 with the red box.
There is no shoes in the failure image, but why is this image retrieved with a top rank?
In the activation map of Fig.
9, three regions of the query image have a high activation on the retrieved image.
And there is a high activation on the purse in the retrieved image corresponding to the shoes in the query image.
This might be because of their similar red color and the arm of the retrieved person may appears like a leg from a speciﬁc viewpoint.
This example also validates the importance of partial activation map generated by our framework for explanation.
C ONCLUSION
We propose a simple yet effective framework for visual explanation of deep metric learning based on the idea of activation decomposition.
The framework is applicable to a host of applications, e.g. image retrieval, face recognition, person re-ID, geo-localization, etc.
Experiments show the importance of visual explanation for metric learning as well as the superiority of both the overall and partial activation map generated by the proposed method.
Furthermore, we introduce two potential applications, i.e. cross-view pattern discovery and interactive retrieval, which highlight the importance of partial activation map.
A A PPENDIX A. W EAKLY S UPERVISED L OCALIZATION D ETAILS
As discussed in Appendix A.
[Table]
the activation map of Grad-CAM on GMP architecture is more scattering, as larger threshold is needed for Grad-CAM to generate comparable result in our experiment.
[Table]
[Table]
A. M ODEL D IAGNOSIS D
[Table]
[Table] [Table]
[Equation]
A. T RANSFORMATION M ATRIX FOR GAP AND GMP
For the feature A ∈ R m × n × p , the GAP (global average pooling) is equivalent to the ﬂattened feature ˆ A ∈ R mnp followed by transformation matrix
T GAP ∈ R p × mnp :
[Equation]
Here ( i, j ) denotes the spatial coordinates.
By reshaping the T GAP to p × m × n × p as T ∗ GAP , the matrix is given by:
[Equation]
The matrix T GAP is simply calculated by reshaping T ∗ GAP to p × mnp .
As shown in Section 4.2,
GMP is equivalent to GAP with a maximum mask in validation phase.
The transformation matrix of GMP (global maximum pooling) is given by:
[Equation]
Here M ∈ R m × n × p is the maximum matrix of A where only the maximum position of each channel has nonzero value as 1 .
T GMP is computed by reshaping T ∗ GMP to p × mnp .
A. G RAD -CAM ON GMP
Considering a classiﬁcation architecture with GMP and FC ( W ∈ R p × l and no bias), the prediction score for class c is given by S c
= (cid:80)
k max i,j ( A : , : ,k ) W k,c
= (cid:80) i,j,k A i,j,k M i,j,k W k,c , where M is the maximum matrix in Appendix A.3.
The decomposition of position ( i, j ) is (cid:80) k
A i,j,k M i,j,k W k,c and only the maximum position of each channel has a nonzero value.
However, for Grad-CAM, the activation map is given by:
[Equation]
where all (i,j) positions have a nonzero weight resulting in a more scattering activation map.
A. G RAD -CAM FOR M ETRIC L EARNING
For metric learning architecture like Section 4.1, the similarity is formulated as S =
E q · E r | E
q || E r | , where
E q ∈ R l and E r ∈ R l are the embedding vector of query and retrieved image.
| x | denotes the L norm and a · b is the inner product of a and b .
The Grad-CAM map of query image is given by:
[Equation]
With the gradient chain rule, the gradient is written as:
[Equation]
[Equation]
By expanding E r as (cid:80) x,y W rx,y A rx,y + B r
(Section 4.1) and merging Eq. 10 with Eq.
11, the
Grad-CAM map is reformulated as:
[Equation]
Here the Z is the normalization term for simplicity.
∂E/ | E | ∂E is the l × l Jacobian matrix given by:
[Equation]
| is the normalization term.
For dominant channel i , the weight (1 − E 2 i | E | 2 ) is small resulting in a more scattering activation map. A. Q UALITATIVE R ESULTS
We provide more qualitative results of using our activation decomposition framework for the two new applications, i.e., cross-view pattern discovery and interactive retrieval.
[Figure]
360°
Partial
Estimated ：91.2°
Ground Truth Angle= 72°
[Figure]
0°
[Figure]
0°
Overall
Estimated ：7.0°
Point-specific
Map 220.8 °
[Figure]
360°
Partial
Estimated ： 199.7°
[Figure]
[Figure]
5°
[Figure]
Overall
Estimated ：57.7°
Point-specific
Map
[Figure]
[Figure]
I NTERACTIVE R ETRIEVAL ON F ACE Eyes
[Figure]
A.6.
I NTERACTIVE R ETRIEVAL
ON
P ERSON R E -
ID
[Figure]
[Figure]
[Figure]
E XPLANATION OF F IGURE
[Figure]
Here we explain Fig.
6 again (Fig. 13) with more details.
We ﬁrst reiterate the application and experiment setting.
The dataset (CVUSA) we used in the experiment is for image-based crossview geo-localization (Zhai et al.
; Hu et al.
and contains image pairs from street and aerial views.
Each matched image pair corresponds to the same GPS location.
The objective of geo-localization is to ﬁnd the best matching GPS-tagged aerial-view image in a reference dataset for a query street-view image.
In the CVUSA dataset (Zhai et al.
(2017)), the image pairs are aligned in terms of orientation.
For example, as shown in Fig. 13, the left two images (streetview panorama and aerial-view images) are the original image pair and the yellow line denotes the 0 ◦ which corresponds to the south direction ( 180 ◦ corresponds to the North direction).
We use [0 , 360] ◦ to denote different angles as marked on these two images.
For the cross-view image matching/retrieval task, if the image pairs are not aligned (e.g., randomly rotate the aerial view images), we discovered that the activation map can be used for orientation estimation.
Speciﬁcally, we train the Siamese image matching model with randomly rotated aerial images so that the model is rotation-invariant and so is the overall activation map for aerial image.
In the example of Fig. 13, the overall activation maps for the original aligned pair are ﬁrst generated to show the highlighted regions, and both views focus on the road area in this example.
The most activated regions are highly relevant in both views and most of them are similar patterns.
When we randomly rotate the aerial image ( 30 ◦ in this example), the aerial view activation map still focuses on the road area which is relevant to the street view.
In this paper, we simply use the pixel with the maximum activation value for orientation estimation, because the most activated areas (highlighted in cyan boxes in Fig. 13) are likely to be relevant from two views.
For the street view image, the selected pixel lies in the angle of 90 .
2 ◦ (the cyan line) based on the angle marks on the left aligned image.
For the rotated aerial view, the selected pixel lies in the angle of 302 . 1 ◦ (the cyan line).
Since the overall map contains multiple activated regions, the selected pixels in both views actually do not correspond to the same object, which reveals one disadvantage by only using the overall activation map.
The estimated angle is 302 . 1 ◦ − 90 .
2 ◦ = 147 .
9 ◦ , which is not correct.
However, for the selected pixel (the one with the maximum activation value) in the street view image, if we generate its corresponding point-speciﬁc (partial) activation map on the aerial view image (as shown on the very right of Fig. 13), the new selected pixel on this point-speciﬁc activation map lies in the angle of 122 .
1 ◦ (the red line).
Then, the estimated angle is calculated as 122 . 1 ◦ − 90 .
2 ◦ = 31 .
9 ◦ , which is very close to the ground truth ( 30 ◦ ).
This demonstrates the advantage of the point-speciﬁc activation decomposition for ﬁnding more ﬁne-grained information in this application.
A.7. H OW TO COMPUTE THE ANGLE ERROR
The angle error is computed by error = ground truth − estimated angle .
We then add or subtract 360 ◦ to this term to make it in the range of [ − 180 , 180] ◦ if the absolute value of the error is greater than 180 ◦ .
For example, when the error is 359 ◦ , we will subtract it by 360 ◦ and get − 1 ◦ degree as the error.
This is a reasonable setting adopted from the previous work (Zhai et al.
A. I NTERACTIVE R ETRIEVAL D ETAILS A.8. H OW TO COMPUTER THE SIMILARITY GIVEN A POINT OF INTEREST
Following Eq.
3, we ﬁrst compute W qi,j A qi,j as the feature of each position on the last convolutional layer, and a bilinear interpolation is adopted to generate the feature for every pixel of the original image.
For a point of interest ( i, j ) , we compute the cosine similarity between the calculated feature on ( i, j ) and the embedding feature of the reference images as the point-speciﬁc similarity, so the embedding features of the reference dataset do not need to be recomputed.
This similarity can be also considered as the summation of the values in the point-speciﬁc map corresponding to ( i, j ) .
In the case of Eq. 2 (CNN+GAP), the similarity can be simpliﬁed as (cid:80) x,y ( (cid:80) k
A qi,j,k A rx,y,k ) .