Paper: Weakly-supervised Camera Localization by Ground-to-satellite Image Registration (ICLR-2024)

Reviewer comment (Experiments):
My main concern is about the setting of the proposed method. Given that the accurate pose label is not available for some places, it is still possible to get small-scale accurate labels at known locations. In this case, a semi-supervised setting could be more suitable than a weakly supervised formulation. The small-scale accurate label could also help improve the performance if properly designed.

Author response:
Thank you for suggesting the semi-supervised application. Our method can be easily adapted to address the scenario when a small amount of training data with accurate pose labels is available by adding additional supervision to the network with this amount of data.

Below, we show the performance of our method when fine-tuned with different portions of the current training dataset with accurate pose labels. Data amount = 0 indicates the original weakly supervised pipeline where no such data is available.


|Data Amount (%)|0|     0.2|0.3     |0.4|0.5|0.6|0.7|0.8|0.9|1|
|-|-|-|-|-|-|-|-|-|-|-|
|Lateral ($d=1$)|62.73|39.49|44.61|55.31|60.96|65.60|71.48|74.61|78.61|83.86|

The results show that fine-tuning leads to better performance when the data amount with accurate pose labels is over 50%. However, when the data amount is smaller, the fine-tuning leads to inferior performance. We suspect this is because the model overfits the limited training data with accurate labels, causing a loss of general inference ability on other data.

We agree a properly designed training method should address this problem, and we would like to explore this further in our future work.

Apart from this semi-supervised scenario, we believe the original weakly-supervised application addressed in this paper is also of high importance. It avoids additional effort to select the portion of accurate data, as most localization algorithms and noisy GPS sensors themselves do not provide a measure of whether its prediction is accurate or not.

Reviewer comment (Experiments):
Some suggestions to think about (Not a weakness): It might also be interesting to consider larger errors for the training images, for example, the satellite-view image has a certain probability of being an incorrect match of the query but is still close to the ground-truth locations.

Author response:
We are sorry that we did not really understand the question. Let us try our best to explain something according to our understanding, and please let us know your further questions.

When the location error of training images is larger, we can increase the coverage of the corresponding satellite image to make it still cover the query image's local surroundings. Our framework is still applicable.

When the satellite image is an incorrect match of the query (the actual location of the camera is outside the region covered by the satellite image, and this is not known), our method still compares the similarity between the ground and satellite image across locations on the satellite image, and find the location corresponds to the largest similarity (though the largest similarity is very small).

When the satellite image covers the local surroundings of the ground camera, while due to time differences between the two-view images, the ground structure might be changed or the illumination is significantly different (day and night), our method behaves similarly as above. It still compares the similarity between the two view images across possible locations on the satellite image and finds the location corresponds to the largest (though very small) similarity.

Reviewer comment (Writing):
There is still room for improvement in the writing. The motivation part of using the proposed approach could be improved:” Leveraging a blackbox network to regress……”. Minor issues: RTK is not explained when it first appears.

Author response:
Thanks for the suggestion. We have rewritten the introduction to make the motivation clearer. The revised content is marked as blue.

Reviewer comment (Writing):
The paper is well structured and readable, however I suggest to improve the introduction by motivating in much more clarity the matching problem and the idea of how you represent the problem of learning the high-level features and the registration via the correlation. The approach seems to be analog to popular representation learning approaches using contrastive learning. By focusing in your introduction and motivation on these problems and novelties, the paper would make clear its contributions to representation learning, the main topic of this conference.

Author response:
Thank you very much for the suggestion. We have rewritten the introduction part and labeled the revised text as blue.

Reviewer comment (Theory):
Page 2: "neural networks are inherently sensitive to rotations ..." — in its generality this statement needs to be refused as e.g. equivariant networks address this problem (https://arxiv.org/pdf/2205.07362.pdf).

Author response:
This description is avoided after we rewrite the introduction.

Reviewer comment (Writing):
Page 2: "... synthesised overhead view image ..." — the sentence gives the reader the impression that satellite images are always "overhead view" orthographic images, which of course is not the case.

Author response:
Thank you for the suggestion. We have added a footnote in this paper, clarifying that the satellite images used *for ground camera localization* are roughly orthographic.

Reviewer comment (Writing):
Page 4: "magnitude of the rotation"? — This phrase is unclear.

Author response:
We have changed this description to "magnitude of the rotation angle".

Reviewer comment (Theory):
Page 4: "the equivariance property of convolution to translations ..." — convolution is invariant to translation.

Author response:
We have changed the description to "the equivariance property of the convolution operation to translations".

Reviewer comment (Theory):
Page 5: Eq. 2 describes the correlation coefficient but later on page 14 a convolution was used.

Author response:
We have changed the terms to make them consistent.

Reviewer comment (Writing):
Page 4: Eq.1 — the rotation is parametrised so why not writing this? R(a).

Author response:
Thanks for the suggestion. We have changed Eq.1 to use the original angle representation instead of $\mathbf{R}$ in the loss. The revised content is marked as blue.

Reviewer comment (Writing):
Page 4: What does |R−R*| mean? L1 of ... — this formulation needs a more precise writing.

Author response:
Thanks for the suggestion. We have changed Eq.1 to use the original angle representation instead of $\mathbf{R}$ in the loss. The revised content is marked as blue.

Reviewer comment (Theory):
Would it not be beneficial to minimise R^(−1)*R → I ?, as R is SO(3).

Author response:
Thanks for the suggestion. We make our network directly output the angle instead of a rotation matrix and thus do not need to apply this constraint to the network output.

Reviewer comment (Experiments):
The satellite images used in this work are 512x512 pixels. In practice satellite images are orders of magnitude larger. To be useful in practice, the method needs to find regions of interest in satellite images in order to perform registration. How could you achieve this?

Author response:
In this work, we consider refining a coarse location of a ground image by ground-to-satellite registration. Thus, given this coarse location and its statistical error, we can extract a satellite image that covers the local surroundings of the query image. In this work, the satellite image coverage is around 100m x 100m (with a resolution of 512 x 512).

When the location ambiguity of a ground image is significantly large, for example, at the city or state level, cross-view retrieval is usually applied. We discuss this line of work in the first paragraph of the Related Work section.

Reviewer comment (Theory):
The bird's-eye view synthesis in Sec. 3 needs to be further investigated and justified. The synthesized bird's-eye view could be dramatically different from the projected bird's-eye view from the ground image due to parallax, occlusion, and appearance changes, making it very difficult to learn feature representations that are invariant to these changes.

Author response:
Our method does not aim to learn viewpoint invariant features robust to parallax and occlusion. Instead, we want to explore the co-visible regions between the two views and weaken other regions' effect in similarity matching. The co-visible regions mainly concentrate on the ground plane and thus are handled by the ground plane's homography projection. For regions that are not co-visible, we resort to the confidence map to lower their weights in similarity matching. The appearance discrepancy for the co-visible areas is handled by the end-to-end training.

We discussed why using this simple projection instead of a powerful yet complex cross-view transformer for overhead-view synthesis in Sec. A of the appendix. The main reason is a balance between batch size and the model complexity. A large batch size is important in self/weakly supervised learning by contrastive learning. We also visualized the projected and corresponding satellite feature maps in Sec. B. The ground structures co-visible by both views are aligned well. More discussions are presented in Sec. C.

We follow previous works for data augmentation: none is applied to ground images, while satellite images are randomly rotated.

Reviewer comment (Evaluation):
In order to convince the audience that the gap between the synthesized and projected bird's-eye views is small, the authors should a) introduce their data augmentation in detail, and b) analyze the feature differences between the projected and actual bird's-eye view images.

Author response:
Our method does not aim to learn viewpoint invariant features robust to parallax and occlusion. Instead, we want to explore the co-visible regions between the two views and weaken other regions' effect in similarity matching. The co-visible regions mainly concentrate on the ground plane and thus are handled by the ground plane's homography projection. For regions that are not co-visible, we resort to the confidence map to lower their weights in similarity matching. The appearance discrepancy for the co-visible areas is handled by the end-to-end training.

We discussed why using this simple projection instead of a powerful yet complex cross-view transformer for overhead-view synthesis in Sec. A of the appendix. The main reason is a balance between batch size and the model complexity. A large batch size is important in self/weakly supervised learning by contrastive learning. We also visualized the projected and corresponding satellite feature maps in Sec. B. The ground structures co-visible by both views are aligned well. More discussions are presented in Sec. C.

We follow previous works for data augmentation: none is applied to ground images, while satellite images are randomly rotated.

Reviewer comment (Novelty):
An important reference is missing: "OrienterNet: Visual Localization in 2D Public Maps with Neural Matching", CVPR '23. The proposed method in this paper is very similar to this missing reference except for the self-supervision. The OrienterNet has achieved better performance than the paper, and more importantly, it is better written with deeper insights.

Author response:
Thank you for pointing out this accidentally missed reference. We have added it back to the paper. 

Both works synthesize an overhead-view feature map from the ground-view image and match it against the reference satellite feature map for translation estimation. The **differences** are:
1. OrienterNet rotates the synthesized overhead view feature map K times and then matches them to the reference satellite image. To avoid the additional search space caused by rotation and obtain continuous rotation estimates, we present a pose regressor for rotation estimation.
2. Our pose regressor leverages the sensitivity of a normal neural network to the rotation of input signals, and we introduce a self-supervised method for this pose regressor training. Our rotation estimation performance is significantly better than OrienterNet.
3. Our method does not require ground truth location labels of ground images for network training.

Reviewer comment (Novelty):
To highlight the difference from OrienterNet, the authors should clearly describe why precise supervision is difficult, which does not seem to be the case since we can get precise reconstructions even from a mobile phone and then register a sequence of images to the satellite images.

Author response:
Precise reconstruction can be obtained from a sequence of ground view images by SfM. However, the GPS of a mobile phone is often noisy. Thus, the absolute geo-location of the reconstructed 3D structure is unknown. Registering reconstructed 3D structures to a satellite image requires complex engineering techniques. From our investigation, there is no readily available technique for this. This effort can be avoided by using the method proposed in this paper. The original noisy GPS signals of ground images can be directly used for network training.

Reviewer comment (Experiments):
To decouple the contribution between model structure and supervision, can we do supervised training and see the upper-bound?

Author response:
The performance of our method with full supervision is as follows:
||Lateral||Longitudinal||Orientation||
|-|-|-|-|-|-|-|
||$d=1$|$d=3$|$d=1$|$d=3$|$\theta=1^\circ$|$\theta=3^\circ$|
|Test-1|83.86|96.69|30.74|58.65|99.66|100.00|         
|Test-2|50.99|79.51|14.62|31.25|99.99|100.00|

The performance improves significantly for the same-area evaluation (Test-1), while it suffers slight degradation on the cross-area evaluation (Test-2). This is similar to the observation on other fully-supervised methods: models tend to overfit on the training area when ground truth poses are available and further demonstrates that the proposed weakly-supervised approach exhibits strong generalization ability.