Sphere2Vec : S ELF -S UPERVISED L OCATION R EPRESENTATION L EARNING ON S PHERICAL S URFACES
Anonymous authors
Paper under double-blind review
A BSTRACT
Location encoding is valuable for a multitude of tasks where both the absolute positions and local contexts (image, text and other types of metadata) of spatial objects are needed for accurate predictions.
However, most existing approaches do not leverage unlabeled data, which is crucial for use cases with limited labels.
Furthermore, the availability of large-scale real-world GPS coordinate data demand representation and prediction at global scales.
Existing location encoding models assume that the input coordinates are in Euclidean space, which can lead to modeling errors due to distortions introduced when mapping coordinates from other manifolds (e.g., spherical surfaces) to Euclidean space.
We introduce Sphere2Vec , a location encoder, which can directly encode spherical coordinates while preserving spherical distances.
Sphere2Vec is trained with a self-supervised learning framework which pre-trains deep location representations from unlabeled geotagged images with contrastive losses, and then ﬁne-tunes to perform supervised geographic object classiﬁcation tasks.
Sphere2Vec achieves the performances of state-of-the-art results on various image classiﬁcation tasks ranging from species,
Point of Interest (POI) facade, to remote sensing.
The self-supervised pretraining signiﬁcantly improves the performance of Sphere2Vec especially when the labeled data is limited.
I NTRODUCTION
Spatial information has become an important component to many machine learning tasks that address the environmental, economic, and societal needs for sustainable development.
Just to name a few, these tasks include geo-aware species ﬁne-grained recognition (Chu et al., 2019; Mac Aodha et al., 2019), Point of Interest (POI) facade image classiﬁcation (Chu et al., 2019; Yan et al., 2018), remote sensing image classiﬁcation (Christie et al., 2018; Ayush et al., 2020), poverty prediction (Jean et al., 2016; 2019), point cloud classiﬁcation and semantic segmentation (Qi et al., 2017a;b), trafﬁc forecasting (Li et al., 2018; Cai et al., 2020), and geographic question answering (Mai et al., 2019; 2020a).
Developing a general model for vector space representations of any point in space, termed a location encoder , would pave the way for many future applications that improves our well-being.
Such a location encoder can operate on a 2D space or on other manifolds; of particular interest is the spherical surface of the Earth.
Here we use geo-aware image classiﬁcation shown in Figure 1 and 2 as an example.
During training we assume a dataset D “ tp I i , x i , y
i q| i “ 1 , ..., N u , where I i is an image, y i P 1 , ...., C is the corresponding class label, x i represents the location (longitude and latitude) and optionally the time the image was taken 1 .
The image information is often processed by an state-of-the-art image classiﬁcation model (e.g., InceptionV3) to predict the class labels as shown in Figure 1.
Although the image itself is the most critical for image classiﬁcation, sometimes it can be insufﬁcient – the image of two distinct classes may look very similar both for the central object and its surrounding environment.
For example, Figure 2a and 2f shows example images of two different fox species: Arctic fox (Vulpes lagopus) and bat-eard fox (Otocyon megalotis), for which the image model can be confused due to the high visual similarity of the two species.
Fortunately, the geo-location information of an image may provide a clue to their real classes.
Figure 2b, 2g show that these two species have distinct spatial distribution patterns, and the image locations provide complementary information to the images.
This
[Figure] [Figure]
insight motivates the development of hybrid classiﬁcation models, which take both the images and their locations as input (Mac Aodha et al., 2019; Mai et al., 2020b).
In spite of their effectiveness, hybrid models have an issue when representing points in non-Euclidean spaces.
The demand on representation and prediction learning at a global scale grows dramatically due to emerging global scale issues, such as the transition path of the latest pandemic (Chinazzi et al., 2020), long lasting issue for malaria (Caminade et al., 2014), under threaten global biodiversity(Di Marco et al., 2019; Ceballos et al., 2020), and numerous ecosystem and social system responses for climate change (Hansen & Cramer, 2015).
However, when applying the state-of-the-art 2D space location encoders (Gao et al., 2019;
Chu et al., 2019; Mac Aodha et al., 2019; Mai et al., 2020b; Zhong et al., 2020) to large-scale real-world GPS coordinate datasets such as species images taken all over the world, a map projection distortion problem (Williamson & Browning, 1973) emerges.
These 2D location encoders are designed for preserving distance in 2D (or 3D)
Euclidean space, while GPS coordinates are in fact on a spherical manifold, e.g., point x “ p λ, φ q on a sphere with longitude λ
P r´ π, π q and latitude φ P r´ π { 2 , ´ π { 2 s (See Figure 1).
Directly applying these 2D location encoders on spherical coordinates will yield a large distortion in the polar regions.
Note that map projection distortion is unavoidable when projecting spherical coordinates into 2D space.
This emphasizes the importance of calculating on a round planet (Chrisman, 2017).
See Appendix 9.1 for more discussion.
From a modeling perspective another key challenge is how to design the self-supervised training objective that effectively transfers the attribute information (e.g., image features) at different locations into a location encoder.
This is very important in practice since collecting and labeling geo-tagged images is particularly costly as annotations often require domain expertise.
However, most existing location encoding approaches (Chu et al., 2019; Mac Aodha et al., 2019; Mai et al., 2020b) are developed and trained in a supervised learning framework while massive unlabeled geographic data cannot be used.
Ayush et al.
(2020) proposed a geographic self-supervised learning framework, but they mainly focus on training image encoders while the usage of geo-location information is limited to data augmentation in the pre-training stage.
In contrast, in this work, we focus on training location encoders in a self-supervised manner so that geo-location information is explicitly use as part of the model input.
In this work, we propose Sphere2Vec as shown in Figure 1, which directly encodes point coordinates on a spherical surface while avoiding the map projection distortion.
The multi-scale encoding method utilizes Double Fourier Sphere basis ( O p S 2 q terms) or a subset ( O p S q terms) of it while still being able to correctly measure the spherical distance.
Furthermore, we develop a self-supervised learning framework which pre-trains deep location representations from unlabeled geo-tagged images by predicting image features or image identities base on their geo locations.
We compare several self-supervised learning objectives including Mean Square Error loss (MSE), contrastive binary classiﬁcation loss (BI), and contrastive multi-classiﬁcation loss (MC).
The pre-trained Sphere2Vec model can be ﬁne-tuned (with an additional output layer) to perform supervised geographic objects classiﬁcation tasks.
We demonstrate the effectiveness of Sphere2Vec on geo-aware image classiﬁcation tasks including ﬁne-grain species recognition (Chu et al., 2019; Mac Aodha et al., 2019; Mai et al., 2020b), POI fascade image classiﬁcation (Tang et al., 2015; Mac Aodha et al., 2019), and remote sensing image recognition (Christie et al., 2018; Ayush et al., 2020).
In summary, the contributions of our work are:
We develop a uniﬁed view of distant preserving location encoding methods on spheres based on Double Fourier Sphere (DFS) (Merilees, 1973; Orszag, 1974).
We propose an effective pre-training strategy for location encoders from unlabeled geotagged images by predicting image features and image identities based on their geo locations.
Sphere2Vec improves the performances of state-of-the-art results on four species recognition datasets (BirdSnap, NABirds, iNat2017, iNat2018), one POI fascade image classiﬁcation dataset (YFCC), and one remote sensing image classiﬁcation dataset (fMoW).
It also demonstrate signiﬁcant advantage in few shot learning settings
P ROBLEM F ORMULATION
Following previous work (Mai et al., 2020b), Euclidean location encoding can be formulated as follows.
Given a set of points (e.g., POI) P “ t p i u in L -D space ( L “ 2 , 3 ), we deﬁne a function e P ,θ p x q : R L Ñ R d ( L ! d ), which is parameterized by θ and maps any coordinate x in space to a vector representation of d dimension.
Each point (e.g., a restaurant) p i “ p x i , v i q is associated with a location x
i and attributes v i (i.e., POI features such as types, names, capacity, images, etc.).
The function e P ,θ p x q encodes the information about Point x which is useful for certain tasks such as predicting a label y .
Attributes and coordinates of points can be seen as analogies to words and word positions in commonly used word embedding models.
Similarly, we can formulate spherical location encoding as follows.
Given a set of points P “ t p i u on the surface of a sphere S 2 , e.g., species occurrences all over the world, where p i “ p x i , v i q and x i “ p λ i , φ i q P S 2 indicates a point with longitude
λ i
P r´ π, π q and latitude φ i
P r´ π { 2 , π { 2 s .
Deﬁne a function e P ,θ p x q : S 2 Ñ R d , which is parameterized by θ and maps any coordinate x in a spherical surface S 2 to a vector representation of d dimension.
See Appendix 9.1 for more discussions about the map projection distortion problem.
M ETHOD
The geo-aware image classiﬁcation task (Chu et al., 2019; Mac Aodha et al., 2019) can be formulated as follows.
Given an image I taken from location/point x , we estimate which category y it belongs to by P p
y
| I , x q .
If we assume I and x are conditionally independent given y , then based on Bayes’ theorem, we have P p
y
| I , x q 9 P p y | x q P p y
| I q (See Mac Aodha et al.
P p
y
| I q can be given by any state-of-the-art image classiﬁer such as Inception V3 (Szegedy et al., 2016) for the species recognition task or MoCo-V2+TP
(Ayush et al., 2020) for the RS image classiﬁcation task.
Figureillustrates the whole training and inference process by using species recognition as the example task.
P p y
| I q and P p
y | x q are given by the image encoder (the green box) and location encoder (the blue box) respectively and are multiplied together for ﬁnal prediction.
In this work, we focus on estimating the geographic prior distribution of class y over the spherical surface P p y | x q 9 σ p e p x q T : ,y q where σ p 9 q is a sigmoid activation function.
T P R d ˆ c is a class embedding matrix where the y th column
T : ,y P R d indicates the class embedding for class y .
In the following we introduce the two main contributions of our work – location encoding on spherical surfaces (the blue box in Figure 1), and a self-supervised training framework for location representation model (the orange box in Figure 1).
In Section 3.1, we will discuss the design of spherical distance-kept location encoder e p x q , Sphere2Vec .
We developed a uniﬁed view of distancereserving encoding on spheres based on Double Fourier Sphere (DFS) (Merilees, 1973; Orszag, 1974).
In Section 3.2 we discuss training strategies for Sphere2Vec including self-supervised pre-training given unlabeled images, and supervised ﬁne tuning.
We use e P ,θ p x q to indicate the spherical location encoding neural network with parameters θ , and will refer it by e p x q for simplicity.
3. Sphere2Vec E NCODER
We want to ﬁnd a function G p sphere q p x q which does a one-to-one mapping from each point x i “ p λ i , φ i q P S 2 to a multi-scale representation with S as the total number of scales such that it satisﬁes the following requirement:
[Equation]
where ∆ D P r 0 , πR s is the spherical surface distance between x 1 , x 2 , R is the radius of this sphere, and f p x q is a strictly monotonically decreasing function for x P r 0 , πR s .
In other words, we expect to ﬁnd a function G p sphere
q p x q such that the resulting multi-scale representation of x preserves the spherical surface distance.
Similar to the case for Euclidean space (Mai et al., 2020b), any point x “ p λ, φ q P S 2 with longitude λ
P r´ π, π q and latitude φ P r´ π 2 , π 2 s , can be explicitly encoded in a multi-scale fashion in the form of e p x q p x q “ NN ffn p G p˚q p x qq where NN ffn pq is a learnable multi-layer perceptron with h hidden layers and k neurons per layer.
G p˚q p x q is a concatenation of multi-scale spherical spatial features of S levels.
p˚q indicates different types of spherical location encoders.
In the following, we call e p x q p x q location encoder and its component G p˚q p x q position encoder .
Let r min , r max be the minimum and maximum scaling factor, and g “ r max r min .sphereDFS
Double Fourier Sphere (DFS) (Merilees, 1973; Orszag, 1974) is a simple yet successful pseudospectral method, which has been applied to efﬁcient analysis of large scale phenomenons such as weather (Sun et al., 2014) and blackholes (Bartnik & Norton, 2000).
Our ﬁrst intuition is to use the base functions of DFS to help decompose x “ p λ, φ q into a high dimensional vector:
[Equation]
where @ s “ 0 , 1 , ..., S ´ 1 , we deﬁne λ s “ λf s , φ s “ φf s , f s “ r min ¨ g s {p S ´ 1 q .
Y means vector concatenation and Ť S ´ 1 s “ 0 indicates vector concatenation through different scales.
It basically lets all the scales of φ terms interact with all the scales of λ terms in the encoder.
This would introduce a position encoder whose output has O p S 2 q dimensions and increase the memory burden in training and hurts generalization (Figure 3a).
[Figure]
An encoder might achieve better result by only using a subset of these terms.
In comparison, the state of art encoder such as grid and wrap (Mac Aodha et al., 2019) deﬁne:
[Equation]
We can see that grid employs a subset of terms from sphereDFS (Figure 3b) and performs poorly at a global scale due to its inability to preserve spherical distances (See Appendix 9.3 for proof).
We explore different subsets of DFS terms while achieving two goals: 1) efﬁcient representation with O p S q dimensions 2) preserving distance measures on spheres.
See Appendix 9.2 for the detail deﬁnition of sphereC , sphereC ` , sphereM , sphereM ` , and Figure 3 for a uniﬁed view.
We hypothesize that encoding these terms in the multi-scale representation would make the training of the encoder easier and the order of output dimension is still O p S q .
In location encoding, the uniqueness of encoding (no two points on sphere having the same position encoding) is very important, G p˚q p x q in the ﬁve proposed methods are one-to-one mapping.
Theorem 1. @˚
P t sphereC, sphereC ` , sphereM, sphereM ` , sphereDFS u , G p˚q p x q is an injective function.
See the proof in Appendix 9.5.
3. T RAINING Sphere2Vec
Self-Supervized Pretraining (the yellow box in Figure 1).
We consider two contrastive objectives.
The ﬁrst is a binary classiﬁcation loss, or so called noise contrastive estimation (Gutmann &
Hyvärinen, 2010), which avoids calculation of the partition function and has been successfully used in word embeddings (Mikolov et al., 2013) and language modeling (Mnih & Teh, 2012):
[Equation]
where BI stands for “binary”, P is a set of positive pairs (denoted as p a , b q ), N is a set of negative pairs (denoted as p a , b ´ q ), s p¨ , ¨q is a similarity function that assigns higher values when the inputs are more similar (such as cosine pq ), and σ p v q “ e v {p 1 ` e v q is the sigmoid function.
The second objective function is the multi-class classiﬁcation loss with temperature, which has been successfully used in unsupervised learning for images (He et al., 2020) and text (Gao et al., 2021):
[Equation]
where MC stands for “multi-class”, N p b | a q obtains a negative pair with ﬁrst entry being a ; P and s p¨ , ¨q are deﬁned as earlier.
The temperature scaling parameter τ determines how soft the softmax is (Hinton et al., 2015).
In practice it help with the trade off between top ranked classes (precision) versus reset of the classes (recall).
In order to learn useful representations, we need to choose appropriate distributions for positive pairs P and negative pairs N .
During the location encoder pretraining stage we use a geo-tagged but unlabeled image set X unlabeled , each image is represented by a embedding generated from a pretrained image extract extractor F 4 such as InceptionV3 or MoCo-V2+TP (Ayush et al., 2020).
Given an unlabeled image p x , I q P X unlabeled , we use the following positive and negative instances:
• Geo-tagged positive P x “ tp e p x q , W F p
I qqu where W is a linear projection layer.
• In-batch negatives N b “ tp e p x q , W F p
I ´ q|p
x ´ , I ´ q P B p x , I qztp x , I quu where B p x , I q represents the batch of examples for which p x , I q is part of during training.
• Sampled negative locations
N s “ tp e p x ´ q , W F p I qqu of size N , where x ´ is one of the N evenly sampled locations from the surface of the sphere for each example x . • Dropout positive P d “ tp e p x q , e 1 p x qqu , where we pass the same input x to the encoder e pq twice and obtain two embeddings as “positive pairs” by applying independently sampled dropout masks.
This is a data augmentation strategy (so called SimCSE), which has been very successful in generating sentence embeddings (Gao et al., 2021).
We use e 1 p x q to denote the embedding from the second mask.
• Dropout negative N d “ tp e p x q , e 1 p x ´ qq|p
x ´ , I ´ q P B p x , I qztp x , I qu , where e 1 p¨q has the same meaning as that deﬁned in P d and B p x , I q indicates the same as N b .
We deﬁne two versions of contrastive losses, which both have three components: in-batch, negative location, and SimCSE.
The self-supervized binary classiﬁcation loss L BI is deﬁned as
[Equation]
where β 1 and β 2 control the contribution of the last two loss components.
The self-supervized multi-class classiﬁcation loss L MC is deﬁned as
[Equation]
where α 1 and α 2 are hyper-parameters.
Supervised Fine-Turning (the blue box in Figure 1).
We directly use image labels in the training objective.
Following Mac Aodha et al.
(2019), we used a presence-absence loss function which converts the multi-class labels into binary multi-labels.
Given a set of training samples X labeled “ tp x , y qu , the loss function L supervized p X q is deﬁned as:
[Equation]
Here, β is a hyperparameter to enlarge the weight of positive samples, and the following positive and negative samples are used:
• Labeled positives P l “ tp e p x q , T : ,y q|p x , y q P X u .
• Labeled negatives
N l “ tp e p x q , T : ,i q|p x , y q P X , i P t 1 ..
c uzt y uu .
• Sampled negative locations N ls “ tp e p x ´ q , T : ,i q|p x , y q P X , i P t 1 ..c uu , where x ´ is one of the N evenly sampled locations from the surface of the sphere for each example x .
R ELATED W ORK
Multi-Scale Position/Location Encoding on Spheres
Recent works in various applications (Zhong et al., 2020; Mai et al., 2020b; Mildenhall et al., 2020; Tancik et al., 2020) found that a simple sinusoidal mapping of input coordinates (“positional encoding”) allows multilayer perceptrons (MLP) to represent higher frequency content.
More speciﬁcally f θ is decomposed into f θ “ NN θ p PE
p x qq , where PE is a deterministic function mapping x from R d into higher dimensional space and NN θ is simply a MLP.
This strategy is compelling since MLPs can be orders of magnitude more compact than grid-sampled representations in order to achieve similar order of details (Tancik et al., 2020).
They show that passing input points through a simple Fourier feature mapping enables a MLP to learn high-frequency functions in low-dimensional problem domains.
This development is consistent with Rahaman et al.
(2019a) who show that deep networks are biased towards learning lower frequency functions, and mapping the inputs to a higher dimensional space using high frequency functions enables better ﬁtting of data that contains high frequency details.
In this work we extend the success of this line of work to none Euclidean spaces such as spherical surface.
We leverage truncated Fourier transformation on spheres to achieve computation efﬁciency while avoiding the error caused by projection distortion.
Unsupervised Representation Learning Unsupervised text encoding models such as transformer (Vaswani et al., 2017; Devlin et al., 2019) has been effectively utilized in many Natural
Language Processing (NLP) tasks.
At its core, a trained model encodes words into vector space representations based on their positions and context in the text.
Following the success in NLP, there has been signiﬁcen recent progress in unsupervised image pretraining (He et al., 2020; Caron et al., 2020; Ayush et al., 2020).
Interestingly almost all of them are based on certain form of contrastive learning (Hadsell et al., 2006), which helps to construct unsupervised classiﬁcation objectives from continuous inputs such as images.
He et al.
(2020) proposes Momentum Contrast (MoCo) for unsupervised visual representation learning.
To increase the number of negative examples in contrastive training, they uses a queue of multiple mini-batches.
Similar strategy has been adopted in NLP (Gao et al., 2021).
To improve the encoding inconsistency between mini batches, they make the target image encoder parameterizes a moving average of the query image encoder.
In this work we are focusing on the pretraining of location encoder with a frozen image encoder.
Our approach is very memory efﬁcient (easily scaling up to 1024 batch size) and therefore avoid the need of multi-batch training.
Ayush et al.
(2020) combines a geo-location classiﬁcation loss and a contrastive image loss similar to MoCo during pre-training.
For geo-location classiﬁcation, a deep image network is trained to predict a coarse geo-location of where in the world the image might come from.
Since the geo-location is not part of the classiﬁer’s input, it is hard for the model to directly leverage it in the prediction process.
Zhai et al.
(2018) learn location representation from unlabeled image-location pairs for image localization.
They apply cross entropy loss to discretized location (or time), which cannot leverage the continuity of the approximated function.
One main difference and also one of our main contributions is our contrastive self-supervised loss, which avoids the need of discretization.
E XPERIMENT 5. T HE E FFECTIVENESS OF S PHERICAL L OCATION E NCODING
To test the effectiveness of Sphere2Vec
we ﬁrst conduct geo-aware image classiﬁcation experiments in a full supervised setting (without the self-supervised pretraining step) on seven large-scale real-world datasets including 5 (animal) species datasets, one POI dataset (YFCC), and one remote sensing dataset (fMoW).
Please refer to Mac Aodha et al.
(2019) and Christie et al.
(2018) for the dataset descriptions.
Table 1 compares the result of ﬁve variants of Sphere2Vec models against nine baseline models (see their detailed descriptions in Appendix 9.6.
We can see that the Sphere2Vec models outperform baselines on all seven datasets, and the variants with linear number of DFS terms works as well as or even better than sphereDFS .
Figure 2c-2e and 2h-2j show the predicted species distributions from different models for two fox species.
We can see that wrap and grid have overgeneralization issue in data sparse area (e.g., Africa) and the North Pole area.
The distributions produced by sphereC ` are more compact and generalizable.
To help understand the impact of Sphere2Vec , we conduct detailed analysis on iNat2017 and iNat2018.
Figure 4a shows the image locations in iNat2017 validation dataset.
We split this dataset into different latitude bands and compare the ∆ MRR between each model to grid .
Figure 4b and 4c show the number of samples and ∆ MRR in each band while Figure 4f shows that the contrast between these two variables for different models.
We can see that Sphere2Vec models have larger ∆ MRR at the North Pole region (Figure 4c).
Moreover, Sphere2Vec has advantages on bands with less data samples, e.g. φ P r´ 30 ˝ , ´ 20 ˝ q .
We also compare sphereC ` and grid at a quantized spatial resolution - in latitude-longitude cells(Figure 4d and 4e).
sphereC ` has more advantages in North
Pole and data sparse cells.
For more details please refer to Appendix 9.12, 9.13, and 9.14.
5. T HE E FFECTIVENESS OF S ELF -S UPERVISED P RETRAINING
Next, we investigate the impact of self-supervised contrastive pretraining (see Section 3.2).
For species recognition and POI image classiﬁcation task, we use the pretrained Inception-V3 model provided by PyTorch as image feature extractor F in the yellow box of Figure 1.
As for fMoW dataset, we use the unsupervised pretrained
MoCo-V2+TP
(Ayush et al., 2020) as F .
Both models are supervised ﬁne-tuned on the respective dataset to act as g p F p¨qq in the green box of Figure 1.
We consider three self-supervised training objectives - mean square error loss MSE between the location embedding e p x q and the image feature F p
I q , noise contrastive binary classiﬁcation loss BI (see Equation 6), and contrastive multi-class classiﬁcation loss MC (See Equation 7).
In order to
[Table]
71.95 74.78 50.76 70.28 rbf (Mai et al., 2020b) 71.
71.35 74.47 51.09 70.65 rff (Rahimi et al., 2007) 71.
74.71 50.67 70.27 grid (Mai et al., 2020b) 71.
72.98 74.96 51.18 70.81 theory (Mai et al., 2020b) 71.
sphereC 72.
73.29 75.33 51.34 71.00 sphereC ` 72.
73.31 75.44 51.28 71.03 sphereM 72.
73.25 75.28 51.35 70.98 sphereM ` 72.
73.72 75.57 51.24 71.10 sphereDFS 71.
[Figure]
Locations
[Figure]
simulate the situations with limited labeled data we ﬁrst perform self-supervised pretraining for e p x q with one of the three objectives using 100% of the unlabeled dataset X unlabeled (e.g., iNatlist 2018).
Then we perform stratiﬁed sampling to obtain Γ portion of the labeled data, denoted as X Γ , while keeping its label distribution the same as the full training set.
Finally we ﬁne-tune the pretrained e p x q on X Γ with the supervised learning objective (Equation 8).
[Figure]
Figure 5a compares the model performance of sphereM ` on iNat2018 with different training objectives L .
Each curve indicates the model performance of one training process in different supervised label ratios Γ with identical e p x q .
From Figure 5a, we can see that compared with Sup.
setting, all three self-supervised objectives can improve the model performances.
The largest improvements happen when Γ is small (i.e., few shot learning setting), and we get a 3.9% performance improvement when adding MC self-supervised training stage for Γ “ 0 . 05 .
BI and MC show competitive performance while MC is slightly better.
MSE is less effective than the two.
Furthermore, in order to show the impact of self-supervised training on different e p x q , we compare the performance of sphereM ` and theory (the 2nd best model on iNat2018 ) under different L (See Figure 5b).
We can see that MC can improve the performance of both e p x q while sphereM ` can outperform theory in either supervised or self-supervised + supervised settings.
We also perform a series of ablation studies on these self-supervised training objectives.
Please refer to Appendix 9.10 for detail analysis.
Moreover, we also compare MC and Sup.
on the fMoW dataset and ﬁnd that MC self-supervised loss is also effective in the label limited learning setting on the fMoW dataset (more details in Figure 8 and Appendix 9.11).
C ONCLUSION
In this work, we propose a multi-scale spherical location encoder - Sphere2Vec which can encode any location on the spherical surface into a high dimension vector.
Moreover, we propose an selfsupervised learning framework which pretrains location encoders based on unlabeled geo-tagged images.
Experiment results on seven large-scale geo-aware image classiﬁcation datasets shows that Sphere2Vec can outperform multiple strong baselines.
Moreover, we show that by adding the self-supervised pretraining stage, the performance of Sphere2Vec can be further improved, especially when labeled data is limited.
In the future, we would like to explore the effectiveness of Sphere2Vec on other machine learning tasks such as image geolocalization, geographic question answering, and so on.
Moreover, we only pretrain the location encoder while keeping the image encoder freeze during the self-supervised training stage.
In the future, we would like to explore ways to pretrain dual encoders at the same time or early fusion of image and location data.
E THICS S TATEMENT
All datasets utilized in this work are publicly available.
We only do some data preprocessing such as image feature extraction.
No new information is collected.
We ﬁnd out that the geographic coverage of most datasets used in this work such as BirdSnap, NABird, iNat2017, iNat2018, YFCC and fMoW are mostly concentrated on North American, Europe while they have poor geographic coverage on other regions and countries such as Russia, China, Africa, and so on (See Figure 4a).
This kind of geographic bias will also affect the machine learning models trained on them as shown in Figure 4d.
This paper shows the our Sphere2Vec is more robust than the state-of-the-art models for such bias.
However, how to systematically handle such geographic bias should be an interesting research direction.
R EPRODUCIBILITY S TATEMENT
The proposed models as well as preprocessed datasets in this paper will be made public available through GitHub.
A PPENDIX 9. T HE M AP P ROJECTION D ISTORTION P ROBLEM
In fact there is no map projection, which can preserve distances in all directions.
The so-called equidistant projection can only preserve distance on one direction, e.g., the longitude direction for the equirectangular projection (See Figure 6d), while the conformal map projections (See Figure 6a) can preserve directions while resulting in a large distance distortion.
For a comprehensive overview of map projections and their distortions, see Mulcahy & Clarke (2001).
This is a well recognized problem in Cartography which shows the importance of calculating on a round planet Chrisman (2017).
Due to the limitations of these 2D location encoders, there is an urgent need for a location encoding method which preserves the spherical distance (e.g., great circle distance 5 ) between two points .
The multi-scale encoding method utilizes Double Fourier Sphere basis ( O p S 2 q terms) or a subset ( O p S q terms) of it while still being able to correctly measure the spherical distance.
This inspires us to explore the most effective subset of Fourier bases on spheres.
9. SPHERE C, SPHERE C+, SPHERE M, SPHERE M+
sphereC
Inspired by the fact that any point p x, y, z q in 3D Cartesian coordinate can be expressed by sin and cos basis of spherical coordinates ( λ , φ plus radius) 6 , we deﬁne the basic form of Sphere2Vec , namely sphereC encoder, for scale s as
[Equation]
[Equation]
It can be shown that when S “ 1 , G p sphereC q p x q directly satisﬁes our expectation in Equation 1 where f p x q “ cos p x R q .
See more detailed analysis and comparison to the grid encoder in Appendix 9.3.
sphereM
Considering the fact that many geographical features are more sensitive to either latitude (e.g., temperature, sunshine duration) or longitude (e.g., timezones, geopolitical borderlines), we might want to focus on increasing the resolution of either φ or λ while the other is hold relatively at large scale.
Therefore, we introduce a multi-scale position encoder sphereM , where interaction terms between φ and λ always have one of them ﬁxed at top scale:
[Equation]
This new encoder ensures that the φ term interact with all the scales of λ terms and λ term interact with all the scales of φ terms.
Note that when S “ 1 , G p sphereM q is equivalent to G p sphereC q .
Both sphereC and sphereM are multi-scale versions of a spherical distance-kept encoder (See Equation 12) and keep that as the main term in their multi-scale representation.
sphereC+ and sphereM+ From the analysis of the two proposed encoders and the state-of-the-art grid encoders (Appendix 9.3), we know that grid pays more attention to the sum of cos difference of latitudes and longitudes, while our proposed encoders pay more attention to the spherical distances.
In order to capture both information, we consider merging grid with each proposed encoders to get more powerful models that encode geographical information from different angles.
[Equation]
9. A NALYSIS OF sphereC AND ITS COMPARISON TO THE grid ENCODER
To illustrate that sphereC is good at capturing spherical distance, we take a close look at its basic case S “ 1 (deﬁne s “ 0 and f s “ 1 ), where the multi-scale encoder degenerates to
[Equation]
These three terms are included in the multi-scale version ( S ą 1 ) and serve as the main terms at the largest scale and also the lowest frequency (when s “ S ´ 1 ).
The high frequency terms are added to help the downstream neuron network to learn the point-feature more efﬁciently.
Interestingly, p
G p sphereC q captures the spherical distance in a very explicit way:
Theorem 2.
Let x 1 , x 2 be two points on the same sphere with radius R , then
[Equation]
[Equation]
where ∆ D is the great circle distance between x 1 and x 2 .
Under this metric,
[Equation]
Moreover, } p G p sphereC q p
[Equation]
´
[Equation]
G
[Equation]
sphereC q p x
[Equation]
q} « ∆ DR ,when ∆ D is small w.r.t.
R .
See the proof in Appendix 9.4.
Since the central angle ∆ δ “ ∆ DR P r 0 , π s and cos p x q is strictly monotonically decrease for x P r 0 , π s , Theorem 2 shows that p G p sphereC q p x q directly satisﬁes our expectation in Equation 1 where f p x q “ cos p xR q .
In comparison, when S “ 1 , the inner product in the output space of grid encoder is
[Equation]
which models the latitude difference and longitude difference of x 1 and x 2 separately rather than spherical distance.
This introduces problems in encoding.
For instance, consider data pairs x 1 “ p λ 1 , φ q and x 2 “ p λ 2 , φ q , the distance between them in output space of grid , } p G p grid q p x 1 q ´ p
G p grid q p x 2 q} 2 “ 2 ´ 2 cos p λ 1 ´ λ 2 q stays as a constant in terms of φ .
However, when φ varies from ´ π 2 to π 2 , the actual spherical distance changes in a wide range, e.g., the actual distance between the data pair at φ “ ´ π 2 (South Pole) is 0 while the distance between the data pair at φ “ 0 (Equator), gets the maximum value.
This issue in measuring distances also has a negative impact on grid ’s ability to model distributions in areas with sparse sample points because it is hard to learn the true spherical distance.
We observe that grid reaches peak performance at much smaller r min than that of
Sphere2Vec encodings (See Appendix ?? ).
Moreover, sphereC outperforms grid near polar regions where grid claims large distance though the spherical distance is small (A, B in Figure 1).
9. P ROOF OF T HEOREM
Proof.
Since p G p sphereC q p x i q “ r sin p φ i q , cos p φ i q cos p λ i q , cos p φ i q sin p λ i qs for i “ 1 , 2 , the inner product
[Equation]
[Equation]
where ∆ δ is the central angle between x 1 and x 2 , and the spherical law of cosines is applied to derive the second last equality.
So,
[Equation]
So
[Equation]
since ∆ D { 2 R P r 0 ,
[Equation]
2 s .
By Taylor expansion, } p G p x
[Equation]
q
[Equation]
when ∆ D is small w.r.t.
R .
9. P ROOF OF T HEOREM
[Equation] [Equation]
[Equation] [Equation]
from s “ 0 terms.
Equation 18 implies φ 1 “ φ 2 .
If φ 1 “ φ 2 “ π { 2 , then both points are at North
Pole, λ 1 “ λ 2 equal to whatever longitude deﬁned at North Pole.
If φ 1 “ φ 2 “ ´ π { 2 , it is similar case at South Pole.
When φ 1 “ φ 2 P p´
[Equation]
[Equation]
q , cos p φ 1 q “ cos p φ 2 q ‰ 0 .
Then from Equation 19 and 20,
[Equation]
(21) which shows that λ 1 “ λ 2 .
In summary, x 1 “ x 2 , so G p˚q is injective.
If
[Equation]
“ G p˚q p x 2 q implies
[Equation]
which proves x 1 “ x 2 and G p˚q is injective directly.
9. B ASELINES
In order to understand the advantage of sphereical-distance-kept location encoders
[Equation]
we compare different versions of Sphere2Vec with multiple baselines:
• No Prior indicates a image classiﬁer without using any location information,
i
[Equation]
e., predicting image labels purely based on image information P p
y
| I q .
• tile divides the study area
A (e.g., the earth surface) into grids with equal intervals along the latitude and longitude direction.
Each grid has an embedding to be used as the encoding for every location x fall into this grid.
This is a common practice by many previous work when dealing with coordinate data (Berg et al., 2014; Adams et al., 2015; Tang et al., 2015).
• wrap is a location encoder model introduced by Mac Aodha et al.
Given a location x “ p λ, φ q , it uses a coordinate wrap mechanism to convert each dimension of x into 2 numbers
[Equation] [Equation]
˝ qs .
Then the results are passed through a multi-layered fully connected neural network NN p wrap q pq which consists of an initial fully connected layer, followed by a series of h residual blocks, each consisting of two fully connected layers ( k hidden neurons) with a dropout layer in between.
We adopt the ofﬁcial code of Mac Aodha et al.
(2019) 7 for this implementation.
• wrap ` ffn is similar to wrap except that it replaces NN p wrap q pq with NN ffn pq , a simple learnable multi-layer perceptron with h hidden layers and k neurons per layer as that Sphere2Vec has.
wrap ` ffn is used to exclude the effect of different NN pq on the performance of location encoders.
In the following, all location encoder baselines use NN ffn pq as the learnable neural network component so that we can directly compare the effect of different position encoding G p¨q on the model performance.
• xyz ﬁrst converts x i “ p λ i , φ i q P S 2 into 3D Cartesian coordinates p x, y, z q centered at the sphere center by following Equation 23 before feeding into a multilayer perceptron NN pq .
Here, we let p x, y, z q to locate on a unit sphere with radius R “ 1 .
xyz can be treated as a special case of sphereC where S “ 1 .
[Equation]
• rbf randomly samples M points from the training dataset as RBF anchor points { x anchorm , m “ 1 ...M }, and use gaussian kernels exp
[Equation]
´ x anchorm (cid:107)
[Equation]
on each anchor points, where σ is the kernel size.
Each point p i has a M -dimension RBF feature vector which is fed into NN ffn pq to obtain the location embedding.
This is a strong baseline for representing ﬂoating number features in machine learning models.
• rff , i.e., Random Fourier Features (Rahimi & Recht, 2008; Nguyen et al., 2017), ﬁrst encodes location x into a D dimension vector -
[Equation]
where ω i i.i.d „ N p 0 , δ 2 I q and b i is uniformly sampled from r 0 , 2 π s .
I is an identity matrix.
Each component of ϕ p x q ﬁrst projects x into a random direction ω i and makes a shift by b i .
Then it wraps this line onto the unit cirle in R 2 with the cosine function.
Rahimi & Recht (2008) show that ϕ p x q T ϕ p x 1 q is an unbiased estimate of the Gaussian kernal K p x , x 1 q .
ϕ p x q is consist of D different estimates to produce a further lower variance approximation.
To make rff comparable to other baselines, we feed ϕ p x q into NN ffn pq to produce the ﬁnal location embedding.
• grid is a multi-scale location encoder on 2D Euclidean space proposed by Mai et al.
(2020b).
Here, we simply treat x “ p λ, φ q as 2D coordinate.
It ﬁrst use G p grid q p x q shown in Equation 3 to encode location x into multi-scale representation and then feed it into NN ffn pq to produce the ﬁnal location embedding.
• theory is another multi-scale location encoder on 2D Euclidean space proposed by Mai et al.
(2020b).
It use a position encoder G p theory q p x q shown in Equation 24..
Here, x s “ r λ s , φ s s “ r λf s , φf s s and a 1 “ r 1 , 0 s T , a 2 “ r´ 1 { 2 , ?
3 { 2 s T , a 3 “ r´ 1 { 2 , ´?
3 { 2 s T P
Rare three unit vectors which orient 2 π { 3 apart from each other.
The encoding results are feei .
nto NN ffn pq to produce the ﬁnal location embedding.
[Equation] [Equation]
9. M ODEL I MPLEMENTATION D ETAIL
All types of Sphere2Vec as well as all baseline models we compared share the same model set up - e p x q p x q “ NN ffn p G p˚q p x qq .
Here, G p˚q p¨q is a position encoder which preprocesses the input coordinate x “ p λ, φ q P S 2 and outputs a position embedding G p˚q p x q .
There are no learnable parameter in G p˚q p x q .
The position encoders G p˚q p¨q used for different Sphere2Vec are discussed in detail in Section 3.1 while G p˚q p¨q of all baselines are discussed in Appendix 9.6.
We can see that G p˚q p¨q of grid , theory and different types of Sphere2Vec encode the input coordinates in a multi-scale fashion by using different sinusoidal functions with different frequencies.
Many previous work call this practice “Fourier input mapping” (Rahaman et al., 2019b; Tancik et al., 2020;
Basri et al., 2020;
Anokhin et al., 2021).
The difference is that grid and theory use the Fourier features from 2D Euclidean space while our Sphere2Vec uses all or the subset of Double Fourier Sphere
Features to take into account the sphereical geometry and the distance distortion it brings.
Except for wrap , NN ffn p¨q used by all the other location encoders is a learnable multi-layer perceptron (MLP) with h hidden layers and k neurons per layer.
We use the original MLP implementation provided by Mai et al.
(2020b).
Here, each layer of NN ffn pq is a linear layer followed by a dropout layer, a nonlinear activation function, a layer normalization layer (Ba et al., 2016), and a skip connection.
Different from other location encoders, wrap uses a different implementation of the full connected layer NN wrapffn p¨q .
Please refer to Appendix 9.6 and Mac Aodha et al.
for the detailed implementation of NN wrapffn p¨q .
All models are implemented in PyTorch.
We use the original implementation of wrap from
Mac Aodha et al.
and the implementation of grid and theory from Mai et al.
(2020b).
We train/evaluate each model on a Ubuntu machine with 2 GeForce GTX Nvidia
GPU cores, each of which has 10GB memory.
9. M ODEL T RAINING D ETAIL AND H YPERPARAMETER T UNING
To train different types of Sphere2Vec as well as baseline models, we use Adam as the optimizer in both unsupervised and supervised training phase.
Basically, we ﬁrst do unsupervised training with an initial learning rate η unsuper by following either Equation 6 or 7 as the unsupervised training objective.
In this setting, we predict the corresponding image embedding based on the input location.
Only location encoder is trained during the unsupervised training phase while we keep the image encoder freezed as shown in Figure 1.
Here, we just use a image encoder pretrained on other image datasets such as ImageNet and use it as a image feature extractor F .
After the model is convergence, we use a new learning rate η super to do supervised training by following Equation 8.
9.8. F ULL S UPERVISED T RAINING S ETTING
In order to test the effectiveness of our spherical encoding method – Sphere2Vec , we train all location encoders in a fully supervised setting.
In order to ﬁnd the best hyperparameter combinations for each model on each dataset, we use grid search to do hyperparameter tuning including supervised training learning rate η super “ r 0 .
00005 s , the number of scales
S “ r 16 , 32 , 64 s , the minimum scaling factor r min “ r 0 .
0001 s , the number of hidden layers and number of neurons used in NN ffn p¨q – h “ r 1 , 2 , 3 , 4 s and k “ r 256 , 512 , 1024 s , the dropout rate in NN ffn p¨q – dropout “ r 0 .
7 s .
We also test multiple options for the nonlinear function used for NN ffn p¨q including ReLU, LeakyReLU, and Sigmoid.
The maximum scaling factor r max can be determined based on the range of latitude φ and longitude λ .
For grid and theory , we use r max “ 360 and for all Sphere2Vec , we use r max “ 1 .
As for rbf and rff , we also tune their hyperparamaters including kernel size σ “ r 0 . 5 , 1 , 2 , 10 s as well as the number of kernels M “ r 100 , 200 , 500 s .
Based on hyperparameter tuning, we ﬁnd out using 0.5 as the dropout rate and ReLU as the nonlinear activation function for NN ffn p¨q works best for every location encoders.
Based on the hyperparameter tuning, the best hyperparameter combinations are selected for different models on different datasets.
The best results are reported in Table 1.
Note that each model has been running fortimes and their mean Top1 score is reported.
Due to the limit of space, the standard deviation of the evaluation result of each model on each dataset is not included in Table 1.
However, we report the standard deviations of all models’ performance on three datasets in Appendix 9.9.
Moreover, we ﬁnd out η super and r min are the most important hyperparameters.
Table 2 shows the best hyperparameter combinations of different Sphere2Vec models on different species image classiﬁcation dataset.
We use a smaller S for sphereDFS since it has O p S 2 q terms while the other models have O p S q terms.
sphereDFS with S “ 8 yield a similar number of terms to the other models wth S “ 32 (See Table 3).
Interestingly, all ﬁrst four Sphere2Vec models ( sphereC , sphereC ` , sphereM , and sphereM ` ) shows the best performance on all ﬁve datasets with the same hyperparamter combinations.
Note that compared with other datasets, iNat2017 and iNat are more up-to-date datasets with more training samples and better geographic coverage.
This indicates that the proposed 4 Sphere2Vec models show similar performance over different hyperparameter combinations.
[Table] [Table]
We also ﬁnd out that using a deeper MLP as NN ffn p¨q
[Table]
i.e.
[Table]
a larger h does not necessarily lead to better classiﬁcation accuracy.
In many cases, one hidden layer – h “ 1 achieve the best performance for many kinds of location encoders.
We discuss this in detail in Appendix 9.9.
9.8. U NSUPERVISED T RAINING S ETTING
In order to test the effectiveness of different unsupervised learning objectives
[Table]
we also conduct another set of experiments by unsupervised training different location encoders before supervised training phase.
In addition to the hyperparameter we discussed in Section 9.8.1, we also ﬁne tune the hyperparameters used in the unsupervised learning phase including the unsupervised training learning rate η unsuper , the negative location loss α 1 , SimCSE loss weight α 2 , the number of negative locations we used | N s | , three temperature τ 0 , τ 1 , and τ 2 .
The hyperparameters we have searched and the hyperparameter tuning results for unsupervised learning setting are shown in Table 5, 6, 7 in
Appedix 9.10.
9. M ORE E XPERIMENTAL R ESULTS ON G EO -A WARE I MAGE C LASSIFICATION T ASKS
Table 4 is a complementary of Table 1 which compares the performance of sphereM ` with all baseline models on the geo-aware image classiﬁcation task.
The results on three datasets are shown here including BirdSnap † , NABirds † , and iNat2018.
For each model, we vary the depth of its
NN ffn pq , i.e., h “ r 1 , 2 , 3 , 4 s .
The best evaluation results with each h are reported.
Moreover, we run each model with one speciﬁc h 5 times and report the standard deviation of the Top1 accuray, indicated in “()”.
Several important observations can be made based on Table 4: 1.
Although the absolute performance improvement between sphereM ` and the best baseline model is not very large – 0.90%, 0.28%, and 0.71% for three datasets respectively, these performance improvements
are statistically signiﬁcant
given the standard deviations of
these Top1 scores
These performance improvements are comparable to those from the previous studies on the same tasks.
In other words, the small margin is due to the nature of these datasets.
For example, Mai et al.
(2020b) showed that grid or theory have 0.79%, 0.44% absolute
Top1 accuracy improvement on BirdSnap † and NABirds † dataset respectively.
Mac Aodha et al.
(2019) showed that wrap has 0.09%, 0%, 0.04% absolute Top1 accuracy improvement on BirdSnap, BirdSnap † and NABirds † dataset.
Here, we only consider the results of wrap that only uses location information, but not temporal information.
Although Mac Aodha et al.
(2019) showed that compared with tile and nearest neighbor methods, wrap has 3.19% and 3.71% performance improvement on iNat2017 and iNat2018 datatset, this large margin is mainly because the baselines they used are rather weak.
When we consider the typical rbf and rff (Rahimi et al., 2007) used in this study, their performance improvement are down to -0.02% and 0.61%.
[Table]
sphereM
[Table]
By comparing the performances of the same model with different depth of its NN ffn pq , i.e., h , we can see that the model performance is not sensitive to h .
In fact, in most cases, one layer NN ffn pq achieve the best result.
This indicates that the depth of the
MLP does not signiﬁcantly affect the model performance and a deeper MLP does not necessarily lead to a better performance .
In other words, the systematic bias (i.e., distance distortion) introduced by grid and theory can not later be compensated by a deep MLP.
It shows the importance of designing a spherical-distance-aware location encoder.
9. A BLATION S TUDY ON D IFFERENT S ELF -S UPERVISED L OSS
Both the BI (Equation 6) and MC (Equation 7) unsupervised loss have three loss component: inbatch, negative location, SimCSE loss component.
How does each of them contribute to the overall unsupervised training?
In order to answering this question, we do ablation studies on these two
[Figure]
objectives by using iNat2018 dataset as an example.
Figure 7a and 7b illustrate the ablation study results.
From Figure 7a, we see that as for BI , adding L negloc BI will signiﬁcantly increase the model performance especially when Γ is large.
Adding L simcse BI also improves the model performance when Γ is small.
As for the ablation study on MC loss as shown in Figure 7b, adding L negloc MC can signiﬁcantly increase the model performance and adding L simcse MC can also make the performance slightly better.
9. E FFECTIVENESS OF U NSUPERVISED P RETRAINING ON F M O W DATASET
9. I MPACT OF MRR BY T HE N UMBER OF S AMPLES AT D IFFERENT L ATITUDE B ANDS
See Figure
[Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Figure] [Figure]
9. P REDICTED D ISTRIBUTIONS I N AT
We plot the predicted species distributions from different models at different geographic regions, and compare them with the training sample locations of the corresponding species, see Figure 10.
We can see that compared with wrap ˚ and grid , in each geographic region with sparse training samples and the North Pole area, the spatial distributions produced by sphereC ` are more compact while the other two have over-generalization issue.
9. E MBEDDING C LUSTERING
We use the location encoder trained on iNat2017 or iNat2018 dataset to produce a location embedding for the center of each small latitude-longitude cell.
Then we do agglomerative clustering 8 on all these embeddings to produce a clustering map.
Figure 11 and 12 show the clustering results for different models with different hyperparameters on iNat2017 and iNat2018 dataset.
[Figure]
[Figure]
9. E XPERIMENTS WITH S YNTHETIC D ATASET 9.15. S YNTHETIC D ATASET G ENERATION
To further investigate the effectiveness of our proposed spherical location encoder Sphere2Vec
[Figure]
we construct a set of synthetic datasets
[Figure] We utilize the von Mises–Fisher distribution ( vMF ) (Izbicki et al., 2019), an analogy of the 2D
Gaussian distribution on the spherical surface S 2 to generate synthetic data points 9 .
The probability density function of vMF is deﬁned as
[Equation]
where χ p x q “ r x, y, z s “ r cos φ cos λ, cos φ sin λ, sin φ s (25)
Here, χ p x q converts x into a coordinates in the 3D Euclidean space, and on the surface of a unit sphere.
A vMF distribution is controlled by two parameters – the mean direction µ P R 3 and concentration parameter κ P R ` .
µ indicates the center of a vMF distribution which is a 3D unit vector.
κ is a positive real number which controls the concentration of vMF .
A higher κ indicates more compact vMF distribution, while κ “ 1 correspond to a vMF distribution with standard deviation covering half of the unit sphere.
To simulate multi-modal distributions, we assume C classes with even prior, and each classes follows a vMF distribution.
To create a dataset we ﬁrst sample C sets of parameters tp µ i , κ i qu ( C “ 50 ).
Then we draw SP samples for each class ( SP “ 100 ).
So in total, each generated synthetic dataset has 5000 data points for 50 balanced classes.
[Figure]
[Figure] [Figure]
[Figure]
The concentration parameter κ i is sampled by ﬁrst drawing r from an uniform distribtuion
U p κ min , κ max q , and then take the square r 2 .
The square helps to avoid sampling many large κ i which yield very concentrated vMF distributions that are rather easy to be classiﬁed.
We ﬁx κ min “ 1 and vary κ max in r 16 , 32 , 64 , 128 s .
For the center parameter µ i we adopt two sampling approaches:
Uniform Sampling : We uniformly sample C centers ( µ i ) from the surface of a unit sphere.
We generate four synthetic datasets (for different values of κ max ) and indicate them as U1, U2, U3, U4.
See Table 8 for the parameters we use to generate these datasets.
Stratiﬁed Sampling :
We ﬁrst evenly divide the latitude range r´ π { 2 , π { 2 s into N µ intervals.
Then we uniformly sample C µ centers ( µ i ) from the spherical surface deﬁned by each latitude interval.
Since the latitude intervals in polar regions have smaller spherical surface area, this stratiﬁed sampling method has higher density in the polar regions.
We keep N µ ˆ C µ “ C “ 50 ﬁxed and varies N µ in r 5 , 10 , 25 , 50 s .
Combined with the 4 κ max choices, this procedure yields 16 different synthetic datasets.
We denote them as Si
j .
See Table 8 for the parameters we use to generate these datasets
Figure 13a-13d visualize the data point distributions of these four synthetic datasets based on the uniform sampling method in 2D space.
Figure 13e visualized the U4 dataset in a 3D Euclidean space.
We can see that when κ max is larger, the variation of point density among different vMF distributions becomes larger.
Some vMF are very concentrated and the resulting data points are easier to be classiﬁed.
Moreover, if we treat these datasets as 2D data points as grid and theory do, vMF distributions in the polar areas will be stretched to very extended shapes making model learning more difﬁcult.
However, this kind of systematic bias can be avoided if we use a spherical location encoder as Sphere2Vec .
[Figure]
Figure 14 visualizes the data distributions of four synthetic datasets with stratiﬁed sampling method.
They have different N µ but the same κ max .
We can see that when N µ increases, a more ﬁne-grain stratiﬁed sampling is carried out.
The resulting dataset has a larger data bias toward the polar areas.
9.15. S YNTHETIC D ATASET E VALUATION R ESULTS
We evaluate all baseline models as well as sphereM ` on those generated 20 syhthetic datasets as described above.
For each model, we do grid search on their hyperparameters for each dataset including supervised learning rate η super , the number of scales S , the minimum scaling factor r min , the number of hidden layers and number of neurons used in NN ffn p¨q – h and k .
The best performance of each model is reported in Table 8.
Some observations can be made from Table 8:
[Figure]
1. sphereM ` is able to outperform all baselines on all 20 synthetic datasets.
The absolute Top1 improvement can go up to 2% and the error rate deduction can go up to 30 .
This shows the robustness of sphereM ` . 2.
When the dataset is fairly easy to classify (i
e., all baseline models can produce 95+% Top accuracy), sphereM ` is still able to further improve the performance and gives a very large error rate reduction (up to 30.8%).
This indicates that sphereM ` is very robust and reliable for datasets with different distribution characteristics.
Comparing the error rate of different stratiﬁed sampling generated datasets (S1.j - S4.j) we can see that when we keep κ max ﬁxed and increase N µ , the relative error reduction ER become larger.
Increasing N µ means we do a more ﬁne-grain stratiﬁed sampling.
The resulting datasets should sample more vMF
distributions
in
the
polar
regions
This
phenomenon shows that
when
the dataset
has a
larger
data
bias
towards
the
polar
area, we expect
sphereM `
will
be
more effective
[Table]