A N I NDUCTIVE B IAS FOR D ISTANCES : N EURAL N ETS THAT R ESPECT THE T RIANGLE I NEQUALITY
A BSTRACT
Distances are pervasive in machine learning.
They serve as similarity measures, loss functions, and learning targets; it is said that a good distance measure solves a task.
When deﬁning distances, the triangle inequality has proven to be a useful constraint, both theoretically—to prove convergence and optimality guarantees— and empirically—as an inductive bias.
Deep metric learning architectures that respect the triangle inequality rely, almost exclusively, on Euclidean distance in the latent space.
Though effective, this fails to model two broad classes of subadditive distances, common in graphs and reinforcement learning: asymmetric metrics and metrics that cannot be embedded into Euclidean space.
To address these problems, we introduce novel architectures that are guaranteed to satisfy the triangle inequality.
We prove our architectures universally approximate norm-induced metrics on R n , and present a similar result for modiﬁed Input Convex Neural Networks.
We show that our architectures outperform existing metric approaches when modeling graph distances and have a better inductive bias than non-metric approaches when training data is limited in the multi-goal reinforcement learning setting
I NTRODUCTION
Many machine learning tasks involve a distance measure over the input domain.
A good measure can make a once hard task easy, even trivial.
In many cases—including graph distances, certain clustering algorithms, and general value functions in reinforcement learning (RL)—it is either known that distances satisfy the triangle inequality, or required for purposes of theoretical guarantees; e.g., speed and loss guarantees in k -nearest neighbors and clustering (Cover and Hart, 1967; Indyk, 1999;
Davidson and Ravi, 2009), or optimality guarantees for A ∗ search (Russell and Norvig, 2016).
This also makes the triangle inequality a potentially useful inductive bias for learning distances.
For these reasons, numerous papers have studied different ways to learn distances that satisfy the triangle inequality (Xing et al., 2003;
Yang and Jin, 2006; Brickell et al., 2008; Kulis et al., 2013).
The usual approach to enforcing the triangle inequality in deep metric learning (Yi et al., 2014;
Hoffer and Ailon, 2015; Wang et al., 2018) is to use a Siamese network (Bromley et al., 1994) that computes a Euclidean distance in the latent space.
Speciﬁcally, the Siamese network models distance d X : X × X
→ R + on domain X by learning embedding φ : X → R n and computing d X ( x, y ) as (cid:107) φ ( x ) − φ ( y ) (cid:107) 2 .
Successful applications include collaborative ﬁltering (Hsieh et al., 2017), few-shot learning (Snell et al., 2017), and multi-goal reinforcement learning (Schaul et al., 2015).
The use of
Euclidean distance, however, has at least two downsides.
First, the Euclidean architecture cannot represent asymmetric metrics, which arise naturally in directed graphs and reinforcement learning.
Second, it is well known that for some metric spaces ( X , d X ) , including large classes of symmetric graphs (e.g., constant-degree expanders and k -regular graphs), there is no embedding φ : X → R n that can model d X precisely using (cid:107) · (cid:107)
(Indyk et al., 2017).
A classic example is shown in Figure 1.
In part due to these issues, some have considered non-architectural constraints.
He et al.
(2016) impose a triangle inequality constraint in RL via an online, algorithmic penalty.
Implementing such a penalty can be expensive, and does not provide any guarantees.
An approach that does guarantee satisfaction of triangle inequality is to ﬁx any violations after learning, as done by Brickell et al.
But this does not scale to large problems or provide an inductive bias during learning.
Cleaned up version
Blah
Blah Blah
Norm
MSE
Euclidean, R n , ∀ n 0.
Deep Norm, R0.
Wide Norm, R0.
Deep Norm
Wide Norm
Mahalanobis  1 1.5 1.
A B
C D 1
A
B
C
D  1 1.0 0.
A B
C
D
Is it possible to impose the triangle inequality architecturally, without the downsides of Euclidean distance?
In response to this question, we present the following contributions: (1) three novel neural network architectures, Deep Norms, Wide Norms and Neural Metrics, which model symmetric and asymmetric norms and metrics, (2) universal approximation theorems for Deep Norms and Wide Norms and modiﬁed Input Convex Neural Networks (Amos et al., 2017), and (3) empirical evaluations of our models on several tasks: modeling norms, metric nearness, modeling shortest path lengths, and learning a general value function (Sutton et al., 2011).
Our models are guaranteed to satisfy the triangle inequality, straightforward to implement, and may be used in place of the usual Euclidean metric should one seek to model asymmetry or increase expressiveness.
M ODELING N ORMS 2. P RELIMINARIES
Our goal is to construct expressive models of metrics and quasi-metrics on domain X .
A metric is a function d : X × X
→ R + satisfying, ∀ x, y, z ∈ X :
M1 (Non-negativity) .
d ( x, y ) ≥ 0 .
M2 (Deﬁniteness) .
d ( x, y ) =
⇐⇒ x = y .
M3 (Subadditivity) .
d ( x, z ) ≤ d ( x, y ) + d ( y, z ) .
M4 (Symmetry) .
d ( x, y ) =
d ( y, x ) .
Since we care mostly about the triangle inequality ( M3 ), we relax other axioms and deﬁne a quasimetric as a function that is M1 and M3 , but not necessarily M2 or M4 .
Given weighted graph
G =
( V , E ) with non-negative weights, shortest path lengths deﬁne a quasi-metric between vertices.
When X is a vector space (we assume over R ), many common metrics, e.g., Euclidean and Manhattan distances, are induced by a norm.
A norm is a function (cid:107)·(cid:107) :
X → R satisfying, ∀ x, y ∈ X , α ∈ R + :
N1 (Pos.
def.) .
(cid:107) x (cid:107) > 0 , unless x = 0 .
N2 (Pos. homo.) .
α (cid:107) x (cid:107) =
(cid:107) αx (cid:107) , for α ≥ 0 .
N3 (Subadditivity) .
(cid:107) x
+ y (cid:107) ≤ (cid:107) x (cid:107)
+ (cid:107) y (cid:107) .
N4 (Symmetry) .
(cid:107) x (cid:107) =
(cid:107) – x (cid:107) .
An asymmetric norm is N1-N3 , but not necessarily N4 .
An (asymmetric ) semi-norm is nonnegative, N2 and N3 (and N4 ), but not necessarily N1 .
We will use the fact that any asymmetric semi-norm (cid:107) · (cid:107) induces a quasi-metric using the rule, d ( x, y )
= (cid:107) x − y (cid:107) , and ﬁrst construct models of asymmetric semi-norms.
Any induced quasi-metric d is translation invariant— d ( x, y )
=
d ( x + z, y + z ) —and positive homogeneous— d ( αx, αy ) =
αd ( x, y ) for α ≥ 0 .
If (cid:107) · (cid:107) is symmetric ( N4 ), so is d ( M4 ).
If (cid:107) · (cid:107) is N1 , d is M2 .
Metrics that are not translation invariant (e.g., Bi et al.
(2015)) or positive homogeneous (e.g., our Neural Metrics in Section 3) cannot be induced by a norm.
A convex function f : X → R is a function satisfying C1: ∀ x, y ∈ X , α ∈ [0 , 1] : f ( αx +(1 − α ) y ) ≤ αf ( x ) + (1 − α )
f ( y ) .
The commonly used ReLU activation, relu( x ) = max(0 , x ) , is convex.
2. D EEP N ORMS
It is easy to see that any N2 and N3 function is convex—thus, all asymmetric semi-norms are convex.
This motivates modeling norms as constrained convex functions, using the following proposition.
Proposition 1.
All positive homogeneous convex functions are subadditive; i.e., C1 ∧ N2 ⇒ N3 .
[Figure]
The proof is straightforward (put α = 12 in C1 and apply N2 to the left side).
To use Proposition 1, we begin with the Input Convex Neural Network (ICNN) (Amos et al., 2017) architecture, which satisﬁes C1 , and further constrain it to be non-negative and satisfy N2 .
The resulting Deep Norm architecture is guaranteed to be an asymmetric semi-norm.
A k -layer Deep Norm is deﬁned as:
[Equation]
with
[Equation]
for i = 1 ... k , where x is the input, h [-
-]0 = 0 , W +1 = 0 , the activation functions g i preserve C1 and N (element-wise), g k is non-negative, W
+ i is a non-negative matrix, and U i is an unconstrained matrix.
As compared to the original ICNN architecture, we have omitted the bias terms from Equation 1 , have constrained the g i to preserve positive homogeneity while also allowing them to be any function that preserves element-wise convexity (this is essential to our universal approximation results), and have required g k to be non-negative.
It is easy to verify that the set of valid element-wise g
i is { g αβ ( x ) =
α relu( x ) + βx | α, β ≥ 0 } .
This includes ReLUs and leaky ReLUs.
But we do not restrict ourselves to element-wise activations.
Inspired by GroupSort (Anil et al., 2018), we use activations that depend on multiple inputs (and preserve element-wise C1 and N2 ).
In particular, we use the pairwise MaxReLU :
[Equation]
Deep Norms are N2 and N3 .
Using the following propositions, we may also impose N1 and N4 .
Proposition 2.
If (cid:107) · | is an asymmetric semi-norm, then (cid:107) x (cid:107)
= (cid:107) x
| + (cid:107) – x | is a semi-norm.
Proposition 3.
If (cid:107) · (cid:107) a is an (asymmetric) semi-norm, (cid:107) · (cid:107) b is a norm (e.g., (cid:107) · (cid:107) b
= (cid:107) · (cid:107) 2 ), and λ > 0 , then (cid:107) x (cid:107) a + λb = (cid:107) x (cid:107) a + λ (cid:107) x (cid:107) b is an (asymmetric) norm.
2. W IDE N ORMS
In addition to Deep Norms, we propose the following alternative method for constructing norms: a
Wide Norm is any combination of (asymmetric) (semi-) norms that preserves N1-N4 .
It is easy to verify that both (1) non-negative sums and (2) max are valid combinations (indeed, these properties were also used to construct Deep Norms), and so the vector-wise MaxMean combination is valid:
[Equation]
Although the family of Wide Norms is broad, for computational reasons to be discussed in Subsection 3.5, we focus our attention on the Wide Mahalanobis norm.
References to “Wide Norms” in the rest of this paper refer to Wide Mahalanobis norms.
The Mahalanobis norm of x ∈ R n , parameterized by W ∈ R m × n , is deﬁned as (cid:107) x (cid:107) W = (cid:107) Wx (cid:107) 2 .
It is easily veriﬁed that (cid:107) · (cid:107) W is a proper norm when
W is a non-singular (square) matrix, and a semi-norm when W is singular or m < n .
A k -component Mixture of Mahalanobis norm (hereafter Wide Norm , or Wide Norm with k Euclidean components) is deﬁned as the maxmean of k Mahalanobis norms:
[Equation]
where W i ∈
R m i [Equation] n with m i ≤ n
[Equation]
Wide Norms are symmetric by default, and must be asymmetrized to obtain asymmetric (semi-) norms.
We use the below property (Bauer et al., 1961) and propositions (proofs in Appendix A).
N5.
(cid:107) · (cid:107) is monotonic in the positive orthant if 0 ≤ x ≤ y (element-wise) implies (cid:107) x (cid:107) ≤ (cid:107) y (cid:107) .
Proposition 4.
If (cid:107) · (cid:107) is an N5 (semi-) norm on R 2 n , then (cid:107) x
| = (cid:107) relu
x :: – x ) (cid:107) , where :: denotes concatenation, is an asymmetric (semi-) norm on R n .
Proposition 5.
The Mahalanobis norm with W = DU , with D diagonal and U non-negative, is N5 .
2. U NIVERSAL A PPROXIMATION OF C ONVEX F UNCTIONS AND N ORMS
How expressive are Deep Norms and Wide Norms?
Although it was empirically shown that ICNNs have “substantial representation power” (Amos et al., 2017), this was never theoretically characterized.
One of our key contributions is a series of universal approximation results for ICNNs (with MaxReLU activations), Deep Norms and Wide Norms.
The next lemma is central to our results.
Lemma 1 (Semilattice Stone-Weierstrass (from below)) .
Let C be a set of continuous functions deﬁned on compact subset K of R n , L be a closed subset of C , and f ∈ C .
If (1) for every x ∈ K , there exists g x ∈ L such that g x ≤ f and g x ( x ) =
f
( x ) , and (2) L is closed under max (i.e., a, b ∈ L ⇒ max( a, b ) ∈ L ), then ∃ h ∈ L with f = h on K .
Intuitively, Lemma 1 and its proof (Appendix A) say we can approximate continuous f arbitrarily well with a family L of functions that is closed under maximums if we can “wrap” f from below using functions g x ∈ L with g x ≤ f .
Our Universal Approximation (UA) results are now straightforward.
Theorem 1 (UA for MICNNs) .
The family M of Max Input Convex Neural Networks (MICNNs) that uses pairwise max-pooling (or MaxReLU) activations is dense in the family C of convex functions.
Proof.
For f ∈ C , x ∈ R n , let g x ∈ M be a linear function whose hyperplane in the graph of f is tangent to f at x .
Then g x satisﬁes condition (1) of Lemma 1 (because f is convex).
The use of pairwise max activations allows one to construct max( h 1 , h 2 ) ∈ M for any two h 1 , h 2 ∈ M by using log 2 ( n ) max-pooling layers, satisfying condition (2) of Lemma 1.
Thus f is in the closure of
M , and the result follows.
This result applies when MaxReLUs are used, since we can set α, β = 0 .
Using MaxReLU (rather than max ) guarantees that MICNNs can imitate regular ICNNs with at most double the parameters.
Theorem 2 (UA for Deep Norms and Wide Norms) .
The families D of Deep Norms (using MaxReLU) and W of Wide Norms (using MaxMean) are dense in the family N of asymmetric semi-norms.
Proof (sketch).
The proof is almost identical to that of Theorem 1, except that here D and W contain all linear functions whose graph is tangent to any f ∈ N since f is N2 .
This is easy to see for functions deﬁned on R 2 .
See Appendix A for more details.
2. M ODELING N ORMS IN 2D
Having shown that Deep Norms and Wide Norms universally approximate norms, we now show that they can successfully learn to approximate random norms on R 2 when trained using gradient descent.
To generate data, we use the below fact, proved in Appendix A.
Proposition 6.
The set of all asymmetric norms on R n is in one-to-one correspondence with the set of all bounded and open convex sets (“unit balls”) containing the origin.
We use Proposition 6 by generating a random point set, computing its convex hull, and using the hull as the unit ball of the generated norm, (cid:107) · (cid:107) .
We then train different models to approximate (cid:107) · (cid:107) using | D | ∈ { 16 , 128 } training samples of form (( xη, yη ) , η ) , where (cid:107) ( x, y ) (cid:107) =
and η ∼ U (0 . 85 , 1 . 15) .
The models are trained to minimize the mean squared error (MSE) between the predicted (scalar) norm value and the ground truth norm value.
See Appendix C.1 for details.
Figure 3 illustrates the learned 2D norm balls of a random symmetric (top) and asymmetric (bottom) norm, for three architectures: Deep Norm, Wide Norm, and an unconstrained, fully connected neural network (MLP).
The blue contours in Figure 3 are the ground truth norm balls for values { 0 . 5 , 1 , 1 . 5 } , and black contours the norm balls of the learned approximations.
With the small dataset, the MLP [Figure] [Table] overﬁts to the training data and generalizes poorly, while Deep Norm and Wide Norm generalize almost perfectly.
With the large dataset, we observe that Deep Norms and Wide Norms generalize to larger and smaller norm balls (due to being N2 ), whereas the MLP is unable to generalize to the 0.
norm ball (indeed, MLP − 1 (0 . 5) = ∅ ).
While the symmetric Wide Norm ﬁts well, the results suggest that the asymmetrized Wide Norm is not as effective as the naturally asymmetric Deep Norm.
See
Appendix C for additional details and visualizations.
M ODELING M ETRICS
Having constructed models of asymmetric semi-norms on R n , we now use them to induce quasimetrics on a domain X .
As the geometry of X ’s raw feature space will not, in general, be amenable to a favorable metric (indeed, the raw features need not be in R n ), we assume the Siamese network approach and learn an embedding function, φ : X → R n .
A Deep Norm or Wide Norm (cid:107) · (cid:107) θ , with parameters θ , is deﬁned on R n and the metric over X is induced as d φ,θ ( x, y )
= (cid:107) φ ( y ) − φ ( x ) (cid:107) θ .
We could also deﬁne (cid:107) · (cid:107) θ using an unconstrained neural network (MLP), but this would not be guaranteed to satisfy the norm axioms and induce a quasi-metric.
To illustrate this approach, we revisit Figure 1.
To get the results shown, we represent the four nodes as one hot vectors and embed them into R 2 using φ ( x ) =
Wx , W ∈ R 2 × 4 .
We then deﬁne the norm on R 2 as either a Mahalanobis norm, a Deep Norm, or a Wide Norm.
Training the norm and φ together, end-to-end with gradient descent, produces the Figure 1 results.
The choices are summarized in Table 1.
While Deep Norm and Wide Norm have similar properties, the depth of the former suggests that they can learn efﬁcient, hierarchical representations (as do deep neural networks); however, Wide Norms have a computational advantage when computing pairwise distances for large minibatches, which we explore in Subsection 3.5.
Since inducing metrics with
Deep Norms or Wide Norms produces a potentially unnecessary homogeneity constraint (due to N2 ), the next Subsection considers Neural metrics, which offer an approach to relaxing this constraint.
3. N EURAL M ETRICS
Instead of inducing a metric with a norm, we could deﬁne a metric directly as a non-negative weighted sum of different metrics.
E.g., as we did for Wide Norms, we can deﬁne a Wide Metric as the mean of k deep Euclidean metrics.
If all components of a Wide Metric are norm-induced, however, it can be induced directly by a single Wide Norm with k components, by setting φ ( x ) to be the concatenation of the φ i ( x ) , and using each W j to select the indices corresponding to φ j .
It follows that for a family of Wide Metrics to be more expressive than the family of norm-induced metrics, it must include components that are either not translation invariant or not positive homogeneous.
We consider the latter, and use the propositions below to modify our norm-induced metrics (proofs in Appendix A).
Proposition 7 (Metric-preserving concave functions) .
If d : X × X → R + is (quasi-) metric, f : R + → R + , f − 1 (0)
= { 0 } , and f is concave (i.e., − f is convex), then f ◦ d is (quasi-) metric.
Proposition 8 (Max preserves metrics) .
If d 1 and d 2 are (quasi-) metric, so too is max( d 1 , d 2 ) .
To use Proposition 7, we note that each unit in the ﬁnal layer of a Deep Norm deﬁnes a metric (assuming g k − 1 is non-negative, as it is when using ReLU or MaxReLU activations), as does each component of a Wide Norm.
Thus, by applying metric-preserving functions f i to these metrics, and then combining them using a MaxMean, mean or max, we obtain a valid metric, which we name a Neural Metric .
Our k -component f
i are parameterized by w i , b i ∈ R
k
as follows:
f i ( x ) =
min j { w ij x + b ij } , where w ij ≥ 0 , b ij ≥ 0 , b 0 = 0 .
The advantage of Neural Metrics over plain Deep Norms and Wide Norms is that one can better model certain metrics, such as d ( x, y )
= min(1 , (cid:107)
x − y (cid:107) ) .
This type of metric might be induced by shortest path lengths in a fully connected graph with some maximum edge weight (for example, a navigation problem with a teleportation device that takes some constant time to operate).
3. A PPLICATION : M ETRIC N EARNESS
[Table]
We now apply our models to the metric nearness problem (Sra et al., 2005; Brickell et al., 2008).
In this problem, we are given a matrix of pairwise nonmetric distances between (ﬁnite) n objects, and it is desired to minimally repair the data to satisfy metric properties (the triangle inequality, M3 , in particular).
Formally, given data matrix D ∈ R n × n , we seek metric solution X ∈ R n × n that minimizes the “distortion” J MN =
[Equation]
(normalized distance) to the original metric.
Sra et al.
(2005) proved this loss function attains its unique global minimum in the set of size n discrete metrics and proposed an O ( n 3 ) triangle ﬁxing (“TF”)
algorithm for the symmetric case, which iteratively ﬁxes M3 violations and is guaranteed to converge to the global minimum.
Table 2 shows that for n = 200 , our models (Deep Norm (DN) and Wide Norm (WN) based Neural Metrics) achieve results comparable to that found by 400 iterations of TF in the symmetric case.
We note that TF, which approaches the solution through the space of all n × n matrices, produces a non-metric approximation, so that the number of M3 violations ( # ¬ M3 ) is greater than 0; this said, it is possible to ﬁx these violations by adding a small constant to all entries of the matrix, which does not appreciably increase the loss J MN .
To test our models in the asymmetric case, we modiﬁed the TF algorithm; although our modiﬁed TF found a solution , our DN model performed signiﬁcantly better.
See Appendix D for complete experimental details.
Although triangle ﬁxing is effective for small n , there is no obvious way to scale it to large datasets or to metric approximations of non-metric distance functions deﬁned on a continuous domain.
Using our models in these settings is natural, as we demonstrate in the next subsection.
3. A PPLICATION : M ODELING G RAPH D ISTANCES
In the previous subsection we sought to “ﬁx” a noisy, but small and fully observable metric; we now test the generalization ability of our models using large ( n > 100 K) metrics.
We do this on the task of modeling shortest path lengths in a weighted graph G =
( V , E ) .
So long as edge weights are positive and the graph is connected, shortest path lengths are discrete quasi-metrics ( n = |V| ), and provide an ideal domain for a comparison to the standard Euclidean approach.
Our experiments are structured as a supervised learning task, with inputs x, y ∈ V and targets d ( x, y ) .
The targets for a random subset of 150 K pairs (of the O ( | V | 2 ) total pairs) were computed beforehand |V| |E| max( d ) σ d Sym? to 278K 611K 145. ↔ 3d 125K 375K 86.
↔ taxi 391K 752K 111. ↔ push 390K 1498K 113. → 3dr 123K 368K 86. → 3dd 125K 375K 97.
→ (a) Graph statistics
Eucl.
WN
DN I
DN N
MLP to 12.
12.3 3d 31.
20.6 taxi 14.
5.8 push 22.
11.3 3dr 22.
25.5 3dd 211.
(b) Final test MSE @
| D | =
[Figure]
using A ∗ search and normalized to have mean 50.
10 K were used as a test set, and the remainder was subsampled to form training sets of size | D | ∈ { 1 K , 2 K , 5 K , 10 K , 20 K , 50 K } .
Nodes were represented using noisy landmark embeddings (see Appendix E).
We compare Wide Norms, Deep
Norms (ICNN style, with ReLU activations, DN I ), and Deep Norm based Neural Metrics (with
MaxReLU and concave activations, DN N ) to the standard Siamese-style Bromley et al.
deep
Euclidean metric and MLPs in six graphs, three symmetric and three not, summarized in Table 3a and described in Appendix E.
Though MLPs do not induce proper metrics, they are expressive function approximators and serve as a relevant reference point.
The results (shown for | D | = 50 K in Table 3b and expanded upon in Appendix E) show that our models tend to outperform the basic Siamese (Euclidean/Mahalanobis) network, and oftentimes the MLP reference point as well.
3. A PPLICATION : L EARNING G ENERAL V ALUE F UNCTIONS
In this section, we evaluate our models on the task of learning a Universal Value Function Approximator (UVFA) in goal-oriented reinforcement learning (Sutton et al., 2011; Schaul et al., 2015).
Figure 4 (left) illustrates the two 11x11 grid world environments, 4-Room and Maze , in which we conduct our experiments.
Each environment has a symmetric version, where the reward is constant (-1), and an asymmetric version with state-action dependent reward.
We create a training set of transitions ( s, s (cid:48) , g, r, d ) for the state, next state, goal, reward, and done ﬂag, and the UVFA V θ
( s, g ) is trained to minimize the temporal difference error via bootstrapped updates with no discounting.
We tested several architectures at different training set sizes, which leaves out a portion of transitions such that only a fraction of states or goals were present during training.
See Appendix F.1 for details.
[Figure] Figure 5 summarizes the ﬁnal generalization performance (on held out ( s, g ) pairs) for each architecture after training for 1000 epochs.
Performance is given in terms of the SPL metric (Anderson et al., 2018), which measures the success rate of reaching the goal weighted by the ratio of agent versus optimal path cost.
We observe that when given only partial training data, Deep Norm and
Wide Norm consistently outperform both MLPs and ICNNs at each sparsity level.
As expected, the
Euclidean metric could not solve the asymmetric environments.
Qualitatively, we plot a heatmap of the squared difference (SE) between the learned and ground truth value function in Figure(center), and observed that our proposed architectures have mostly lower SE than the MLP and ICNN architectures.
Figure 4 (right) illustrates that WideNorm architecture is able to reach more cells (with almost optimal trajectory) in the environment when using a greedy policy with respect to the learned value function.
Additional visualizations and results are in Appendix F.2.
3. C OMPUTATIONAL C ONSIDERATIONS
[Figure]
[Figure]
Several deep metric learning algorithms, such as semihard triplet mining Schroff et al. (2015), involve computing a pairwise distance matrix for large minibatches.
For example, Schroff et al.
use mini-batches of size 1800.
Computing this matrix for Euclidean distances can be done efﬁciently by taking advantage of the identity (cid:107) x
− y (cid:107)
= (cid:107) x (cid:107)
+ (cid:107) y (cid:107)
− 2 x T y
(see
Section 4 of Oh Song et al.
(2016) for details).
This same identity can be applied to each component of a Wide Norm.
There is no obvious way, however, to compute pairwise Deep Norms more efﬁciently than the naive O ( n 2 ) approach.
To quantify, we recorded the pairwise distance computation time for our implementations (Table 6).
Thus, although our previous experiments often found that Deep Norms performed slightly better (see, e.g., Figureand Tables 2 and 3b), these results suggest that only Wide Norms are practical for large mini-batches.
D ISCUSSION AND O THER R ELATED W ORK
Deep Norms, Wide Norms, and Neural Metrics all respect the triangle inequality while universally approximating ﬁnite-dimensional asymmetric semi-norms.
This allows them to represent metrics that the deep Euclidean Siamese architecture cannot, no matter how deep its embedding function is (see Figure 1).
Our models thus provide a more expressive, non-Euclidean alternative for learning distances that satisfy the triangle inequality.
As noted in the Introduction, this may useful for providing running time and error rate guarantees in clustering (Davidson and Ravi, 2009) and as an inductive bias to improve generalization performance (Figure 5; Hsieh et al.
A line of theoretical work, surveyed by Indyk et al.
(2017), characterizes the representational capacity of Euclidean space (and, more generally, (cid:96) p space) by examining the asymptotic “distortion” of embedding n -point metric spaces into Euclidean space.
Here the distortion of an embedding φ : X → X (cid:48) of metric space ( X , d X ) into metric space ( X (cid:48) , d X (cid:48) ) is at most c ≥ 1 if there exists r > 0 such that for all x, y ∈ X , r · d X ( x, y ) ≤ d X (cid:48)
( φ ( x ) , φ ( y )) ≤ cr · d X ( x, y ) .
The pioneering work of Bourgain (1985) bounds worst case distortion by O (log n ) , and Linial et al.
(1995) shows that this bound is tight (i.e., Θ(log n ) ) for graph distances in n -point constant-degree expanders.
This applies regardless of the dimensionality of the embedding space, and is true for all (cid:96) p with 1 ≤ p < ∞ .
Future work might investigate the asymptotic distortion of our proposed architectures.
On account of the above limitations, others have also proposed non-Euclidean alternatives to learning and representing metrics.
To improve expressivity, some have proposed non-parametric algorithms that learn distances directly.
For instance, Biswas and Jacobs (2015) propose to frame clustering as a metric nearness problem and applying a quadratic programming algorithm; see also Gilbert and
Jain (2017) and Veldt et al.
Others learn parametric distances, as we do.
Bi et al.
(2015) parameterize and learn Cayley-Klein metrics, which are not translation-invariant.
Yang and Jin (2006) and Nielsen et al.
(2016) propose metrics that are similar to our Wide Norm, in that they use non-negative sums of several components (binary codes and Cayley-Klein metrics).
As for symmetry, several papers have used asymmetric measures such as KL divergence (Vilnis and McCallum, 2014;
Vendrov et al., 2015; Chen et al., 2016; Ou et al., 2016).
To our knowledge, we are the ﬁrst to propose a parametric measure that is both asymmetric and satisﬁes the triangle inequality.
Another common approach in deep “metric” learning is to forgo the triangle inequality altogether and use non-metric similarity measures such as cosine similarity (Bromley et al., 1994;
Yi et al., 2014).
This can be sensible, as proper metrics are not always required.
In image recognition, for example, the triangle inequality is questionable: why should d ( C AT , W OLF ) ≤ d ( C AT , D OG )
+ d ( D OG , W OLF ) ?
In this case, Scheirer et al.
ﬁnd that non-metric similarities often perform better, concluding that “good recognition is non-metric”.
Thus, one should apply our models with care, in settings where the triangle inequality is thought to be a useful inductive bias (e.g., Hsieh et al.
Such applications we are excited about include learning search heuristics (Russell and Norvig, 2016) and scaling our UVFA models to more complex reinforcement learning problems (Plappert et al., 2018).
C ONCLUSION
This paper proposed three novel architectures for modeling asymmetric semi-norms and metrics.
They can be used instead of the usual Euclidean metric to increase expressiveness while respecting the triangle inequality.
We showed that our models outperform Euclidean metrics when used with a
Siamese-style deep metric learning architecture to (1) solve the metric nearness problem, (2) model shortest path lengths, and (3) learn general value functions in small reinforcement learning domains.
Future work should explore larger scale applications such as facial recognition (Schroff et al., 2015), collaborative metric learning (Hsieh et al., 2017) and continuous control (Plappert et al., 2018). A CKNOWLEDGMENTS
A P ROOFS
In this Appendix, we restate each proposition from the main text and provide a short proof.
Proposition 1.
All positive homogeneous convex functions are subadditive; i.e., C1 ∧ N2 ⇒ N3 .
Proof.
Putting α = 0 .
5 in the deﬁnition of C1 gives f (0 .
5 x +0 .
5 y ) ≤ 0 .
5 f ( x )+0 .
5 f ( y ) .
Applying
N2 on the left and multiplying by 2 gives f ( x + y ) ≤ f ( x ) + f ( y ) as desired.
Proposition 2.
If (cid:107) · | is an asymmetric semi-norm, then (cid:107) x (cid:107)
= (cid:107) x
| + (cid:107) – x | is a semi-norm.
Proof.
(cid:107) x (cid:107) =
(cid:107) x
| + (cid:107) – x | = (cid:107) – x (cid:107) , so N4 is satisﬁed.
N2 - N3 and non-negativity are similarly trivial.
Proposition 3.
If (cid:107) · (cid:107) a is an (asymmetric) semi-norm, (cid:107) · (cid:107) b is a norm (e.g., (cid:107) · (cid:107) b
= (cid:107) · (cid:107) 2 ), and λ > 0 , then (cid:107) x (cid:107) a + λb = (cid:107) x (cid:107) a + λ (cid:107) x (cid:107) b is an (asymmetric) norm.
Proof.
This follows from the positive deﬁniteness of λ (cid:107) x (cid:107) b and the easily veriﬁed fact that nonnegative weighted sums of semi-norms are semi-norms.
Proposition 4.
If (cid:107) · (cid:107) is an N5 (semi-) norm on R 2 n , then (cid:107) x
| = (cid:107) relu
x :: – x ) (cid:107) , where :: denotes concatenation, is an asymmetric (semi-) norm on R n .
Proof. N1 and N2 are easily veriﬁed.
To see that (cid:107) · | is not necessarily symmetric, choose (cid:107) · (cid:107) to be a Mahalanobis norm parameterized by a diagonal W with W ii
=
i (but NB that if (cid:107) · (cid:107) is invariant to element-wise permutations, as are the L p norms, (cid:107) · | will be symmetric).
For N3 , we have:
[Equation]
The ﬁrst inequality holds because (cid:107)·(cid:107) is N3 .
The second holds because (cid:107)·(cid:107) is N5 , and we have either element-wise equality (when sign( x i ) and sign( y i ) agree) or domination (when they don’t).
Proposition 5.
The Mahalanobis norm with W = DU , with D diagonal and U non-negative, is N5 .
Proof.
Let (cid:107) · (cid:107) be (cid:107) · (cid:107) 2 , and 0 ≤ x ≤ y = x
+ (cid:15) .
We have:
[Equation]
The ﬁrst inequality is Cauchy-Schwarz.
The second holds as all elements of x , U T D T DU , and (cid:15) are non-negative.
Lemma 1 (Semilattice Stone-Weierstrass (from below)) .
Let C be a set of continuous functions deﬁned on compact subset K of R n , L be a closed subset of C , and f ∈ C .
If (1) for every x ∈ K , there exists g x ∈ L such that g x ≤ f and g x ( x ) =
f
( x ) , and (2) L is closed under max (i.e., a, b ∈ L ⇒ max( a, b ) ∈ L ), then ∃ h ∈ L with f = h on K .
Proof.
Fix (cid:15) > 0 and consider the sets U x = { y | y ∈ K, f ( y ) − (cid:15) <
g x ( y ) } , for all x ∈ K .
The U x form an open cover of K , so there is a ﬁnite subcover { U x 1 , U x 2 , . . .
, U x n } .
Let g = max
( g x 1 , g x 2 , . .
g x n ) ∈ L .
We have f − (cid:15) ≤ g ≤ f , and the result follows since L is closed.
Theorem 1 (UA for MICNNs) .
The family M of Max Input Convex Neural Networks (MICNNs) that uses pairwise max-pooling (or MaxReLU) activations is dense in the family C of convex functions.
Proof.
For f ∈ C , x ∈ R n , let g x ∈ M be a linear function whose hyperplane in the graph of f is tangent to f at x .
Then g x satisﬁes condition (1) of Lemma 1 (because f is convex).
The use of pairwise max activations allows one to construct max( h 1 , h 2 ) ∈ M for any two h 1 , h 2 ∈ M by using log 2 ( n ) max-pooling layers, satisfying condition (2) of Lemma 1.
Thus f is in the closure of
M , and the result follows.
Theorem 2 (UA for Deep Norms and Wide Norms) .
The families D of Deep Norms (using MaxReLU) and W of Wide Norms (using MaxMean) are dense in the family N of asymmetric semi-norms.
Proof.
The proof is almost identical to that of Theorem 1.
The only subtlety is that D and W do not contain all linear functions in R n ; they do, however, contain all linear functions whose hyperplane is tangent to any f ∈ N , since f is N2 .
This is easy to see for functions deﬁned on R 2 .
To generalize the intuition to R n , consider the ray R x
= { ( αx :: α (cid:107) x (cid:107) ) | α ≥ 0 } ⊂ R n +1 deﬁned for each x ∈ R n .
By N2 , this ray is a subset of the graph g ⊂ R n +1 of f .
Furthermore, any hyperplane tangent to one point on this ray is tangent to the entire ray and contains all points on the ray, since the ray is linear from the origin—therefore the hyperplane contains the origin.
But any hyperplane tangent to g at x is tangent to a point ( x ) on the ray R x , and so contains the origin.
Since D and W contain all linear functions containing the origin, it follows that they contain all linear functions whose graph is tangent to f , in satisfaction of condition (1) of Lemma 1 (because f is convex).
For Deep Norms, the use of pairwise max activations allows one to construct a global max operation, as in the proof of Theorem 1, satisfying condition (2) of Lemma 1.
Wide Norms using MaxMean have direct access to a global max, and so satisfy condition (2) of Lemma 1.
It follows that f is in the closures of D and W .
Proposition 6.
The set of all asymmetric norms on R n is in one-to-one correspondence with the set of all bounded and open convex sets (“unit balls”) containing the origin.
Proof.
Given asymmetric norm (cid:107) · (cid:107) on R n , its unit ball B 1 :=
{ x | (cid:107) x (cid:107) < 1 } is convex since, ∀ x, y ∈ B 1 , λ ∈ [0 , 1] , we have (cid:107) (1 − λ ) x
+ λy (cid:107) ≤ (1 − λ ) (cid:107) x (cid:107) + λ (cid:107) y
(cid:107) ≤ 1 (using N3 & N2 ).
Conversely, given open and bounded convex set B 1 containing the origin, let (cid:107) x (cid:107) =
inf { α > 0 | xα ∈
B 1 } .
N1 and N2 are straightforward and N3 follows by noting that for x, y ∈ R n , α, β > 0 , such that xα , yβ ∈ B 1 , we have αα + β xα + βα + β yβ = x
+ y α + β ∈ B 1 , so that inf { γ > 0 | x + yγ ∈ B 1 } ≤ inf { α > 0 | xα ∈ B 1 } + inf { β > 0 | yβ ∈ B 1 } .
Proposition 7 (Metric-preserving concave functions) .
If d : X × X → R + is (quasi-) metric, f : R + → R + , f − 1 (0)
= { 0 } , and f is concave (i.e., − f is convex), then f ◦ d is (quasi-) metric.
Proof.
This proposition is proven for metrics by Doboš (1998)
(Chapter 1, Theorem 3).
The extension to quasi-metrics is immediate, as the proof does not require symmetry.
Proposition 8 (Max preserves metrics) .
If d 1 and d 2 are (quasi-) metric, so too is max( d 1 , d 2 ) .
Proof.
M1 and M2 are trivial.
For M3 , let d = max( d 1 , d 2 ) .
Given some x, y, z , we have both d 1 ( x, y )+ d 1 ( y, z ) ≥ d 1 ( x, z ) and d 2 ( x, y )+ d 2 ( y, z ) ≥ d 2 ( x, z ) , so that d ( x, y )+ d ( y, z ) ≥ d 1 ( x, z ) and d ( x, y )
+ d ( y, z ) ≥ d 2 ( x, z ) .
Therefore, d ( x, y )+
d ( y, z ) ≥ max( d 1 ( x, z ) , d 2 ( x, z ))
=
d ( x, z ) .
Because if an element is larger than two other elements, it is also larger than their max.
B I MPLEMENTATION D ETAILS
Except where otherwise noted, these implementation details are common throughout our experiments.
Our Deep Norm implementation constrains each W i for i < k to be non-negative by clipping the parameter matrix after each gradient update.
For W k , we use either a simple mean or MaxMean (see
Subsection 2.3).
We set U k = 0 .
We use either ReLU or MaxReLU for our activations g i , as noted.
Since the output layer is always scalar (size 1), we refer to Deep Norms in terms of their hidden layers only.
Thus, a Deep Norm with k = 3 layers of sizes (400, 400, 1) is a “2x400” Deep Norm.
Our Wide Norm implementation avoids parameterizing the α i in Equation 3 by absorbing them into the weight matrices, so that (cid:107) x (cid:107) = 1 k (cid:80) i (cid:107) U
i x (cid:107) 2 ,
where U
i = α i kW i .
Our asymmetric Wide Norm constrains matrix U of Proposition 5 by clipping negative values and does not use matrix D (i.e., we set D
= I ).
Neither our Deep Norm nor Wide Norm implementations impose positive deﬁniteness.
This simpliﬁes our architectures, does not sacriﬁce representational power, and allows them to be used on pseudometric problems (where d ( x, y )
= 0 is possible for distinct x, y ∈ X ).
Neural Metrics involve only a very small modiﬁcation to Deep Norms and Wide Norms: before applying the mean or MaxMean global activation to obtain the ﬁnal output, we apply an element-wise concave activation function, as described in Subsection 3.1.
C 2 D N ORMS A DDITIONAL D ETAILS
Dataset generation To generate data, we use Proposition 6 by generating a random point set, computing its convex hull, and using the hull as the unit ball of the generated norm, (cid:107) · (cid:107) .
Having obtained the unit ball for a random norm, we sample a set of N = 500 unit vectors in R 2 (according to L 2 ) in random directions, then scale the vectors until they intersect the convex hull: { x } N .
We use these vectors for testing data, deﬁned as D test = { ( x (1) , 1) , ..., ( x ( N ) , 1) } .
For training data, we sample a size | D | = { 16 , 128 } subset of these vectors and multiply them by random perturbations (cid:15) ( i ) ∼ U (0 . 85 , 1 . 15) to get { ˆ x | ˆ x ( i ) =
x ( i ) (cid:15) ( i ) } k .
The training data is then deﬁned as
D train =
{ (ˆ x (1) , (cid:15) (1) ) , ..., (ˆ x ( k ) , (cid:15) ( k ) ) } .
Models We compare 4 model types on 4 types of norms.
The models are (1) Mahalanobis, (2)
Deep
Norms of depth ∈ { 2 , 3 , 4 , 5 } and layer size ∈ { 10 , 50 , 250 } , (3) Wide Norms of width ∈ { 2 , 10 , 50 } and number of components ∈ { 2 , 10 , 50 } , and (4) unconstrained, fully connected neural networks with ReLU activations of depth ∈ { 2 , 3 , 4 , 5 } and layer size ∈ { 10 , 50 , 250 } (MLPs).
Although MLPs do not generally satisfy N1-N4 , we are interested in how they generalize without the architectural inductive bias.
Norms
The norms we experiment on are random (1) symmetric and (2) asymmetric norms, constructed as above, and (3) square ( L ∞ ), and (4) diamond ( L 1 ) norms.
Training The models, parameterized by θ , are trained to minimize the mean squared error (MSE) between the predicted (scalar) norm value (cid:107) ˆ x ( i ) (cid:107) and the label norm value (cid:15) ( i ) :
L ( θ ) = 1 M (cid:80) Mi
( (cid:107) ˆ x ( i ) (cid:107) θ − (cid:15) ( i ) ) 2 , where M = 16 is the batch size.
We use Adam Optimizer Kingma and Ba (2014), with learning rate 1e-3 , and train for a maximum of 5000 epochs.
Results Tables 5-7 show the test MSE for the best conﬁgurations for each target norm and training data size.
Deep Norms performed orders of magnitude better than Mahalanobis and Wide Norms on the random symmetric and asymmetric norms, but Wide Norms performed better on the square and diamond norms.
The MLP performed worse than our models, except on the asymmetric norm with small training data.
Figure 3 illustrates the learned norm balls for random symmetric and asymmetric norms, when trained with small ( k = 16 ) and large ( k = 128 ) data sizes.
Appendix C includes additional visualizations.
From the red contours, we observe that Deep Norms and Wide Norms generalize to larger and smaller norm balls (due to being N2 ), whereas the MLP is unable to generalize to the 0.5 norm ball.
C. A DDITIONAL D ETAILS ON N ORM G ENERATION
To generate point sets in R 2 , we generated c ∈ [3 , 10] clusters, each with n i ∈ [5 , 50] random points.
The points for each cluster were sampled from a truncated normal distribution with µ i ∼ U ( − 0 .
5 , 0 . 5) and σ ( i ) ∼ U (0 .
2 , 0 . 6) , truncated to 2 standard deviations.
For symmetric sets, we considered only those points with positive x -coordinates and also included their reﬂection about the origin.
To ensure that the origin was inside the resulting convex hull, we normalized the points to have mean zero before computing the hull.
The convex hull was generated by using the SciPy library Jones et al.
(2001) ConvexHull implementation.
[Table]
[Table]
[Table]
C. A DDITIONAL V ISUALIZATIONS
Figure 8 and Figure 7 further visualizes the training data and the Mahalanobis architecture in addition to the MLP, Deep Norm, and Wide Norm architectures, for the random symmetric, asymmetric, square ( L ∞ ) and the diamond ( L 1 ) unit ball shape
[Figure]
a) Small Dataset ( k = 16 )
[Figure]
[Figure]
D M ETRIC N EARNESS A DDITIONAL D ETAILS
Dataset creation For the symmetric metric nearness dataset, we generated the data as was found in
Sra et al.
(2005): a random matrix in R 200 × 200 was generated with values drawn from the uniform distribution ranging between 0 and 5.
This matrix was added to its transpose to make it symmetric.
Random uniform noise was added to each entry between 0 and 1, then the diagonal was removed.
For the asymmetric case, a more complex dataset creation strategy was employed due to the fact that a random asymmetric matrix generated as above was too difﬁcult a task for triangle ﬁxing.
Instead, we generated a random directed lattice graph with random weights generated from the exponential function applied to a random uniform sample between -1 and 1.
The distances were calculated using
Dijkstra’s A ∗ algorithm.
Then, again, random uniform noise between 0 and 4 was added to the output, multiples of 10 removed to make the scale comparable to before, and the diagonals removed.
The reason for the extra noise was due to the fact that the data matrix was already much closer to a metric.
Architectures used For the symmetric metric nearness experiment, the Deep Norm based Neural
Metric architecture used two layers of 512 neurons (we found that larger layer sizes learned much faster), with a 512-dimensional embedding function, MaxReLU activations, 5-unit concave activation functions, and a MaxMean global pooling layer.
The Wide Norm based Neural Metric consisted of 128 Mahalanobis components of size 48, with a 512-dimensional embedding function, 5-unit concave activation functions, and MaxMean global pooling.
The deep Euclidean architecture used a 1024-dimensional embedding function.
For the asymmetric case, the architectures were the same except that we used the asymmetric equivalents for Deep Norm and Wide Norm.
Training Regime
The triangle ﬁxing algorithm was allowed to go up to 400 iterations or convergence.
For the symmetric case we reimplemented Sra et al.
in Python using their C++ code as a template.
For the asymmetric version, we used the same code but we removed their symmetrization step.
For the training of all networks, we did 1500 epochs in total split up in to 500 epoch chunks were the learning rate decreased from 1e-3 to 3e-4 to 1e-4 at the end, using a batch size of 1000.
We used the Adam optimizer Kingma and Ba (2014) with default hyperparameters.
Results were averaged over 10 seeds.
E G RAPH E XPERIMENTS A DDITIONAL D ETAILS
Symmetric graph descriptions
The ﬁrst symmetric graph, to , consists of a symmetrized road network extracted from openStreetMap (OSM, www.openstreetmap.org).
The second, 3d , represents navigation in a 50x50x50 cubic 3d gridworld, where the agent can move one step at a time indirections, and movement wraps around the sides of the cube.
The third, taxi , represents a 25x 2d taxi environment, where the agent can move in four directions and there is a passenger that the agent can pick up and drop.
Unlike in 3d , there is no wraparound in taxi .
Weights for edges in to correspond to the OSM distances, whereas weights for edges in 3d and taxi are randomly sampled from { 0 . 01 , 0 .
(but ﬁnal distances are normalized so that the mean distance is 50).
Asymmetric graph descriptions
The ﬁrst, push , is a 25x25 2d pusher environment, where the agent can move in four directions, and there is a box that the agent can push by moving into it.
If the box is pushed into a wall, it switches places with the agent.
The second, 3dd , is a directed version of 3d , where all paths in three of the six movement directions have been pruned (the same three directions for all nodes).
The third, 3dr , is a randomly pruned version of 3d , where three random movement directions were pruned at each node, and any inaccessible portions of the resulting graph were pruned.
Our experiments are structured as a supervised learning task, with inputs x, y ∈ V and targets d ( x, y ) .
The targets for a random subset of 150 K pairs (of the O ( | V | 2 ) total pairs) were computed beforehand using A ∗ search and normalized to have mean 50.
10 K were used as a test set, and the remainder was subsampled to form training sets of size | D | ∈ { 1 K , 2 K , 5 K , 10 K , 20 K , 50 K } .
Nodes were represented using noisy landmark embeddings.
A subset of 32 landmark nodes was randomly chosen, and the distances to and from all other nodes in the graph were computed using Dijkstra’s algorithm to form 32 base landmark features for symmetric graphs and 32 base landmark features for asymmetric graphs.
These base landmark features were then normalized to have mean 0 and standard deviation 1, and noise sampled from N (0 , 0 . 2) was added.
To add additional noise through distractor features, 96 normally distributed features were concatenated with the noisy landmark features to obtain node embeddings with 128 dimensions for symmetric graphs and 160 dimensions for asymmetric graphs.
We also tested node2vec embeddings
Grover and Leskovec (2016), but found that our noisy landmark approach was both faster to run and produced results that were an order of magnitude better for all algorithms.
Architectures We compare a 128-dimensional Mahalanobis metric (equivalent to a deep Euclidean metric with an additional layer), a Wide Norm based Neural Metric (32 components of 32 units, with 5-unit concave activations , and MaxMean global pooling , a plain
Deep Norm layers of 128 units ICNN style with ReLU activations, no concave activations, and average pooling, DN I ), and a Deep
Norm based Neural Metric (3 layers of 128 units, with MaxReLU and 5-unit concave activations, and MaxMean pooling, DN N ) and an MLP (3 layers of 128 units).
We train each algorithm withdifferent embedding functions φ , each a fully connected, feed-forward neural network with ReLU activations
The depths of tested φ ranged from 0 to 3 layers, all with 128 units
No regularization was used besides the size /depth of the layers.
Training Training was done end-to-end with Adam Optimizer Kingma and Ba (2014), using an initial learning rate of 1e-3 and a batch size of 256.
Networks were each trained for 1000 total epochs, and the learning rate was divided by 5 every 250 epochs.
Results The complete results/learning curves are displayed in Figures 10 and 11 below.
F G ENERAL V ALUE F UNCTIONS A DDITIONAL D ETAILS F. E XPERIMENTAL D ETAILS
4-Room and Maze environment description The fully observable grid environment has a state and goal representation of a binary tensor with dimensions 11 × 11 × 3 .
Each cell in the 2D grid is represented by a 1-hot vector with 3 dimensions, indicating whether the cell is (1) empty, (2) wall, (3) agent.
The agent can move up to 4 cardinal directions (North, South, East, West).
If a wall is present in the direction then the agent cannot take that move.
The 4-Room environment and Maze environment has a total of 4556 and 2450 training ( s, g ) pairs, respectively.
For each environment, the agent has access to a neighbour function N ( s ) which returns a list of possible next states, corresponding to the empty cells adjacent to the current agent location.
This is equivalent to having environment transition model p ( s (cid:48) | s, a ) over all the actions.
Transition reward and episode termination For symmetric environments, every transition has a reward of -1, i.e., R ( s, s (cid:48) )
= − 1 .
Note that reward is not a function of the goal, but the termination is.
For asymmetric environments, we add a noise µ ∼ U (0 , − 5) to the base − 1 reward for a transition if the direction of movement is west (left) or south (down).
The done ﬂag is set to 1 when the neighbouring state equals to the goal, i.e. D ( s, s (cid:48) , g ) =
[ s (cid:48)
=
g ] .
Choosing Training-Test Split Dataset Given the full training data set of ( s, g ) pairs, we usetypes of splitting into train and test datasets: (1) goal, and (2) state.
We denote the training fraction η = ∈ [0 , 1] as the fraction of the total data points used for the training set.
For goal splitting, we pick a subset of goals { g ∗ } ⊆
g with |{ g ∗ }| ≈ (1 − η ) |{ g }| , and remove all ( s, g ) pairs where g ∈ g ∗ , from the training set.
These goals were chosen to be the percentage of empty cells from the bottom of the grid, when numbering the empty cells from left to right, and top-down (i.e. reading order of words in a page).
This corresponds to approximately a fraction of the bottom rows of goals.
In this case, the training set observes all possible states, and hence also all the transition rewards.
We are interested in whether it can generalize to new unseen goals.
For state splitting, we perform a similar procedure but instead with a subset of states which are removed, with the states corresponding to the agent being at the bottom rows of the grid.
While all the goals are included in the training dataset, in the state splitting, some states and transition rewards are unknown to the agent, hence much more difﬁcult to generalize.
Architecture The architectures used for the experiments consist of a shared feature extractor φ for the state s and goal g , followed by a function f θ on the difference of the features:
[Equation]
The feature extractor φ is composed of 2 convolutional layers with 3 × 3 kernel size, and 32 and 62 ﬁlters with ReLU activations and a stride of 1.
We then ﬂatten the feature maps and follow by 2 fully connected layers of hidden size 128 and 64, with ReLU activation on the ﬁrst layer only.
The variants of f θ are summarized in table 8.
Euclidean simply computes the L2 norm between the feature embeddings: (cid:107) φ ( g ) − φ ( s ) (cid:107) 2 :
[Table]
Training The objective function for training the models is the Temporal Difference (TD) Error
L ( φ, θ ) :
[Equation]
otherwise
[Equation]
Where N ( s ) refers to the set of next states of s after applying different actions at that state.
Note that we did not apply a discount factor on the value of the next state (i.e. the discount factor γ = 1 ).
We make use of target networks ¯ φ, ¯ θ when computing the target y .
The target networks are updated once every epoch of training, via exponential moving average (Polyak Averaging) with the main networks ( φ, θ ):
[Equation]
(7) where, α = 0 .
95 is the update fraction.
We use Adam Kingma and Ba (2014) with learning rate 0.0001, batch size 128, for 1000 epoch.
We evaluate SPL metric every 200 epoch.
Evaluation Metrics We evaluate the learned value functions on several metrics: MSE, Policy
Success, and SPL Anderson et al.
The MSE was calculated on on the ground truth value of the held out test set V ( s, g ) .
For policy success and SPL, we select N = 100 random ( s, g ) pairs in the test set, initialize the agent at s , and follow a greedy policy to visit the neighbour s (cid:48) =
arg max (cid:48) s r
( s, s (cid:48) )
+ V ( s (cid:48) )
for up to T = 121 timesteps ( 11 × 11 ).
If the agent reaches the desired goal within the episode i then the binary success indicator S i = 1 and the episode terminates, and 0 otherwise.
Let l i be the ground truth V ( s, g ) optimal episode cumulative reward (i.e. path cost), and
p i the cumulative reward in the episode by the agent’s trajectory.
We then deﬁne the Policy
Success rate and SPL as:
[Equation]
F. A DDITIONAL E VALUATIONS AND V ISUALIZATIONS
We visualize additional heatmaps for (1) learned value function (Figure 15),
(2) Squared Error (SE) between the ground truth value and the learned value (Figure 16), and (3) SPL metric for individual ( s 0 , s ) and ( s, s 0 ) pairs (Figure 17).
These results are for agents trained with training fraction η = 0 .
In comparing the performance versus training fraction, we also plot the ﬁnal performance afterepoch of training: (1) training set SPL versus training fraction (Figure 13a), (2) Train (Figure 14a) and test (Figure 14b) mean squared error (MSE) with the groundtruth values.
We note that while our Deep Norm and Wide Norm appears to have higher MSE than the baselines, in practice when utilizing the value function with greedy policy, our architectures were able to achieve better success than the baseline architectures.
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure]
[Figure] [Figure]
[Figure]
[Figure] [Figure]
[Figure]
[Figure] [Figure]
[Figure]
a z
[Figure]
S y
[Figure]
T e s t : g o a l
[Figure]
z
[Figure]
y
[Figure]
T [Figure] t a t e
[Figure]
4 - R oo m A s y m , T e s t : g o a l
SPL Euclidean
SPL MLP
SPL ICNN
SPL DeepNorm Asym SPL DeepNorm Sym
SPL WideNorm Asym
SPL WideNorm Sym 0.0 0.
4 - R oo m A s y m , T e s t : s t a t e
SPL Euclidean
SPL MLP
SPL ICNN
SPL DeepNorm Asym SPL DeepNorm Sym
SPL WideNorm Asym
SPL WideNorm Sym 0.0 0.
(a
4 - R oo m S y m , T e s t : g o a l
SPL Euclidean
SPL MLP
SPL ICNN
SPL DeepNorm Asym SPL DeepNorm Sym
SPL WideNorm Asym
SPL WideNorm Sym 0.0 0.
4 - R oo m S y m , T e s t : s t a t e
SPL Euclidean
SPL MLP
SPL ICNN
SPL DeepNorm Asym SPL DeepNorm Sym
SPL WideNorm Asym
SPL WideNorm Sym 0.0 0.
(b
M a z e A s y m ,
T e s t : g o a l
SPL Euclidean
SPL MLP
SPL ICNN
SPL DeepNorm Asym SPL DeepNorm Sym
SPL WideNorm Asym
SPL WideNorm Sym 0.0 0.
M a z e A s y m ,
T e s t : s t a t e
SPL Euclidean
SPL MLP
SPL ICNN
SPL
[Figure]
M a z e S y m ,
T e s t : g o a l
SPL Euclidean
[Figure]
MLP
[Figure]
ICNN
[Figure]
M a z e S y m ,
T e s t : s t a t e
SPL Euclidean
[Figure]