Original: R ELATIVE REPRESENTATIONS ENABLE ZERO - SHOT LATENT SPACE COMMUNICATION
Revised: R ELATIVE REPRESENTATIONS ENABLE ZERO - SHOT LATENT SPACE COMMUNICATION

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Neural networks embed the geometric structure of a data manifold lying in a high-dimensional space into latent representations.
Revised: Neural networks embed the geometric structure of a data manifold lying in a high-dimensional space into latent representations.

Original: Ideally, the distribution of the data points in the latent space should depend only on the task, the data, the loss, and other architecture-specific constraints.
Revised: Ideally, the distribution of the data points in the latent space should depend only on the task, the data, the loss, and other architecture-specific constraints.

Original: However, factors such as the random weights initialization, training hyperparameters, or other sources of randomness in the training phase may induce incoherent latent spaces that hinder any form of reuse.
Revised: However, factors such as the random weights initialization, training hyperparameters, or other sources of randomness in the training phase may induce incoherent latent spaces that hinder any form of reuse.

Original: Nevertheless, we empirically observe that, under the same data and modeling choices,distinct latent spaces typically differ by an unknown quasi-isometric transformation: that is, in each space, the distances between the encodings  do not change.
Revised: Nevertheless, we empirically observe that, under the same data and modeling choices, the angles between the encodings within distinct latent spaces do not change.

Original: In this work, we propose the latent similarity between each sample and a fixed set of anchors as an alternative data representation, demonstrating  it can enforce the desired invariances without any additional training.
Revised: In this work, we propose the latent similarity between each sample and a fixed set of anchors as an alternative data representation, demonstrating that it can enforce the desired invariances without any additional training.

Original: We show how neural architectures can leverage these relative representations to guarantee, in practice,  latent isometry invariance, effectively enabling latent space communication: from zero-shot model stitching to latent space comparison between diverse settings.
Revised: We show how neural architectures can leverage these relative representations to guarantee, in practice, invariance to latent isometries and rescalings, effectively enabling latent space communication: from zero-shot model stitching to latent space comparison between diverse settings.

Original: We extensively validate the generalization capability of our approach on different datasets, spanning various modalities (images, text, graphs), tasks (e.g., classification, reconstruction) and architectures (e.g., CNNs, GCNs, transformers).
Revised: We extensively validate the generalization capability of our approach on different datasets, spanning various modalities (images, text, graphs), tasks (e.g., classification, reconstruction) and architectures (e.g., CNNs, GCNs, transformers).

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Neural Networks (NN) learn to transform high dimensional data into meaningful representations that are helpful for solving downstream tasks.
Revised: Neural Networks (NN) learn to transform high dimensional data into meaningful representations that are helpful for solving downstream tasks.

Original: Typically, these representations are seen as elements of a vector space, denoted as latent space, which corresponds to the constrained output (explicitly or implicitly) of a key component of the NN, e.g., the bottleneck in an Autoencoder (AE), or the word embedding space in an NLP task.
Revised: Typically, these representations are seen as elements of a vector space, denoted as latent space, which corresponds to the constrained output (explicitly or implicitly) of a key component of the NN, e.g., the bottleneck in an Autoencoder (AE), or the word embedding space in an NLP task.

Original: The underlying assumption is that the learned latent spaces should be an optimal encoding given the data distribution, the downstream task, and the network constraints.
Revised: The underlying assumption is that the learned latent spaces should be an optimal encoding given the data distribution, the downstream task, and the network constraints.

Original: In practice, however, the learned latent spaces are subject to changes even when the above assumptions remain fixed.
Revised: In practice, however, the learned latent spaces are subject to changes even when the above assumptions remain fixed.

Original: We illustrate this phenomenon in Figure 1, where we show the latent spaces produced by an AE with a two-dimensional bottleneck, trained on the MNIST dataset several times from scratch.
Revised: We illustrate this phenomenon in Figure 1, where we show the latent spaces produced by an AE with a two-dimensional bottleneck, trained on the MNIST dataset several times from scratch.

Original: These spaces differ from one another, breaking the fundamental assumptions made above.
Revised: These spaces differ from one another, breaking the fundamental assumptions made above.

Original: The distribution of the latent embeddings is affected by several factors, such as the random initialization of the network weights, the data shuffling, hyperparameters, and other stochastic processes in the training phase.
Revised: The distribution of the latent embeddings is affected by several factors, such as the random initialization of the network weights, the data shuffling, hyperparameters, and other stochastic processes in the training phase.

Original: Although different, the learned representations in Figure 1 are intrinsically similar: the distances between the embedded representations are approximately the same across all spaces, even if their absolute coordinates differ.
Revised: Although different, the learned representations in Figure 1 are intrinsically similar: the distances between the embedded representations are approximately the same across all spaces, even if their absolute coordinates differ.

Original: Indeed, the learned latent spaces are the same up to a nearly isometric transformation.
Revised: Indeed, the learned latent spaces are the same up to a nearly isometric transformation.

Original: This symmetry is a consequence of the implicit biases underlying the optimization process (Soudry et al., 2018) forcing the model to generalize and, therefore, to give similar representations to similar samples with respect to the task.
Revised: This symmetry is a consequence of the implicit biases underlying the optimization process (Soudry et al., 2018) forcing the model to generalize and, therefore, to give similar representations to similar samples with respect to the task.

Original: There exist infinitely many spatial arrangements complying with these similarity constraints, each associated with a different isometry.
Revised: There exist infinitely many spatial arrangements complying with these similarity constraints, each associated with a different isometry in the example of Figure 1.

Original: But while the resulting models will be equally good in terms of  the task, one still encounters several practical problems. For example, it is notoriously challenging to compare latent spaces across different trainings or across
Revised: To the best of our knowledge, the first to acknowledge this behavior was Olah (2015) in a blogpost.

Original: [Figure]
Revised: [Figure]

Revised: But while the resulting models will be equally good in terms of the task, one still encounters several practical problems.

Original: As we show in Figure 5, this property holds even for high-dimensional latent spaces  different NNs; perhaps more importantly, re-using neural components trained on different embeddings of the same data becomes impossible, since they are incompatible.
Revised: For example, it is notoriously challenging to compare latent spaces across different trainings or across different NNs; perhaps more importantly, re-using neural components trained on different embeddings of the same data becomes impossible, since they are incompatible.

Original: To overcome this, we propose adopting a local coordinate system defined by the data itself.
Revised: To overcome this, we propose adopting a local coordinate system defined by the data itself.

Original: Each data point becomes a set of coefficients that encode the point as a function of other data samples, instead of an independent point in R d .
Revised: Each data point becomes a set of coefficients that encode the point as a function of other data samples, instead of an independent point in R d .

Original: The proposed relative representation directly encodes the intrinsic information underlying the data, and can be made fully invariant to isometries by construction.
Revised: The proposed relative representation directly encodes the intrinsic information underlying the data, and only depends on the angles between embeddings by construction.

Original: Remarkably, this enables a form of compositionality between learning models; it allows, for instance, to stitch together an encoder trained on ImageNet with a decoder trained on CIFAR, as we showcase in our experiments.
Revised: Remarkably, this enables a form of compositionality between learning models; it allows, for instance, to stitch together an encoder trained on ImageNet with a decoder trained on CIFAR, as we showcase in our experiments.

Original: Our main contributions can be summarized as follows:
Revised: Our main contributions can be summarized as follows:

Original: • We show that the representations learned by NNs are subject to change due to severalfactors in the training process, and model these changes via latent space isometries.
Revised: • We show that the representations learned by NNs are subject to change due to several training factors; nonetheless, the angles between latent embeddings are preserved.

Original: • We introduce a novel relative representation for latent embeddings, that is invariant by construction to the isometries induced by stochastic factors in the training process.
Revised: • We introduce a novel relative representation for latent embeddings, that is invariant by construction to the transformations induced by stochastic factors in the training process.

Original: • For the first time, we successfully demonstrate zero-shot stitching of neural components produced by distinct training regimens, e g., due to different seeds or different neural architectures; we validate our findings on different data modalities (e g. images, text).
Revised: • For the first time, we successfully demonstrate zero-shot stitching of neural components produced by distinct training regimens, e.g., due to different seeds or different neural architectures; we validate our findings on different data modalities (e.g. images, text).

Original: • Our framework also provides a quantitative measure of performance while training neural models, which is differentiable, does not need any labeled data, and is correlated with standard performance measures such as accuracy.
Revised: • Our framework also provides a quantitative measure of performance while training neural models, which is differentiable, does not need any labeled data, and is correlated with standard performance measures such as accuracy

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Representation similarity.
Revised: Representation similarity.

Original: Recently, there has been growing agreement that good networks learn similar representations across a variety of architectures, tasks and domains (Morcos et al., 2018;
Revised: Recently, there has been growing agreement that good networks learn similar representations across a variety of architectures, tasks and domains (Morcos et al., 2018;

Original: Li et al., 2016; Kornblith et al., 2019; Bonheme & Grzes, 2022; Tsitsulin et al., 2020; Barannikov et al., 2022; Vuli´c et al., 2020; Lample et al., 2018; Lenc & Vedaldi, 2015; Mikolov et al., 2013b;
Revised: Li et al., 2016; Kornblith et al., 2019; Bonheme & Grzes, 2022; Tsitsulin et al., 2020; Barannikov et al., 2022; Vuli´c et al., 2020; Lample et al., 2018; Lenc & Vedaldi, 2015; Mikolov et al., 2013b;

Original: Antonello et al., 2021), although this is still debated (Wang et al., 2018) and missing strong theoretical justifications.
Revised: Antonello et al., 2021), although this is still debated (Wang et al., 2018) and missing strong theoretical justifications.

Revised: Similar observations have been made in the context of biological models Laakso & Cottrell (2000); Kriegeskorte et al.

Revised: ; Raizada & Connolly (2012).

Original: Supported by the empirical evidence widely reported in these works, our method assumes that well-performing neural networks trained on similar tasks and data produce similar latent spaces, which allows us to define a representation that unifies all these spaces.
Revised: Supported by the empirical evidence widely reported in these works, our method assumes that well-performing neural networks trained on similar tasks and data produce similar latent spaces, which allows us to define a representation that unifies all these spaces.

Original: Model stitching.
Revised: Model stitching.

Original: Lenc & Vedaldi (2015) introduced trainable stitching layers that allow swapping parts of different networks, while Bansal et al.
Revised: Lenc & Vedaldi (2015) introduced trainable stitching layers that allow swapping parts of different networks, while Bansal et al.

Original: ; Csisz ´ arik et al.
Revised: ; Csisz´arik et al.

Original: (2021) employed stitching to quantitatively verify statements such as “good networks learn similar representations” and “more data, width or time is better”.
Revised: (2021) employed stitching to quantitatively verify statements such as “good networks learn similar representations” and “more data, width or time is better”.

Original: Other works, such as Gygli et al.
Revised: Other works, such as Gygli et al.

Original: ; Biondi et al.
Revised: ; Biondi et al.

Original: ; Yaman et al.
Revised: ; Yaman et al.

Original: ; Bianchi et al.
Revised: ; Bianchi et al.

Original: (2020), tried to directly produce compatible and reusable network components without stitching layers; more generally, stitching has been adopted in the literature to analyze neural networks.
Revised: (2020), tried to directly produce compatible and reusable network components without stitching layers; more generally, stitching has been adopted in the literature to analyze neural networks.

Original: In our work, we sidestep the need for trainable stitching layers and propose zero-shot model stitching to effectively reuse models.
Revised: In our work, we sidestep the need for trainable stitching layers and propose zero-shot model stitching to effectively reuse models.

Original: Relative information.
Revised: Relative information.

Original: The attention mechanism (Vaswani et al., 2017) and its variants (Kossen et al., 2021) exploit the relationship between features to extract meaningful representations.
Revised: The attention mechanism (Vaswani et al., 2017) and its variants (Kossen et al., 2021) exploit the relationship between features to extract meaningful representations.

Original: Prototypical Networks (Snell et al., 2017) learn a metric space where the classification can be performed by measuring the distances to prototype representations.
Revised: Prototypical Networks (Snell et al., 2017) learn a metric space where the classification can be performed by measuring the distances to prototype representations.

Original: Shalam & Korman (2022) proposed the Self Optimal Transport feature transform to enrich the sample representations with higher order relations between the instance features, while Alvarez-Melis et al.
Revised: Shalam & Korman (2022) proposed the Self Optimal Transport feature transform to enrich the sample representations with higher order relations between the instance features, while Alvarez-Melis et al.

Original: (2019) proposed a general formulation of the optimal transport that accounts for global invariances in the underlying feature spaces.
Revised: (2019) proposed a general formulation of the optimal transport that accounts for global invariances in the underlying feature spaces.

Original: Mathematically, our method bears resemblance to a kernel method (Hofmann et al., 2008) as it employs inner products of embedded features as a core ingredient.
Revised: Mathematically, our method bears resemblance to a kernel method (Hofmann et al., 2008) as it employs inner products of embedded features as a core ingredient.

Original: However, differently from kernel methods, we do not introduce learnable parameters and, crucially, we compute the representations explicitly without resorting to a kernel trick.
Revised: However, differently from kernel methods, we do not introduce learnable parameters and, crucially, we compute the representations explicitly without resorting to a kernel trick.

Original:  M ETHOD
Revised:  M ETHOD

Original: Given a training set X , standard NNs learn an embedding function E θ : X → R d , parametrized by θ , which maps each sample x ( i )
Revised: Given a training set X , standard NNs learn an embedding function E θ : X → R d , parametrized by θ , which maps each sample x ( i )

Original: ∈
Revised: ∈

Original: X to its latent representation, or absolute representation , e x ( i ) =
Revised: X to its latent representation, or absolute representation , e x ( i ) =

Original: E θ
Revised: E θ

Original: ( x ( i ) ) .
Revised: ( x ( i ) ) .

Original: This representation is then exploited to solve downstream tasks, such as classification, reconstruction or generation, optimizing over some objective function of the general form:
Revised: This representation is then exploited to solve downstream tasks, such as classification, reconstruction or generation, optimizing over some objective function of the general form:

Original: [Equation]
Revised: [Equation]

Original: Here, E X denotes the expectation over the training distribution, and Reg ( θ ) encodes additional constraints on the weights θ .
Revised: Here, E X denotes the expectation over the training distribution, and Reg ( θ ) encodes additional constraints on the weights θ .

Original: As previously discussed, we argue that the learned weights θ ∗ are not only a function of X and of the specific loss appearing in Equation 1, but in practice they are also affected by the optimization process used to train the network due to weight initialization, data shuffling, hyperparameters, and other stochastic factors.
Revised: As previously discussed, we argue that the learned weights θ ∗ are not only a function of X and of the specific loss appearing in Equation 1, but in practice they are also affected by the optimization process used to train the network due to weight initialization, data shuffling, hyperparameters, and other stochastic factors.

Original: We denote these factors collectively by ϕ .
Revised: We denote these factors collectively by ϕ .

Original: In particular, as shown in Figure 1, changing these factors induces a transformation T over the latent space, i.e., ϕ → ϕ ′ implies E θ ( x ( i ) )
Revised: In particular, as shown in Figure 1, changing these factors induces a transformation T over the latent space, i.e., ϕ → ϕ ′ implies E θ ( x ( i ) )

Original: → TE θ ( x ( i ) ) .
Revised: → TE θ ( x ( i ) ) .

Original: We make the core assumption that T is an isometric transformation, preserving the distances between elements of the latent space, namely d ( e x ( i ) , e x ( j ) ) =
Revised: We make the core assumption that T preserves the angles between elements of the latent space, namely ∠ ( e x ( i ) , e x ( j ) ) =

Original: d ( T e x ( i ) , T e x ( j ) ) for every ( x ( i ) , x ( j ) ) ∈ X .
Revised: ∠ ( T e x ( i ) , T e x ( j ) ) for every ( x ( i ) , x ( j ) ) ∈ X .

Original: While the isometry assumption might seem too restrictive, in practice it arises in several real scenarios as we show in the sequel.
Revised: While this assumption might seem too restrictive, in practice it arises in several real scenarios as we show in the sequel.

Original: 3. R ELATIVE R EPRESENTATIONS
Revised: 3. R ELATIVE R EPRESENTATIONS

Original: To build our representation, we start by selecting a subset A of the training data X , which we denote as anchor samples.
Revised: To build our representation, we start by selecting a subset A of the training data X , which we denote as anchor samples.

Original: Every sample in the training distribution will be represented with respect to the embedded anchors e a ( j ) =
Revised: Every sample in the training distribution will be represented with respect to the embedded anchors e a ( j ) =

Original: E ( a ( j ) ) with a ( j ) ∈ A .
Revised: E ( a ( j ) ) with a ( j ) ∈ A .

Original: As a measure capturing the relation between the anchors and the other samples, we consider a generic similarity function sim : R d × R d → R , yielding a scalar score r between two absolute representations r = sim ( e x ( i ) , e x ( j ) ) .
Revised: As a measure capturing the relation between the anchors and the other samples, we consider a generic similarity function sim : R d × R d → R , yielding a scalar score r between two absolute representations r = sim ( e x ( i ) , e x ( j ) ) .

Original: Given the anchors A in an arbitrary ordering a (1) , . . .
Revised: Given the anchors A in an arbitrary ordering a (1) , . . .

Original: , a ( | A | ) , we define the relative representation of x ( i ) ∈ X as:
Revised: , a ( | A | ) , we define the relative representation of x ( i ) ∈ X as:

Original: [Equation]
Revised: [Equation]

Original:  Figure 2 illustrates the key differences between absolute and relative representations.⇒ ⇒
Revised: [Equation] Figure 2 illustrates the key differences between absolute and relative representations.

Original: Choice of the anchors.
Revised: Choice of the anchors.

Original: Anchors directly affect the expressivity of the relative representation space, and are related to the task at hand.
Revised: Anchors directly affect the expressivity of the relative representation space, and are related to the task at hand.

Original: For example, in a classification task, we should sample anchors from each class in the training set, in order to well represent each data sample in X .
Revised: For example, in a classification task, we should sample anchors from each class in the training set, in order to well represent each data sample in X .

Original: One case of interest arises when the data comes from different domains  X , Y , and we are given a partial correspondence Γ : P X (cid:55)→ P Y mapping from a subset of X to a subset of Y .
Revised: One case of interest arises when the data comes from different domains or modalities X , Y , and we are given a partial correspondence Γ : P X (cid:55)→ P Y mapping from a subset of X to a subset of Y .

Original: In this case, we can sample anchors A X ⊆ P X and obtain corresponding anchors on the other domain directly as Γ( A ) .
Revised: In this case, we can sample anchors A X ⊆ P X and obtain corresponding anchors on the other domain directly as Γ( A ) (Norelli et al., 2022).

Original: We refer to these as parallel anchors .
Revised: We refer to these as parallel anchors .

Original: We show an example of parallel anchors in Section 5.2, where X and Y represent Amazon reviewswritten in two different languages.
Revised: We show an example of parallel anchors in Section 5.2, where X and Y are Amazon reviews in two different languages.

Original: The choice of the anchors is not restricted to elements in the training distribution.
Revised: The choice of the anchors is not restricted to elements in the training distribution.

Original: Given an encoder pre-trained on a fixed training distribution, we can pick elements from a set ˜
Revised: Given an encoder pre-trained on a fixed training distribution, we can pick elements from a set ˜

Original: A that is out-of-domainw.r.t.
Revised: A that is out-of-domain

Original:  X , and build therelative representations on topof ˜ A .
Revised: ⇒ ⇒ w t. X , and build the top ˜ A .

Original: We refer to these as OOD anchors and exploit them, e.g., to solve domain adaptation tasks where we do not have access to a correspondence, and have scarce data labels.
Revised: We refer to these as OOD anchors and - ploit them, e.g., to solve domain adaptation tasks where we do not have access to a correspondence, and have scarce data labels.

Original: We refer again to the Sections 5.2 and 5.3 for real-world examples and to Appendix A.2 for a preliminary analysisof different selection strategies.
Revised: We refer again to the Sections 5.2 and real world examples and to Appendix A preliminary analysis strategies.

Original: Isometry invariance.
Revised: Achieving latent invariance.

Original: In this work, we choose the cosine similarity as the similarity function due to the properties it induces on the relative representation.
Revised: In this work, we choose the cosine similarity as the similarity function due to the properties it induces on the relative representation.

Original: The cosine similarity S C is the dot product of unit vectors, corresponding to the cosine of the angle θ between the two:
Revised: The cosine similarity S C is the dot product of unit vectors, corresponding to the cosine of the angle θ between the two:

Original: [Equation]
Revised: [Equation]

Original: Importantly, cos θ does not change if we apply the same orthogonal transformation T to two vectors a and b , i.e., the cosine similarity is invariant to rotations, reflections, and rescaling.
Revised: Importantly, cos θ does not change if we apply the same angle-preserving transformation T to two vectors a and b , i.e., the cosine similarity is invariant to rotations, reflections, and rescaling.

Original: While this is not true for translations, NNs commonly employ normalization techniques (e.g., InstanceNorm (Ulyanov et al., 2016)) to center the latent spaces around zero.
Revised: While this is not true for translations, NNs commonly employ normalization techniques (e.g., InstanceNorm (Ulyanov et al., 2016)) to center the latent spaces.

Original: Under this assumption, cosine similarity guarantees a relative representation r x ( i ) invariant to isometric transformations.
Revised: Under this assumption, cosine similarity guarantees a relative representation r x ( i ) invariant to angle-preserving transformations.

Original: This means we have the freedom to change the embedding function E θ with any other function ˜
Revised: This means we have the freedom to change the embedding function E θ with any other function ˜

Original: E that produces different but isometric representations, i.e.:
Revised: E that produces different representations with same angles, i.e.:

Original: [Equation]
Revised: [Equation]

Original: where ˜ e x ( i ) =
Revised: where ˜ e x ( i ) =

Original: ˜ E ( x ( i ) ) =
Revised: ˜ E ( x ( i ) ) =

Original: T E ( x ( i ) ) and T is an arbitrary isometric transformation.
Revised: T E ( x ( i ) ) and T is an arbitrary angle-preserving transformation.

Original: A practical application of this invariance is the possibility of comparing latent spaces across multiple trainings, and re-using models as demonstrated in Sections 4 and 5.
Revised: A practical application of this invariance is the possibility of comparing latent spaces across multiple trainings, and re-using models as demonstrated in Sections 4 and 5.

Original: Quasi-isometry invariance.

Original: We remark that other choices of similarity function can be made to enforce different invariances into the representation.
Revised: We remark that other choices of similarity function can be made to enforce different invariances into the representation.

Original: For example,to relax the isometry assumption, one may impose invariance to non-isometric deformations with bounded distortion.
Revised: For example, one may impose invariance to non-isometric deformations with bounded distortion.

Original: We did not find this to be necessary in our experiments, as typically NNs that generalize sufficiently well can handle small perturbations.
Revised: We did not find this to be necessary in our experiments, as typically NNs that generalize sufficiently well can handle small perturbations.

Original: Nevertheless, this invariance can be enforced by design with vector quantization algorithms.
Revised: Nevertheless, this invariance can be enforced by design with vector quantization algorithms.

Original: Figure 7 shows a preliminary investigation, leaving further exploration to future work.
Revised: Figure 7 shows a preliminary investigation, leaving further exploration to future work.

Original:  L ATENT S PACE C OMMUNICATION
Revised:  L ATENT S PACE C OMMUNICATION

Original: In this section, we demonstrate how our relative representations can effectively be used to produce latent spaces that are stable under a variety of factors.
Revised: In this section, we demonstrate how our relative representations can effectively be used to produce latent spaces that are stable under a variety of factors.

Original: Our main question is the following:
Revised: Our main question is the following:

Original: Given two different learning models that are trained separately on different data, can we compare their latent embeddings?
Revised: Given two different learning models that are trained separately on different data, can we compare their latent embeddings?

Original: In asking this, we assume that the two models are trained on a similar phenomenon, e.g., on two different samplings of the English language or on two different modalities.
Revised: In asking this, we assume that the two models are trained on a similar phenomenon, e.g., on two different samplings of the English language or on two different modalities.

Original: We answer in the positive, showing the gained invariance enables effective communication between different, but semantically equivalent latent spaces.
Revised: We answer in the positive, showing the gained invariance enables effective communication between different, but semantically equivalent latent spaces.

Original: In particular, we analyze how different word embedding spaces, once projected onto relative representations, are intrinsically the same (Section 4.1); we then show how the similarity between the relative counterparts of two or more embedding spaces is a surprisingly good predictor of model performance (Section 4.2); finally, we confirm that relative representations in the training phase are not detrimental to performance (Section 4.3).
Revised: In particular, we analyze how different word embedding spaces, once projected onto relative representations, are intrinsically the same (Section 4.1); we then show how the similarity between the relative counterparts of two or more embedding spaces is a surprisingly good predictor of model performance (Section 4.2); finally, we confirm that relative representations in the training phase are not detrimental to performance (Section 4.3).

Original: 4. W ORD E MBEDDINGS
Revised: 4. W ORD E MBEDDINGS

Original: Experimental setting.
Revised: Experimental setting.

Original: We select two different word embeddings on the English language, namely
Revised: We select two different word embeddings on the English language, namely

Original: FastText (Bojanowski et al., 2017) and Word2Vec (Mikolov et al., 2013a).
Revised: FastText (Bojanowski et al., 2017) and Word2Vec (Mikolov et al., 2013a).

Original: Both models are pretrained on different data, but partly share a vocabulary from which we extract ≈ 20 K words.
Revised: Both models are pre-trained on different data, but partly share a vocabulary from which we extract ≈ 20 K words.

Original: For each embedding space, we convert their absolute embeddings to corresponding relative representations, using 300 randomly drawn parallel anchors.
Revised: Using 300 randomly drawn parallel anchor, we convert each embedding space to a relative one.

Original: In Table 1 (left), we show the originalembeddings and theresulting relative ones.
Revised: In Table 1 (left) [Table] we show the original and the relative embeddings.

Original: For each word w , we consider its corresponding encodings x and y in the source and target space.
Revised: For each word w , we consider its corresponding encodings x and y in the source and target space.

Original: We apply three different metrics to measure their similarity (following a setting similar to Vuli´c et al.
Revised: We apply three different metrics to measure their similarity (in a setting similar to Vuli´c et al.

Original: (2020)): (i)
Revised: (2020)): (i)

Original: Jaccard : the discrete Jaccard similarity between the set of word neighborsx has in source, and the set ofword neighbors xhas in  target; (ii) Mean Reciprocal Rank : measures the (reciprocal) ranking of w among the top-k neighbors of x in the target space; (iii) Cosine : measures the cosine similarity between x and y .
Revised: Jaccard : the discrete Jaccard similarity between the set of word neighbors of x in source and target; (ii) Mean Reciprocal Rank : measures the (reciprocal) ranking of w among the top-k neighbors of x in the target space; (iii) Cosine : measures the cosine similarity between x and y .

Original: Additional details in Appendix A.5.
Revised: Additional details in Appendix A.5.

Original: FastText Word2Vec
Revised: FastText Word2Vec

Original: A

Original: [Table]
Revised: [Table]

Original: u [Table] e
Revised: u t e

Original: R e l a ti v e
Revised: R e l a ti v e

Original: [Table]
Revised: [Table]

Original: embeddings using absolute and relative representations.

Original: PCA is applied only for visualization.

Original: All metrics are calculated with K = 10 averaged over 20k words and across 10 different random seeds.

Original: See Figure 9 for other dimensionality reductions and Table 8 and fig.

Original: 10 for the same experiment on CIFAR-10 .

Original: Result analysis.
Revised: Result analysis.

Original: Table 1 (left) highlights clusters of semantically similar words and shows that the absolute representations are incoherent across the two latent spaces, while the relative embeddings are highly similar.
Revised: Table 1 (left) highlights clusters of semantically similar words and shows that the absolute representations are incoherent across the two latent spaces, while the relative embeddings are highly similar.

Original: The average Jaccard distance reported in Table 1 (right) , says that the word neighborhoods of the relative representations are matched exactly 34% of the time in one direction, and 39% of the time in the other one (the missing 61% is due to semantic differences, that are not taken into account by the discrete nature of the Jaccard metric).
Revised: The average Jaccard distance reported in Table 1 (right) , says that the word neighborhoods of the relative representations are matched exactly 34% of the time in one direction, and 39% of the time in the other one (the missing 61% is due to semantic differences, that are not taken into account by the discrete nature of the Jaccard metric).

Original: By contrast, the absolute embeddings are never matched exactly (Jaccard score equal to zero); for a match to happen, it would mean that the FastText and Word2Vec embeddings of a given English word are almost the same, which is highly unlikely.
Revised: By contrast, the absolute embeddings are never matched exactly (Jaccard score equal to zero); for a match to happen, it would mean that the FastText and Word2Vec embeddings of a given English word are almost the same, which is highly unlikely.

Original: MRR, close to a perfect score for the relative representations, shows that the most-similar word to a given one is usually itself, even if their cosine similarity doesn’t reach 1.
Revised: MRR, close to a perfect score for the relative representations, shows that the most-similar word to a given one is usually itself, even if their cosine similarity doesn’t reach 1.

Original: Overall, these results show that relative representations are preserved across different word embedding models, validating ourisometry assumptions.
Revised: Overall, these results show that relative representations are preserved across different word embedding models, validating our assumptions.

Original: 4. L ATENT DISTANCE AS A PERFORMANCE PROXY
Revised: 4. L ATENT DISTANCE AS A PERFORMANCE PROXY

Original: Experimental setting.
Revised: Experimental setting.

Original: In this experiment, we consider a node classification task on the Cora graph dataset (Sen et al., 2008).
Revised: In this experiment, we consider a node classification task on the Cora graph dataset (Sen et al., 2008).

Original: We first train a reference model that achieves good accuracy on a validation set.
Revised: We first train a reference model that achieves good accuracy on a validation set.

Original: Then, we train ≈ 2000 models with various combinations of seed, number of epochs, number of layers, dropout probability, activation functions, optimizer type, learning rate or type of graph embedder (Table 10).
Revised: Then, we train ≈ 2000 models with various combinations of seed, number of epochs, number of layers, dropout probability, activation functions, optimizer type, learning rate or type of graph embedder (Table 10).

Original: All the models are classically trained using absolute representations, which are converted to relative post-training by projecting the embeddings onto 300 randomly drawn but fixed anchors.
Revised: All the models are classically trained using absolute representations, which are converted to relative post-training by projecting the embeddings onto 300 randomly drawn but fixed anchors.

Original: For each model, we measure its classification accuracy and compute the similarity of its space with the reference one.
Revised: For each model, we measure its classification accuracy and compute the similarity of its space with the reference one.

Original: This similarity is computed as the average cosine similarity between the node embeddings produced by a given model and the corresponding embeddings in the reference.
Revised: This similarity is computed as the average cosine similarity between the node embeddings produced by a given model and the corresponding embeddings in the reference.

Original: Result analysis.
Revised: Result analysis.

Original: The scatter plot in Figure 3 (left) shows that better-performing models tend to be the ones with the latent spaces most similar to the reference. 
Revised: The scatter plot in Figure 3 (left) shows that better-performing models tend to be the ones with the latent spaces most similar to the reference. The performance-similarity correlation [Figure] m a n ce

Original: The performance-similarity correlation also holds over time, as shown in Figure 3 (right) .
Revised: Similarity P e rf o r m a n ce Epochs S i m il a r it y also holds over time, as shown in Figure 3 (

Original: Additional correlation examples are in Figure 8.
Revised: Additional correlation examples are in Figure 9.

Original: Interestingly, this metric is differentiable, enabling an explicit supervision signal on the latent space, which does not require labeled data and could be readily exploited in a teacher-student framework.[Figure] Epochs [Figure]
Revised: Interestingly, this metric is differentiable, enabling an explicit supervision signal on the latent space, which does not require labeled data and could be readily exploited in a teacher-student framework.

Original: y Overall, these results suggest that the similarity between the relative representations of latent spaces is a remarkably good proxy to evaluate model performance.
Revised: Overall, these results suggest that the similarity between the relative representations of latent spaces is a remarkably good proxy to evaluate model performance.

Original: 4. T RAINING WITH A BSOLUTE VS . R ELATIVE REPRESENTATIONS
Revised: 4. T RAINING WITH A BSOLUTE VS . R ELATIVE REPRESENTATIONS

Original: Experimental setting.
Revised: Experimental setting.

Original: Finally, we compare architectures that do or do not employ the relative representation while training.
Revised: Finally, we compare architectures that do or do not employ the relative representation while training.

Original: In these experiments, the models vary slightly according to the dataset; however, the relative and absolute versions are always comparable in terms of architecture, number of learnable parameters and hyperparameters.
Revised: In these experiments, the models vary slightly according to the dataset; however, the relative and absolute versions are always comparable in terms of architecture, number of learnable parameters and hyperparameters.

Original: We refer to the supplementary material and the open-source code for further details on their implementation.
Revised: We refer to the supplementary material and the open-source code for further details on their implementation.

Original: In this section we consider classification tasks on several datasets, spanning the image domain (Lecun et al., 1998; Xiao et al., 2017;
Revised: In this section we consider classification tasks on several datasets, spanning the image domain (Lecun et al., 1998; Xiao et al., 2017;

Original: Krizhevsky, 2009) and the graph domain (Yang et al., 2016).
Revised: Krizhevsky, 2009) and the graph domain (Yang et al., 2016).

Original: [Table]
Revised: [Table]

Original: Result analysis.
Revised: Result analysis.

Original: The results, reported in Table 2, show that relative representations, when used at training time, are not detrimental to performance in general.
Revised: The results, reported in Table 2, show that relative representations, when used at training time, are not detrimental to performance in general.

Original: This is further shown in Tables 3 toand 15 to 18, where a subset of the results compares the absolute and relative representations on a variety of domains, datasets and tasks.
Revised: This is further shown in Tables 3 toand 15 to 18, where a subset of the results compares the absolute and relative representations on a variety of domains, datasets and tasks.

Original: Overall, these results show that relative representations are effective when involved in end-to-end training, without significant performance drops.
Revised: Overall, these results show that relative representations are effective when involved in end-to-end training, without significant performance drops.

Original:  Z ERO -S HOT M ODEL S TITCHING
Revised:  Z ERO -S HOT M ODEL S TITCHING

Original: In this section, we illustrate how the latent space communication demonstrated in Section 4 enables zero-shot interoperability of pre-trained neural components.
Revised: In this section, we illustrate how the latent space communication demonstrated in Section 4 enables zero-shot interoperability of pre-trained neural components.

Original: In previous works, such as Lenc & Vedaldi (2015); Bansal et al.
Revised: In previous works, such as Lenc & Vedaldi (2015); Bansal et al.

Original: (2021), stitching layers are trainable linear projections that allow swapping parts of different networks.
Revised: (2021), stitching layers are trainable linear projections that allow swapping parts of different networks.

Original: Instead, on relative representations unlocks the possibility of zero-shot stitching different neural components, treating them as frozen black-box modules.
Revised: Instead, on relative representations unlocks the possibility of zero-shot stitching different neural components, treating them as frozen black-box modules.

Original: We define a generic stitched model as the composition of an encoder, that embeds data, plus a relative decoder specialized in a downstream task (classification, reconstruction).
Revised: We define a generic stitched model as the composition of an encoder, that embeds data, plus a relative decoder specialized in a downstream task (classification, reconstruction).

Original: The stitching operation is always performed without training or fine-tuning, in a zero-shot fashion.
Revised: The stitching operation is always performed without training or fine-tuning, in a zero-shot fashion.

Original: Hereafter, we showcase stitching capabilities across combinations of different stochasticity sources (Figureand table 3), neural architectures (Tables 4 and 5) or datasets (Table 6).
Revised: Hereafter, we showcase stitching capabilities across combinations of different stochasticity sources (Figureand table 3), neural architectures (Tables 4 and 5) or datasets (Table 6).

Original: Finally, we present strong real-world applications in NLP (Section 5.2) and CV (Section 5.3), e.g. zero-shot predictions on novel languages.
Revised: Finally, we present strong real-world applications in NLP (Section 5.2) and CV (Section 5.3), e.g. zero-shot predictions on novel languages.

Original: Additional implementation details are given in the supplementary materials.
Revised: Additional implementation details are given in the supplementary materials.

Original: S
Revised: S

Original: A b s .
Revised: A b s .

Original: R e l .
Revised: R e l .

Original: A b s .
Revised: A b s .

Original: R e l .
Revised: R e l .

Original: [Figure]
Revised: [Figure]

Original: 5. I MAGE R ECONSTRUCTION
Revised: 5. I MAGE R ECONSTRUCTION

Original: Experimental setting.
Revised: Experimental setting.

Original: We perform zero-shot stitching with AEs and VAEs trained with relative representations end-to-end on several datasets.
Revised: We perform zero-shot stitching with AEs and VAEs trained with relative representations end-to-end on several datasets.

Original: For each combination of model and dataset, we perform 5 trainings with different seeds, and zero-shot stitch together the resulting encoders and decoders.
Revised: For each combination of model and dataset, we perform 5 trainings with different seeds, and zero-shot stitch together the resulting encoders and decoders.

Original: Result analysis.
Revised: Result analysis.

Original: In Figure 4 the stitched models that employ absolute representations ( Abs. ) produce erroneous predictions, since the latent spaces obtained from distinct trainings are incompatible.
Revised: In Figure 4 the stitched models that employ absolute representations ( Abs. ) produce erroneous predictions, since the latent spaces obtained from distinct trainings are incompatible.

Original: Interestingly, although the absolute VAE does not produce compatible latent spaces, it is regularized, thus all embeddings produced by the encoders correspond to wrong but semantically meaningful reconstructions.
Revised: Interestingly, although the absolute VAE does not produce compatible latent spaces, it is regularized, thus all embeddings produced by the encoders correspond to wrong but semantically meaningful reconstructions.

Original: Relative representations ( Rel. ) exhibit almost indistinguishable reconstructions between the models trained end-to-end and the stitched ones.
Revised: Relative representations ( Rel. ) exhibit almost indistinguishable reconstructions between the models trained end-to-end and the stitched ones.

Original: Quantitative results are in Table 3.
Revised: Quantitative results are in Table 3.

Original: Overall, these results support our claim that relative representations are empirically invariant to training stochasticity.
Revised: These results support our claim that relative representations are empirically invariant to training stochasticity.

Original: [Table]
Revised: is computed over 5 different seeds.

Original: [Figure]
Revised: Stitching with our relative representations yields an error up to two orders of magnitude less than the absolute counterpart

Original: [Table]
Revised: [Table]

Original: 5. T EXT C LASSIFICATION
Revised: 5. T EXT C LASSIFICATION

Original: In this Section
Revised: In this Section

Original: [Table]
Revised: [Table]

Original: we show practical examples of the use of parallel anchors (Sec 3.1)
Revised: we show practical examples of the use of parallel anchors (Sec 3.1)

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Experimental setting.
Revised: Experimental setting.

Original: We consider two different text classification settings.
Revised: We consider two different text classification settings.

Original: Cross-lingual : given a review predict the associated star rating, done on multi-lingual data from the Amazon Reviews dataset (Keung et al., 2020).
Revised: Cross-lingual : given a review predict the associated star rating, done on multi-lingual data from the Amazon Reviews dataset (Keung et al., 2020).

Original: Following the original paper, we work on a binarized version of the task, with FScore and MAE as metrics.
Revised: Following the original paper, we work on a binarized version of the task, with FScore and MAE as metrics.

Original: In the supplementary material, we report results on the fine-grained formulation.
Revised: In the supplementary material, we report results on the fine-grained formulation.

Original: We adopt four different pre-trained languagespecific RoBERTa transformers (Liu et al., 2019) and evaluate their zero-shot stitching performance on languages never seen by the classifier.
Revised: We adopt four different pre-trained languagespecific RoBERTa transformers (Liu et al., 2019) and evaluate their zero-shot stitching performance on languages never seen by the classifier.

Original: We use parallel anchors in two modalities: i)
Revised: We use parallel anchors in two modalities: i)

Original: Translated : consider English reviews translated 2 into the other languages; ii) Wikipedia : adopt an external corpus, WikiMatrix (Schwenk et al., 2021), providing parallel sentences extracted from Wikipedia.
Revised: Translated : consider English reviews translated 2 into the other languages; ii) Wikipedia : adopt an external corpus, WikiMatrix (Schwenk et al., 2021), providing parallel sentences extracted from Wikipedia.

Original: Cross-architecture : assessed on three different datasets: TREC (coarse) (Hovy et al., 2001),
Revised: Cross-architecture : assessed on three different datasets: TREC (coarse) (Hovy et al., 2001),

Original: DBpedia (Zhang et al., 2015), Amazon Reviews (English split).
Revised: DBpedia (Zhang et al., 2015), Amazon Reviews (English split).

Original: We adopt two different pretrained BERT (Devlin et al., 2019) transformers (cased and uncased version), ELECTRA (Clark et al., 2020) and RoBERTa.
Revised: We adopt two different pretrained BERT (Devlin et al., 2019) transformers (cased and uncased version), ELECTRA (Clark et al., 2020) and RoBERTa.

Original: Result analysis.
Revised: Result analysis.

Original: Tables 4 and 5 show for the first time that it is possible to learn to solve a downstream task on a specific language or transformer and perform predictions on another.
Revised: Tables 4 and 5 show for the first time that it is possible to learn to solve a downstream task on a specific language or transformer and perform predictions on another.

Original: Stitching with absolute representations yields performances comparable to random guessing across the board, proving that relative representations are a key element for the success of this kind of zero-shot stitching.
Revised: Stitching with absolute representations yields performances comparable to random guessing across the board, proving that relative representations are a key element for the success of this kind of zero-shot stitching.

Original: Moreover, Table 4 highlights the robustness that relative representations have on the choice of anchors, even when they are noisy ( Translated case), or their distribution differs from the one of the downstream task ( Wikipedia case), as long as their encoding can be handled correctly by the encoder.
Revised: Moreover, Table 4 highlights the robustness that relative representations have on the choice of anchors, even when they are noisy ( Translated case), or their distribution differs from the one of the downstream task ( Wikipedia case), as long as their encoding can be handled correctly by the encoder.

Original: In our case, the encoder is pre-trained to represent a variety of texts in a specific language, thus, even if WikiMatrix has a completely different domain from Amazon
Revised: In our case, the encoder is pre-trained to represent a variety of texts in a specific language, thus, even if WikiMatrix has a completely different domain from Amazon

Original: Reviews , the transformer still computes a meaningful and comparable representation with those of the reviews.
Revised: Reviews , the transformer still computes a meaningful and comparable representation with those of the reviews.

Original: We report in Tables 15 and 16 complete results on all languages combination, and in Table 17 the performance obtained by a multi-lingual transformer.
Revised: We report in Tables 15 and 16 complete results on all languages combination, and in Table 17 the performance obtained by a multi-lingual transformer.

Original: To the best of our knowledge, it is the only alternative for obtainining compatible representations across languages.
Revised: To the best of our knowledge, it is the only alternative for obtainining compatible representations across languages.

Original: According to these results, relative representations show invariance to different architectures and data distribution shifts (e.g., different train languages).
Revised: According to these results, relative representations show invariance to different architectures and data distribution shifts (e.g., different train languages).

Original: 5. I MAGE C LASSIFICATION
Revised: 5. I MAGE C LASSIFICATION

Original: In this Section
Revised: In this Section

Original: [Table]
Revised: [Table]

Original: we show practical examples of the use of OOD anchors (Sec 3.1)
Revised: we show practical examples of the use of OOD anchors (Sec 3.1).

Original: [Table]
Revised: [Table]

Original: [Table]

Original: Experimental setting.
Revised: Experimental setting.

Original: We consider a classification task on ImageNet1k and CIFAR-100 with coarse labels (20), and 4 different pre-trained image encoders: three variants of the ViT transformer (Dosovitskiy et al., 2020) and RexNet (Han et al., 2020).
Revised: We consider a classification task on ImageNet1k and CIFAR-100 with coarse labels (20), and 4 different pre-trained image encoders: three variants of the ViT transformer (Dosovitskiy et al., 2020) and RexNet (Han et al., 2020).

Original: Result analysis.
Revised: Result analysis.

Original: The results in Table 6 highlight how the relative representations allow stitching modules with different encoding dimensionality, since the decoder receives a relative representation with guaranteed equal size.
Revised: The results in Table 6 highlight how the relative representations allow stitching modules with different encoding dimensionality, since the decoder receives a relative representation with guaranteed equal size.

Original: Further, the results demonstrate the ability to generalize and perform zero-shot stitching on CIFAR-100 , although that data was never seen by the encoder since it is a frozen transformer trained on ImageNet1k .
Revised: Further, the results demonstrate the ability to generalize and perform zero-shot stitching on CIFAR-100 , although that data was never seen by the encoder since it is a frozen transformer trained on ImageNet1k .

Original: Interestingly, rexnet-100 is the only transformer whose latent dimensionality is higher than the number of anchors, and the biggest drop in stitching performance happens when the decoder is trained on it.
Revised: Interestingly, rexnet-100 is the only transformer whose latent dimensionality is higher than the number of anchors, and the biggest drop in stitching performance happens when the decoder is trained on it.

Original: This suggests the number of anchors is an important hyperparameter; we refer to Figure 6 for a deeper analysis.
Revised: This suggests the number of anchors is an important hyperparameter; we refer to Figure 6 for a deeper analysis.

Original: Overall, these results prove that relative representations can bridge general-purpose encoders and pre-trained task-specific decoders.
Revised: Overall, these results prove that relative representations can bridge general-purpose encoders and pre-trained task-specific decoders.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this work, we introduced the concept of relative representations to enable zero-shot latent space communication, with several practical consequences as showcased in our discussion and experiments.
Revised: In this work, we introduced the concept of relative representations to enable zero-shot latent space communication, with several practical consequences as showcased in our discussion and experiments.

Original: Our work proves that a latent semantic correspondence between data domains, when present, can be exploited through a simple representation shift, without resorting to sophisticated processing or heavy training.
Revised: Our work proves that a latent semantic correspondence between data domains, when present, can be exploited through a simple representation shift, without resorting to sophisticated processing or heavy training.

Original: Limitations and future work.
Revised: Limitations and future work.

Original: Our work is open to several follow-up directions.
Revised: Our work is open to several follow-up directions.

Original: While in this paper we considered the cosine similarity, different functions can enforce additional invariances in the relative representation.
Revised: While in this paper we considered the cosine similarity, different functions can enforce additional invariances in the relative representation.

Original: The study of invariant latent spaces as a general direction has the potential to lead to further impact; in Figure 7 we showed preliminary results of this possibility, obtaining representations that are invariant to near-isometries with guaranteed bounded distortion via vector quantization.
Revised: The study of invariant latent spaces as a general direction has the potential to lead to further impact; in Figure 7 we showed preliminary results of this possibility.

Original: Another interesting line of research to improve the representation expressivity would be to estimate geodesic distances over the data manifold instead of adopting Euclidean approximations.
Revised: Another interesting line of research to improve the representation expressivity would be to estimate geodesic distances over the data manifold instead of adopting Euclidean approximations.

Original: Similarly, we believe that the connections between the composition of the anchors set A and the expressivity of relative representations demands additional research.
Revised: Similarly, we believe that the connections between the composition of the anchors set A and the expressivity of relative representations demands additional research.

Original: For example, the training cost is directly affected by the number and update frequency of the anchors.
Revised: For example, the training cost is directly affected by the number and update frequency of the anchors.

Original: Finally, the stitching procedure may be extended to multiple layers, promoting reusable network components.
Revised: Finally, the stitching procedure may be extended to multiple layers, promoting reusable network components.

Revised: A CKNOWLEDGMENTS

Revised: This work is supported by the ERC Starting Grant No. 802554 (SPECGEO).

Original: R EPRODUCIBILITY S TATEMENT
Revised: R EPRODUCIBILITY S TATEMENT

Original: We describe in detail the relative representation computation in Section 3.1.
Revised: We describe in detail the relative representation computation in Section 3.1.

Original: We describe the experimental settings for the various scenarios, and refer to the supplementary material for further implementation details (Appendix A.5).
Revised: We describe the experimental settings for the various scenarios, and refer to the supplementary material for further implementation details (Appendix A.5).

Original: Moreover, we release a well-documented and modular codebase, with the relative representation layer being implemented as a stand-alone PyTorch module.
Revised: Moreover, we release a well-documented and modular codebase, with the relative representation layer being implemented as a stand-alone PyTorch module.

Original: All the checkpoints used in the experiments are versioned with DVC (Kuprieiev et al., 2022) to easily reproduce all the figures and tables.
Revised: All the checkpoints used in the experiments are versioned with DVC (Kuprieiev et al., 2023) to easily reproduce all the figures and tables.

Original: The stand-alone module allows the integration of the relative representations in any existing neural network effortlessly.A PPENDIX A.
Revised: The stand-alone module allows the integration of the relative representations in any existing neural network effortlessly.

Original: L ATENT S PACE I SOMETRIES

Original: In Figure 1, multiple trainings of the same two-dimensional AE produce isometric latent spaces; in

Original: Figure 5 we show this property also holds on AEs with a high-dimensional bottleneck.

Original: In the first row, PCA is fitted indipendently in each column, and since the PCA transformation produces the same output everywhere the latent spaces are isometric.

Original: In the second row, PCA is fitted only on the first latent space; since in this case the PCA transformations produces different outputs, the latent spaces although isometric are extrinsically different.

Original: A. A NCHORS ANALYSIS
Revised: A. A NCHORS ANALYSIS

Original: The cardinality of the anchors set A and the choice of specific anchors is crucial to the quality of the relative representations.
Revised: The cardinality of the anchors set A and the choice of specific anchors is crucial to the quality of the relative representations.

Original: At the extreme, selecting one single anchor or the same repeated data points for all anchors, will produce collapsed relative representations.
Revised: At the extreme, selecting one single anchor or the same repeated data points for all anchors, will produce collapsed relative representations.

Original: We believe that additional research is required to obtain a better understanding on the optimal choice for A .
Revised: We believe that additional research is required to obtain a better understanding on the optimal choice for A .

Original: Questions like “Are anchors set composed only by stopwords worse than the ones composed by meaningful and diverse words?” require empirical evidence and could help revealing the semantics of the latent space.
Revised: Questions like “Are anchors set composed only by stopwords worse than the ones composed by meaningful and diverse words?” require empirical evidence and could help revealing the semantics of the latent space.

Original: Indeed, each anchor is associated with a dimension in a relative representation; one could inspect the anchor data point to get a sense of the meaning of that latent dimension.
Revised: Indeed, each anchor is associated with a dimension in a relative representation; one could inspect the anchor data point to get a sense of the meaning of that latent dimension.

Original: Anchor number.
Revised: Anchor number.

Original: Below, we report a preliminary study on the performance sensitivity against the cardinality of the anchors set.
Revised: Below, we report a preliminary study on the performance sensitivity against the cardinality of the anchors set.

Original: In Figure 6 we report the performance on the node classification task on Cora , with a model trained end-to-end adopting the relative representations while training, and on image classification tasks on CIFAR-100 , with a frozen encoder.
Revised: In Figure 6 we report the performance on the node classification task on Cora , with a model trained end-to-end adopting the relative representations while training, and on image classification tasks on CIFAR-100 , with a frozen encoder.

Original: The performance improves monotonically as the number of anchors increase when the absolute representations are frozen (right) .
Revised: The performance improves monotonically as the number of anchors increase when the absolute representations are frozen right) .

Original: Differently, training models end-to-end proves to be more susceptible to model collapse and instabilities, as increasing the number of anchors does not always improve the performance (left) .
Revised: Differently, training models end-to-end proves to be more susceptible to model collapse and instabilities, as increasing the number of anchors does not always improve the performance (left) .

Original: Further research on the relation between the absolute latent space dimensionality and the relative representation dimensionality (i.e., the number of anchors) is needed to clarify how the two quantities impact the performance, when training end-to-end or not.
Revised: Further research on the relation between the absolute latent space dimensionality and the relative representation dimensionality (i.e., the number of anchors) is needed to clarify how the two quantities impact the performance, when training end-to-end or not.

Original: Anchor selection.
Revised: Anchor selection.

Original: In Tables 7 and 8, we analyze different anchor selection strategies under an experimental setting analogous to the one described in Section 4.1:
Revised: In Tables 7 and 8, we analyze different anchor selection strategies under an experimental setting analogous to the one described in Section 4.1:

Original: [Figure]
Revised: [Figure]

Original: • uniform The first selection strategy is the same adopted in the main manuscript.
Revised: • uniform The first selection strategy is the same adopted in the main manuscript.

Original: We randomly select the anchors with a uniform probability distribution over all the available samples; • fps We select the anchors according to a farthest point sampling strategy; • kmeans We select the anchors as the words more close to the centroids of K-means clustering with K = number of anchors; • top { k } We select the anchors as the k most frequent words, after skipping the firstwhich are mostly stopwords.
Revised: We randomly select the anchors with a uniform probability distribution over all the available samples; • fps We select the anchors according to a farthest point sampling strategy; • kmeans We select the anchors as the words more close to the centroids of K-means clustering with K = number of anchors; • top { k } We select the anchors as the k most frequent words, after skipping the firstwhich are mostly stopwords.

Original: We expect strategies that better cover the absolute space with anchors to be the most effective ones.
Revised: We expect strategies that better cover the absolute space with anchors to be the most effective ones.

Original: Indeed, the results are comparable across selection strategies, but fps reaches everywhere the best
Revised: Indeed, the results are comparable across selection strategies, but fps reaches everywhere the best

Original: Jaccard and MRR scores while k-means the best Cosine ones.
Revised: Jaccard and MRR scores while k-means the best Cosine ones.

Original: We attribute this behavior to their different nature: they both rely on the geometry of the latent spaces they are applied to, but kmeans also favors high-density regions, and this can become a negative bias for the task at hand.
Revised: We attribute this behavior to their different nature: they both rely on the geometry of the latent spaces they are applied to, but kmeans also favors high-density regions, and this can become a negative bias for the task at hand.

Original: In general, the uniform sampling is the most straightforward to apply, since it does not require additional computation for the selection process, and still achieves good performances.
Revised: In general, the uniform sampling is the most straightforward to apply, since it does not require additional computation for the selection process, and still achieves good performances.

Original: A.Q UASI -I SOMETRY I NVARIANCE WITH GUARANTEED BOUNDS
Revised: A. I NVARIANCE WITH GUARANTEED BOUNDS

Original: In this section, we explore a slightly modified version of the similarity function adopted in the main paper.
Revised: In this section, we explore a slightly modified version of the similarity function adopted in the main paper.

Original: The experimental setting is the same as in Section 4.1.
Revised: The experimental setting is the same as in Section 4.1.

Original: We want to measure the similarity between pairs of absolute embeddings and their relative counterparts.
Revised: We want to measure the similarity between pairs of absolute embeddings and their relative counterparts.

Original: To get some kind of quantitative measure, we add a similarity score calculated as the pairwise cosine distance between the two embedding types, averaged.
Revised: To get some kind of quantitative measure, we add a similarity score calculated as the pairwise cosine distance between the two embedding types, averaged.

Original: Therefore, a lower score indicates the spaces are more similar.
Revised: Therefore, a lower score indicates the spaces are more similar.

Original: On top of the standard relative representations, the ones computed with sim = S C , here we try to improve the similarity measure with guaranteed robustness to quasi-isometries up to a desired distortion bound.
Revised: On top of the standard relative representations, the ones computed with sim = S C , here we try to improve the similarity measure with guaranteed robustness to bounded distortion.

Original: In Figure 7 we report preliminary results that adopt this technique: a vector-quantized similarity function produces relative representations which are more similar (they have a lower score).
Revised: In Figure 7 we report preliminary results that adopt this technique: a vector-quantized similarity function produces relative representations which are more similar (they have a lower score).

Original: The vector-quantization is done through agglomerative clustering on the absolute embeddings at various thresholds t .
Revised: The vector-quantization is done through agglomerative clustering on the absolute embeddings at various thresholds t .

Original: We leave to future works the study of the trade-off betweena guaranteedbound on the near-isometries invariance  and the expressiveness of the resulting representations.
Revised: We leave to future works the study of the trade-off between guaranteed invariance to arbitrary bounded distorsion and the expressiveness of the resulting representations.

Original: A. D ATASET I NFORMATION
Revised: A. D ATASET I NFORMATION

Original: In Table 9 we summarize the datasets utilized in our work, and for each one, we specify the number of classes, to give an idea about the classification difficulty.
Revised: In Table 9 we summarize the datasets utilized in our work, and for each one, we specify the number of classes, to give an idea about the classification difficulty.

Original: FastText
Revised: FastText

Original: Word2Vec
Revised: Word2Vec

Original: Abs.
Revised: Abs.

Original: Rel. Rel.
Revised: Rel. Rel.

Original: Qnt.
Revised: Qnt.

Original: t =
Revised: t =

Original: Rel.
Revised: Rel.

Original: Qnt.
Revised: Qnt.

Original: t = 1 .
Revised: t = 1 .

Original: Rel.
Revised: Rel.

Original: Qnt.
Revised: Qnt.

Original: t =
Revised: t =

Original: Score ↓ 3 .1 .1 .1 .1 .
Revised: Score ↓ 3 .1 .1 .1 .1 .

Original: A. I MPLEMENTATION D ETAILS
Revised: A. I MPLEMENTATION D ETAILS

Original: In this Section, following the corresponing sections in the main paper, we report implementation details for all the experimental settings considered.
Revised: In this Section, following the corresponing sections in the main paper, we report implementation details for all the experimental settings considered.

Original: Tools & Technologies
Revised: Tools & Technologies

Original: In all the experiments presented in this work, the following tools were used:
Revised: In all the experiments presented in this work, the following tools were used:

Original: •  PyTorch Lightning , to ensure reproducible results while also getting a clean and modular codebase; • Weights and Biases , to log experiments and compare runs across huge sweeps; • Transformers by HuggingFace , to get ready-to-use transformers for both text and images; • Datasets by HuggingFace , to access most of the NLP datasets and ImageNet for CV; • DVC , for data versioning; • PyTorch Geometric , to handle graph datasets and get ready-to-use GNN architectures.
Revised: • NN-Template GrokAI (2021), to easily bootstrap the project and enforce best practices; • PyTorch Lightning (Falcon & The PyTorch Lightning team, 2019), to ensure reproducible results while also getting a clean and modular codebase; • Weights and Biases (Biewald, 2020), to log experiments and compare runs across huge sweeps; • Transformers by HuggingFace (Wolf et al., 2020), to get ready-to-use transformers for both text and images; • Datasets by HuggingFace (Lhoest et al., 2021), to access most of the NLP datasets and ImageNet for CV; • DVC (Kuprieiev et al., 2023), for data versioning; • PyTorch Geometric (Fey & Lenssen, 2019), to handle graph datasets and get ready-to-use GNN architectures.

Original: A.5. W ORD E MBEDDINGS
Revised: A.5. W ORD E MBEDDINGS

Original: For both the Figure and the Table in Section 4.1, the number of anchors is set to 300 for a fair comparison with the dimensionality of the original spaces.
Revised: For both the Figure and the Table in Section 4.1, the number of anchors is set to 300 for a fair comparison with the dimensionality of the original spaces.

Original: For visualization purposes, we needed the figure to both show an easy clusterable and restricted set of word embeddings.
Revised: For visualization purposes, we needed the figure to both show an easy clusterable and restricted set of word embeddings.

Original: They are obtained by subsampling the shared vocabulary with the following procedure: we select 4 random pivot words, and for each of them we consider the top-200 words in their neighborhood.
Revised: They are obtained by subsampling the shared vocabulary with the following procedure: we select 4 random pivot words, and for each of them we consider the top-200 words in their neighborhood.

Original: This results in a total of 800 points divided in 4 clusters, the ones used only for the visualization part.
Revised: This results in a total of 800 points divided in 4 clusters, the ones used only for the visualization part.

Original: For the quantitative part (table results), we select 20K random words from the shared vocabulary with a fixed seed for reproducibility purposes.
Revised: For the quantitative part (table results), we select 20K random words from the shared vocabulary with a fixed seed for reproducibility purposes.

Original: For the computer vision counterpart (Figure 10 and table 8), the procedure is similar but with the following differences: i) the number of anchors is set to 500 to balance between the different encoding dimensions of the two transformers (384 for ViT-small and 768 for ViT-base); ii) the subsampling for visualization purposes is done by selecting 4 classes and randomly picking 200 samples for each of them;
Revised: For the computer vision counterpart (Figure 11 and table 8), the procedure is similar but with the following differences: i) the number of anchors is set to 500 to balance between the different encoding dimensions of the two transformers (384 for ViT-small and 768 for ViT-base); ii) the subsampling for visualization purposes is done by selecting 4 classes and randomly picking 200 samples for each of them;

Original: Evaluation metrics Consider the set of ≈ 20k samples S (words for the NLP test, images for the CV one) and the source space X and target space Y and any sample s ∈ S , we compute its representation in X and Y through the functions f X : S → X and f Y :
Revised: Evaluation metrics Consider the set of ≈ 20k samples S (words for the NLP test, images for the CV one) and the source space X and target space Y and any sample s ∈ S , we compute its representation in X and Y through the functions f X : S → X and f Y :

Original: S
Revised: S

Original: → Y and define the metrics as follows:
Revised: → Y and define the metrics as follows:

Original: [Equation]
Revised: [Equation]

Original: where KNN A k ( v ) is a function that returns the k -top similar samples (according to cosine similarity) to v in the space A , and Rank A ( v , u ) is a function that returns the index at which u is found in the ordered KNN A k ( v ) .
Revised: where KNN A k ( v ) is a function that returns the k -top similar samples (according to cosine similarity) to v in the space A , and Rank A ( v , u ) is a function that returns the index at which u is found in the ordered KNN A k ( v ) .

Original: The final score for each metric is the mean over each s ∈ S .
Revised: The final score for each metric is the mean over each s ∈ S .

Original: A.5. L ATENT DISTANCE AS A PERFORMANCE PROXY
Revised: A.5. R ELATIVE REPRESENTATION SPACE CORRELATIONS

Revised: In this section, we analyze how similarities in absolute and relative spaces are correlated.

Revised: Let us consider two spaces alignable in the relative space.

Revised: We denote elements of the spaces with A ∈ R m 1 × n 1 and B ∈ R m 2 × n 2 and corresponding relative embeddings with C ∈ R m 1 × d , D ∈ R m 2 × d .

Revised: Examples of A and B can be the FastText and Word2Vec word embedding spaces.

Revised: We already observed in Table 1 how the spaces A

Revised: and B are well aligned in the relative space.

Revised: We can go further and analyze how self similarities in each space are preserved by the relative transform.

Revised: In

Revised: Figure 8, we show that relative representations not only facilitate latent space communication, but also preserve the underlying (absolute) latent space metric up to a certain degree.

Revised: A.5. L ATENT DISTANCE AS A PERFORMANCE PROXY

Original: The hyperperameters used in Section 4.2 are summarized in Table 10.
Revised: The hyperperameters used in Section 4.2 are summarized in Table 10.

Original: A.5. T RAINING WITH A BSOLUTE VS . R ELATIVE R EPRESENTATIONS
Revised: A.5. T RAINING WITH A BSOLUTE VS . R ELATIVE R EPRESENTATIONS

Original: The models trained on relative representations do not backpropagate through the anchors, which encourages a smoother optimization of the anchors’ representations.
Revised: The models trained on relative representations do not backpropagate through the anchors, which encourages a smoother optimization of the anchors’ representations.

Original: Image Classification
Revised: Image Classification

Original: The architecture is a standard deep CNN.
Revised: The architecture is a standard deep CNN.

Original: We run a sweep for each dataset where we vary only the random seed (over 10 possible in total).
Revised: We run a sweep for each dataset where we vary only the random seed (over 10 possible in total).

Original: We then aggregate by dataset and encoding type to obtain the final results with their standard deviation.
Revised: We then aggregate by dataset and encoding type to obtain the final results with their standard deviation.

Original: Graph Classification We run a sweep identical to the one in Table 10 for the reference model, except that we sweep on the “Number of layers” with two values: 32 and 64.
Revised: Graph Classification We run a sweep identical to the one in Table 10 for the reference model, except that we sweep on the “Number of layers” with two values: 32 and 64.

Original: Each configuration is repeated with 10 different seeds, then we aggregate by dataset and encoding type to obtain the final results with their standard deviation.
Revised: Each configuration is repeated with 10 different seeds, then we aggregate by dataset and encoding type to obtain the final results with their standard deviation.

Original: A.5. I MAGE R ECONSTRUCTION
Revised: A.5. I MAGE R ECONSTRUCTION

Original: The relative and absolute models appearing in Figure 4 are vanilla AEs and VAEs, the same for all the datasets, and have a comparable number of trainable parameters.
Revised: The relative and absolute models appearing in Figure 4 are vanilla AEs and VAEs, the same for all the datasets, and have a comparable number of trainable parameters.

Original: Their architecture is composed by simple convolutions, deconvolutions and mean squared error as reconstruction loss.
Revised: Their architecture is composed by simple convolutions, deconvolutions and mean squared error as reconstruction loss.

Original: The number of anchors is 500 and the latent dimensionality of the absolute representations is 500 .
Revised: The number of anchors is 500 and the latent dimensionality of the absolute representations is 500 .

Original: A.5. T EXT C LASSIFICATION
Revised: A.5. T EXT C LASSIFICATION

Original: We report in Tables 11 to 13 details on the transformers and anchors adopted in Section 5.2.
Revised: We report in Tables 11 to 13 details on the transformers and anchors adopted in Section 5.2.

Original: Preprocessing Following the original work in which the Amazon Reviews dataset was proposed (Keung et al., 2020), we utilize both the title and body of each review.
Revised: Preprocessing Following the original work in which the Amazon Reviews dataset was proposed (Keung et al., 2020), we utilize both the title and body of each review.

Original: We differ in not using the category and in how we merge them; namely, we add the title as prefix for the body and add a full stop as separator when needed (avoiding duplicates).
Revised: We differ in not using the category and in how we merge them; namely, we add the title as prefix for the body and add a full stop as separator when needed (avoiding duplicates).

Original: To obtain a single latent encoding for each sample, with fixed shape, we take the last hidden state and select the representation corresponding to the [CLS] token.
Revised: To obtain a single latent encoding for each sample, with fixed shape, we take the last hidden state and select the representation corresponding to the [CLS] token.

Original: Wikipedia anchors We use WikiMatrix, a corpus of sentences extracted from Wikipedia.
Revised: Wikipedia anchors We use WikiMatrix, a corpus of sentences extracted from Wikipedia.

Original: The sentences are parallel between pairs of languages (i.e., same sentences translated in two languages), and since we are looking for a collection of parallel anchors between all 4 languages, we decided to use the English language as a pivot to compute the intersection.
Revised: The sentences are parallel between pairs of languages (i.e., same sentences translated in two languages), and since we are looking for a collection of parallel anchors between all 4 languages, we decided to use the English language as a pivot to compute the intersection.

Original: To get the final results, we considered only the sentences with margin score ≥ 1 . 06 , getting high-quality sentence alignments.
Revised: To get the final results, we considered only the sentences with margin score ≥ 1 . 06 , getting high-quality sentence alignments.

Original: In Table 13 we show the total number of parallel sentences when computing the intersections.
Revised: In Table 13 we show the total number of parallel sentences when computing the intersections.

Original: We randomly selected 768 samples to use as anchors.
Revised: We randomly selected 768 samples to use as anchors.

Original: A.5. I MAGE C LASSIFICATION
Revised: A.5. I MAGE C LASSIFICATION

Original: The details of the transformers used in Section 5.3 are summarized in Table 14.
Revised: The details of the transformers used in Section 5.3 are summarized in Table 14.

Original: A. A DDITIONAL RESULTS
Revised: A. A DDITIONAL RESULTS

Original: In this section we report additional results on the correlation between latent similarity and performance in Figure 8, results on the multilingual stitching both with Amazon coarse-grained in Table 15 and fine-grained in Table 16, results on the image classification stitching on CIFARfine-grained in Table 18.
Revised: In this section we report additional results on the correlation between latent similarity and performance in Figure 9, results on the multilingual stitching both with Amazon coarse-grained in Table 15 and fine-grained in Table 16, results on the image classification stitching on CIFARfine-grained in Table 18.

Original: Moreover, we evaluate the stitching performance of a multilingual transformer in Table 17.FastText Word2Vec A b s o l u t e
Revised: Moreover, we evaluate the stitching performance of a multilingual transformer in Table 17.

Original: R e l a ti v e

Original: FastText

Original: Word2Vec

Original: [Figure]
Revised: [Figure]

Original: l

Original: [Figure]

Original: ti v e
Revised: ti v e

Original: [Figure]
Revised: [Figure]

Original: [Table]

Original: [Table]
Revised: [Table]

Original:  o p
Revised: t o p

Original: [Table]
Revised: [Table]

Original: t o p
Revised: t o p

Original: [Table]
Revised: [Table]

Original:  [Table]
Revised: u [Table] [Table] [Table]

Revised: CiteSeer

Original: [Table]
Revised: [Table]

Revised: [Table]

Revised: ) — 50 (fine)

Revised: [Table]

Revised: [Table]

Original: 2 (coarse) — 5 (
Revised: 2 (coarse) — 5 (fine)

Original: [Table]
Revised: [Table]

Original: [Table] [Table] 
Revised: [Table] [Table] [Table]

Original: [Table] [Table] 
Revised: [Table] [Table] [Table]

Original: [Table] [Table] 
Revised: [Table] [Table] [Figure]

Revised: u

Original: [Table]
Revised: [Figure]

Revised: a

Original: [Table]
Revised: [Figure]

Revised: v

Original: [Table]
Revised: [Figure]