Original: Design of the topology for contrastive visual-textual alignment
Revised: ESIGN OF THE TOPOLOGY FOR CONTRASTIVE VISUAL - TEXTUAL ALIGNMENT

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: Abstract
Revised: A BSTRACT

Original: Pre-training weakly related image-text pairs in the contrastive style shows great power in learning semantic aligning cross-modal models.
Revised: Pre-training weakly related image-text pairs in the contrastive style shows great power in learning semantic aligning cross-modal models.

Original: The common choice to measure the distance between the feature representations of the image-text pairs is the cosine similarity, which can be considered as the negative inner product distance of features embedded on a sphere, mathematically.
Revised: The common choice to measure the distance between the feature representations of the image-text pairs is the cosine similarity, which can be considered as the negative inner product distance of features embedded on a sphere, mathematically.

Original: However, empirically, aligning image-text pairs on the spherical topology is vulnerable to the semantic ambiguity phenomenon resulting from the noise in the pre-training datasets.
Revised: However, empirically, aligning image-text pairs on the spherical topology is vulnerable to the semantic ambiguity phenomenon resulting from the noise in the pre-training datasets.

Original: Specifically, under the noisy training data, instead of the optimal alignment-uniformity solution, the system would achieve an equilibrium (a gap between distances of positive and negative pairs), when the gradients for attraction and repulsion are neutralized.
Revised: Specifically, under the noisy training data, instead of the optimal alignmentuniformity solution, the system would achieve an equilibrium (a gap between distances of positive and negative pairs), when the gradients for attraction and repulsion are neutralized.

Original: Although intuitively, the model should always find this equilibrium given a suﬀiciently long training scheme, its numerical values  might be out of the distance range ( e.g. [-1, 1] for the cosine similarity).
Revised: Although intuitively, the model should always find this equilibrium given a sufficiently long training scheme. However, since the contrastive loss is implemented using softmax and cross-entropy loss, which makes the required numerical values for equilibrium much larger, where the equilibrium might be out of the distance range ( e.g. [-1, 1] for the cosine similarity).

Original: In the practice of former studies, this problem is partly tackled by introducing a learnable softmax temperature parameter, in other words, by explicitly scaling the range of the distance function.
Revised: In the practice of former studies, this problem is partly tackled by introducing a learnable softmax temperature parameter, in other words, by explicitly scaling the range of the distance function.

Original: In this work, we alternatively design the topology of embedding space and its endowed distance function.
Revised: In this work, we alternatively design the topology of embedding space and its endowed distance function.

Original: Motivated by studies that make use of Riemannian geometry for visual tasks, we propose a rather simple solution to address the aforementioned equilibrium problem.
Revised: Motivated by studies that make use of Riemannian geometry for visual tasks, we propose a rather simple solution to address the aforementioned equilibrium problem.

Original: That is, we map the feature representations onto the oblique manifold endowed with the negative inner product as the distance function.
Revised: That is, we map the feature representations onto the oblique manifold endowed with the negative inner product as the distance function.

Original: In the experimental analysis, we show that we can improve the baseline performance by a large margin ( e.g. 4% in the zero-shot image to text retrieval task) by changing only two lines of the training codes.
Revised: Furthermore, we propose a multi-token implementation of the oblique manifold. With this configuration, we outperform the officially released CLIP-ViT/L-14 model using a ViT/B-16 visual backbone on the zero-shot image classification and image-to-text retrieval tasks.

Original:  Introduction
Revised:  I NTRODUCTION

Original: Learning visual and textual feature representations that are semantically aligned in their embedding space is an ordinary problem in the vision-language cross-modal tasks (Frome et al., 2013; Karpathy & Fei-Fei, 2015; Romera-Paredes & Torr, 2015; Wang et al., 2016; Faghri et al., 2017; Xian et al., 2016).
Revised: Learning visual and textual feature representations that are semantically aligned in their embedding space is an ordinary problem in the vision-language cross-modal tasks (Frome et al., 2013; Karpathy & Fei-Fei, 2015; Romera-Paredes & Torr, 2015; Wang et al., 2016; Faghri et al., 2017; Xian et al., 2016).

Original: In early works that employ feature representations from deep neural networks, e.g. Frome et al.
Revised: In early works that employ feature representations from deep neural networks, e.g. Frome et al.

Original: , the alignment is often achieved by a fundamental metric learning approach with the hinge rank loss.
Revised: , the alignment is often achieved by a fundamental metric learning approach with the hinge rank loss.

Original: That is, the similarity between a visual feature vector u and a textual feature vector v is calculated as u T W v , where W are the learnable weight parameters.
Revised: That is, the similarity between a visual feature vector u and a textual feature vector v is calculated as u T W v , where W are the learnable weight parameters.

Original: Thanks to the revolutionary advances in computational power, we can now achieve this in a more effective and practical approach termed contrastive learning, where we align quantities of positive samples and push their negative samples away simultaneously in a large mini-batch (Radford et al., 2021; Singh et al., 2022;Jia et al., 2021; Pham et al., 2021;
Revised: Thanks to the revolutionary advances in computational power, we can now achieve this in a more effective and practical approach termed contrastive learning, where we align quantities of positive samples and push their negative samples away simultaneously in a large mini-batch (Radford et al., 2021; Singh et al., 2022;

Original:  Yuan et al., 2021).
Revised: Jia et al., 2021; Pham et al., 2021; Yuan et al., 2021).

Original: The standard choice of the distance measure between an image-text pair for the contrastive learning algorithm is the Cosine Similarity (in both uni-modal Chen et al.
Revised: The standard choice of the distance measure between an image-text pair for the contrastive learning algorithm is the Cosine Similarity (in both uni-modal Chen et al.

Original: (2020a)
Revised: (2020a)

Original: ; Caron et al.
Revised: ; Caron et al.

Original: ; Chen et al.
Revised: ; Chen et al.

Original: (2020b)
Revised: (2020b)

Original: and cross-modal Radford et al.
Revised: and cross-modal Radford et al.

Original: ; Jia et al.
Revised: ; Jia et al.

Original: ; Singh et al.
Revised: ; Singh et al.

Original: (2022) scenarios).
Revised: (2022) scenarios).

Original: Mathematically, the Cosine Similarity computes the negative inner product value between feature representation vectors mapped onto the unit sphere embedding space.
Revised: Mathematically, the Cosine Similarity computes the negative inner product value between feature representation vectors mapped onto the unit sphere embedding space.

Original: The spherical embedding space is advantageous in aligning visual and textual feature rep- resentations in the following two aspects.
Revised: The spherical embedding space is advantageous in aligning visual and textual feature representations in the following two aspects.

Original: First, calculating the inner product consumes low computational resources during both forward and backward propagation.
Revised: First, calculat- ing the inner product consumes low computational resources during both forward and backward propagation.

Original: Second, we have a proper definition of uniformity on the sphere.
Revised: Second, we have a proper definition of uniformity on the sphere.

Original: The uniformity is a demanded property in optimizing the contrastive loss, by which the feature representations would preserve maximal information of the data.
Revised: The uniformity is a demanded property in optimizing the contrastive loss, by which the feature representations would preserve maximal information of the data.

Original: However, since the data for large-scale contrastive alignment are internet-collected noisy image-text pairs, we often find pairs of semantically related images and texts labeled as “negative” and verse visa, which we term “semantic ambiguity”.
Revised: However, since the data for large-scale contrastive alignment are internet-collected noisy image-text pairs, we often find pairs of semantically related images and texts labeled as “negative” and verse visa, which we term “semantic ambiguity”.

Original: Because of the ambiguity, it is impossible to achieve the perfect alignment and uniformity conditions of sample embeddings for the system.
Revised: Because of the ambiguity, it is impossible to achieve the perfect alignment and uniformity conditions of sample embeddings for the system.

Original: More specifically, during the training, the false negative samples are pushed away from each other (repulsion), while the false positive samples are pulled together (attraction).
Revised: More specifically, during the training, the false negative samples are pushed away from each other (repulsion), while the false positive samples are pulled together (attraction).

Original: As a consequence, the system will gradually find an equilibrium when the noisy samples’ gradients for attraction and repulsion are neutralized.
Revised: As a consequence, the system will gradually find an equilibrium when the noisy samples’ gradients for attraction and repulsion are neutralized.

Original: In other words, we say the training progress is converged under the given hyper-parameters.
Revised: In other words, we say the training progress is converged under the given hyper-parameters.

Original: To be more concretely, owing to the fact that the gradient is eventually back-propagated from the difference between the positive and negative distances.
Revised: To be more concrete, owing to the fact that the gradient is eventually back-propagated from the difference between the positive and negative distances.

Original: Given suﬀicient model capacity, the numerical values between the distances of positive and negative pairs of samples will be optimized to fit the noisy level of the dataset.
Revised: Given sufficient model capacity, the numerical values between the distances of positive and negative pairs of samples will be optimized to fit the noisy level of the dataset.

Original: For instance, if there is a reasonable amount of false negative samples, then the model would learn a smaller negative distance for not being punished too hard when encountering false negative samples.
Revised: For instance, if there is a reasonable amount of false negative samples, then the model would learn a smaller negative distance for not being punished too hard, when encountering false negative samples in another mini-batch.

Original: Furthermore, the triangular inequality (or a “relaxed” version) of the distance within a group of semantically similar samples will pull the positive pairs of samples away from each other (See Section 3.2).
Revised: Furthermore, the triangular inequality (or a “relaxed” version) of the distance within a group of semantically similar samples will pull the positive pairs of samples away from each other (See Section 3.2).

Original: Finally, the model reaches the equilibrium of compromised positive and negative distances, which minimizes the contrastive loss regardless of the semantic ambiguity.
Revised: Finally, the model reaches the equilibrium of compromised positive and negative distances, which minimizes the contrastive loss regardless of the semantic ambiguity.

Original: Therefore, we say the equilibrium is essentially a property of the dataset, referring to its noise level, under a certain embedding space and its distance function.
Revised: Therefore, we say the equilibrium is essentially a property of the dataset, referring to its noise level, under a certain embedding space and its distance function.

Revised: Here, the problem is that, since we implement the contrastive loss with the combination of softmax and cross-entropy, which makes the required numerical values for equilibrium exponentially larger.

Original: Here, the problem is that the numerical values of distances  at equilibrium might be out of the distance range ( e.g. [-1, 1] for the cosine similarity).
Revised: The numerical values of distances required by softmax at equilibrium might be out of the distance range ( e.g. [-1, 1] for the cosine similarity).

Original: Therefore, in the practice of former studies (Wu et al., 2018; Radford et al., 2021), researchers have to scale the range with a learnable softmax temperature parameter.
Revised: Therefore, in the practice of former studies (Wu et al., 2018; Radford et al., 2021), researchers have to scale the range with a learnable softmax temperature parameter.

Original: Although the learnable temperature somehow fixes the range problem, it still has two drawbacks that need to be addressed.
Revised: Although the learnable temperature somehow fixes the range problem, it still has two drawbacks that need to be addressed.

Original: Firstly, the learnable temperature delays the learning progress.
Revised: Firstly, the learnable temperature delays the learning progress.

Original: Empirically, we observe that the model trends to acquire a proper scaling for the distance range earlier than achieving a good alignment.
Revised: Empirically, we observe that the model trends to acquire a proper scaling for the distance range earlier than achieving a good alignment.

Original: Secondly, a large temperature is numerically unstable for the backpropagation, especially for the low-bit precision computation.
Revised: Secondly, a large temperature is numerically unstable for the back-propagation, especially for the low-bit precision computation.

Original: In this work, we alternatively design the topology for embedding vectors and its endowed distance function.
Revised: In this work, we alternatively design the topology for embedding vectors and its endowed distance function.

Original: Motivated by the utilization of Riemannian geometry for visual tasks, we propose a relatively simple solution to address the aforementioned out-of-range equilibrium problem.
Revised: Motivated by the utilization of Riemannian geometry for visual tasks, we propose a relatively simple solution to address the aforementioned out-of-range equilibrium problem.

Original: Our contributions can be summarized as follows:
Revised: Our contributions can be summarized as follows:

Original: We reveal that the learnable softmax temperature is essentially a scaling factor for the distance range, which indicates the noise level of the dataset in the contrastive visual-textual alignment.
Revised: We reveal that the learnable softmax temperature is essentially a scaling factor for the distance range, which indicates the noise level of the dataset in the contrastive visual-textual alignment.

Revised: We also observe that the model learns a suitable temperature before representations, which degrades the performance.

Original: We tackle the out-of-range equilibrium problem  by designing the topology of embedding space, that is, we employ the oblique manifold endowed with the negative inner product as distance functions.
Revised: We tackle the out-of-range equilibrium problem resulting from the softmax cross-entropy loss, by designing the topology of embedding space. That is, we employ the oblique manifold endowed with the negative inner product as distance functions.

Original: We demonstrate that the proposed approach can be non-painfully implemented by changing only two lines of the training code, whilst improving the baseline performance in the zero-shot image-text retrieval tasks.
Revised: We demonstrate that the proposed approach can be non-painfully implemented by changing only two lines of the training code, whilst improving the baseline performance in the zeroshot image-text retrieval tasks.

Revised: In the larger scale experiment, we have learned a ViT-B/ model that outperforms the officially released ViT-L/14 model.

Original:  Preliminary
Revised:  P RELIMINARY

Original: Notations: We start with notation and review mathematical expressions of the basic building blocks used in our analysis.
Revised: Notations: We start with notation and review mathematical expressions of the basic building blocks used in our analysis.

Original: In this work, we denote scalars by italic letters, e.g. , n, m, B, D ∈ R , and denote vectors and higher-order tensors by boldface letters, e.g., x = [ x 0 , x 1 , . . .
Revised: In this work, we denote scalars by italic letters, e.g. , n, m, B, D ∈ R , and denote vectors and higher-order tensors by boldface letters, e.g., x = [ x 0 , x 1 , . . .

Original: , x n − 1 ] ⊤ ∈ R n and Y ∈ R N × D .
Revised: , x n − 1 ] ⊤ ∈ R n and Y ∈ R N × D .

Original: We denote sets by calligraphic letters, e.g. , U =
Revised: We denote sets by calligraphic letters, e.g. , U =

Original: { U 1 , U 2 , . . . } .
Revised: { U 1 , U 2 , . . . } .

Original: We also employ italic letters to define functions, with subscripts denote their parameters, e.g. , f θ ( · ) .
Revised: We also employ italic letters to define functions, with subscripts denote their parameters, e.g. , f θ ( · ) .

Original: The operation ∥ · ∥ p denotes the ℓ p norm of a vector and | · | denotes the absolute value of a scalar.
Revised: The operation ∥ · ∥ p denotes the ℓ p norm of a vector and | · | denotes the absolute value of a scalar.

Original: For any integer K , we use [ K ] to denote the set of integers from 1 to K .
Revised: For any integer K , we use [ K ] to denote the set of integers from 1 to K .

Original: Visual-Textual Pre-trained Model:
Revised: Visual-Textual Pre-trained Model:

Original: Given a set of semantically related image-text pairs S
Revised: Given a set of semantically related image-text pairs S

Original: = { ( U 1 , V 1 ) , ( U 2 , V 2 ) , . . .
Revised: = { ( U 1 , V 1 ) , ( U 2 , V 2 ) , . . .

Original: , ( U K , V K ) } , where U is an image of size H × W × C , V is a tokenized text of length L .
Revised: , ( U K , V K ) } , where U is an image of size H × W × C , V is a tokenized text of length L .

Original: The goal is to simultaneously learn a pair of encoders f θ : U → u , g ϕ : V → v to map the image and text into an embedding space, u , v are called embedding vectors of samples.
Revised: The goal is to simultaneously learn a pair of encoders f θ : U → u , g ϕ : V → v to map the image and text into an embedding space, u , v are called embedding vectors of samples.

Original: A well-optimized visual-textual pre-trained model aligns the embedding vectors across the visual and textual models.
Revised: A well-optimized visual-textual pre-trained model aligns the embedding vectors across the visual and textual models.

Original: That is, the embedding vectors extracted semantically related image-text pairs earn higher similarity scores than the non-related ones.
Revised: That is, the embedding vectors extracted semantically related image-text pairs earn higher similarity scores than the non-related ones.

Original: To generalize the problem, we view the embedding vectors as points on specified typologies, and the similarity score between embedding vectors is an endowed distance function that evaluates the distance between the points.
Revised: To generalize the problem, we view the embedding vectors as points on specified typologies, and the similarity score between embedding vectors is an endowed distance function that evaluates the distance between the points.

Original: For instance, the commonly employed cosine similarity calculates the inner product of the normalized embedding vectors on the unit sphere as the (negative) distance between the sample pairs.
Revised: For instance, the commonly employed cosine similarity calculates the inner product of the normalized embedding vectors on the unit sphere as the (negative) distance between the sample pairs.

Original: To this end, we further consider the encoders as compositions of functions that i) map the inputs into the euclidean space and ii) map the input vectors in euclidean space on specified typologies.
Revised: To this end, we further consider the encoders as compositions of functions that i) map the inputs into the euclidean space and ii) map the input vectors in euclidean space on specified typologies.

Original: We denote this twostep mapping as: f θ = ¯ f θ · f , where ¯ f θ : U → ¯ u is the encoder with learnable parameters, ¯ u ∈ R d is the output in the d -dimensional euclidean space.
Revised: We denote this two-step mapping as: f θ = ¯ f θ · f , where ¯ f θ : U → ¯ u is the encoder with learnable parameters, ¯ u ∈ R d is the output in the d -dimensional euclidean space.

Original: f : ¯ u → u , u ∈ M is an specified operator (without learnable parameters) maps the representations onto a topology M , which is usually considered as a manifold embedded in the d -dimensional euclidean space.
Revised: f : ¯ u → u , u ∈ M is an specified operator (without learnable parameters) maps the representations onto a topology M , which is usually considered as a manifold embedded in the d -dimensional euclidean space.

Original: Contrastive Learning:
Revised: Contrastive Learning:

Original: Following the definition in Oord et al.
Revised: Following the definition in Oord et al.

Original: (2018); Wang & Isola (2020);
Revised: (2018); Wang & Isola (2020);

Original: Chen et al.
Revised: Chen et al.

Original: (2021a)
Revised: (2021a)

Original: ; Radford et al.
Revised: ; Radford et al.

Original: (2021), we formulate the contrastive loss as
Revised: (2021), we formulate the contrastive loss as

Original: L c ( f θ , g ϕ ; τ, S ) :=
Revised: L c ( f θ , g ϕ ; τ, S ) :

Original: [Equation]
Revised: [Equation]

Original: where τ is the temperature term, we write it as a multiplier for simplicity.
Revised: where τ is the temperature term, we write it as a multiplier for simplicity.

Original: d ( · , · ) is the distance function between two points.
Revised: d ( · , · ) is the distance function between two points.

Original: M ∈ Z + is a fixed number of negative samples.
Revised: M ∈ Z + is a fixed number of negative samples.

Original: Briefly, optimizing this loss term minimizes the distance between positive image-text pairs and maximizes the distance between negative image-text pairs.
Revised: Briefly, optimizing this loss term minimizes the distance between positive image-text pairs and maximizes the distance between negative image-text pairs.

Original: It is worth mentioning that, in recent studies Radford et al.
Revised: It is worth mentioning that, in recent studies Radford et al.

Original: ; Chen et al.
Revised: ; Chen et al.

Original: (2021b), the contrastive loss is usually implemented as the cross-entropy between one-hot labels and the class probability obtained by softmax within a mini-batch S M .
Revised: (2021b), the contrastive loss is usually implemented as the cross-entropy between one-hot labels and the class probability obtained by softmax within a mini-batch S M .

Original: We also employ this implementation in this work, which can be formulated as
Revised: We also employ this implementation in this work, which can be formulated as

Original: [Equation]
Revised: [Equation]

Original: where U i = {− τd ( f θ (  U i ) , g ϕ ( V j )) } j ∈
Revised: where H ( ·|· ) is the cross-entropy loss, U i =

Original: [ M ] , V i = {− τd ( f θ ( U j ) , g ϕ
Revised: [Equation] = {− τd ( f θ ( U j ) , g ϕ

Original: ( V i ))
Revised: ( V i ))

Original: } j ∈
Revised: } j ∈

Original: [ M ] are the (negative) distance between an image/text to all the texts/images in the mini-batch.
Revised: [ M ] are the (negative) distance between an image/text to all the texts/images in the mini-batch.

Original: σ is the softmax function, q i is the one-hot label vectors of i .
Revised: σ is the softmax function, q i is the one-hot label vectors of i .

Original: Oblique Manifold: Briefly, the oblique manifold Ob ( n, n ) is the set of matrices of size n × m with unit-norm columns.
Revised: Oblique Manifold: Briefly, the oblique manifold Ob( n, n ) is the set of matrices of size n × m with unit-norm columns.

Original: We follow the definition in Absil et al.
Revised: We follow the definition in Absil et al.

Original: (2009), that is,
Revised: (2009), that is,

Original: [Equation]
Revised: [Equation]

Original: where the diag ( · ) is the diagonal entries of the matrix, I n is the identity matrix of size n .
Revised: where the diag( · ) is the diagonal entries of the matrix, I n is the identity matrix of size n .

Original: The geometry of an oblique manifold is exactly the same as that of the product manifold of spheres [Equation] | {z } m copies , where S n − 1 is the sphere manifold embedded in R n .[Figure]
Revised: The geometry of an oblique manifold is exactly the same as that of the product manifold of spheres S n − 1 × · · · × S n −(cid:124) (cid:123)(cid:122) (cid:125) m copies , where S n − 1 is the sphere manifold embedded in R n .

Original:  Method
Revised:  M ETHOD

Original: In this section, we introduce how we implement the proposed approach.
Revised: In this section, we introduce how we implement the proposed approach.

Original: That is, employing the oblique manifold as the embedding space, with the (negative) inner product as its distance function.
Revised: That is, employing the oblique manifold as the embedding space, with the (negative) inner product as its distance function.

Original: 3.1 Proposed Approach
Revised: # image_encoder - vision transformer # text_encoder - text transformer # U - mini-batch of aligned images, [n, h, w, c] # V - mini-batch of aligned texts, [n, l] # Ob_m - m of the oblique manifold # Ob_n - n of the oblique manifold # t - learned temperature parameter 8# extract features of each modality u_bar = image_encoder(U) #[n, d] v_bar = text_encoder(V) #[n, d] 12''' Lines 15-16 are the only codes that need to change compared to a standard CLIP algorithm. '''

Revised: # map features onto Ob(n,m) u

Revised: = u_bar.reshape('n d -> n Ob_m Ob_n').l2_normalize(axis=-1).reshape('-> n d') v = v_bar.reshape('n d -> n Ob_m Ob_n').l2_normalize(axis=-1).reshape('-> n d') 17# scaled pairwise inner product distance, [n, n] neg_distances = u @ v.t() * t.exp() 20# symmetric loss function

Original:  Basic Implementation: To map the feature vectors onto the oblique manifold Ob ( n, m ) , we first reshape the feature vectors ¯ u , ¯ v of size d to a matrix of shape m × n , then we ℓ 2 − normalize the columns to obtain u , v .
Revised: [Figure] [Figure] 3. P ROPOSED A PPROACH Basic Implementation: To map the feature vectors onto the oblique manifold Ob( n, m ) , we first reshape the feature vectors ¯ u , ¯ v of size d to a matrix of shape m × n , then we ℓ 2 − normalize the columns to obtain u , v .

Original: Finally, the distance is defined as the negative value of the trace of the matrix product, i.e. d ( u , v ) = − tr ( u T v ) .
Revised: Finally, the distance is defined as the negative value of the trace of the matrix product, i.e. d ( u , v ) = − tr( u T v ) .

Original: We provide the pythonstyle pseudo-code of the proposed approach in Figure 1.
Revised: We provide the python-style pseudo-code of the proposed approach in Figure 1.

Original: It is worth mentioning that, in the implementation, we vectorize the embedding vectors back to their original size d , then apply the vector inner product to compute the distance.
Revised: It is worth mentioning that, in the implementation, we vectorize the embedding vectors back to their original size d , then apply the vector inner product to compute the distance.

Original: Moreover, we employ the term “neg_distances” to avoid reduplicated calculation of the negative operation.
Revised: Moreover, we employ the term “neg distances” to avoid reduplicated calculation of the negative operation.

Original: Multi-Token Implementation: We provide another implementation of the proposed topology and distance functions by utilizing the [CLS] token  of the transformer architecture.
Revised: Multi-Token Implementation: We provide another implementation of the proposed topology and distance functions by utilizing the [CLS] token (the token used to reperent global information in ViT, see the implementation details in Dosovitskiy et al. (2020)) of the transformer architecture.

Original: Briefly, we attach multiple randomly initialized [CLS] tokens in the first embedding layers.
Revised: Briefly, we attach multiple randomly initialized [CLS] tokens in the first embedding layers.

Original: At the output layer, we consider each token embedding as a sub-sphere of the oblique manifold.
Revised: At the output layer, we consider each token embedding as a sub-sphere of the oblique manifold.

Original: We then compute the inner product of the vectorized visual and textual embeddings.
Revised: We then compute the inner product of the vectorized visual and textual embeddings.

Original: Unlike the standard implementation, the multi-token implementation requires more computational resources in the backbone.
Revised: Unlike the standard implementation, the multi-token implementation requires more computational resources in the backbone.

Original: On the contrary, the sub-sphere could benefit from the global attention operation and provide more representative feature embeddings.
Revised: On the contrary, the sub-sphere could benefit from the global attention operation and provide more representative feature embeddings.

Original: 3.2 Rethinking The Properties of Topology
Revised: 3. R ETHINKING T HE P ROPERTIES OF T OPOLOGY

Original: In this section, we discuss the motivation of our proposed approach in detail.
Revised: In this section, we discuss the motivation for our proposed approach in detail.

Original: We discuss the properties of the proposed approach by comparing it with three reference configurations of different topologies and distance functions.
Revised: Besides the former discussed distance range, we further discuss the properties of the proposed approach by comparing it with three reference configurations of different topologies and distance functions.

Original: Specifically, we consider i) the sphere S d −endowed with the inner product as distance, ii) the euclidean space R d endowed with ℓdistance; iii) the oblique manifold Ob ( d / m, m ) endowed with the minimizing geodesic as distance, which is denoted as Geo ( u , v ) = tr 1 2 ( arccos 2 ( u T v )) .
Revised: Specifically, we consider i) the sphere S d − 1 endowed with the inner product as distance, ii) the euclidean space R d endowed with ℓ 2 distance; iii) the oblique manifold Ob( d/m, m ) endowed with the minimizing geodesic as distance, which is denoted as Geo( u , v ) = tr 12 (arccos 2 ( u T v )) .

Original: The overview of the compar- [Table] ison is summarized in Table 1.
Revised: The overview of the comparison is summarized in Table 1.

Original: The comparison summarizes the three important properties of the embedding topology, which are the major motivation for our proposed approach.
Revised: The comparison summarizes the three important properties of the embedding topology in addition to the distance range. Keeping these properties favored for contrastive learning is important in the design of embedding topology. [Table]

Original:  i. Low computational resource: In the contrastive learning algorithm, the logits (or distance/similarity) matrix often costs the most computational resource in large-scale training.
Revised: [ − 1 , 1] [0 , + ∞ ) [0 , mπ ] [ − m, m ] i. Low computational resource: In the contrastive learning algorithm, the logits (or distance/similarity) matrix often costs the most computational resource in large-scale training.

Original: For instance, given a mini-batch of sample pairs of size b with d − dimensional output, the computation of the inner product achieves a complexity of O ( b 2 d ) and storage usage of O ( b 2 ) .
Revised: For instance, given a mini-batch of sample pairs of size b with d − dimensional output, the computation of the inner product achieves a complexity of O ( b 2 d ) and storage usage of O ( b 2 ) .

Original: However, since the back-propagation of the ℓ 2 − norm requires intermediate results that cannot be “inplace” calculated, the ℓ 2 − norm (or any ℓ p − norm based distance) in euclidean space requires a storage usage of O ( b 2 d ) .
Revised: However, since the back-propagation of the ℓ 2 − norm requires intermediate results that cannot be “inplace” calculated, the ℓ 2 − norm (or any ℓ p − norm based distance) in euclidean space requires a storage usage of O ( b 2 d ) .

Original: On the contrary, the geodesic distance cache m curve lengths and hence requires O ( b 2 m ) storage usage, while the proposed approach only needs one matrix multiplication after re-vectorization.
Revised: On the contrary, the geodesic distance cache m curve lengths and hence requires O ( b 2 m ) storage usage, while the proposed approach only needs one matrix multiplication after re-vectorization.

Original: ii.
Revised: ii.

Original: Proper definition of uniformity: As explained by Wang & Isola (2020), contrastive loss is a combination of two objects a) alignment of features from positive sample pairs and b) the distribution of the features encouraged to match.
Revised: Proper definition of uniformity: As explained by Wang & Isola (2020), contrastive loss is a combination of two objects a) alignment of features from positive sample pairs and b) the distribution of the features encouraged to match.

Original: Naturally, with the loss form defined in Equation (2), the distribution object will result in a uniform distribution on the sphere.
Revised: Naturally, with the loss form defined in Equation (2), the distribution object will result in a uniform distribution on the sphere.

Original: Although it is not essential for the distribution of samples to be uniform as discovered by Chen et al.
Revised: Although it is not essential for the distribution of samples to be uniform as discovered by Chen et al.

Original: (2021a), it is necessary to define a proper prior distribution for samples to match via optimal transport algorithms ( e.g. sliced Wasserstein distance), which is undoubtedly a computational burden.
Revised: (2021a), it is necessary to define a proper prior distribution for samples to match via optimal transport algorithms ( e.g. sliced Wasserstein distance), which is undoubtedly a computational burden.

Original: Both the sphere and oblique manifold have a proper uniform distribution defined as the surface area measure, while the unbounded euclidean space does not.
Revised: Both the sphere and oblique manifold have a proper uniform distribution defined as the surface area measure, while the unbounded euclidean space does not.

Original: In practice, the ℓ 2 − norm defined distance between the samples grows larger along training and eventually overflows.
Revised: In practice, the ℓ 2 − norm defined distance between the samples grows larger along training and eventually overflows.

Original: iii.
Revised: iii.

Original: “Relaxed” triangular inequality: Assume that we have a “well-optimized” model f ∗ θ , g ∗ ϕ , where for a positive pair ( U , V ) + , their distance ϵ + :=
Revised: where for a positive pair ( U , V ) + , their distance ϵ + :

Original:  d ( u ∗ , v ∗ )
Revised: = d ( u ∗ , v ∗ )

Original: =
Revised: =

Original: d ( f ∗ θ ( U ) , g ∗ ϕ ( V )) is reasonable small, and the distance ϵ − for negative pairs ( U , V ) − is reasonable large.
Revised: d ( f ∗ θ ( U ) , g ∗ ϕ ( V )) is reasonable small, and the distance ϵ − for negative pairs ( U , V ) − is reasonable large.

Original: Let us consider the following scenario, in S ±
Revised: Let us consider the following scenario, in S ±

Original: = { ( U 1 , V 1 ) , ( U 2 , V 2 ) } , the pair ( U 1 , V 2 ) − is also semantically correlated despite of being recognized as a negative sample.
Revised: = { ( U 1 , V 1 ) , ( U 2 , V 2 ) } , the pair ( U 1 , V 2 ) − is also semantically correlated despite of being recognized as a negative sample.

Original: For the “welloptimized” model, it will predict a distance of ϵ + for this pair instead of ϵ − .
Revised: For the “well-optimized” model, it will predict a distance of ϵ + for this pair instead of ϵ − .

Original: If the distance function d is a metric , then according to the triangle inequality axiom of metric, we have the following inequality,
Revised: If the distance function d is a metric , then according to the triangle inequality axiom of metric, we have the following inequality,

Original: [Equation]
Revised: [Equation]

Original: We can see that the negative distance is bounded by three times the positive distance, despite the fact that the positive distance should be as small as possible.
Revised: We can see that the negative distance is bounded by three times the positive distance, despite the fact that the positive distance should be as small as possible.

Original: Given the fact that, in a mini-batch of suﬀiciently large size, it is very likely to have negative pairs of image and text that match each other equally well as the positive ones.
Revised: Given the fact that, in a mini-batch of sufficiently large size, it is very likely to have negative pairs of image and text that match each other equally well as the positive ones.

Original: Therefore, we need to “relax” the triangular inequality to alleviate the effects of ambiguity of the positive/negative pairs.
Revised: Therefore, we need to “relax” the triangular inequality to alleviate the effects of ambiguity of the positive/negative pairs.

Original: In practice, we have the following observations: i)
Revised: In practice, we have the following observations: i)

Original: The positive pairs of samples usually have a much larger distance than that in the perfect alignment condition, ii) d ( u ∗ 1 , u ∗ 2 ) may become exceptionally small since none of the loss terms regularize it, resulting in a further tightened bound of the negative distance; iii)
Revised: The positive pairs of samples usually have a much larger distance than that in the perfect alignment condition, ii) d ( u ∗ 1 , u ∗ 2 ) may become exceptionally small since none of the loss terms regularize it, resulting in a further tightened bound of the negative distance; iii)

Original: Although the inner product distance is not a metric , it still obeys an “relaxed” triangular inequality because we can yield a metric on the sphere by ArcCos (the geodesic), see Schubert (2021) for more information.
Revised: Although the inner product distance is not a metric , it still obeys an “relaxed” triangular inequality because we can yield a metric on the sphere by ArcCos (the geodesic), see Schubert (2021) for more information.

Original: In Table 1, we characterize both the minimizing geodesic and the ℓ 2 − norm distances are metrics and hence obey the restricted triangular inequality.
Revised: In Table 1, we characterize both the minimizing geodesic and the ℓ 2 − norm distances are metrics and hence obey the restricted triangular inequality.

Original:  Experimental Analysis 4.1 Experimental Settings
Revised:  E XPERIMENTAL A NALYSIS 4. E XPERIMENTAL S ETTINGS

Original: Datasets: For the experimental analysis in Section 4.2, we employ the 15M subset (Cui et al., 2022) of the YFCC100M dataset (Thomee et al., 2016) as the training dataset, which contains roughly 15.3 million internet collected weakly related image-text pairs.
Revised: Datasets: For the experimental analysis in Section 4.2, we employ the 15M subset (Cui et al., 2022) of the YFCC100M dataset (Thomee et al., 2016) as the training dataset, which contains roughly 15.3 million internet collected weakly related image-text pairs.

Original: We evaluate the proposed methods with two types of vision tasks: i) Zero-Shot image-to-text and text-to-image retrieval on Flickr30k (Plummer et al., 2015) and ii) Zero-Shot and Linear Probe object classification on ImageNet-1K (Russakovsky et al., 2015).
Revised: We evaluate the proposed methods with two types of vision tasks: i) Zero-Shot image-to-text and text-to-image retrieval on Flickr30k (Plummer et al., 2015) and ii) Zero-Shot and Linear Probe object classification on ImageNet-1K (Russakovsky et al., 2015).

Original: Furthermore, we employ the RedCaps (Desai et al., 2021) dataset as the out-domain data for visualizing the distributions of sample distances.
Revised: Furthermore, we employ the RedCaps (Desai et al., 2021) dataset as the out-domain data for visualizing the distributions of sample distances.

Original: For the experimental analysis in Section 4.3, we follow the configurations of the reference models, employ a collection of CC3M (Sharma et al., 2018),
Revised: For the experimental analysis in Section 4.2, we collect data from publicly available datasets (Schuhmann et al. ; Changpinyo et al. ; Sharma et al.

Original: Coco captions (Chen et al., 2015), Visual genome (Krishna et al., 2017) and SBU (Ordonez et al., 2011) as the pre-training dataset, which contains roughly 4 million annotated imagetext pairs.
Revised: ; Chen et al. Krishna et al.

Original: The models are then evaluated using Flickr30k (Plummer et al., 2015) and Coco captions (Chen et al., 2015).
Revised: (2017); Plummer et al. ; Chen et al. ; Russakovsky et al. ; Desai et al.

Original: Models: Due to the limited computational resources, we adopt a moderate scaling of the models.
Revised: Kuznetsova et al. ; Li et al. (2017), etc.). We also have clawed weakly related image-text pairs from the web, resulting a total of 420 million individual images and roughly 500 million image-text pairs.

Revised: This dataset is comparable to the one employed in the official CLIP paper (Radford et al., 2021) and another open source re-implementation (Ilharco et al., 2021).

Revised: To further remove the bias caused by datasets, we also re-implement the naive clip algorithm for reference.

Revised: Models: Due to the limited computational resources, we adopt a moderate scaling of the models.

Original: Specifically,  we employ the original ViT-S/16 architecture for our image encoders (Dosovitskiy et al., 2020), with an input image resolution of 224, resulting inimage tokens.
Revised: Specifically, For the ablation experiments, we employ the original ViT-S/16 architecture for our image encoders (Dosovitskiy et al., 2020), with an input image resolution of 224, resulting inimage tokens.

Revised: For large-scale training, we employ the ViT-B/16 as our image encoders.

Original: For our text encoders, we employ Ernie-2.0-en-base (Sun et al., 2020), which is literally a Bert model (Devlin et al., 2018) of 12 layers and 512 hidden neuron sizes with a customized vocabulary of 30,522 tokens, and the maximum context length is set to be 77.
Revised: For our text encoders, we employ Ernie-2.0-en-base (Sun et al., 2020), which is literally a Bert model (Devlin et al., 2018) of 12 layers and 512 hidden neuron sizes with a customized vocabulary of 30,522 tokens, and the maximum context length is set to be 77.

Original: We project the feature representation ([CLS] token) from the top layer of transformers to a  512-dimensional embedding space.
Revised: We project the feature representation ([CLS] token) from the top layer of transformers to a (sum of) 512-dimensional embedding space.

Original: All the parameters except the temperature are optimized from random initialization.
Revised: All the parameters except the temperature are optimized from random initialization.

Original: The default initialization of the project matrix employs the Gaussian initializer of zero mean, and standard deviation equal reversed square root of the input size ( a.k.a.
Revised: The default initialization of the project matrix employs the Gaussian initializer of zero mean, and standard deviation equal reversed square root of the input size ( a.k.a.

Original: Kaiming initialization).
Revised: Kaiming initialization).

Original: For the temperature, we initialize it with e t for t = e 0 , e 1 , e 2 .
Revised: For the temperature, we initialize it with e t for t = e 0 , e 1 , e 2 .

Original: Hyper-parameters employed for training are provided in the appendix.
Revised: Hyper-parameters employed for training are provided in the appendix.

Revised: The details of the hyperparameters are provided in Table 7 in the appendix.

Original: Evaluation: For zero-shot retrieval on Flickr30K, we employ the logits (distance) computed by the distance function and report the image-text pairs with the top- k shortest distance as the retrieval results.
Revised: Evaluation: For zero-shot retrieval on Flickr30K and Coco, we employ the logits (distance) computed by the distance function and report the image-text pairs with the top- k shortest distance as the retrieval results.

Original: For zero-shot classification on ImageNet.
Revised: For zero-shot classification on ImageNet.

Original: We employ multiple prompt templates described in Radford et al.
Revised: We employ multiple prompt templates described in Radford et al.

Original: (2021), while we first compute the distances between image and text embeddings, then average the distances.
Revised: (2021), while we first compute the distances between image and text embeddings, then average the distances.

Original: For linear probe classification on ImageNet, we remove the learned projection head (no topological structure is preserved), then attach a random initialized linear projector to map the feature representation to the 1,000 class logits.
Revised: For linear probe classification on ImageNet, we remove the learned projection head (no topological structure is preserved), then attach a random initialized linear projector to map the feature representation to the 1,000 class logits.

Original: 4.2 Experiments Results
Revised: 4. E XPERIMENTS R ESULTS

Original: Comparison between different configurations: In Table 2, we report the performance of the configurations described in Table 1.
Revised: Comparison between different configurations: In Table 2, we report the performance of the configurations described in Table 1.

Original: We employ the oblique manifold structure of n = 64 , m = 8 for our proposed approach, denoted as Ob(64,8) in the table.
Revised: We employ the oblique manifold structure of n = 64 , m = 8 for our proposed approach, denoted as Ob(64,8) in the table.

Original: The proposed approach provides +4.0%/1.44% improvement over the baseline for the retrieval tasks.
Revised: The proposed approach provides +4.0%/1.44% improvement over the baseline for the retrieval tasks.

Original: It is worth noting that the geodesic distance version also improves the baseline reasonably, despite the fact that it obeys the restricted triangular inequality.
Revised: It is worth noting that the geodesic distance version also improves the baseline reasonably, despite the fact that it obeys the restricted triangular inequality.

Original: This suggests that a more elaborate topology structure helps the alignment of the features from different modalities.
Revised: This suggests that a more elaborate topology structure helps the alignment of the features from different modalities.

Original: The Euclidean configuration performs the worst, which could be the drawback of having no properly defined uniformity.
Revised: The Euclidean configuration performs the worst, which could be the drawback of having no properly defined uniformity.

Original: For the zero-shot and linear probe classification accuracy on ImageNet, the oblique configurations provide better performance when the temperature initialization is low.
Revised: For the zero-shot and linear probe classification accuracy on ImageNet, the oblique configurations provide better performance when the temperature initialization is low.

Original: We suggest the reason is that the equilibrium temperature for the oblique is lower than that for the sphere; hence the model could focus more on learning the representations of the visual modality.
Revised: We suggest the reason is that the equilibrium temperature for the oblique is lower than that for the sphere; hence the model could focus more on learning the representations of the visual modality.

Original: Effects of temperature initialization:
Revised: Effects of temperature initialization:

Original: In Table 2, we also demonstrate the results using different temperature initialization.
Revised: In Table 2, we also demonstrate the results using different temperature initialization.

Original: Although it can be seen that the final performance is not largely impacted by the initialization, a large or small temperature at the start of the training still cause diﬀiculties in optimizing. It is also interesting to see that when the
Revised: Although it can be seen that the final performance is not largely impacted by the initialization, a large or small temperature at the start of the training still cause difficulties

Original: [Table]
Revised: [Table]

Original:  temperature is fixed at 1.0, the sphere one (narrow distance range) and the geodesic one (restrict triangular inequality) fail to learn meaningful feature embeddings for contrastive alignment.
Revised: in optimizing. It is also interesting to see that when the temperature is fixed at 1.0, the sphere one (narrow distance range) and the geodesic one (restrict triangular inequality) fail to learn meaningful feature embeddings for contrastive alignment.

Original: The performance of the proposed approach is also largely reduced.
Revised: The performance of the proposed approach is also largely reduced.

Original: However, since the euclidean topology does not have an upper bound of the distance, the optimizer can still reach the equilibrium.
Revised: However, since the euclidean topology does not have an upper bound of the distance, the optimizer can still reach the equilibrium.

Original: We further draw the trend of the temperature during the training progress in Figure 2.
Revised: We further draw the trend of the temperature during the training progress in Figure 2.

Original: From the figures, we can confirm that: i)
Revised: From the figures, we can confirm that: i)

Original: Given a bounded distance range, the temperature is an inherent property of the datasets, depicting the noise level of the datasets; ii)
Revised: Given a bounded distance range, the temperature is an inherent property of the datasets, depicting the noise level of the datasets; ii)

Original: The temperature will first converge to an equilibrium regardless of initialization, then raise gradually along the optimization progress; iii) It takes longer training iterations for the temperature to converge on the oblique embedding space, while the final temperature is smaller than that in the sphere embedding space.
Revised: The temperature will first converge to an equilibrium regardless of initialization, then raise gradually along the optimization progress; iii) It takes longer training iterations for the temperature to converge on the oblique embedding space, while the final temperature is smaller than that in the sphere embedding space.

Original: The results suggest that border distance range and topology structure help the model focus more on aligning images and texts rather than finding the equilibrium.
Revised: The results suggest that border distance range and topology structure help the model focus more on aligning images and texts rather than finding the equilibrium.

Revised: Retrieval performance on Flickr and Coco of our implemented TCL model and the variant using our proposed method.

Revised: The numbers in brackets are the performance obtained using the contrastive alignment head.

Original: Ablation on oblique structures and multi-token implementation: In Table 3, we modify the structure of the oblique manifold under fixed total dimensions.
Revised: Ablation on oblique structures and multi-token implementation: In Table 3, we modify the structure of the oblique manifold under fixed total dimensions.

Original: It can be seen that a higher mvalue ( i.e. the number of product sub-spheres) is more likely to obtain a
Revised: It can be seen that a higher m

Original: [Table] [Table]
Revised: [Table] [Table]

Original:  better zero-shot text-to-image retrieval recall.
Revised: denote the implementation from Radford et al. Ilharco et al value ( i.e. the number of product sub-spheres) is more likely to obtain a better zero-shot text-toimage retrieval recall.

Original: However, an over-complicated structure such as Ob(4,128) could ruin the performance.
Revised: However, an over-complicated structure such as Ob(4,128) could ruin the performance.

Original: We conjecture that, since the sphere has one redundant dimension, the larger number of product sub-spheres reduces the representation capacity of the topology.
Revised: We conjecture that, since the sphere has one redundant dimension, the larger number of product sub-spheres reduces the representation capacity of the topology.

Original: And because we employ a textual encoder that is gently larger than the visual one, therefore the moderate reduction of the capacity helps overcome the overfitting on the textual side.
Revised: And because we employ a textual encoder that is gently larger than the visual one, therefore the moderate reduction of the capacity helps overcome the overfitting on the textual side.

Original: We also provide the results using the multi-tokens implementation discussed in Section 3, denoted as Multi ( · , · ) .
Revised: We also provide the results using the multi-tokens implementation discussed in Section 3, denoted as Multi ( · , · ) .

Original: For the linear probe, we concatenate all the representations together before projecting them to 1,000 class logit.
Revised: For the linear probe, we concatenate all the representations together before projecting them to 1,000 class logit.

Original: This implementation outperforms other configurations by a large margin at the cost of computational resources.
Revised: This implementation outperforms other configurations by a large margin at the cost of computational resources.

Original: Training the largest Multi (16 , 32) models takes approximately 14% more time.
Revised: Training the largest Multi (16 , 32) models takes approximately 14% more time.

Original: 4.3 Additional Results
Revised: 4. R ESULTS USING LARGE - SCALE DATASET

Original: We combine our proposed method with the TCL model Yang et al. (2022), which is one of the state-of-the-art vision-language retrieval models that employ contrastive visual-textual alignment in its earlier stage. During the pre-training, the TCL induces a mixture of inmodal and cross-modal contrastive losses, while conducting the masked language modeling (MLM) and image-text matching tasks simultaneously.

Original: During the testing, the cross-modal contrastive alignment head first lists sample pairs with high similarity scores, then these pairs are fed into the matching head to obtain the final matching scores. We alternate the topologies ofall the embedding spaces with Ob(128,2); more precisely, we change the normalization function as shown in Section 3.1. The results are shown in Table 4.
Revised: In Table 4, we compare the performance of the proposed method using a larger scale configuration.

Original: Since our method does not affect the matching head, we also report the performance of the contrastive alignment head. In general, our method improves the average recall performance, but the improvement isnot significant. We consider the reasons as i) The method (or recent similarmethods) employs pre-trained vision and language models, as well as a matching head and an MLM head, hence it is less sensitive to the gradients from the contrastive alignment; ii)
Revised: It can be seen that using the ViT-B/16 as the visual bone, our implementation of the naive model performance is similar to the publicly released ones.

Original: The datasets employed for training contain less noise, while the training is scheduled withan overlength scheme (the zero-shot performance does not increase in the last 5 epochs).
Revised: On the other hand, the model equipped with the multi-token implementation of (16,32) significantly outperforms the other ViT-B/16 models, with only less than 8% more computational costs.

Original:  Related Works
Revised:  R ELATED W ORKS

Original: Momentum distillation:
Revised: Momentum distillation:

Original: In recent works such as Cheng et al. (2021); Li et al.
Revised: In recent works such as Cheng et al. (2021); Li et al.

Original: (2021a), the momentum (self-)distillation is introduced to mitigate the semantic noise in the sample pairs.
Revised: (2021a), the momentum (self-)distillation is introduced to mitigate the semantic noise in the sample pairs.

Original: That is, a momentum version of the model is updated by the moving average of the model’s historical parameters.
Revised: That is, a momentum version of the model is updated by the moving average of the model’s historical parameters.

Original: Then, the cross entropy between the softmax logits computed by the model and its momentum version is used as an additional loss for supervision.
Revised: Then, the cross entropy between the softmax logits computed by the model and its momentum version is used as an additional loss for supervision.

Original: The authors claim that the pseudo-targets of the momentum (self-)distillation will not penalize the model for matching negative samples that are reasonably similar.
Revised: The authors claim that the pseudo-targets of the momentum (self-)distillation will not penalize the model for matching negative samples that are reasonably similar.

Original: Here, we consider that the pseudo-targets do “relax” the triangular inequality restriction implicitly by letting the distance of alignment be reasonably large.
Revised: Here, we consider that the pseudo-targets do “relax” the triangular inequality restriction implicitly by letting the distance of alignment be reasonably large.

Original: Hence, it could be much easier for the optimizer to find the equilibrium discussed in Section 3.2.
Revised: Hence, it could be much easier for the optimizer to find the equilibrium discussed in Section 3.2.

Original: Other implemention of non-metric distance:
Revised: Other implemention of non-metric distance:

Original: In Yao et al.
Revised: In Yao et al.

Original: (2021), the authors proposed a so-called fine-grained contrastive learning scheme that matches all the visual and textual tokens using a maximum-average operator.
Revised: (2021), the authors proposed a socalled fine-grained contrastive learning scheme that matches all the visual and textual tokens using a maximum-average operator.

Original: Concretely, for each visual token, it finds the textual token with maximum similarity, then takes the average over the visual tokens as the similarity of the image to a text and vice versa.
Revised: Concretely, for each visual token, it finds the textual token with maximum similarity, then takes the average over the visual tokens as the similarity of the image to a text and vice versa.

Original: Using our framework, this work can be explained as embedding samples onto the product manifold S d − 1 × · · · × S d − 1 endowed with the maximum-average distance, which is a non-metric distance.
Revised: Using our framework, this work can be explained as embedding samples onto the product manifold S d − 1 × · · · × S d − 1 endowed with the maximum-average distance, which is a non-metric distance.

Original: While the authors employ the sub-manifold S d − 1 to represent local information.
Revised: At the same time, the authors employ the sub-manifold S d − 1 to represent local information.

Original: The effects of softmax temperature: In Wang & Liu (2021), the authors draw the uniformity of the embedding distribution and the tolerance to semantically similar samples of learned models under different temperatures.
Revised: The effects of softmax temperature: In Wang & Liu (2021), the authors draw the uniformity of the embedding distribution and the tolerance to semantically similar samples of learned models under different temperatures.

Original: From the observations, the authors claim that “a good choice of temperature can compromise these two properties properly to both learn separable features and tolerant to semantically similar samples, improving the feature qualities and the downstream performances”.
Revised: From the observations, the authors claim that “a good choice of temperature can compromise these two properties properly to both learn separable features and tolerant to semantically similar samples, improving the feature qualities and the downstream performances”.

Original: Unlike our work, this work is done under uni-modal contrastive learning, where the semantic correlation of the negative samples is not a property of the datasets but rather a drawback of the larger mini-batch size.
Revised: Unlike our work, this work is done under uni-modal contrastive learning, where the semantic correlation of the negative samples is not a property of the datasets but rather a drawback of the larger mini-batch size.

Original: Uni-modal side tasks:
Revised: Uni-modal side tasks:

Original: In works such as Mu et al. (2021); Li et al.
Revised: In works such as Mu et al. (2021); Li et al.

Original: (2021b)
Revised: (2021b)

Original: ; Yang et al.
Revised: ; Yang et al.

Original: (2022), authors combine cross-modal contrastive loss with other uni-modal tasks, for instance, visual/textual self-supervised contrastive learning, masked image/language modeling.
Revised: , authors combine cross-modal contrastive loss with other uni-modal tasks, for instance, visual/textual self-supervised contrastive learning, masked image/language modeling.

Original: These combined methods demonstrate superior performance in downstream tasks such as zero-shot classification empirically.
Revised: These combined methods demonstrate superior performance in downstream tasks such as zero-shot classification empirically.

Original: Although these works do not overlap with this one, we find that the uni-modal tasks provide reasonable uniformity within the visual/textual feature embedding, contrary to the cross-modal contrastive shown in Section 3.2.
Revised: Although these works do not overlap with this one, we find that the uni-modal tasks provide reasonable uniformity within the visual/textual feature embedding, contrary to the cross-modal contrastive shown in Section 3.2.

Original: Therefore, the model could obtain a more “numerically relaxed” triangular inequality when dealing with noisy pairs of samples.
Revised: Therefore, the model could obtain a more “numerically relaxed” triangular inequality when dealing with noisy pairs of samples.

Revised: Other works that employ oblique manifold: It is notable that learning representations embedded on the oblique manifold for computer vision tasks have been explored by former studies.

Revised: For instance, in Qi et al.

Revised: (2021), the authors employ the oblique topology for few-shot learning.

Revised: However, different from these works, our paper mainly tackles the noisy database problem in the contrastive image-text alignment task.

Revised: We employ the oblique topology with a non-metric distance function to tackle the out-of-range equilibrium problem.

Original: Conclusion
Revised: Hyperbolic embedding space: The hyperbolic topology is another popular choice for hierarchical representation embedding space, in both NLP (Nickel & Kiela, 2017), CV (Zhang et al., 2022; Liu et al., 2020; Khrulkov et al., 2020), and cross-modal (Guo et al., 2021) tasks. However, the hyperbolic topology has unfavored properties similar to Euclidean space. That is, the resource required for computing distance is high, difficult to define/implement uniformity in terms of numerical stability, and the geodesic distance has restricted triangular inequality.

Revised: Therefore, we do not consider this topology in this study.

Revised:  C ONCLUSION

Original: Summary:
Revised: Summary:

Original: In this work, we discuss the important properties of the feature embedding space for contrastive alignment.
Revised: In this work, we discuss the important properties of the feature embedding space for contrastive alignment.

Original: We show that the most commonly adopted cosine similarity has disadvantages in dealing with noisy data and training stability.
Revised: We show that the most commonly adopted cosine similarity has disadvantages in dealing with noisy data and training stability.

Original: Therefore, we propose to combine the oblique manifold with the negative inner product distance to tackle these problems.
Revised: Therefore, we propose to combine the oblique manifold with the negative inner product distance to tackle these problems.

Original: The proposed approach demonstrates better performance in the image-text retrieval tasks practically.
Revised: The proposed approach demonstrates better performance in the image-text retrieval tasks practically.

Original: Limiation: Due to remarkably limited computational resources, we cannot conduct experiments on a larger scale in terms of batch size,numbers of training data, andnumbers of parameters in the neural network (and time). 
Revised: Limiation: First, due to remarkably limited computational resources, we cannot conduct experiments on a larger scale in terms of batch size, training data, and parameters in the neural network (and time). Second, in recent studies, besides the contrastive alignment, more pre-training tasks are appended to the head of the model using the non-normalized full token embedding.

Original: Therefore, we do not know if the conclusions of this study can be applied to scenarios of near-infinite resources.
Revised: Such as imagetext matching (Li et al., 2021a; Yang et al., 2022), image captioning (Yu et al., 2022), or masked modeling that do not employ the contrastive alignment (Wang et al., 2022). In these configurations, the performance improvement resulting from a better contrastive alignment could be marginal. And hence leave future work on designing the topology of the full token embedding.

Original: A Appendix A.1 Train hyper-parameters:
Revised: A A PPENDIX A. T RAIN HYPER - PARAMETERS :

Original: We train with a batch size of 2,048 and the AdamW optimizer in all our experiments.
Revised: We train with a batch size of 2,048 and the AdamW optimizer in all our experiments.

Original: Following the hyper-parameters that improve training stability in the original CLIP (Radford et al., 2021) paper [Table] we set the β 1 = 0 .
Revised: Following the hyper-parameters that improve training stability in the original CLIP (Radford et al., 2021) paper, we set the β 1 = 0 .

Original: 9 , β 2 = 0 .
Revised: 9 , β 2 = 0 .

Original: 98 , ϵ = 1 e - [Table] and weight decay = 0 .
Revised: 98 , ϵ = 1 e - 8 , and weight decay = 0 .

Original: We also employ the gradient clip operation with a global norm of 1.0.
Revised: We also employ the gradient clip operation with a global norm of 1.0.

Original: We did not employ mixed precision to reduce the possible variance introduced by random initialization.
Revised: We did not employ mixed precision to reduce the possible variance introduced by random initialization.

Original: We apply a compact training scheme that updates the model for 108,000 iterations

Original: [Table] which is roughly equal to training the model for 15 epochs of the dataset [Table] and takes approximately 1 day to train one model using 32 Nvidia V100-16G GPUs.
Revised: We apply a compact training scheme that updates the model for 108,000 iterations, which is roughly equal to training the model for 15 epochs of the dataset, and takes approximately 1 day to train one model using 32 Nvidia V100-16G GPUs.

Original: We use the cosine learning rate decay scheme of peak learning rate equal to 5 e - 4 , combined with a warmup period of 5,000 iterations.
Revised: We use the cosine learning rate decay scheme of peak learning rate equal to 5 e - 4 , combined with a warmup period of 5,000 iterations.

Original: For data augmentation, we only apply the RandomResizedCrop with a scale range of [0 . 5 , 1 . 0] .
Revised: For data augmentation, we only apply the RandomResizedCrop with a scale range of [0 . 5 , 1 . 0] .

Original: In the linear probe evaluation, the hyperparameters follow the setup of MoCo v3 (Chen et al., 2021b).
Revised: In the linear probe evaluation, the hyperparameters follow the setup of MoCo v3 (Chen et al., 2021b).

Original: Concretely, we use SGD without momentum and no weight decay.
Revised: Concretely, we use SGD without momentum and no weight decay.

Original: The learning rate is schemed by cosine decay with a peak learning rate equal to 1.0, combined with a warmup period of 5 epochs.
Revised: The learning rate is schemed by cosine decay with a peak learning rate equal to 1.0, combined with a warmup period of 5 epochs.

Original: We train for 100 epochs and augment the image using the RandomResizedCrop with a scale range of [0 . 75 , 1 . 0] and AutoAugment with the code rand-m9-mstd0.5-inc1 . 
Revised: We train for 100 epochs and augment the image using the RandomResizedCrop with a scale range of [0 . 75 , 1 . 0] and AutoAugment with the code rand-m9-mstd0.5-inc1 . Temp. Init. Temp.

Original: A.2 Distribution of Learned Distance
Revised: Final ConvergeStep Zero-Shot I2TR@

Revised: Zero-Shot

Revised: T2IR@

Revised: Zero-Shot

Revised: Cls.

Revised: Acc.

Revised: LinearCls.

Revised: Acc.

Revised: Topology: Sphere,

Revised: Distance: − u T v 2.

Revised: 18k 48.

Revised: 39k 46.

Revised: 22k 49.

Revised: Detach 5.

Revised: Topology: Euclidean,

Revised: Distance: ∥ u − v ∥2.

Revised: 21k 47.

Revised: 1k 48.

Revised: 25k 47.

Revised: Detach 47.

Revised: Topology: Ob(64, 8),

Revised: Distance: Geo( u , v ) 2.

Revised: 20k 50.

Revised: 55k 50.

Revised: 42k 49.

Revised: Detach 4.

Revised: Topology: Ob(64, 8),

Revised: Distance: − tr( u T v ) 2.

Revised: 24k 52.

Revised: 57k 50.

Revised: 36k 50.

Revised: Detach 30.

Revised: “Temp

Revised: [Table]

Revised: Init.”

Revised: denotes the values for initializing temperature; “Temp.

Revised: Final” denotes the final temperature at the end of training; “Converge Step” denotes the number of steps for tempearture starts to converge (changes less than 2% for an epoch.)

Revised: A. T ABLE 2 FROM THE V IEW OF T OPOLOGIES

Revised: In Table 5, we review Table 2 by the topologies.

Revised: We further provide the final temperature at the end of training and at what step the temperature converges (changes less than 2% for an epoch, also see

Revised: Figure 2).

Revised: It can be seen that the performance of the Euclidean topology is only slightly affected by the initialization of the temperatures, and even though the temperature is detached from learning, it still performs reasonably well because of the unlimited distance range.

Revised: While the spherical and oblique topologies are affected by how the temperature is initiated.

Revised: However, a rough trend can be

Revised: [Figure]

Revised: seen that the faster the temperature converges, the better performance the model achieves, which means the learnable temperature delays the learning of the methods.

Revised: The model needs first to find a proper temperature and then begin to learn representations well.

Revised: A. D ISTRIBUTION OF L EARNED D ISTANCE

Original: We depict the distribution of distance for pairs of samples in Figure 3.
Revised: We depict the distribution of distance for pairs of samples in Figure 3.

Original: As argued in Section 3.2 of the main manuscripts, since the cross-modal contrastive loss does not handle the unimodal data distributions, the distance between negative pairs of images and texts could be much smaller than that of a positive image-text pair, resulting in a tighter distance bound.
Revised: As argued in Section 3.2 of the main manuscripts, since the cross-modal contrastive loss does not handle the uni-modal data distributions, the distance between negative pairs of images and texts could be much smaller than that of a positive image-text pair, resulting in a tighter distance bound.

Original: Also, we can see this phenomenon is much more severe in out-domain data, which could reduce the transferability of the feature embeddings to downstream tasks.
Revised: Also, we can see this phenomenon is much more severe in out-domain data, which could reduce the transferability of the feature embeddings to downstream tasks.

Revised: It is also notable that, the oblique endowed with the negative inner product as the distance function learns similar distributions compared to the sphere reference, while the numerical values of distances between samples are inherently larger without having multiplied with temperature.

Original: A.3 Additional Ablation on Oblique Structure
Revised: A. A DDITIONAL A BLATION ON O BLIQUE S TRUCTURE

Original: We provide more ablation results regarding the structure of the oblique manifold under fixed total dimensions in Table 5.
Revised: We provide more ablation results regarding the structure of the oblique manifold under fixed total dimensions in Table 6.

Original: We can observe that the Ob(32, 32) configuration performs the best in general, while the sphere with more 1024-dimensional embedding has slightly better linear probe performance.
Revised: We can observe that the Ob(32, 32) configuration performs the best in general, while the sphere with more 1024-dimensional embedding has slightly better linear probe perfor- [Table]

Original:  We also notice that a more complicated structure provides better text-to-image retrieval results.
Revised: mance. We also notice that a more complicated structure provides better text-to-image retrieval results.

Original: A.4 Additional Notes on TCL
Revised: A. D ETAILED M ODEL H YPERPARAMETERS OF OUR CLIP MODEL

Revised: [Table] A. A DDITIONAL R ESULTS We combine our proposed method with the TCL model Yang et al.

Original:  We provide the comparison results with oﬀicially released checkpoints.
Revised: (2022), which is one of the state-of-the-art vision-language retrieval models that employ contrastive visual-textual alignment in its earlier stage. During the pre-training, the TCL induces a mixture of in-modal and crossmodal contrastive losses, while conducting the masked language modeling (MLM) and image-text matching tasks simultaneously. During the testing, the cross-modal contrastive alignment head first lists sample pairs with high similarity scores, then these pairs are fed into the matching head to obtain the final matching scores. We alternate the topologies of all the embedding spaces with Ob(128,2); more precisely, we change the normalization function as shown in Section 3.1.

Original: It can be seen that our implementation performs 0.5-1.0% worse than the oﬀicial checkpoints.
Revised: For the experimental analysis in this subsection, we follow the configurations of the reference models, employ a collection of CC3M (Sharma et al., 2018), Coco captions (Chen et al., 2015), Visual genome (Krishna et al., 2017) and SBU (Ordonez et al., 2011) as the pre-training dataset, which contains roughly 4 million annotated image-text pairs. The models are then evaluated using Flickr30k (Plummer et al., 2015) and Coco captions (Chen et al., 2015). The results are shown in Table 8. Since our method does not affect the matching head, we also report the performance of the contrastive alignment head.

Original: On the other
Revised: In general, our method improves the average recall performance, but the improvement is not significant. We consider the reasons as i) The method (or recent similar methods) employs pre-trained vision and language models, as well as a matching head and an MLM head, hence it is less sensitive to the gradients from the contrastive alignment; ii) The datasets employed for training contain less noise, while the training is scheduled with an overlength scheme (the zero-shot performance does not increase in the last 5 epochs).

Original: [Figure]
Revised: [Table]

Revised: Additional Notes on TCL We also provide the comparison results with officially released checkpoints.

Revised: It can be seen that our implementation performs 0.5-1.0% worse than the official checkpoints.

Original:  hand, our implementation has better alignment head performance.
Revised: On the other hand, our implementation has better alignment head performance.

Original: Since we are employing the codes released in the oﬀicial repository, the reason might be the following: i) Datasets difference, that we have ∼ 3000 fewer images in the SBU dataset while owning 5000 more images in the CC3M dataset; ii)
Revised: Since we are employing the codes released in the official repository, the reason might be the following: i) Datasets difference, that we have ∼ 3000 fewer images in the SBU dataset while owning 5000 more images in the CC3M dataset; ii)

Original: We resize the CC3M dataset to short edge 500 pixels, while the oﬀicial repository does not clearly provide the pre-processing approach; iii) We implicitly have a short training time or smaller matching loss weight than the oﬀicial checkpoints due to the difference in the framework.[Table]
Revised: We resize the CC3M dataset to short edge 500 pixels, while the official repository does not clearly provide the pre-processing approach; iii) We implicitly have a short training time or smaller matching loss weight than the official checkpoints due to the difference in the framework.

Original: performance obtained using the contrastive alignment head.