Original: Robust Graph Structure Learningover Images via Multiple Statistical Tests
Revised: Robust Graph Structure Learning via Multiple Statistical Tests

Original: Abstract
Revised: Abstract

Original: Graph structure learning aims to learn connectivity in a graph from data.
Revised: Graph structure learning aims to learn connectivity in a graph from data.

Original: It is particularly important for many computer vision related tasks since no explicit graph structure is available for images for most cases.
Revised: It is particularly important for many computer vision related tasks since no explicit graph structure is available for images for most cases.

Original: A natural way to construct a graph among images is to treat each image as a node and assign pairwise image similarities as weights to corresponding edges.
Revised: A natural way to construct a graph among images is to treat each image as a node and assign pairwise image similarities as weights to corresponding edges.

Original: It is well known that pairwise similarities between images are sensitive to the noise in feature representations, leading to unreliable graph structures.
Revised: It is well known that pairwise similarities between images are sensitive to the noise in feature representations, leading to unreliable graph structures.

Original: We address this problem from the viewpoint of statistical tests.
Revised: We address this problem from the viewpoint of statistical tests.

Original: By viewing the feature vector of each node as an independent sample, the decision of whether creating an edge between two nodes based on their similarity in feature representation can be thought as a single statistical test.
Revised: By viewing the feature vector of each node as an independent sample, the decision of whether creating an edge between two nodes based on their similarity in feature representation can be thought as a single statistical test.

Original: To improve the robustness in the decision of creating an edge, multiple samples are drawn and integrated by multiple statistical tests to generate a more reliable similarity measure, consequentially more reliable graph structure.
Revised: To improve the robustness in the decision of creating an edge, multiple samples are drawn and integrated by multiple statistical tests to generate a more reliable similarity measure, consequentially more reliable graph structure.

Original: The corresponding elegant matrix form named B -Attention is designed for efficiency.
Revised: The corresponding elegant matrix form named B -Attention is designed for efﬁciency.

Original: The effectiveness of multiple tests for graph structure learning is verified both theoretically and empirically on multiple clustering and ReID benchmark datasets.
Revised: The effectiveness of multiple tests for graph structure learning is veriﬁed both theoretically and empirically on multiple clustering and ReID benchmark datasets.

Original: Source codes are available at https://github.com/Thomas-wyh/B-Attention .
Revised: Source codes are available at https://github.com/Thomas-wyh/B-Attention .

Original:  Introduction
Revised:  Introduction

Original: Graph structure learning plays an important role in Graph Convolutional Networks (GCNs).
Revised: Graph structure learning plays an important role in Graph Convolutional Networks (GCNs).

Original: It seeks to discover underlying graph structures for better graph representation learning when graph structures are unavailable, noisy or corrupted.
Revised: It seeks to discover underlying graph structures for better graph representation learning when graph structures are unavailable, noisy or corrupted.

Original: These issues are typical in the field of vision tasks [58, 63, 62, 20, 42, 50, 57, 13, 39, 67, 47].
Revised: These issues are typical in the ﬁeld of vision tasks [58, 63, 62, 20, 42, 50, 57, 13, 39, 67, 47].

Original: As an example, there is no explicit graph structure between photos in your phone albums, so the graph-based approaches [20, 42, 57] in photo management softwares seek to learn graph structures.
Revised: As an example, there is no explicit graph structure between photos in your phone albums, so the graph-based approaches [20, 42, 57] in photo management softwares seek to learn graph structure.

Original: A popular research direction on it is to consider graph structure learning as similarity measure learning upon the node embedding space [60].
Revised: A popular research direction on it is to consider graph structure learning as similarity measure learning upon the node embedding space [60].

Original: They learn weighted adjacency matrices [20, 42, 9, 10, 43, 64, 28, 8, 25, 35] as graphs and feed them to GCNs to learn representations.
Revised: They learn weighted adjacency matrices [20, 42, 9, 10, 43, 64, 28, 8, 25, 35] as graphs and feed them to GCNs to learn representations.

Original: However, the main problem of the above weighted-based methods is that weights are not always accurate and reliable: two nodes with smaller associations may have a higher weight score than those with larger associations.
Revised: However, the main problem of the above weighted-based methods is that weights are not always accurate and reliable: two nodes with smaller associations may have a higher weight score than those with larger associations.

Original: The root of this problem is closely related to the noise in the high dimensional feature vectors extracted by the deep learning model, as similarities between nodes are usually decided based on their feature vectors.
Revised: The root of this problem is closely related to the noise in the high dimensional feature vectors extracted by the deep learning model, as similarities between nodes are usually decided based on their feature vectors.

Original: The direct consequence of using an inaccurate graph structure is so called feature pollution [57], leading to a performance degradation in downstream tasks.
Revised: The direct consequence of using an inaccurate graph structure is so called feature pollution [57], leading to a performance degradation in downstream tasks.

Original: In our empirical studies, we found that close to 1 / 3 of total weights are assigned to noisy edges when unnormalized cosine similarities between feature vectors are used to calculate weights.
Revised: In our empirical studies, we found that close to 1 / 3 of total weights are assigned to noisy edges when unnormalized cosine similarities between feature vectors are used to calculate weights.

Original: Many methods have been developed to improve similarity measure between nodes in graph, including adding learnable parameters [9, 28, 8], introducing nonlinearity [10, 43, 64], Gaussian kernel functions [25, 35] and self-attention [53, 12, 20, 42].
Revised: Many methods have been developed to improve similarity measure between nodes in graph, including adding learnable parameters [9, 28, 8], introducing nonlinearity [10, 43, 64], Gaussian kernel functions [25, 35] and self-attention [53, 12, 20, 42].

Original: Despite the progress, they are limited to modifying similarity function, without explicitly addressing the noise in feature vectors, a more fundamental issue for graph structure learning.
Revised: Despite the progress, they are limited to modifying similarity function, without explicitly addressing the noise in feature vectors, a more fundamental issue for graph structure learning.

Original: One way to capture the noise in feature vectors is to view the feature vector of each node as an independent sample from some unknown distribution associated with each node.
Revised: One way to capture the noise in feature vectors is to view the feature vector of each node as an independent sample from some unknown distribution associated with each node.

Original: As a result, the similarity score between two nodes can be thought as the expectation of a test to decide if two nodes share the same distribution.
Revised: As a result, the similarity score between two nodes can be thought as the expectation of a test to decide if two nodes share the same distribution.

Original: Since, only one vector is sampled for each node in the above methods, the statistical test is essentially based on a single sample from both nodes, making the calculated weights inaccurate and unreliable.
Revised: Since, only one vector is sampled for each node in the above methods, the statistical test is essentially based on a single sample from both nodes, making the calculated weights inaccurate and unreliable.

Original: An possible solution towards addressing the noise in feature vectors is to run multiple tests, a common approach to reduce variance in sampling and decision making [4, 11, 26].
Revised: An possible solution towards addressing the noise in feature vectors is to run multiple tests, a common approach to reduce variance in sampling and decision making [4, 11, 26].

Original: Multiple samples are drawn for each node, and each sample can be used to run an independent test to decide if two nodes share the same distribution.
Revised: Multiple samples are drawn for each node, and each sample can be used to run an independent test to decide if two nodes share the same distribution.

Original: Based on the results from multiple tests , we develop a novel and robust similarity measure that significantly improves the robustness and reliability of calculated weights.
Revised: Based on the results from multiple tests , we develop a novel and robust similarity measure that signiﬁcantly improves the robustness and reliability of calculated weights.

Original: This paper will show how to accomplish multiple tests in the absence of graph structures and theoretically prove why the proposed approach can produce a better similarity measure under specified conditions.
Revised: This paper will show how to accomplish multiple tests in the absence of graph structures and theoretically prove why the proposed approach can produce a better similarity measure under speciﬁed conditions.

Original: This similarity measure is also crafted into an efficient and elegant form of matrix, named B -Attention .
Revised: This similarity measure is also crafted into an efﬁcient and elegant form of matrix, named B -Attention .

Original: B -Attention utilizes the fourth order statistics of the features, which is a significant difference with popular attentions.
Revised: B -Attention utilizes the fourth order statistics of the features, which is a signiﬁcant difference with popular attentions.

Original: Benefiting from a better similarity measure produced by multiple tests, graph structures can be significantly improved, resulting in better graph representations and therefore boosting the performance of downstream tasks.
Revised: Beneﬁting from a better similarity measure produced by multiple tests, graph structures can be signiﬁcantly improved, resulting in better graph representations and therefore boosting the performance of downstream tasks.

Original: We empirically verify the effectivness of the proposed method over the task of clustering and ReID.
Revised: We empirically verify the effectivness of the proposed method over the task of clustering and ReID.

Original: In summary, this paper makes the following three major contributions: (1) To the best of our knowledge, this is the first paper to address inaccurate edge weights problem by statistical tests on the task of graph structure learning over images.
Revised: In summary, this paper makes the following three major contributions: (1) To the best of our knowledge, this is the ﬁrst paper to address inaccurate edge weights problem by statistical tests on the task of graph structure learning over images.

Original: (2) A novel and robust similarity measure is proposed based on multiple tests, proven theoretically to be superior and designed in an elegant matrix form to improve graph structure quality.
Revised: (2) A novel and robust similarity measure is proposed based on multiple tests, proven theoretically to be superior and designed in an elegant matrix form to improve graph structure quality.

Original: (3) Extensive experiments further verify the robustness of the graph structure learning method against noise.
Revised: (3) Extensive experiments further verify the robustness of the graph structure learning method against noise.

Original: State-of-the-art (SOTA) performances are achieved on multiple clustering and ReID benchmarks with the learned graph structures.
Revised: State-of-the-art (SOTA) performances are achieved on multiple clustering and ReID benchmarks with the learned graph structures.

Original:  Methodology
Revised:  Methodology

Original: To handle the noise in feature vectors, we model the feature vector of each node as a sample drawn from the underlying distribution associated with that node, and the similarity score between two nodes as the expectation of a statistical test.
Revised: To handle the noise in feature vectors, we model the feature vector of each node as a sample drawn from the underlying distribution associated with that node, and the similarity score between two nodes as the expectation of a statistical test.

Original: The test is to compare their similarity to a given threshold to decide if two nodes share the same distribution and that decision can be captured by a binary
Revised: The test is to compare their similarity to a given threshold to decide if two nodes share the same distribution and that decision can be captured by a binary

Original: Bernoulli random variable.
Revised: Bernoulli random variable.

Original: The comparison in statistical test may lead to error edges, i.e., noisy edges (an edge connects two nodes of different categories) or missing edges (two nodes of the same category should be connected but are not).
Revised: The comparison in statistical test may lead to error edges, i.e., noisy edges (an edge connects two nodes of different categories) or missing edges (two nodes of the same category should be connected but are not).

Original: The superiority of the similarity measure can be revealed by comparing the probability of creating an error edge.
Revised: The superiority of the similarity measure can be revealed by comparing the probability of creating an error edge.

Original: The proposed similarity measure will be introduced in Section 2.2, and its further design in matrix form will be presented in Section 2.3.
Revised: The proposed similarity measure will be introduced in Section 2.2, and its further design in matrix form will be presented in Section 2.3.

Original: 2.1 Preliminary
Revised: 2.1 Preliminary

Original: Chernoff Bounds.
Revised: Chernoff Bounds.

Original: Sums of random variables have been studied for a long time [4, 11, 26].
Revised: Sums of random variables have been studied for a long time [4, 11, 26].

Original: Chernoff
Revised: Chernoff

Original: Bounds [11] provide sharply decreasing bounds that are in the exponential form on tail distributions of sums of independent random variables.
Revised: Bounds [11] provide sharply decreasing bounds that are in the exponential form on tail distributions of sums of independent random variables.

Original: It is a widely used fundamental tool in mathematics and computer science [32, 41].
Revised: It is a widely used fundamental tool in mathematics and computer science [32, 41].

Original: An often used and looser form of Chernoff Bounds is as follows.
Revised: An often used and looser form of Chernoff Bounds is as follows.

Original: Lemma 1 (Chernoff Bounds [11]) .
Revised: Lemma 1 (Chernoff Bounds [11]) .

Original: Suppose X i is a binary random variable taking value in set { 0 , 1 } with P ( X i = 1)
Revised: Suppose X i is a binary random variable taking value in set { 0 , 1 } with P ( X i = 1)

Original: = p i .
Revised: = p i .

Original: Define X =
Revised: Deﬁne X =

Original: (cid:80) n 1 X i where { X 1 , X 2 , · · · , X n } are independently sampled.
Revised: P n 1 X i where { X 1 , X 2 , · · · , X n } are independently sampled.

Original: Then for any 0 < ϵ < 1 ,
Revised: Then for any 0 < ✏ < 1 ,

Original: [Equation]
Revised: [Equation]

Original: It shows that with a high probability 1 − δ , we have X ≤ (1+ ϵ ) µ , or X ≥ (1 − ϵ ) µ , where δ = e − ϵ 2 3 .
Revised: It shows that with a high probability 1   , we have X  (1+ ✏ ) µ , or X  (1  ✏ ) µ , where  = e  ✏ 2 3 .

Original: GCNs.
Revised: GCNs.

Original: A GCN network normally consists of multiple GCN layers.
Revised: A GCN network normally consists of multiple GCN layers.

Original: Given an input graph G ( X , A ) with L vertices, the output of a normal GCN layer [34] is:
Revised: Given an input graph G ( X , A ) with L vertices, the output of a normal GCN layer [34] is:

Original: [Equation]
Revised: [Equation]

Original: where ˜A =
Revised: where ˜A =

Original: A + I , ˜D ii =
Revised: A + I , ˜D ii =

Original: L(cid:88)
Revised: L

Original:  j
Revised: X j

Original: = ˜A i,j , (1)
Revised: = ˜A i,j , (1)

Original: X ∈ R L × M is input vertex features with M dimensions, A ∈ R L × L is their adjacency matrix;
Revised: X 2 R L ⇥ M is input vertex features with M dimensions, A 2 R L ⇥ L is their adjacency matrix;

Original: I ∈ R L × L is an identity matrix, ˜D ∈ R L × L is the diagonal degree matrix of ˜A ; W l ∈ R M × M ′ is a trainable matrix, and σ ( · ) is an activation function such as ReLU [19]; X ′ ∈ R L × M ′ is the output vertex features with M ′ dimensions, M ′{+
+} =
Revised: I 2 R L ⇥ L is an identity matrix, ˜D 2 R L ⇥ L is the diagonal degree matrix of ˜A ; W l 2 R M ⇥ M 0 is a trainable matrix, and  ( · ) is an activation function such as ReLU [19]; X 0 2 R L ⇥ M 0 is the output vertex features with M 0 dimensions, M {+
+} =

Original: M in this work.
Revised: M in this work.

Original: 2.2 Multiple tests on similarity measure
Revised: 2.2 Multiple tests on similarity measure

Original: Assumption .

Original: Let Vi,j = { v 1 , v 2 , ..., v m } be thecommon neighbours collection of nodes v i and v j .
Revised: Let V = { v 1 , v 2 , ..., v N } be the collection of N nodes.

Original: Each node is assumed to be in one of two categories: +1 or − 1 for simplicity.Among the nodes in
Revised: Each node is assumed to be in one of two categories: +1 or  1 for simplicity.

Original: V i,j , αm nodes share the same category as v i and (1 − α ) m nodes belong to the opposite category, with α > 12 .

Original: For any node v j ∈ V i , its probability of being connected with v i , according to the statistical test, is q ∈ [0 , 1] if C ( v i )

Original: ̸ = C ( v j ) , and the probability increases to 1 ≥ p > q if C ( v i ) =

Original: C ( v j ) .

Original: This section will show that, if m is sufficiently large (i.e., enough number of tests can be run), there will exist an appropriate threshold for the Sim-M such that the number of error edges can be reduced significantly compared to Sim-S.

Original: For each node v i , the subset of candidate nodes to connect it is denoted by V i .
Revised: For each node v i , the subset of candidate nodes to connect it is denoted by V i .

Original: This subset can be decided by a simple criterion such as by k nearest neighbours.
Revised: This subset can be decided by a simple criterion such as by k nearest neighbours.

Original: When calculating the proposed similarity between v i and v j ∈ V i , the common candidate nodes between v i and v j , i.e., V i,j = V i∩V
Revised: When calculating the proposed similarity between v i and v j 2 V i , the common candidate nodes between v i and v j , i.e., V i,j = V i

Original:  j are first identified.
Revised: \ V j are ﬁrst identiﬁed.

Original: The nodes in the V i,j are treated as the multiple samples drawn for the comparison of v i and v j .
Revised: The nodes in the V i,j are treated as the multiple samples drawn for the comparison of v i and v j .

Original: Then, each v k ∈ V i,j performs single test twice, one for v k and v i and one for v k and v j .
Revised: Then, each v k 2 V i,j performs single test twice, one for v k and v i and one for v k and v j .

Original: The single test here is a comparison using the pairwise similarity based on the features of v k and v i , or v k and v j .
Revised: The single test here is a comparison using the pairwise similarity based on the features of v k and v i , or v k and v j .

Original: The pairwise similarity can be cosine similarity, Gaussian kernel similarity or any other form, and is denoted by Sim-S .
Revised: The pairwise similarity can be cosine similarity, Gaussian kernel similarity or any other form, and is denoted by Sim-S .

Original: The proposed similarity score between v i and v j increases by one if the two tests yield C ( v i ) =
Revised: The proposed similarity score between v i and v j increases by one if the two tests yield C ( v i ) =

Original: C ( v k ) and C ( v k ) =
Revised: C ( v k ) and C ( v k ) =

Original: C ( v j ) , and zero otherwise, where C ( v ) is the category of node v .
Revised: C ( v j ) , and zero otherwise, where C ( v ) is the category of node v .

Original: It contains multiple single tests, hence the name multiple tests , corresponding similarity measure denoted by Sim-M .
Revised: It contains multiple single tests, hence the name multiple tests , corresponding similarity measure denoted by Sim-M .

Original: For the convenience of analysis, a few assumptions are further introduced.
Revised: For the convenience of analysis, a few assumptions are further introduced.

Original: First, the size of V i,j is m , and among the nodes in V i,j , αm nodes share the same category as v i and (1 − α ) m nodes belong to the opposite category, with α > 12 .
Revised: First, the size of V i,j is m , and among the nodes in V i,j , ↵ m nodes share the same category as v i and (1  ↵ ) m nodes belong to the opposite category, with ↵ > 12 .

Original: For any node v j ∈ V i , its probability of being connected with v i , according to the statistical test, is q ∈ [0 , 1] if C ( v i ) ̸ = C ( v j ) , and the probability increases to 1 ≥ p > q if C ( v i ) =
Revised: For any node v j 2 V i , its probability of being connected with v i , according to the statistical test, is q 2 [0 , 1] if C ( v i ) 6 = C ( v j ) , and the probability increases to 1  p > q if C ( v i ) =

Original: C ( v j ) .
Revised: C ( v j ) .

Original: This section will show that, if m is sufficiently large (i.e., enough number of tests can be run), there will exist an appropriate threshold for the Sim-M such that the number of error edges can be reduced significantly compared to Sim-S.
Revised: This section will show that, if m is sufﬁciently large (i.e., enough number of tests can be run), there will exist an appropriate threshold for the Sim-M such that the number of error edges can be reduced signiﬁcantly compared to Sim-S.

Original: Proposition 1.
Revised: Proposition 1.

Original: For the single test method, in expectation, the number of noisy edges E { N noisy } =
Revised: For the single test method, in expectation, the number of noisy edges E { N noisy } =

Original: (1 − α ) qm and the number of missing edges E { N miss } =α (1 − p )
Revised: (  ↵ ) qm and the number of missing edges E { N miss } =

Original:  m .
Revised: ↵ (1  p ) m .

Original: The theorem below shows that both E { N noisy } and E { N miss } can be reduced by a significant portion when m is large enough.
Revised: The theorem below shows that both E { N noisy } and E { N miss } can be reduced by a signiﬁcant portion when m is large enough.

Original: Theorem 1.
Revised: Theorem 1.

Original: Suppose γ > 1 and
Revised: Suppose  > 1 and

Original: [Equation]
Revised: [Equation]

Original: γ

Original: [Equation] We have E { N noisy } =
Revised: We have E { N noisy } =

Original: (1 − α ) qm/γ and E { N miss } =
Revised: (1  ↵ ) qm/  and E { N miss } =

Original: α (1 − p ) m/γ .
Revised: ↵ (1  p ) m/  .

Original: Proof.
Revised: Proof.

Original: Consider node v i and one of its candidate node v j ∈ V i .
Revised: Consider node v i and one of its candidate node v j 2 V i .

Original: S ki,j is denoted as the similarity score between v i and v j based on the statistical test with respect to v k .
Revised: S ki,j is denoted as the similarity score between v i and v j based on the statistical test with respect to v k .

Original: As shown in Figure 1, v k may be of the same category as v i or not (denoted by v ′ k for a better show).
Revised: As shown in Figure 1, v k may # " $% … # $ $ # % & &,! $ = (∗#$+ 1−( ∗$# =#$,./ 1 " & ≠1(" ! ) ! ! ! " # … # $ $ # % & &,!

Revised: $ =(∗# ( + 1−( ∗$ ( ,./ 1 " & =1(" ! ) " $%

Revised: Self Attention

Revised: Features of

Revised: [Figure]

Revised: dot

Revised: [Figure]

Revised: addition

Revised: [Figure]

Revised: be of the same category as v i or not (denoted by v 0 k for a better show).

Original: Evidently, E [ S ki,j ] =
Revised: Evidently, E [ S ki,j ] =

Original: pq if
Revised: pq if

Original: C ( v i ) {+
+}
Revised: C ( v i ) {+
+}

Original: ̸ = C ( v j ) , and E [ S ki,j ] =
Revised: = C ( v j ) , and E [ S ki,j ] =

Original: αp 2 + (1 − α ) q 2 if C ( v i ) =
Revised: ↵ p 2 + (1  ↵ ) q 2 if C ( v i ) =

Original: C ( v j ) .
Revised: C ( v j ) .

Original: The Sim-M between v i and v j is defined as:
Revised: The Sim-M between v i and v j is deﬁned as:

Original: [Equation]
Revised: [Equation]

Original: Using the Chernoff Bounds [11] [Equation] we have with a probability 1 − δ , [Equation]
Revised: Using the Chernoff Bounds [11], we have with a probability 1   , 8< :

Original:  [Equation]
Revised: [Equation]

Original: By assuming that m is large enough, i.e.,
Revised: By assuming that m is large enough, i.e.,

Original: [Equation]
Revised: [Equation]

Original: by choosing the similarity threshold
Revised: by choosing the similarity threshold

Original: [Equation]
Revised: [Equation]

Original: only with a probability δ , we will miss a correct edge, and with a probability δ we will introduce a noisy edge.
Revised: only with a probability  , we will miss a correct edge, and with a probability  we will introduce a noisy edge.

Original: We complete the proof by substituting δ in Equation 5 with
Revised: We complete the proof by substituting  in Equation 5 with

Original: [Equation]
Revised: [Equation]

Original: We can guarantee that the Sim-M between two nodes from the same category is strictly larger than that for two nodes from different categories, leading to the distillation of all noisy edges.
Revised: We can guarantee that the Sim-M between two nodes from the same category is strictly larger than that for two nodes from different categories, leading to the distillation of all noisy edges.

Original: 2.3 Multiple tests on GCNs
Revised: 2.3 Multiple tests on GCNs

Original: With a better similarity measure prototype above, we further craft it into an elegant and efficient matrix form for adaptation of graph structure learning in the real-world.
Revised: With a better similarity measure prototype above, we further craft it into an elegant and efﬁcient matrix form for adaptation of graph structure learning in the real-world.

Original: The analysis in Theorem 1 can be easily extended to real similarities, instead of binary simiarities, by using the generalized version of
Revised: The analysis in Theorem 1 can be easily extended to real similarities, instead of binary simiarities, by using the generalized version of

Original: Chernoff inequality detailed in Section E of Appendix.
Revised: Chernoff inequality detailed in Section E of Appendix.

Original: Given the L2 normalized original node features
Revised: Given the L2 normalized original node features

Original: X , the idea of Sim-S can be implemented by inner product of features, i.e. A X =
Revised: X , the idea of Sim-S can be implemented by inner product of features, i.e. A X =

Original: XX T .
Revised: XX T .

Original: The twicerun tests on each node can be achieved by A X multiplying itself.
Revised: The twicerun tests on each node can be achieved by A X multiplying itself.

Original: Inspired by the design of popular
Revised: Inspired by the design of popular

Original: Transformers [53], some learnable parameters are also introduced.
Revised: Transformers [53], some learnable parameters are also introduced.

Original: It contains the fourth-order statistics of features , which is a significant difference, since self-attention only has second-order.
Revised: It contains the fourth-order statistics of features , which is a signiﬁcant difference, since self-attention only has second-order.

Original: If self-attention is regarded as Duet of Attention , our method is analogous to Quartet of Attention ( QAttention for short).
Revised: If self-attention is regarded as Duet of Attention , our method is analogous to Quertet of Attention ( QAttention for short).

Original: The vectorized representation of Equation 3, Q -Attention
Revised: The vectorized representation of Equation 3, Q -Attention

Original: A qart ∈ R L × L , is then implemented as follows:
Revised: A qart 2 R L ⇥ L , is then implemented as follows:

Original: [Equation]
Revised: [Equation]

Original: 𝒗 ! 𝒗 𝒊 … 𝒗 𝒌 𝑣 $% … 𝑝 𝑞 𝑞 𝑝 𝐸 𝑆 &,!
Revised: qart

Original: $ = 𝛼∗𝑝𝑞+ 1−𝛼 ∗𝑞𝑝 =𝑝𝑞,𝑖𝑓 𝐶 𝑣 & ≠𝐶(𝑣 ! )
Revised: [Equation]

Original: 𝒗 ! 𝒗 𝒊 … 𝒗 𝒌 … 𝑝 𝑞 𝑞 𝑝 𝐸 𝑆 &,!{+W Qqart 2 R L ⇥ L 0 and W Kqart 2 R L ⇥ L 0 are learnable weights, L 
+}
Revised: {+W Qqart 2 R L ⇥ L 0 and W Kqart 2 R L ⇥ L 0 are learnable weights, L 
+}

Original: $ =𝛼∗𝑝 ( + 1−𝛼 ∗𝑞 ( ,𝑖𝑓 𝐶 𝑣 & =𝐶(𝑣 ! ) 𝑣 $%
Revised: = L in this work. L2 normalization is also applied to A X .

Original: Self Attention Features of nodes dot product addition -Attention Motivated by the further enhancements from combinations of intrinsic graph structures and implicit graph structures in literature [35, 8, 36], the self-attention can play an intrinsic role to fuse with the
Revised: Motivated by the further enhancements from combinations of intrinsic graph structures and implicit graph structures in literature [35, 8, 36], the self-attention can play an intrinsic role to fuse with the

Original: Q -Attention in vision tasks, although no intrinsic graph structure exists between images at all.
Revised: Q -Attention in vision tasks, although no intrinsic graph structure exists between images at all.

Original: The self-attention part is the same as that in Transformers [53].
Revised: The self-attention part is the same as that in Transformers [53].

Original: The attention map A self ∈ R L × L is:
Revised: The attention map A self 2 R L ⇥ L is:

Original: [Equation]
Revised: [Equation]

Original:  where Q self = XWQself , K self
Revised: , where Q self = XW

Original: [Equation]
Revised: [Equation]

Original: Kself
Revised: , K

Original: [Equation]
Revised: [Equation]

Original: W Qself ∈ R M × M d and W Kself ∈ R M × M d are the learnable weights, M d is the dimension of query and key features.
Revised: W Qself 2 R M ⇥ M d and W Kself 2 R M ⇥ M d are the learnable weights, M d is the dimension of query and key features.

Original: The combined form blends the duet and quartet of attention, hence the name Band of Attention ( B -Attention for short).
Revised: The combined form blends the duet and quartet of attention, hence the name Band of Attention ( B -Attention for short).

Original: The overview of B -Attention mechanism is shown in Figure 2.
Revised: The overview of B -Attention mechanism is shown in Figure 2.

Original: The fusion of A self and A qart is defined as (where softmax ( · ) is a softmax function):
Revised: The fusion of A self and A qart is deﬁned as (where softmax ( · ) is a softmax function):

Original: [Equation]
Revised: [Equation]

Original: When combined with the neural network layer in Equation 1, the final form is:
Revised: When combined with the neural network layer in Equation 1, the ﬁnal form is:

Original: [Equation]
Revised: [Equation]

Original: In real implementation, considering the graph size can be very large for large-scale datasets, sub-graph sampling strategy [57, 55] is adopted to make it more computationally efficient and scalable.
Revised: In real implementation, considering the graph size can be very large for large-scale datasets, sub-graph sampling strategy [57, 55] is adopted to make it more computationally efﬁcient and scalable.

Original: In particular, k seed nodes are first selected as seeds from a dataset in each sampling step.
Revised: In particular, k seed nodes are ﬁrst selected as seeds from a dataset in each sampling step.

Original: Then the seed nodes together with their k nearest neighbours ( k NN) [14] form the nodes of a sub-graph.
Revised: Then the seed nodes together with their k nearest neighbours ( k NN) [14] form the nodes of a sub-graph.

Original: As a result,
Revised: As a result,

Original: Equations 8 and 9 are applied to the sub-graph during training and inference.
Revised: Equations 8 and 9 are applied to the sub-graph during training and inference.

Original: For a dataset of size
Revised: For a dataset of size

Original: N d , it takes about N d /k seed iterations to process the whole dataset on average.
Revised: N d , it takes about N d /k seed iterations to process the whole dataset on average.

Original: The pseudo code for self-attention and Q -Attention is detailed in Section D of Appendix.
Revised: The pseudo code for self-attention and Q -Attention is detailed in Section D of Appendix.

Original:  Experiments
Revised:  Experiments

Original: Experiments are designed with three levels.
Revised: Experiments are designed with three levels.

Original: Firstly, experiments are conducted to analyse the advantage of Sim-M over that of Sim-S and its robustness to noise.
Revised: Firstly, experiments are conducted to analyse the advantage of Sim-M over that of Sim-S and its robustness to noise.

Original: Secondly, armed with multiple tests on
Revised: Secondly, armed with multiple tests on

Original: GCNs, i.e., B -Attention is further investigated in comparison with the self-attention and a commonly used form of GCNs on robustness and superiority.
Revised: GCNs, i.e., B -Attention is further investigated in comparison with the self-attention and a commonly used form of GCNs on robustness and superiority.

Original: Ablation experiments are then conducted to study how each part of B -Attention contributes to and influences the performance.
Revised: Ablation experiments are then conducted to study how each part of B -Attention contributes to and inﬂuences the performance.

Original: Finally, with the robust graph structure learned by B -Attention , comparison experiments in downstream graph-based clustering and ReID tasks are conducted to evaluate the benefits of the proposed approach.
Revised: Finally, with the robust graph structure learned by B -Attention , comparison experiments in downstream graph-based clustering and ReID tasks are conducted to evaluate the beneﬁts of the proposed approach.

Original: 3.1 Experimental setups
Revised: 3.1 Experimental setups

Original: Datasets Experiments are conducted on commonly used visual datasets 4 with three different types of objects, i.e., MS-Celeb [21] (human faces), MSMT17 [59] (human bodies) and VeRi776 [37] (vehicles), to verify the generalization of the proposed method.
Revised: Datasets Experiments are conducted on commonly used visual datasets 4 with three different types of objects, i.e., MS-Celeb [21] (human faces), MSMT17 [59] (human bodies) and VeRi776 [37] (vehicles), to verify the generalization of the proposed method.

Original: In these datasets, each sample has a specific category, and most categories have rich samples.
Revised: In these datasets, each sample has a speciﬁc category, and most categories have rich samples.

Original: The tasks on these datasets are popular and representative in computer vision.
Revised: The tasks on these datasets are popular and representative in computer vision.

Original: It is worth noting that MS-Celeb is divided intoparts by identities (Part0-9): Part0 for training and Part1-9 for testing, to maintain comparability and consistency with previous works [63, 63, 20, 57, 42, 50].
Revised: It is worth noting that MS-Celeb is divided intoparts by identities (Part0-9): Part0 for training and Part1-9 for testing, to maintain comparability and consistency with previous works [63, 63, 20, 57, 42, 50].

Original: Details of the dataset are in Section A.1.
Revised: Details of the dataset are in Section A.1.

Original: Metrics The Area Under the Curve ( AUC ) is adopted to directly evaluate the discriminatory power of similarity measures between node pairs.
Revised: Metrics The Area Under the Curve ( AUC ) is adopted to directly evaluate the discriminatory power of similarity measures between node pairs.

Original: Edge Noise Rate ( ENR ) illustrates the amount of noise in graphs quantitatively.
Revised: Edge Noise Rate ( ENR ) illustrates the amount of noise in graphs quantitatively.

Original: For node v i , ENR = N noisei N alli , where N noisei is the number of noisy edges and
Revised: For node v i , ENR = N noisei N alli , where N noisei is the number of noisy edges and

Original: N alli is the amount of all edges for v i .
Revised: N alli is the amount of all edges for v i .

Original: The average ENR is defined as the average of the ENRs of all nodes.
Revised: The average ENR is deﬁned as the average of the ENRs of all nodes.

Original: The feature quality of GCNs output is investigated to prove the effectiveness of graph structure learning:
Revised: The feature quality of GCNs output is investigated to prove the effectiveness of graph structure learning:

Original: The mean Average Precision ( mAP ) [68] is used from the perspective of identification and
Revised: The mean Average Precision ( mAP ) [68] is used from the perspective of identiﬁcation and

Original: ROC curves are shown from the perspective of verification.
Revised: ROC curves are shown from the perspective of veriﬁcation.

Original: The higher the quality of the features the better the graph structure learning.
Revised: The higher the quality of the features the better the graph structure learning.

Original: In downstream ReID tasks, mAP R and Rank1 accuracy ( R1 ) are used.
Revised: In downstream ReID tasks, mAP R and Rank1 accuracy ( R1 ) are used.

Original: Compared to mAP, mAP R additionally takes into account the camera
Revised: Compared to mAP, mAP R additionally takes into account the camera

Original: id information, i.e., samples from the same camera as the probe will not be counted in evaluation.
Revised: id information, i.e., samples from the same camera as the probe will not be counted in evaluation.

Original: BCubed ( F B ) [3, 1] and Pairwise ( F P ) [51] F-scores are used for evaluating clustering tasks.
Revised: BCubed ( F B ) [3, 1] and Pairwise ( F P ) [51] F-scores are used for evaluating clustering tasks.

Original: 3.2 Effectiveness of the Sim-M
Revised: 3.2 Effectiveness of the Sim-M

Original: [Table]
Revised: [Table]

Original: To verify the robustness of Sim-M, the study is first conducted on MS-Celeb (Part1) by evaluating the discriminatory ability on different average ENRs comparing with Sim-S. The Sim-S uses the cosine similarity here for simplicity and without loss of generality.
Revised: To verify the robustness of Sim-M, the study is ﬁrst conducted on MS-Celeb (Part1) by evaluating the discriminatory ability on different average ENRs comparing with Sim-S. The Sim-S uses the cosine similarity here for simplicity and without loss of generality.

Original: For the sake of sparsity and efficiency it brings, the graphs are built based on k NN via the approximate nearest neighbour algorithm.
Revised: For the sake of sparsity and efﬁciency it brings, the graphs are built based on k NN via the approximate nearest neighbour algorithm.

Original: The average ENR of graphs is controlled by using different k values when searching neighbours.
Revised: The average ENR of graphs is controlled by using different k values when searching neighbours.

Original: AUC is adopted to evaluate the similarity score between node pairs which have k NN relations.
Revised: AUC is adopted to evaluate the similarity score between node pairs which have k NN relations.

Original: Table 1 presents that as the k increases, more and more noise is added (ENR gets larger).
Revised: Table 1 presents that as the k increases, more and more noise is added (ENR gets larger).

Original: For Sim-M, however, more samples can be drawn and more tests can be run, so the advantage of Sim-M over Sim-S becomes increasingly obvious.
Revised: For Sim-M, however, more samples can be drawn and more tests can be run, so the advantage of Sim-M over Sim-S becomes increasingly obvious.

Original: This demonstrates the robustness of Sim-M to noise.
Revised: This demonstrates the robustness of Sim-M to noise.

Original: This phenomenon also conforms to the condition and conclusion in Theorem 1.
Revised: This phenomenon also conforms to the condition and conclusion in Theorem 1.

Original: However, as k increases and ENR gets larger, too much noise can not be handled by multiple tests, so the performance of Sim-M begins to drop.
Revised: However, as k increases and ENR gets larger, too much noise can not be handled by multiple tests, so the performance of Sim-M begins to drop.

Original: This downward trend can be slowed down by the careful design of B -Attention .
Revised: This downward trend can be slowed down by the careful design of B -Attention .

Original: The same experiments are conducted on the MSMT and VeRi-776, with similar findings in Section A.2.
Revised: The same experiments are conducted on the MSMT17 and VeRi-776, with similar ﬁndings in Section A.2.

Original: 3.3 Effectiveness of the B -Attention
Revised: 3.3 Effectiveness of the B -Attention

Original: To demonstrate the robustness of B -Attention mechanism, the study is first conducted on MSCeleb (Part1) by evaluating the performance of B -Attention with different average ENRs, comparing it to other baselines.
Revised: To demonstrate the robustness of B -Attention mechanism, the study is ﬁrst conducted on MSCeleb (Part1) by evaluating the performance of B -Attention with different average ENRs, comparing it to other baselines.

Original: Then, comparison experiments are also conducted on MSMT17, VeRi-776 and also larger scales (Part3-9) of MS-Celeb with the best ENR settings for each dataset.
Revised: Then, comparison experiments are also conducted on MSMT17, VeRi-776 and also larger scales (Part3-9) of MS-Celeb with the best ENR settings for each dataset.

Original: Comparison baselines include the original features of each dataset, a commonly used form of GCN [34, 57],
Revised: Comparison baselines include the original features of each dataset, a commonly used form of GCN [34, 57],

Original: Transformer, GCN with the self-attention (GCN with A self ), Transformer with the B -Attention mechanism (Trans with A band ), and GCN with B -Attention mechanism (GCN with A band ).
Revised: Transformer, GCN with the self-attention (GCN with A self ), Transformer with the B -Attention mechanism (Trans with A band ), and GCN with B -Attention mechanism (GCN with A band ).

Original: Transformer here is only used as an encoder for structure learning, no positional encoding is required.
Revised: Transformer here is only used as an encoder for structure learning, no positional encoding is required.

Original: Network architectures and settings The model is composed of multiple GCN layers with BAttention , and an extra fully connected layer with PReLU [23] activation.
Revised: Network architectures and settings The model is composed of multiple GCN layers with BAttention , and an extra fully connected layer with PReLU [23] activation.

Original: The output features have a dimension of 2048 for all datasets.
Revised: The output features have a dimension of 2048 for all datasets.

Original: The number of GCN layers is also tuned respectively for different datasets: MS-Celeb uses three GCN layers; MSMT17 and VeRi-776 use one GCN layer.
Revised: The number of GCN layers is also tuned respectively for different datasets: MS-Celeb uses three GCN layers; MSMT17 and VeRi-776 use one GCN layer.

Original: Each node acts as the probe to search its k NN to construct the graph.
Revised: Each node acts as the probe to search its k NN to construct the graph.

Original: The k value is tuned respectively for different datasets: 120 for MS-Celeb, 30 for MSMT17, and 80 for VeRi-776.
Revised: The k value is tuned respectively for different datasets: 120 for MS-Celeb, 30 for MSMT17, and 80 for VeRi-776.

Original: Each node in a dataset can be enhanced multiple times, as either a probe or a neighbour.
Revised: Each node in a dataset can be enhanced multiple times, as either a probe or a neighbour.

Original: The Hinge Loss [45, 57] is used to classify node pairs.
Revised: The Hinge Loss [45, 57] is used to classify node pairs.

Original: The initial learning rate is 0.008 with the cosine annealing strategy.
Revised: The initial learning rate is 0.008 with the cosine annealing strategy.

Original: Performance on different average ENRs Models are trained on Part0 and tested on Part1 of
Revised: Performance on different average ENRs Models are trained on Part0 and tested on Part1 of

Original: MS-Celeb.
Revised: MS-Celeb.

Original: All baselines use three GCN/Transformer layers for MS-Celeb, except the GCN with
Revised: All baselines use three GCN/Transformer layers for MS-Celeb, except the GCN with

Original: A self which uses two GCN layers (observed to have better performance than three GCN layers).
Revised: A self which uses two GCN layers (observed to have better performance than three GCN layers).

Original: For a certain ENR, the same k value is used in training and testing.
Revised: For a certain ENR, the same k value is used in training and testing.

Original: Figure 3 shows how the output feature qualities (mAP) change with different average ENRs for all baselines.
Revised: Figure 3 shows how the output feature qualities (mAP) change with different average ENRs for all baselines.

Original: It shows that the GCN and
Revised: It shows that the GCN and

Original: Transformer with A band work the best in most average ENRs in terms of both mAP values and their robustness to different ENRs.
Revised: Transformer with A band work the best in most average ENRs in terms of both mAP values and their robustness to different ENRs.

Original: The GCN has the worst results, and the performances of the ones with
Revised: The GCN has the worst results, and the performances of the ones with

Original: A self are in between.
Revised: A self are in between.

Original: This validates that a commonly used form of GCN is sensitive to noisy edges in input graphs; the proposed B -Attention can well alleviate the influence of noisy edges and show the strongest robustness when learning graph structure; the A self also has some power of dealing with noisy edges, but it is worse than A band .
Revised: This validates that a commonly used form of GCN is sensitive to noisy edges in input graphs; the proposed B -Attention can well alleviate the inﬂuence of noisy edges and show the strongest robustness when learning graph structure; the A self also has some power of dealing with noisy edges, but it is worse than A band .

Original: When k value is too small (e.g., 5, 10), the performances of
Revised: When k value is too small (e.g., 5, 10), the performances of

Original: A band and A self are similar.
Revised: A band and A self are similar.

Original: This indicates that the Q -Attention part of B -Attention can hardly bring a positive effect when there are too few context neighbours, i.e. the number of single tests gathered is insufficient.
Revised: This indicates that the Q -Attention part of B -Attention can hardly bring a positive effect when there are too few context neighbours, i.e. the number of single tests gathered is insufﬁcient.

Original: In addition, GCN and Transformer have very similar performance, either with
Revised: In addition, GCN and Transformer have very similar performance, either with

Original: A band or A self .
Revised: A band or A self .

Original: This indicates that base networks are not critical, and both attention mechanisms work for the two base networks.
Revised: This indicates that base networks are not critical, and both attention mechanisms work for the two base networks.

Original: The performance degradation of B -Attention with very large ENR (e.g., 0.64) indicates that the current form B -Attention still has improvement room, although works much better than the self-attention.
Revised: The performance degradation of B -Attention with very large ENR (e.g., 0.64) indicates that the current form B -Attention still has improvement room, although works much better than the self-attention.

Original: Some empirical case studies on self attention, Q -Attention and
Revised: Some empirical case studies on self attention, Q -Attention and

Original: B -Attention are shown in Section F of Appendix to better understand the effect of multiple tests.
Revised: B -Attention are shown in Section F of Appendix to better understand the effect of multiple tests.

Original: [Figure]
Revised: [Table]

Revised: [Table]

Original: [Table]
Revised: [Table]

Original: Performance on different types of objects To further validate the B -Attention mechanism, comparisons are also made on datasets with other types of objects: MSMT17 and VeRi-776.
Revised: Performance on different types of objects To further validate the B -Attention mechanism, comparisons are also made on datasets with other types of objects: MSMT17 and VeRi-776.

Original: In these two datasets, k values and network depths are also tuned for all baselines, aiming for their best performances.
Revised: In these two datasets, k values and network depths are also tuned for all baselines, aiming for their best performances.

Original: For both datasets, one GCN layer is used for GCN, GCN with A band , and Transformer with A band ; three GCN layers are used for GCN or Transformer with A self .
Revised: For both datasets, one GCN layer is used for GCN, GCN with A band , and Transformer with A band ; three GCN layers are used for GCN or Transformer with A self .

Original: In MSMT17, k = for GCN and “GCN w/ A self ”; k =20 for Transformer; k =30 for the two B -Attention methods.
Revised: In MSMT17, k = for GCN and “GCN w/ A self ”; k =20 for Transformer; k =30 for the two B -Attention methods.

Original: In
Revised: In

Original: VeRi-776, k =80 for “GCN w/ A band ” and “Trans w/ A band ”, and k =20 for the rest of baselines.
Revised: VeRi-776, k =80 for “GCN w/ A band ” and “Trans w/ A band ”, and k =20 for the rest of baselines.

Original: The best mAP results of all baselines are listed in Table 2, where the results for MS-Celeb are the peak values in Figure 3.
Revised: The best mAP results of all baselines are listed in Table 2, where the results for MS-Celeb are the peak values in Figure 3.

Original: From Table 2, the B -Attention works the best in all three datasets; A self works better than GCN; and all output features are much better than the original ones, although the original features of MSMT17 and VeRi-776 are current SOTA ones.
Revised: From Table 2, the B -Attention works the best in all three datasets; A self works better than GCN; and all output features are much better than the original ones, although the original features of MSMT17 and VeRi-776 are current SOTA ones.

Original: The best performances are achieved by the GCN with A band in all datasets.
Revised: The best performances are achieved by the GCN with A band in all datasets.

Original: Their ROC curves are shown in Figure 4, where the same observations can be seen from verification perspective.
Revised: Their ROC curves are shown in Figure 4, where the same observations can be seen from veriﬁcation perspective.

Original: As shown in Figure 4 and Table 2, the performance differences among those baselines with attention mechanisms are less significant in MSMT17 where k values are relatively small and close across baselines.
Revised: As shown in Figure 4 and Table 2, the performance differences among those baselines with attention mechanisms are less signiﬁcant in MSMT17 where k values are relatively small and close across baselines.

Original: This observation is similar to that in Figure 3: the performance margin between A band and A self is less significant when 
Revised: This observation is similar to that in Figure 3: the performance margin between A band and A self is less signiﬁcant when average ENRs (i.e., k ) are relatively small. This further shows that the beneﬁt of B -Attention is limited when with a small number of neighbours, as not many single tests to run and leverage. Better performance can not be achieved by enlarging k values for MSMT17, as its average sample amount for each individual is only 31. Our analysis assumes that for any node v , there are more nodes in the k nearest neighbors sharing the same category as v than those sharing different categories (i.e.

Original: [Figure]
Revised: ↵ > 12 ).

Original: positive rate 0. T r u e p o s i
Revised: By signiﬁcantly enlarging k , we will end up with ↵ < 12 in k nearest neighbors, which fails the assumption of our algorithm.

Original: t i v e r a t e GCN [Figure]
Revised: Too large k values would result in too big ENR which can not be handled by the current form of B -Attention as discussed on Figure 3.

Original: [Figure] [Figure] w/ A self TransGCNOriginal (a)
Revised: Experiments are also conducted to compare these baselines on larger scales of MS-Celeb (i.e.,

Original: [Figure] Part [Figure] [Figure]
Revised: Part3-9) in Section A.3 of Appendix.

Original: [Figure]

Original: [Figure] rate [Figure] [Figure]
Revised: Results in Table 6 and Figure 5 show that all baselines

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: p

Original: [Figure]i [Figure]
Revised: [Figure]

Original: i

Original: [Figure]

Original: [Figure] TransGCNOriginal ( [Figure] MSMT
Revised: Dataset

Original: [Figure]
Revised: [Table]

Original: rate

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: T r u e p

Original: o s i t i v e r a t e
Revised: Part

Original: GCN w/ A band Trans w/ A band GCN w/ A self TransGCNOriginal (c) VeRi- [Table]
Revised: Part

Original: Metrics
Revised: Metrics

Original: F P
Revised: F P

Original: F B
Revised: F B

Original: F P
Revised: F P

Original: F B
Revised: F B

Original: F P
Revised: F P

Original: F B
Revised: F B

Original: F P
Revised: F P

Original: F B
Revised: F B

Original: F P
Revised: F P

Original: [Table] B
Revised: F B

Original: K-Means 79.
Revised: K-Means 79.

Original: 69.42 HAC 70.
Revised: 69.42 HAC 70.

Original: 66.96 DBSCAN 67.
Revised: 66.96 DBSCAN 67.

Original: [Table]
Revised: Clusformer 88.

Original: average ENRs (i.e., k ) are relatively small.
Revised: 79.95 STAR-FC 91. 81.47 Ada-NETS 92.

Original: This further shows that the benefit of B -Attention is limited when with a small number of neighbours, as not many single tests to run and leverage. Better performance can not be achieved by enlarging k values for MSMT17, as its average sample amount for each individual is only 31.
Revised: Original+G-cut 69.

Original: Our analysis assumes that for any node v , there are more nodes in the k nearest neighbors sharing the same category as v than those sharing different categories (i.e. α > 12 ). By significantly enlarging k , we will end up with α < 12 in k nearest neighbors, which fails the assumption of our algorithm. Too large k values would result in too big ENR which can not be handled by the current form of B -Attention as discussed on Figure 3. Experiments are also conducted to compare these baselines on larger scales of MS-Celeb (i.e.,
Revised: 58.54 Ours+G-cut 93.

Original: Part3-9) in Section A.3 of Appendix.
Revised: Original+Infomap 93.

Original: Results in Table 6 and Figure 5 show that all baselines have performance degradation when the scale enlarges while the degradation of that “w/ A band ” is much smaller than baselines.
Revised: [Table] have performance degradation when the scale enlarges while the degradation of that “w/ A band ” is much smaller than baselines.

Original: These results for the cases with different ENRs, on different types of objects and larger scales of testsets, together validate the robustness and effectiveness of proposed
Revised: These results for the cases with different ENRs, on different types of objects and larger scales of testsets, together validate the robustness and effectiveness of proposed

Original: B -Attention in graph structure learning.
Revised: B -Attention in graph structure learning.

Original: Ablation experiments on the design of the topological architecture and attention fusion method are also conducted in Section A.4 of Appendix.
Revised: Ablation experiments on the design of the topological architecture and attention fusion method are also conducted in Section A.4 of Appendix.

Original: 3.4 Effectiveness of the downstream tasks
Revised: 3.4 Effectiveness of the downstream tasks

Original: The improvement of graph structure learning can also be proven through the downstream tasks by exploiting the features it produces.
Revised: The improvement of graph structure learning can also be proven through the downstream tasks by exploiting the features it produces.

Original: The better the graph structure is learned, the better the performance on downstream tasks.
Revised: The better the graph structure is learned, the better the performance on downstream tasks.

Original: Graphs have important applications in large-scale clustering and re-reranking in ReID.
Revised: Graphs have important applications in large-scale clustering and re-reranking in ReID.

Original: However, they suffer from inaccurate weight problems when learning and working with the graph structure.
Revised: However, they suffer from inaccurate weight problems when learning and working with the graph structure.

Original: Armed the GCN with A band , many SOTA results are achieved.
Revised: Armed the GCN with A band , many SOTA results are achieved.

Original: Performance on clustering tasks The clustering tasks are evaluated on MS-Celeb and MSMT17.
Revised: Performance on clustering tasks The clustering tasks are evaluated on MS-Celeb and MSMT17.

Original: Both of them have a large number of classes and are suitable as clustering benchmarks.
Revised: Both of them have a large number of classes and are suitable as clustering benchmarks.

Original: The same features output by “GCN w/ A band ” are used as that in Section A.3 for MS-Celeb and Section 3.
Revised: The same features output by “GCN w/ A band ” are used as that in Section A.3 for MS-Celeb and Section 3.

Original: for MSMT17.
Revised: for MSMT17.

Original: The same clustering method (denoted “G-cut”) is adopted as that in [20, 57], where pairs of nodes are linked when their similarity scores are larger than a tuned threshold, and clustering is finally completed by transitively merging all links via a union-find algorithm [2].
Revised: The same clustering method (denoted “G-cut”) is adopted as that in [20, 57], where pairs of nodes are linked when their similarity scores are larger than a tuned threshold, and clustering is ﬁnally completed by transitively merging all links via a union-ﬁnd algorithm [2].

Original: The threshold is tuned for each dataset, aiming for the best balance between BCubed and Pairwise F-Scores.
Revised: The threshold is tuned for each dataset, aiming for the best balance between BCubed and Pairwise F-Scores.

Original: In addition, map equation based unsupervised clustering algorithm Infomap [46, 61] is also adopted here.
Revised: In addition, map equation based unsupervised clustering algorithm Infomap [46, 61] is also adopted here.

Original: Their BCubed and Pairwise F-scores are listed in Table 3 and Table 10, where the results of current SOTA GCN/Transformer methods (Clusformer [42], STAR-FC [50] and Ada-NETS [57]) and three conventional clustering approaches (K-Means [38], HAC [52] and DBSCAN [17]) are also listed.
Revised: Their BCubed and Pairwise F-scores are listed in Table 3 and Table 10, where the results of current SOTA GCN/Transformer methods (Clusformer [42], STAR-FC [50] and Ada-NETS [57]) and three conventional clustering approaches (K-Means [38], HAC [52] and DBSCAN [17]) are also listed.

Original: The results show that the features enhanced via our approach have the best clustering results in all datasets, either with G-cut or Infomap, both of which outperform the current SOTA methods
Revised: The results show that the features enhanced via our approach have the best clustering results in all datasets, either with G-cut or Infomap, both of which outperform the current SOTA methods

Original: Ada-NETS and STAR-FC
Revised: Ada-NETS and STAR-FC

Original: with large margins.
Revised: with large margins.

Original: In comparison, the results of the ones with Infomap are better than those with G-cut
Revised: In comparison, the results of the ones with Infomap are better than those with G-cut.

Original: [Table][Table]
Revised: [Table]

Original: Performance on ReID tasks
Revised: Performance on ReID tasks

Original: The
Revised: The

Original: ReID tasks is evaluated on two ReID datasets: MSMT17 and VeRi-776.
Revised: ReID tasks is evaluated on two ReID datasets: MSMT17 and VeRi-776.

Original: The enhanced features used are the same as those in Table 2.
Revised: The enhanced features used are the same as those in Table 2.

Original: mAP R and R1 are evaluated following the standard probe and gallery protocols of MSMT17 and
Revised: mAP R and R1 are evaluated following the standard probe and gallery protocols of MSMT17 and

Original: VeRi-776.
Revised: VeRi-776.

Original: The results are shown in Table 4.
Revised: The results are shown in Table 4.

Original: As the output feature of learned graph can also be understood as a rerank process in ReID, comparisons are also made on commonly used rerank methods: QE [13], LBR [39], KR [67] and
Revised: As the output feature of learned graph can also be understood as a rerank process in ReID, comparisons are also made on commonly used rerank methods: QE [13], LBR [39], KR [67] and

Original: ECN [47].
Revised: ECN [47].

Original: Table 4 shows that the “GCN w/ A band ” achieves the best or comparable performance, and all enhanced features are much better than the original
Revised: Table 4 shows that the “GCN w/ A band ” achieves the best or comparable performance

Original:  SOTA ones.
Revised: [Table] and all enhanced features are much better than the original SOTA ones.

Original: In MSMT17, similar to the results in Section 3.3, the baselines with attention mechanism have very close performance, but work much better than the GCN.
Revised: In MSMT17, similar to the results in Section 3.3, the baselines with attention mechanism have very close performance, but work much better than the GCN.

Original: In VeRi-776, from the mAP R perspective, B -Attention works better than A self , and both of them are better than the GCN, but their R1 results are comparable.
Revised: In VeRi-776, from the mAP R perspective, B -Attention works better than A self , and both of them are better than the GCN, but their R1 results are comparable.

Original: These results still support the superiority of the proposed approach, although R1 results of VeRi-776 are comparable, as mAP R is a more comprehensive indication of feature quality for ReID.
Revised: These results still support the superiority of the proposed approach, although R1 results of VeRi-776 are comparable, as mAP R is a more comprehensive indication of feature quality for ReID.

Original: The experiments on the combination of “GCN w/ A band ” with these commonly used rerank methods are also conducted in Section A.6, achieving more SOTA performances and proving the complementarity between them.
Revised: The experiments on the combination of “GCN w/ A band ” with these commonly used rerank methods are also conducted in Section A.6, achieving more SOTA performances and proving the complementarity between them.

Original:  Related Work
Revised:  Related Work

Original: Graph Convolutional Networks.
Revised: Graph Convolutional Networks.

Original: GCNs [34, 22, 54] extends the operation of convolution from the grid-data (e.g. images, videos) to non-grid data which is more common in the real world.
Revised: GCNs [34, 22, 54] extends the operation of convolution from the grid-data (e.g. images, videos) to non-grid data which is more common in the real world.

Original: The classic GCN [34] is proposed based on the spectral theory and achieves promising results on citation network datasets: Citeseer, Cora and Pubmed [49].
Revised: The classic GCN [34] is proposed based on the spectral theory and achieves promising results on citation network datasets: Citeseer, Cora and Pubmed [49].

Original: GraphSAGE [22] further extends GCNs from transductive framework to inductive framework and obtains better results.
Revised: GraphSAGE [22] further extends GCNs from transductive framework to inductive framework and obtains better results.

Original: GAT [54] introduces the attention mechanism in GCN, making it more expressive.
Revised: GAT [54] introduces the attention mechanism in GCN, making it more expressive.

Original: Fast-GCN [6] interprets graph convolutions as integral transforms of embedding functions under probability measures, which not only is efficient for training but also generalizes well for inference.
Revised: Fast-GCN [6] interprets graph convolutions as integral transforms of embedding functions under probability measures, which not only is efﬁcient for training but also generalizes well for inference.

Original: Some research works [48, 66, 27, 56, 7] also advance GCNs for relational data modeling.
Revised: Some research works [48, 66, 27, 56, 7] also advance GCNs for relational data modeling.

Original: However, these GCNs are all investigated on relational datasets that have explicit graph structures.
Revised: However, these GCNs are all investigated on relational datasets that have explicit graph structures.

Original: This work aims to extend the capacity of GCNs to work on datasets (e.g., image objects) which have no explicitly defined graph structures.
Revised: This work aims to extend the capacity of GCNs to work on datasets (e.g., image objects) which have no explicitly deﬁned graph structures.

Original: Graph Structure Learning.
Revised: Graph Structure Learning.

Original: The literature [60] suggests that research on graph structure learning for GCNs can be roughly divided into discrete-based and weighted-based ones.
Revised: The literature [60] suggests that research on graph structure learning for GCNs can be roughly divided into discrete-based and weighted-based ones.

Original: Some discretebased methods [16, 31, 40, 18] treat the graph structures as random distributions and sample binary adjacency matrices from them.
Revised: Some discretebased methods [16, 31, 40, 18] treat the graph structures as random distributions and sample binary adjacency matrices from them.

Original: Another way [57] constructs discrete graphs based on heuristic proxy objectiveness aiming for clean yet rich neighbours for each node.
Revised: Another way [57] constructs discrete graphs based on heuristic proxy objectiveness aiming for clean yet rich neighbours for each node.

Original: The binary matrices methods are difficult to optimize because of the gradient breakage, so such methods resort to variational inference (e.g., approaches in [16, 40]), reinforcement learning (e.g., approaches in [31]) or optimization in stages (e.g., approaches in [57]), which may lead to sub-optimal solutions.
Revised: The binary matrices methods are difﬁcult to optimize because of the gradient breakage, so such methods resort to variational inference (e.g., approaches in [16, 40]), reinforcement learning (e.g., approaches in [31]) or optimization in stages (e.g., approaches in [57]), which may lead to sub-optimal solutions.

Original: In addition, the parameter size of distributions [18] can be very large (e.g., O ( Nk ) where N is the number of nodes and k is the number of neighbours for each node) and therefore not suitable for large datasets.
Revised: In addition, the parameter size of distributions [18] can be very large (e.g., O ( Nk ) where N is the number of nodes and k is the number of neighbours for each node) and therefore not suitable for large datasets.

Original: The weighted-based methods [9, 10, 43, 64, 28, 8, 25, 35, 20, 42], which are the focus of this paper, learn weighted adjacency matrices so that it can be optimized by SGD techniques [44, 33, 5].
Revised: The weighted-based methods [9, 10, 43, 64, 28, 8, 25, 35, 20, 42], which are the focus of this paper, learn weighted adjacency matrices so that it can be optimized by SGD techniques [44, 33, 5].

Original: The main difference between these works [9, 10, 43, 64, 28, 8, 25, 35] is in the similarity measure, which can be classified into attention-based [9, 10, 43, 64, 28, 8] and kernelbased [25, 35] roughly.
Revised: The main difference between these works [9, 10, 43, 64, 28, 8, 25, 35] is in the similarity measure, which can be classiﬁed into attention-based [9, 10, 43, 64, 28, 8] and kernelbased [25, 35] roughly.

Original: To enhance the discriminatory power of the similarity measure, they have adopted various forms: adding learnable parameters [9, 28, 8], introducing nonlinearity [10, 43, 64],
Revised: To enhance the discriminatory power of the similarity measure, they have adopted various forms: adding learnable parameters [9, 28, 8], introducing nonlinearity [10, 43, 64],

Original: Gaussian kernel functions [25, 35] or self-attention [20, 42, 53, 12] etc.
Revised: Gaussian kernel functions [25, 35] or self-attention [20, 42, 53, 12] etc.

Original: However, these methods are sensitive to the noise from feature representations, particular when each node is represented by a high dimensional vector, leading to the problem of inaccurate weights and vulnerable graph structure.
Revised: However, these methods are sensitive to the noise from feature representations, particular when each node is represented by a high dimensional vector, leading to the problem of inaccurate weights and vulnerable graph structure.

Original: This work aims to propose a multiple samples based similarity measure to improve the robustness of graph structure learning over images.
Revised: This work aims to propose a multiple samples based similarity measure to improve the robustness of graph structure learning.

Original: Another approach uses the low-rank, sparsity and feature smoothness properties of the real-world graph to optimize the graph structure [29], but these priors are suitable for community networks rather than graph of image objects.
Revised: Another approach uses the low-rank, sparsity and feature smoothness properties of the real-world graph to optimize the graph structure [29] but these priors are suitable for community networks rather than graph of image objects.

Original: Some work [30] learns to augment the given graph with new edges by reinforcement learning to improve the performance of downstream tasks, but existing noisy edges problem in graphs cannot be handled.
Revised: Some work [30] learns to augment the given graph with new edges by reinforcement learning to improve the performance of downstream tasks, but existing noisy edges problem in graphs cannot be handled.

Original:  Conclusion
Revised:  Conclusion

Original: The problem of inaccurate edge weights is identified in graph structure learning, analysed from the perspective of noise in feature vectors, and handled based on several tests.
Revised: The problem of inaccurate edge weights is identiﬁed in graph structure learning, analysed from the perspective of noise in feature vectors and handled based on several tests.

Original: A novel and robust similarity measure Sim-M is proposed, proven to be superior and verified in experiments.
Revised: A novel and robust similarity measure Sim-M is proposed, proven to be superior and veriﬁed in experiments.

Original: Then it is designed in an elegant matrix form, i e., B -Attention mechanism, to improve the graph structure quality in GCNs.
Revised: Then it is designed in an elegant matrix form, i e., B -Attention mechanism, to improve the graph structure quality in GCNs.

Original: The superiority and robustness of B -Attention are validated by comprehensive experiments and achieve many SOTA results on clustering and ReID benchmarks.
Revised: The superiority and robustness of B -Attention are validated by comprehensive experiments and achieve many SOTA results on clustering and ReID benchmarks

Original: Acknowledgments and Disclosure of Funding
Revised: Acknowledgments and Disclosure of Funding

Original: We thank many colleagues at DAMO Academy of Alibaba, in particular, Yuqi Zhang, Weihua Chen and Hao Luo for useful discussions.
Revised: We thank many colleagues at DAMO Academy of Alibaba, in particular, Yuqi Zhang, Weihua Chen and Hao Luo for useful discussions

Original: This research was partially funded by the National Science Foundation of China (No. 62172443) andHunan Provincial Natural Science Foundation of China (No. 2022JJ30053)
Revised: This research was partially funded by the National Science Foundation of China (No. 62172443) and

Original: A Experiments A.1 Details of the dataset

Original: MS-Celeb [21] is a dataset with 10M face images of nearly 100K individuals.

Original: A cleaned version [15] with about 5.8M face images of 86K identities is used in this paper.

Original: The protocol and features are the same as that in [63], where the dataset is divided into 10 parts by identities (Part0-9): Part0 for training and Part1-9 for testing.

Original: The feature dimension is 256.

Original: The data amounts of Part 0-9 (0, 1, 3, 5, 7 and 9) are respectively: 584K, 584K, 1.74M, 2.89M, 4.05M and 5.21M.

Original: MSMT17 [59] is the current largest ReID dataset.

Original: It contains 32,621 images of 1,041 individuals for training and 93,820 images of 3,060 individuals for testing.

Original: These images are captured by 15 cameras under different time periods, weather and light conditions.

Original: This work uses the SOTA features from

Original: AGW [65] with a dimension of 2048.

Original: VeRi-776 [37] is a commonly used vehicle ReID benchmark, which contains over 40K bounding boxes of 619 vehicles captured by 20 cameras in unconstrained traffic scenes.

Original: The dataset is divided into two parts: 37,715 images of 576 vehicles for training, and 13,257 images of 200 vehicles for testing.

Original: This work uses the features extracted by the SOTA ViT-B/16 Baseline in TransReID

Original: [24], with 768 dimensions.

Original: A.2 Effectiveness of the Sim-M

Original: The experiments on MSMT17 and VeRi-776 also show that the advantage of Sim-M over Sim-S is increasingly obvious as k increases and then drops for introduction of too much noise as in Table 5.

Original: [Table]

Original: A.3 Performance on larger scales of MS-Celeb

Original: Experiments are also conducted to compare these baselines on larger scales of MS-Celeb (i.e.,

Original: Part3-9).

Original: The same model for MS-Celeb Part1 is used as in Section 3.3, but tested on Part3-9.

Original: The results in Table 6 show that all baselines have performance degradation when the scale enlarges.

Original: The

Original: B -Attention obtains the best results in all parts; A self works better than GCN; all output features are much better than the original ones.

Original: Among the baselines with attention mechanisms, the ones using GCN-based networks have better performances.

Original: The GCN with A band has the highest mAP scores.

Original: Figure 5 shows the ROC curves of GCN, “GCN w/ A self ” and “GCN w/ A band ” across all parts of MS-Celeb, from which we can also see the performance degradation with larger testsets.

Original: The performance degradation of “GCN w/ A band ” is much smaller than the other two baselines, which is shown more clearly from the differences of AUC in Table 7.

Original: In Table 7, differences on

Original: AUC of two adjacent parts (e.g., part1 - part3, part3 - part5, part5 - part7, part7 - part9) and their summation for each method are shown.

Original: It can be observed that the summation of the differences of “GCN w/ A band ” is 1.49, which is only 58.30% and 67.26% of the ones of GCN (2.56) and “GCN with A self " (2.22).

Original: This validates that the features from “GCN w/ A band ” have a much smaller performance degradation than the other two baselines with larger testsets from the perspective of verification.

Original: Similar observations can also be seen in Table 6, where the performance degradation of the baselines with attention mechanisms is much smaller than GCN and original features.

Original: 1.0 False positive rate 0.

Original: T r u e p

Original: o

Original: s i

Original: t i v e r a t

Original: e Part1Part3Part5Part7Part (a) GCN 0. 1.0 False positive rate 0. T r u e p
Revised: Hunan Provincial Natural Science Foundation of China (No. 2022JJ30053)

Original: o

Original: s i

Original: t i v e r a t

Original: e

Original: Part1Part3Part5Part7Part (b) GCN with A self 0.

Original: 1.0 False positive rate 0.

Original: T r u e p

Original: o

Original: s i

Original: t i v e r a t

Original: e

Original: Part1Part3Part5Part7Part (c) GCN with A band

Original: Partition

Original: [Table]

Original: [Table]

Original: [Table]

Original: GCN 90.

Original: [Table]

Original: [Table]

Original: [Table]

Original: Transformer

Original: [Table]

Original: GCN w

Original: [Table]

Original: Trans w/

Original: A band 95.

Original: GCN w/ A band 95.

Original: [Table]

Original: [Table]

Original: A.4 Ablation study on B -Attention

Original: Ablation experiments are conducted mainly on MS-Celeb to study the design of the topological architecture and attention fusion method of B -Attention .

Original: Similar to Section 3.3, models are trained on Part0 and tested on Part1.

Original: All settings other than the factor being investigated are the same as that in Section 3.3 for “GCN with A band ” with k = 120 .

Original: A.4.1 topological architecture design

Original: This study compares the topological architecture of the B -Attention with three other designs: (1) one with only the A qart of B -Attention ; (2) one using only the Q -Attention part but with the A X replaced by A self in Equation 8, denoted as (cid:101)

Original: A qart ; (3) and one with B -Attention but with the A qart replaced by (cid:101)

Original: A qart , denoted as (cid:101)

Original: A band .

Original: Table 8 shows their mAP results, where the results of “GCN w/ A band ” and one with only A self are also listed for comparison.

Original: For the purpose of ablation study, the one with only A self here has the same depth (i.e., three layers) with “GCN w/ A band ”, which is different from that in Table 2 (two layers).

Original: The comparisons are made on two conditions: with or without W qart (i.e., W Q qart and W K qart ) in Equation 8.

Original: For the cases without W qart , W Q qart and W Kqart are both identity matrices, i.e., Q qart =

Original: K qart =

Original: A X .

Original: Their ROC curves are shown in Figure 6.

Original: It can be seen that the B -Attention design achieves the best results in both mAP and

Original: ROC; A qart has better performance than A self ; (cid:101)

Original: A qart and (cid:101)

Original: A band are worse than A qart and A band respectively.

Original: This shows that the combination of A self and A qart and the use of A X (instead of

Original: A self ) in A qart both contribute to the performance improvement.

Original: In addition, the baselines with

Original: W qart have better performances (higher mAP scores and better ROC) than those without, indicating that W Qqart and W Kqart are also critical for better graph structure learning.

Original: [Table]

Original: The ablation experiments are also conducted on attention map fusion method in Section A.4.

Original: in Appendix .

Original: The results in Table 9 and Figure 7 together will prove that the proposed form in

Original: Equation 11 has the best performance across all datasets.

Original: A.4.2 Attention map fusion method

Original: The attention map fusion method is studied by comparing it to two other designs: direct element-wise addition without weighting ( A qart + A self ) and element-wise multiplication ( A qart ⊙ A self ) in

Original: Equation 11.

Original: The results are shown in Table 9 and Figure 7: The θ qart A qart + θ self A self design works the best, and the performance of A qart ⊙ A self is very close to θ qart A qart + θ self A self .

Original: However, results of MSMT17 and VeRi-776 in Table 9 shows our proposed method has the best performance across all datasets.

Original: [Table]

Original: T r u e p

Original: o

Original: s i

Original: t i v e r

Original: [Figure]

Original: [Table]

Original: A.5 Clustering performance on MSMT

Original: The clustering tasks are also evaluated on MSMT17 and the results are in Table 10.

Original: A.6 Ensemble Performance on ReID tasks

Original: In addition to directly comparing with the commonly used rerank methods (e.g., QE [13], LBR [39],

Original: KR [67] and ECN [47]) in Section 4.4, the “GCN with A band " can also be combined with them.

Original: The commonly used rerank methods can utilize the enhanced features from “GCN with A band " for better ranking.

Original: The results are shown in Table 11.

Original: Since mAP R can provide a more comprehensive evaluation of the feature quality than R1 for ReID, the mAP R is discussed more often.

Original: It can be observed that new SOTA performances are obtained on both datasets.

Original: The mAP R on MSMT17 is as high as 79.67 by “ A band +KR" and on VeRi-776 is as high as 85.92 by “ A band +LBR".

Original: For most of the commonly used rerank methods, the mAP R of reranking with A band is higher than with original features and also higher than “GCN w/ A band ”.

Original: The only exception is “ A band +ECN" which is higher than “GCN w/ A band ” but only comparable with “original+ECN".

Original: The results show that there is a complementary relationship between “GCN with A band " and commonly used rerank methods.

Original: The “GCN with A band " can provide better features so that the commonly used rerank methods can be further improved, although the R 1 results are only comparable.

Original: [Table]

Original: A.7 Declaration of the type of computing resources

Original: All experiments are conducted on a single machine, with 54 E5-2682 v4 CPUs, 8 NVIDIA P cards and 400G memory.

Original: B Limitations

Original: Our approach achieves promising results in graph structure learning, but it still has the following limitations:

Original: • The B -Attention requires more computation and number of parameters as shown in Section 2.3.

Original: However, the baseline methods can not achieve the same performance even adding their network depth.

Original: The layer number of network in each method is tuned in all experiments.

Original: • The superiority of B -Attention is not obvious enough on datasets with few samples in each category compared with self attention as revealed in Section 3.3.

Original: • The performance of B -Attention will also degrade when with large ENR as discussed in Section 3.3, indicating the current form still has improvement room on the robustness against noise to further investigate.

Original: • The outputs B -Attention will vary depending on the choice of nearest neighbours.

Original: However, the superiority of this method can be guaranteed as long as the conditions in Section 2.2 are met.

Original: C Potential Negative Societal Impacts

Original: The proposed graph structure learning method inherently is harmless just like many other AI technologies.

Original: This technology may have negative societal impacts if someone uses it for malicious applications such as surveillance, personal information collections.

Original: As authors, we advocate for proper technology usage.

Original: D Pseudocode for self-attention and Q -Attention

Original: # X: the input feature matrix, with shape (N, D) def self_attention(X):

Original: Q_self=

Original: torch.matmul(X, W_self_Q)

Original: K_self =

Original: torch.matmul(X, W_self_K)

Original: self_attn = torch.matmul(Q_self, K_self.transpose(-2, -1)) return self_attn def Q_attention(X):

Original: A_X=torch.matmul(X, X.transpose(-2, -1))

Original: Q_qart =

Original: torch.matmul(A_X, W_qart_Q)

Original: K_qart = torch.matmul(A_X, W_qart_K)

Original: Q_attn = torch.matmul(Q_qart, K_qart.transpose(-2, -1)) return Q_attn

Original: E Detailed Proof of with generalized version of Chernoff Bounds

Original: Below is a full analysis for the practical algorithm in Section 2.3 that does not assume a binary relationship between two nodes in A X .

Original: Let X i denote a random variable with no distribution assumptions such that a ≤ X i ≤ b for all i .

Original: Define X = (cid:80) m 1 X

Original: i , µ = E ( X ) .

Original: Then for all 0 < ϵ < 1 , we have the general version of Chernoff inequality as follows.

Original: ( i ) Upper Tail: P ( X ≥ (1 + ϵ ) µ )

Original: ≤ e − ϵ 2 µ 2 m ( b − a )2 ; ( ii ) Lower Tail: P ( X ≤ (1 − ϵ ) µ )

Original: ≤ e − ϵ 2 µ 2 m ( b − a )2 .

Original: The above show that with probability at least 1 − δ , (1 − ϵ ) µ ≤ X ≤ (1 + ϵ ) µ where δ = e − ϵ 2 µ

Original: [Equation]

Original: b − a

Original: [Equation]

Original: or equivalently ϵ = (cid:113) m

Original: ( b − a

Original: [Equation]

Original: 2 log (1 /

Original: [Equation]

Original: µ.

Original: Instead of assuming p and q for making connections in the case of C ( v i )

Original: =

Original: C ( v j ) and C ( v i )

Original: ̸ = C ( v j ) , we assume that the similarity score s i,j is sampled from a bounded distribution (bounded between -1 and +1) with mean of s + when C ( v i ) =

Original: C ( v j ) , and s i,j is sampled from a bounded distribution (bound between -1 and +1) with mean of s − when C ( v i )

Original: =

Original: C ( v j ) .

Original: We of course assume s + > s − .

Original: In expectation, the number of edges connected under S ki,j on V

Original: i,j is

Original: [Equation]

Original: [Equation]

Original: where C ( v ) is the category of node v .

Original: It should be noted that ( b − a ) 2 = 4 under our assumption.

Original: Therefore, when v i , v j are sampled from different categories, we have, with probability at least 1 − δ ,

Original: [Equation]

Original: When v i

Original: [Equation]

Original: v j are sampled from the same category

Original: [Equation]

Original: we have

Original: [Equation]

Original: with probability at least 1 − δ ,

Original: [Equation]

Original: The two distributions can be well separated if the above two bounds do not overlap .

Original: That is, we require that

Original: [Equation]

Original: On the other hand, it is obvious s + s − < αs 2+ + (1 − α )

Original: s 2 − , so we know

Original: [Equation]

Original: [Equation]

Original: Set ϵ

Original: =

Original: ϵ − to have a larger step and a stricter condition

Original: [Equation]

Original: we have

Original: [Equation]

Original: [Equation]

Original: Because α > 12 and s + > s − ,

Original: [Equation]

Original: So we have

Original: [Equation]

Original: We complete the proof by substituting δ with

Original: [Equation]

Original: [Equation]

Original: where γ > 1 is a constant.

Original: Finally, we prove that

Original: [Equation]

Original: γ min(

Original: [Equation]

Original: [Equation]

Original: −

Original: [Equation]

Original: [Equation]

Original: ) (cid:19) .

Original: We can guarantee that when the above condition is met, the proposed similarity measure between two nodes from the same category is strictly larger than that for two nodes from different categories.

Original: It is able to reduce the number of noisy edges and missing edges of the single test method by 1 /γ in expectation.

Original: F Case Studies on self attention, Q -Attention and B -Attention

Original: To better understand the effect of multiple tests (i.e. the fourth order of statistics), we illustrate the effect of self-attention, Q -Attention and B -Attention with the help of the tool BertViz 5 .

Original: [Figure]

Original: As shown in the left panel of Figure 8, self-attention introduces a number of noisy connections between nodes belonging to different categories.

Original: In contrast, according to the middle panel of

Original: Figure 8, Q -Attention is able to introduce many more connections between nodes from the same category, and at the same time significantly reduces the weights for those noisy connections.

Original: In the last panel, the combination self-attention with the fourth order statistics, The B -Attention further removes those noisy connections while keeps most of the clean connections between nodes.