Paper: CTRL: Graph condensation via crafting rational trajectory matching (ICLR-2024)

Reviewer comment (Novelty):
The core techniques introduced in this paper, including the Euclidean distance component and replacing the random initialization of the synthetic data by selecting nodes from sub-clusters formulated by K-means, have all been previously adopted for dataset condensation [1][2]. This makes the proposed method appear to be a straightforward mash-up of multiple existing methods in dataset condensation, resulting in limited novelty.

Author response:
Thank you very much for your comments on our method.

In summary, our approach differs significantly from previous methods, not merely as a straightforward mash-up. While there might be some formal similarities, our methodology is grounded in **entirely different principles and analytical processes**. Furthermore, we provide robust experimental evidence to support our claims.

First of all, DosCond [A] does not provide a detailed explanation for the choice of using Euclidean distance. In practice, although the paper employs Euclidean distance, it is worth noting that the official code on GitHub utilizes MSE distance for experimentation. In contrast, our decision to use Euclidean distance **stems from** pre-experiments demonstrating **the strong correlation** between gradient magnitude measured with Euclidean distance and the frequency domain distribution of the graph. Furthermore, experimental results confirm that our approach better preserves the frequency domain distribution of the graph.

In addition, the current initialization methods within the realm of graph compression exhibit several issues. For instance, both real data sampling and core-set-based initialization pose challenges. The former is overly simplistic and may result in **uneven sampling**, while the latter necessitates training the GNN on the **entire dataset** to obtain improved embeddings for core-set selection, which is particularly challenging for large datasets. In contrast, our method only requires clustering of data within each class, significantly reducing the computational resource requirements. Experimental results also demonstrate that our approach effectively captures the feature distribution of the original data, offering an enhanced starting point for synthetic graph optimization with almost no additional computational cost.

To further illustrate the distinctions between our approach and previous methods, we conducted experiments by directly integrating our method with existing approaches (improving GCOND by the sampling method in Dream[B], and altering the gradient matching method based on MSE distance, respectively). The results indicate that this direct combination led to counterproductive outcomes. In contrast, our paper shows that both components of CTRL can effectively improve the performance of existing methods.

Note that all results in the table are averaged over three repetitions for accuracy and consistency.

|                            | GCond | GCond+Dream           |       GCond(mse)              |
| - | - | - | - |
| Cora (ratio = 1.30%)       | 79.9  | 78.8±1.1  ${\downarrow}$1.1 | 78.4±0.8  ${\downarrow}$1.3 |
| Cora (ratio = 2.60%)       | 80.0  | 78.0±0.3  ${\downarrow}$2.1 | 72.4±0.5  ${\downarrow}$7.6 |
| Cora (ratio = 5.20%)       | 79.1  | 77.8±0.2  ${\downarrow}$1.2 | 74.2±0.1  ${\downarrow}$4.9 |
| Citeseer (ratio = 0.90%)   | 70.7  | 68.3±0.3  ${\downarrow}$2.4 | 64.8±0.5  ${\downarrow}$5.9 |
| Citeseer (ratio = 1.80%)   | 70.9  | 68.1±1.1  ${\downarrow}$2.8 | 62.3±1.0  ${\downarrow}$8.6 |
| Citeseer (ratio = 3.60%)   | 70.3  | 68.0±1.1  ${\downarrow}$2.3 | 63.9±0.8  ${\downarrow}$6.4 |
| Flickr (ratio = 0.10%)     | 46.5  | 43.6±0.2  ${\downarrow}$2.9 | 46.3±0.3  ${\downarrow}$0.2 |
| Flickr (ratio = 0.50%)     | 47.1  | 44.1±0.3  ${\downarrow}$3.0 | 46.7±0.2  ${\downarrow}$2.4 |
| Flickr (ratio = 1.0%)      | 47.1  | 44.7±0.1  ${\downarrow}$2.4 | 46.1±0.1  ${\downarrow}$1.1 |
| Ogbn-XRT (ratio = 0.05%)   | 55.1  | 52.8±0.4  ${\downarrow}$2.3 | 49.4±0.3  ${\downarrow}$5.8 |
| Ogbn-XRT (ratio = 0.25%)   | 68.1  | 64.6±0.7  ${\downarrow}$3.5 | 60.7±0.7  ${\downarrow}$7.4 |
| Ogbn-XRT (ratio = 0.50%)   | 69.3  | 66.1±0.5  ${\downarrow}$3.2 | 63.3±0.4  ${\downarrow}$6.1 |
| Ogbn-arxiv (ratio = 0.05%) | 60.2  | 56.6±0.7  ${\downarrow}$3.6 | 55.3±0.4  ${\downarrow}$4.9 |
| Ogbn-arxiv (ratio = 0.25%) | 63.4  | 59.3±0.6  ${\downarrow}$4.1 | 62.7±0.7  ${\downarrow}$0.7 |
| Ogbn-arxiv (ratio = 0.50%) | 64.8  | 60.9±0.5  ${\downarrow}$3.9 | 63.4±0.5  ${\downarrow}$1.4 |
| Reddit (ratio = 0.01%)     | 88.0  | 85.9±1.1  ${\downarrow}$2.1 | 80.9±1.0  ${\downarrow}$7.1 |
| Reddit (ratio = 0.05%)     | 89.6  | 84.8±0.7  ${\downarrow}$5.2 | 83.9±0.9  ${\downarrow}$6.4 |
| Reddit (ratio = 0.50%)     | 90.1  | 85.3±0.6  ${\downarrow}$4.8 | 86.6±0.9  ${\downarrow}$3.5 |

Reviewer comment (Theory):
Although the authors repeatedly emphasize that their method considers the impact of matching gradient magnitude rather than only matching directions, in actual implementation, they introduced a regularization term based on the Euclidean distance between gradients. This is somewhat inconsistent with their statement and might be misleading, as the Euclidean distance also considers differences beyond just the magnitude of vectors.

Author response:
We appreciate your attention to detail on this point.

To sum up, the role of the Euclidean distance is not that of a regularization term. Rather, it serves as **a crucial metric** for assessing gradient differences and is instrumental in fitting frequency domain distributions. And we don't find sufficient literature demonstrating that the Euclidean distance captures information beyond gradient distances.

Firstly, the CTRL method represents an enhancement of existing gradient matching approaches. The role of Euclidean distance is to collaborate with cosine distance for **improved gradient measurement and matching in the frequency domain**. It is not merely a regularization term but rather serves as a means to better quantify gradients and perform matching in the frequency domain. The utilization of Euclidean distance in the improvement of previously researched gradient matching methods based solely on cosine distance is not contradictory to our claims, and it does not lead to any misleading interpretations. Secondly, we acknowledge that Euclidean distance, as a **widely recognized method for measuring gradient magnitude**, raises questions about the additional information it considers. We opted for using Euclidean distance to measure gradient magnitude because our preliminary experiments demonstrated a strong correlation between the gradient magnitude measured by Euclidean distance and the frequency domain distribution of the graph. Many studies involving the measurement of gradient magnitude employed Euclidean distance without explicit evidence of considering additional information. While experimental results do indicate that using Euclidean distance can yield better results than MSE distance, this is largely attributed to the superior ability of Euclidean distance to measure the magnitude of gradients effectively.

Reviewer comment (Evaluation):
From the presented results, it seems that the proposed method does not offer significant improvements over the compared state-of-the-art.

Author response:
In comparison to GCond and DosCond, the primary baselines for gradient matching, CTRL demonstrates average improvements of approximately **2.4% and 1.9%**, with maximum improvements reaching **6% and 6.2%**, respectively. 

Notably, even when compared to the latest method SFGC, CTRL still achieves superior results, while SFGC exhibits excellent performance in node classification, it incurs significant computational costs. For instance, the computation time for the buffer may even exceed that used in the distillation process. In contrast, CTRL achieves **better results at a lower cost**. Furthermore, it is crucial to highlight that the SFGC method lacks the capability, as demonstrated by CTRL, to perform dataset distillation in graph classification tasks. These aspects underscore the significance of the CTRL method in the field of graph dataset distillation.

Reviewer comment (Theory):
The empirical results of this paper show that minimizing the Euclidean distance between model gradients can align the frequency distribution between synthetic and real graphs. However, there is no theoretical analysis of the relationship between gradient magnitude and frequency distribution. Additionally, it is unclear to me why matching the frequency distribution of graphs helps the distillation process.

Author response:
In summary, due to the **strong correlation** between gradient magnitudes and the frequency domain distribution of the graph, it is possible to indirectly fit the frequency domain distribution by optimizing gradient magnitudes. Our experimental results also **validate this assumption**.

Existing research has already demonstrated the significant impact of both frequency domain distribution and graph structure in graph datasets on the performance of Graph Neural Networks (GNNs) [C][D]**.** This motivates us to investigate whether enhancing the distillation of datasets can be achieved by fitting the frequency domain information of synthesized graphs to that of the original graphs.

However, explicitly optimizing graph structural information in the domain of graph data generation is particularly challenging because of the substantial differences in scale between the two and the matrix decomposition involved in many indicators of frequency distribution, making it a computationally intensive process. In our investigation of the correlation between gradient magnitude and the frequency domain distribution of graphs, we found a strong correlation between the two, giving rise to the idea of utilizing gradient magnitude matching to fit frequency distributions.

The core rationale behind matching the frequency distribution lies in the strong correlation between gradient magnitude and the frequency distribution within the graph. By **optimizing one** of these aspects, it becomes possible to **implicitly enhance the other**. In other words, through matching gradient magnitudes, we guide the synthesis of graphs to possess frequency domain distributions that are closer to those of the original graphs. Our experimental results further affirm that this method leads to **better preservation** of the original graph's structural information, ultimately resulting in improved performance.

Reviewer comment (Experiments):
There is no ablation study to evaluate the roles of direction-based matching, magnitude-based matching, and their combination.

Author response:
In brief, the ablation experiments demonstrate that the **combined utilization** of cosine distance and Euclidean distance for gradient matching **surpasses** employing **either of them** individually.

We further supplemented the ablation study to assess the performance differences associated with direction-based matching, magnitude-based matching (including mse-based and norm-based), and CTRL, as illustrated in the table below. Note that all results in the table are based on the same initialization method, i.e. the random true initialization employed in GCond for fair comparison, and all CTRL methods are subjected to the same loss function.

Note that all results in the table are averaged over three repetitions for accuracy and consistency.

|                            | GCond(paper) | GCond(reproduction) | GCond(mse) | GCond(norm) |
| -------------------------- | ------------ | ------------------- | ---------- | ----------- |
| Cora (ratio = 1.30%)       | 79.8±1.3     | 79.9±1.1            | 78.4±0.8   | 79.2±0.4    |
| Cora (ratio = 2.60%)       | 80.1±0.6     | 80.0±0.3            | 72.4±0.5   | 80.8±0.4    |
| Cora (ratio = 5.20%)       | 79.3±0.3     | 79.1±0.2            | 74.2±0.1   | 79.3±0.6    |
| Citeseer (ratio = 0.90%)   | 70.5±0.5     | 70.7±0.3            | 64.8±0.5   | 71.7±0.3    |
| Citeseer (ratio = 1.80%)   | 70.6±0.9     | 70.9±1.0            | 62.3±1.0   | 72.5±0.7    |
| Citeseer (ratio = 3.60%)   | 69.8±1.4     | 70.3±1.1            | 63.9±0.8   | 72.7±0.4    |
| Flickr (ratio = 0.10%)     | 46.5±0.4     | 46.6±0.3            | 46.3±0.3   | 45.9±0.2    |
| Flickr (ratio = 0.50%)     | 47.1±0.1     | 47.1±0.3            | 46.7±0.2   | 46.8±0.1    |
| Flickr (ratio = 1.0%)      | 47.1±0.1     | 47.1±0.1            | 46.1±0.1   | 47.0±0.1    |
| Ogbn-XRT (ratio = 0.05%)   | -            | 55.3±0.4            | 49.4±0.3   | 53.2±0.3    |
| Ogbn-XRT (ratio = 0.25%)   | -            | 68.2±0.7            | 60.7±0.7   | 66.5±0.4    |
| Ogbn-XRT (ratio = 0.50%)   | -            | 69.1±0.6            | 63.3±0.4   | 68.8±0.3    |
| Ogbn-arxiv (ratio = 0.05%) | 59.2±1.1     | 60.3±0.6            | 55.3±0.4   | 58.7±0.5    |
| Ogbn-arxiv (ratio = 0.25%) | 63.2±0.3     | 63.2±0.4            | 62.7±0.7   | 64.4±0.8    |
| Ogbn-arxiv (ratio = 0.50%) | 64.0±1.4     | 64.9±0.3            | 63.4±0.5   | 64.9±0.4    |
| Reddit (ratio = 0.01%)     | 88.0±1.8     | 88.3±1.3            | 80.9±1.0   | 85.6±0.6    |
| Reddit (ratio = 0.05%)     | 89.6±0.7     | 89.2±0.8            | 83.9±0.9   | 87.1±0.7    |
| Reddit (ratio = 0.50%)     | 90.1±0.5     | 90.1±0.7            | 86.6±0.9   | 88.3±0.6    |

Reviewer comment (Experiments):
This paper does not report of performance of SFGC [2], which is a strong graph condensation baseline, in Tables 3 and 4.

Author response:
Thank you very much for your insightful suggestion regarding the inclusion of SFGC in our work.

SFGC indeed serves as a robust baseline, demonstrating outstanding performance in tasks such as node classification. However, we did not compare our approach with SFGC in Table 3 and Table 4, primarily due to the following considerations: 

1. It follows a fundamentally **distinct technological trajectory** based on parameter matching compared to CTRL, which relies on gradient matching. Much like the disparity between IDC and MTT in CV DD, the commendable cross-architecture performance of SFGC is predicated on extensive time and memory-consuming training of expert models. Given equivalent computational and memory resources, a gradient matching-based approach can feasibly distill corresponding synthetic datasets for each model architecture. 
2. SFGC didn't present experimental results for the Versatility experiment in Table 4 and the Neural Architecture Search experiment in Table 5. Owing to constraints in time and computational resources, we are unable to undertake a corresponding reproduction for comparative purposes.

Reviewer comment (Writing):
Minor Typo. In several instances, the authors mistakenly write ‘Ogbn-arxiv’ as ‘Ogbn-arvix’.

Author response:
We have carefully reviewed the entire manuscript and corrected all typos. Thank you again for your meticulous review and comments!

Reviewer comment (Novelty):
The overall method is similar to the recently proposed graph condensation methods (especially GCond as it also uses a gradient matching scheme). The novelty is somewhat limited, although the struggle to capture the feature distribution in this problem is interesting.

Author response:
To sum up, gradient matching is a commonly employed technique, and it has gradually become outdated in the field of CV DD. Our approach not only **reveals the shortcomings** of the current criterion of gradient matching in the realm of GRAPH DD from a frequency domain perspective, but also demonstrates **the further research value** of gradient matching in GRAPH DD.

In Data Distillation, gradient matching is a very general scheme. In CV DD, DC[B] is one of the earliest methods based on gradient matching. Many other methods, such as DSA[C] and IDC[D], are improved methods based on DC. Similarly, we explore further to improve the gradient matching approach in graph condensation. 

To delineate our contributions more precisely, we can categorize them into three main facets:

1. Unlike previous studies, we **depart from** **frequency domain distribution** to investigate the relationship between gradient magnitude and graph structure. Furthermore, we elaborate on the **rationale** behind introducing Euclidean distance in gradient matching. It is noteworthy that we implemented similar enhancements on the IDC, but the improvement was not nearly as significant as observed in the graph dataset distillation. For instance, the average improvement is about 0.2% and the maximum improvement is about 1.1%, while compared to GCond and DosCond, the average improvements of CTRL are about 2.4% and 1.9%, up to 6% and 6.2%, respectively.

2. In the realm of CV DD, the advent of the parameter-matching method MTT[E] has seemingly rendered gradient matching outdated. However, our research effectively demonstrates that in the GRAPH DD domain, even in the absence of the time and space-consuming buffer process, a gradient-matching approach can outperform parameter-matching methods by incorporating frequency domain distribution matching. Our research demonstrates **the potential of gradient-based methods in the GRAPH DD domain**.

3. Additionally, many initialization methods commonly employed in CV DD cannot be directly applied in GRAPH DD. For example, the selection of images that can be correctly classified may not be directly transferable to the context of graph datasets, while the current initialization methods within the realm of graph compression are overly simplistic. Therefore, we propose an initialization method that can capture the feature distribution of original data, and provides a better optimization starting point for synthetic graph while introducing almost no additional computational cost. This is an area that **has not been explored in previous research**.


We demonstrate the effectiveness of CTRL's two components through extensive experimentation. Additionally, the two approaches are easy to follow and can be effortlessly integrated into other gradient-matching-based graph condensation methods.

Reviewer comment (Experiments):
Can the authors explain in detail why leveraging gradient magnitude matching is correlated with frequency distribution? Did the authors also measure the correlation between gradient direction matching (cosine similarity between gradients) and frequency distribution? These correlations can be investigated through running some experiments.

Author response:
In brief, due to the **strong correlation** between gradient magnitudes and the frequency domain distribution of the graph, it is possible to indirectly fit the frequency domain distribution by optimizing gradient magnitudes. Our experimental results also **validate this assumption**.

Existing research has already demonstrated the significant impact of both frequency domain distribution and graph structure in graph datasets on the performance of Graph Neural Networks (GNNs) [F][G]. This motivates us to investigate whether enhancing the distillation of datasets can be achieved by fitting the frequency domain information of synthesized graphs to that of the original graphs.

However, explicitly optimizing graph structural information in the domain of graph data generation is particularly challenging because of the substantial differences in scale between the two and the matrix decomposition involved in many indicators of frequency distribution, making it a computationally intensive process. In our investigation of the correlation between gradient magnitude and the frequency domain distribution of graphs, we found a strong correlation between the two, giving rise to the idea of utilizing gradient magnitude matching to fit frequency distributions.

The core rationale behind leveraging gradient magnitude matching lies in the strong correlation between gradient magnitude and the frequency distribution within the graph. By **optimizing one** of these aspects, it becomes possible to **implicitly enhance the other**. In other words, through matching gradient magnitudes, we guide the synthesis of graphs to possess frequency domain distributions that are closer to those of the original graphs. Our experimental results further affirm that this method leads to **better preservation** of the original graph's structural information, ultimately resulting in improved performance



To sum up, experimental results demonstrate that the variation in gradient direction exhibits **a much lower correlation**  (average of 0.569 compared to an average of 0.905)  with the frequency domain distribution of the graph compared to the gradient magnitudes.

If the question referring to whether matching gradients can preserve certain frequency domain information, i.e., whether the synthetic images generated by the GCond method can exhibit frequency distribution similar to that of the original images, as shown in Table 6, it may achieve this to some extent, but not as effective as our proposed method.  We conducted additional experiments and found **a relatively weak correlation between gradient direction and frequency domain distribution**, compared to the result in Table 9. This also explains why the gradient matching method based on cosine distance can also generate a composite graph with some information in the frequency domain of the original graph.

The table below shows the Spearman correlation coefficients of the high-frequency area and gradient direction under different graph models and GNN architectures, where the gradient direction is measured by the variance of an array of cosine distance between gradients of the previous epoch and the next epoch.

|                 | SGC   | Cheby | APPNP | GCN   |
| - | - | - | - | - |
| Erdos-Renyi     | 0.675 | 0.755 | 0.513 | 0.614 |
| Barabasi-Albert | 0.667 | 0.577 | 0.758 | 0.313 |
| Watts-Strogatz  | 0.386 | 0.422 | 0.489 | 0.661 |

Reviewer comment (Novelty):
Novelty of the proposed methods is limited, which seems a simple adaptation of an existing method from CV literature, and the proposed methods do not appear to be specifically designed for graphs. Firstly, the loss formula (Equation 3) that considers gradient magnitude is identical to Equation 9 in [1]. Secondly, the initialization step using k-means is performed on node features, which overlooks the influence of graph structure. For instance, if two nodes have the same features but significantly differ in their neighbors, the approach described in the paper would not select both nodes simultaneously, even though they would have completely different embeddings after message passing.

Author response:
**Weaknesses1-1: Novelty of the proposed methods is limited, which seems a simple adaptation of an existing method from CV literature, and the proposed methods do not appear to be specifically designed for graphs**.

**A1：**

We thank the reviewer, and will clarify our technical novelty and significance during revision.

To sum up, the utilization of Euclidean distance in [A] and our approach **stems from entirely distinct motivations, effects, and theoretical foundations.** The introduction of matching gradient magnitudes in GRAPH DD effectively captures the frequency domain distributions of both the original and synthetic graphs. Conversely, such a phenomenon is not observed in [A]. Our experimental results also demonstrate that **seemingly similar approaches can lead to markedly different outcomes.**

First and foremost, a notable distinction from [A] lies in our extended **analysis of the rationale** behind incorporating Euclidean distance in GRAPH DD. Specifically, the frequency domain distribution of the graph encompasses structural information, our experiments show that matching Euclidean distance effectively preserves the high and low-frequency information of the original graph(which is important for GNN training), as shown in Table 6, thereby enhancing the performance of the synthesized dataset.

While in Data Distillation of Computer Vision (CV DD), there is a lack of similar research conclusions. Specifically, [A] proposed a method that introduces Euclidean distance into gradient matching, yielding only a marginal improvement of around 0.5%. In comparison to the improvements introduced in GRAPH DD, the performance is significantly lower. Additionally, we conducted experiments on the state-of-the-art (among those based on gradient matching)  method IDC in CV DD, as shown in the table below, IDC(cos) indicates the original IDC method and IDC(norm) means the IDC method based on Euclidean distance matching.

As shown in the table below, IDC(norm) demonstrated a **significantly lower performance** compared to IDC(cos). Conversely, in GRAPH DD, the Euclidean distance-based matching method, GCond(norm) even exhibited **slightly higher performance** than the cosine distance-based method GCond(cos) in some experiments. This observation further substantiates our claim that by matching gradient magnitudes, we can better approximate the frequency domain distribution of the original graph with synthetic graphs, thereby achieving superior performance.

Note: we use ${\downarrow}$ and ${\uparrow}$ to denote the reduction and increase of Acc, all results in the table are averaged over three repetitions for accuracy and consistency.

|              | IPC  | IDC (cos) | IDC (norm)              |
| - | - | - | - |
| MNIST        | 1    | 94.2      | 90.2  ${\downarrow}$4.0 |
| MNIST        | 10   | 98.4      | 93.2  ${\downarrow}$5.2 |
| MNIST        | 50   | 99.1      | 94.3  ${\downarrow}$4.9 |
| FashionMNIST | 1    | 81.0      | 79.6  ${\downarrow}$1.4 |
| FashionMNIST | 10   | 86.0      | 83.1  ${\downarrow}$2.9 |
| FashionMNIST | 50   | 86.2      | 84.3  ${\downarrow}$1.9 |
| SVHN         | 1    | 68.5      | 61.5  ${\downarrow}$7.0 |
| SVHN         | 10   | 87.5      | 81.4  ${\downarrow}$6.1 |
| SVHN         | 50   | 90.1      | 87.1  ${\downarrow}$3.0 |
| CIFAR-10     | 1    | 50.6      | 45.5  ${\downarrow}$4.9 |
| CIFAR-10     | 10   | 67.5      | 63.5  ${\downarrow}$4.0 |
| CIFAR-10     | 50   | 74.5      | 72.3  ${\downarrow}$2.2 |

|  | Ratio | GCond(cos) | GCond(norm) |
| - | - | - | - |
| Cora  | 1.30% | 79.9 | 79.2${\downarrow}$0.7 |
| Cora  | 2.60% | 80.0 | 80.9$\uparrow$0.9 |
| Cora  | 5.20% | 79.1 | 79.3$\uparrow$0.2 |
| Citeseer | 0.90% | 70.7 | 71.7$\uparrow$1.0 |
| Citeseer | 1.80% | 70.9 | 72.4$\uparrow$1.5 |
| Citeseer | 3.60% | 70.3 | 72.7$\uparrow$2.4 |
| Flickr | 0.10% | 46.5 | 45.9${\downarrow}$0.6 |
| Flickr | 0.50% | 47.1 | 46.8${\downarrow}$0.3 |
| Flickr | 1.0% | 47.1 | 47.0${\downarrow}$0.1 |
| Ogbn-XRT | 0.05% | 55.1 | 53.2${\downarrow}$1.9 |
| Ogbn-XRT | 0.25% | 68.1 | 66.5${\downarrow}$1.6 |
| Ogbn-XRT | 0.50% | 69.3 | 68.9${\downarrow}$0.4 |
| Ogbn-arxiv | 0.05% | 60.2 | 58.7${\downarrow}$1.5 |
| Ogbn-arxiv | 0.25% | 63.4 | 64.4$\uparrow$1.0 |
| Ogbn-arxiv | 0.50% | 64.8 | 64.9$\uparrow$0.1 |
| Reddit | 0.01% | 88.0 | 85.6${\downarrow}$2.4 |
| Reddit | 0.05% | 89.6 | 87.1${\downarrow}$2.5 |
| Reddit | 0.50% | 90.1 | 88.3${\downarrow}$1.8 |

Reviewer comment (Experiments):
The setup of the ablation experiments is not clearly defined. It is not specified which loss was used to train the models in Figure 3 (c) and (d). It is not specified what initialization was used for the model in Figure 3 (e).

Author response:
**Q2：It is not specified which loss was used to train the models in Figure 3 (c) and (d). What is the value of** **$\beta$** **in the training of models in Figure 3 (c) and (d)? How is the models in Figure 3 (e) initialized?**

**A4:**

Thanks for pointing out this specific matter within the paper.

The values of $\beta$ in the training of models in Figure 3 (c) and (d) are 0.1 and 0.15, respectively. We incorporate this information into the article.

In Figure 3 (e), we employ the specific initialization method(K-MEANS-based) in CTRL for Ogbn-arxiv, and random sampling for cora and citeseer, as shown in A.2.

We supplement a complete ablation experiment here, as follows, all the results are obtained by repeating experiments for 3 times.

|                            | GCond(paper) | GCond(reproduction) | GCond+initlization | CTRL w/o initlization    |
| - | - | - | - | - |
| Cora (ratio = 1.30%)       | 79.8±1.3     | 79.9±1.1            | 79.4±0.8           | 81.9±0.4($\beta$ = 0.7)  |
| Cora (ratio = 2.60%)       | 80.1±0.6     | 80.0±0.3            | 80.4±0.6           | 81.8±0.5($\beta$ = 0.05) |
| Cora (ratio = 5.20%)       | 79.3±0.3     | 79.1±0.2            | 79.2±0.3           | 81.8±0.6($\beta$ = 0.15) |
| Citeseer (ratio = 0.90%)   | 70.5±0.5     | 70.7±0.3            | 70.8±0.3           | 73.3±0.3($\beta$ = 0.9)  |
| Citeseer (ratio = 1.80%)   | 70.6±0.9     | 70.9±1.0            | 71.1±1.1           | 73.5±0.8($\beta$ = 0.9)  |
| Citeseer (ratio = 3.60%)   | 69.8±1.4     | 70.3±1.1            | 70.9±0.9           | 73.4±0.6($\beta$ = 0.9)  |
| Flickr (ratio = 0.10%)     | 46.5±0.4     | 46.6±0.3            | 46.5±0.3           | 47.1±0.2($\beta$ = 0.7)  |
| Flickr (ratio = 0.50%)     | 47.1±0.1     | 47.1±0.3            | 46.8±0.3           | 47.4±0.1($\beta$ = 0.05) |
| Flickr (ratio = 1.0%)      | 47.1±0.1     | 47.1±0.1            | 47.1±0.2           | 47.5±0.1($\beta$ = 0.15) |
| Ogbn-XRT (ratio = 0.05%)   | -            | 55.3±0.4            | 57.2±0.3           | 61.1±0.3($\beta$ = 0.3)  |
| Ogbn-XRT (ratio = 0.25%)   | -            | 68.2±0.7            | 69.7±0.5           | 69.4±0.4($\beta$ = 0.15) |
| Ogbn-XRT (ratio = 0.50%)   | -            | 69.1±0.6            | 69.3±0.5           | 70.4±0.4($\beta$ = 0.3)  |
| Ogbn-arxiv (ratio = 0.05%) | 59.2±1.1     | 60.3±0.6            | 62.3±0.7           | 64.9±0.7($\beta$ = 0.3)  |
| Ogbn-arxiv (ratio = 0.25%) | 63.2±0.3     | 63.2±0.4            | 64.7±0.6           | 66.7±0.6($\beta$ = 0.9)  |
| Ogbn-arxiv (ratio = 0.50%) | 64.0±1.4     | 64.9±0.3            | 65.4±0.7           | 67.3±0.3($\beta$ = 0.3)  |
| Reddit (ratio = 0.01%)     | 88.0±1.8     | 88.3±1.3            | 88.9±1.1           | 88.9±0.5($\beta$ = 0.1)  |
| Reddit (ratio = 0.05%)     | 89.6±0.7     | 89.2±0.8            | 89.9±0.9           | 90.1±0.8($\beta$ = 0.45) |
| Reddit (ratio = 0.50%)     | 90.1±0.5     | 90.1±0.7            | 90.6±0.3           | 91.3±0.7($\beta$ = 0.45) |

Reviewer comment (Writing):
There are quite a few typos in the paper where "ogbn-arxiv" is misspelled.

Author response:
**Q4: Minor Typo**

**A6:**

Thanks for the comment.

We have carefully reviewed the entire manuscript and corrected all typos. Thank you again for your meticulous review and comments!

Reviewer comment (Writing):
Are there any differences in applying CTRL to graphs compared to its application in computer vision?

Author response:
**Q1: Are there any differences in applying CTRL to graphs compared to its application in Computer Vision?**

**A3：**

Thanks for the comment. 

In summary, directly using our method in CV DD **does not yield as favorable results** as seen in GRAPH DD. As previously mentioned, the effective rationale behind introducing Euclidean distance in graph DD does not apply in this context.

To further demonstrate the difference between DD in GRAPH and CV yields,  we tried to add CTRL to CV DD, which didn't achieve such good improvement. For instance, we conduct a series of experiments based on two kinds of gradient matching schemes, [A] (We introduce $\beta$) and IDC[B] (We introducing Euclidean distance), the average improvement is about **0.2%** and the maximum improvement is about **1.1%**, while compared to GCond and DosCond, the average improvements of CTRL are about **2.4% and 1.9%**, up to **6% and 6.2%**, respectively.

Note that introducing $\beta$ can slightly improve the method in [A], as shown in the table below, we tried three values of​ $\beta$, 0.1, 0.5, and 0.9, and show the results that have improvement higher than 0.1%. Where MD denotes the method proposed in [A] using both cosine distance and Euclidean distance(equal weight), while MDA is the improved version of MD in [A], all results in the table are averaged over three repetitions for accuracy and consistency.

|              | IPC  | MD       | MD+CTRL                 | MDA      | MDA+CTRL                |
| - | - | - | - | - | - |
| FashionMNIST | 10   | 85.4±0.3 | 85.4±0.3                | 84.2±0.3 | 84.7±0.1  $\uparrow$0.5 |
| FashionMNIST | 50   | 87.4±0.2 | 87.6±0.2  $\uparrow$0.2 | 87.9±0.2 | 87.9±0.2                |
| SVHN         | 10   | 75.9±0.7 | 75.9±0.4                | 75.9±0.7 | 76.3±0.2  $\uparrow$0.4 |
| SVHN         | 50   | 82.9±0.2 | 83.0±0.1  $\uparrow$0.1 | 83.2±0.3 | 83.4±0.3  $\uparrow$0.2 |
| CIFAR-10     | 1    | 30.0±0.6 | 31.0±0.4  $\uparrow$1.0 | -        | -                       |
| CIFAR-10     | 10   | 49.5±0.5 | 49.5±0.3                | 50.2±0.6 | 51.3±0.6  $\uparrow$1.1 |

Reviewer comment (Reproducibility):
What is the value of beta in the training of models in Figure 3 (c) and (d)? How is the models in Figure 3 (e) initialized?

Author response:
**Q2：It is not specified which loss was used to train the models in Figure 3 (c) and (d). What is the value of** **$\beta$** **in the training of models in Figure 3 (c) and (d)? How is the models in Figure 3 (e) initialized?**

**A4:**

Thanks for pointing out this specific matter within the paper.

The values of $\beta$ in the training of models in Figure 3 (c) and (d) are 0.1 and 0.15, respectively. We incorporate this information into the article.

In Figure 3 (e), we employ the specific initialization method(K-MEANS-based) in CTRL for Ogbn-arxiv, and random sampling for cora and citeseer, as shown in A.2.

We supplement a complete ablation experiment here, as follows, all the results are obtained by repeating experiments for 3 times.

|                            | GCond(paper) | GCond(reproduction) | GCond+initlization | CTRL w/o initlization    |
| - | - | - | - | - |
| Cora (ratio = 1.30%)       | 79.8±1.3     | 79.9±1.1            | 79.4±0.8           | 81.9±0.4($\beta$ = 0.7)  |
| Cora (ratio = 2.60%)       | 80.1±0.6     | 80.0±0.3            | 80.4±0.6           | 81.8±0.5($\beta$ = 0.05) |
| Cora (ratio = 5.20%)       | 79.3±0.3     | 79.1±0.2            | 79.2±0.3           | 81.8±0.6($\beta$ = 0.15) |
| Citeseer (ratio = 0.90%)   | 70.5±0.5     | 70.7±0.3            | 70.8±0.3           | 73.3±0.3($\beta$ = 0.9)  |
| Citeseer (ratio = 1.80%)   | 70.6±0.9     | 70.9±1.0            | 71.1±1.1           | 73.5±0.8($\beta$ = 0.9)  |
| Citeseer (ratio = 3.60%)   | 69.8±1.4     | 70.3±1.1            | 70.9±0.9           | 73.4±0.6($\beta$ = 0.9)  |
| Flickr (ratio = 0.10%)     | 46.5±0.4     | 46.6±0.3            | 46.5±0.3           | 47.1±0.2($\beta$ = 0.7)  |
| Flickr (ratio = 0.50%)     | 47.1±0.1     | 47.1±0.3            | 46.8±0.3           | 47.4±0.1($\beta$ = 0.05) |
| Flickr (ratio = 1.0%)      | 47.1±0.1     | 47.1±0.1            | 47.1±0.2           | 47.5±0.1($\beta$ = 0.15) |
| Ogbn-XRT (ratio = 0.05%)   | -            | 55.3±0.4            | 57.2±0.3           | 61.1±0.3($\beta$ = 0.3)  |
| Ogbn-XRT (ratio = 0.25%)   | -            | 68.2±0.7            | 69.7±0.5           | 69.4±0.4($\beta$ = 0.15) |
| Ogbn-XRT (ratio = 0.50%)   | -            | 69.1±0.6            | 69.3±0.5           | 70.4±0.4($\beta$ = 0.3)  |
| Ogbn-arxiv (ratio = 0.05%) | 59.2±1.1     | 60.3±0.6            | 62.3±0.7           | 64.9±0.7($\beta$ = 0.3)  |
| Ogbn-arxiv (ratio = 0.25%) | 63.2±0.3     | 63.2±0.4            | 64.7±0.6           | 66.7±0.6($\beta$ = 0.9)  |
| Ogbn-arxiv (ratio = 0.50%) | 64.0±1.4     | 64.9±0.3            | 65.4±0.7           | 67.3±0.3($\beta$ = 0.3)  |
| Reddit (ratio = 0.01%)     | 88.0±1.8     | 88.3±1.3            | 88.9±1.1           | 88.9±0.5($\beta$ = 0.1)  |
| Reddit (ratio = 0.05%)     | 89.6±0.7     | 89.2±0.8            | 89.9±0.9           | 90.1±0.8($\beta$ = 0.45) |
| Reddit (ratio = 0.50%)     | 90.1±0.5     | 90.1±0.7            | 90.6±0.3           | 91.3±0.7($\beta$ = 0.45) |

Reviewer comment (Experiments):
Some of the compared methods, such as GCond, had their code updated recently. Has the performance of these methods been re-evaluated, specifically regarding the results presented in Table 1?

Author response:
**Q3: Some of the compared methods, such as GCond, had their code updated recently. Has the performance of these methods been re-evaluated, specifically regarding the results presented in Table 1?**

**A5:**

Thanks for the reviewer’s suggestion.

In brief, we conducted a reproduction using the latest codebase, and the reproduced results **fall within the margin of error** proposed in the original paper.

We replicated the experiments using the latest publicly released code and corresponding commands for GCond and evaluated the final saved synthetic graphs. The results are presented below, and we believe that these results fall within the error range stated in the paper. To prove we do not ignore **the** **use of MSE distance** for gradient matching as mentioned in the code repository. We also conducted experiments under equivalent settings.

Note that all results in the table are averaged over three repetitions for accuracy and consistency.

|                            | GCond(paper) | GCond(reproduction) | GCond(mse) | GCond(norm) |
| - | - | - | - | - |
| Cora (ratio = 1.30%)       | 79.8±1.3     | 79.9±1.1            | 78.4±0.8   | 79.2±0.4    |
| Cora (ratio = 2.60%)       | 80.1±0.6     | 80.0±0.3            | 72.4±0.5   | 80.8±0.4    |
| Cora (ratio = 5.20%)       | 79.3±0.3     | 79.1±0.2            | 74.2±0.1   | 79.3±0.6    |
| Citeseer (ratio = 0.90%)   | 70.5±0.5     | 70.7±0.3            | 64.8±0.5   | 71.7±0.3    |
| Citeseer (ratio = 1.80%)   | 70.6±0.9     | 70.9±1.0            | 62.3±1.0   | 72.5±0.7    |
| Citeseer (ratio = 3.60%)   | 69.8±1.4     | 70.3±1.1            | 63.9±0.8   | 72.7±0.4    |
| Flickr (ratio = 0.10%)     | 46.5±0.4     | 46.6±0.3            | 46.3±0.3   | 45.9±0.2    |
| Flickr (ratio = 0.50%)     | 47.1±0.1     | 47.1±0.3            | 46.7±0.2   | 46.8±0.1    |
| Flickr (ratio = 1.0%)      | 47.1±0.1     | 47.1±0.1            | 46.1±0.1   | 47.0±0.1    |
| Ogbn-XRT (ratio = 0.05%)   | -            | 55.3±0.4            | 49.4±0.3   | 53.2±0.3    |
| Ogbn-XRT (ratio = 0.25%)   | -            | 68.2±0.7            | 60.7±0.7   | 66.5±0.4    |
| Ogbn-XRT (ratio = 0.50%)   | -            | 69.1±0.6            | 63.3±0.4   | 68.8±0.3    |
| Ogbn-arxiv (ratio = 0.05%) | 59.2±1.1     | 60.3±0.6            | 55.3±0.4   | 58.7±0.5    |
| Ogbn-arxiv (ratio = 0.25%) | 63.2±0.3     | 63.2±0.4            | 62.7±0.7   | 64.4±0.8    |
| Ogbn-arxiv (ratio = 0.50%) | 64.0±1.4     | 64.9±0.3            | 63.4±0.5   | 64.9±0.4    |
| Reddit (ratio = 0.01%)     | 88.0±1.8     | 88.3±1.3            | 80.9±1.0   | 85.6±0.6    |
| Reddit (ratio = 0.05%)     | 89.6±0.7     | 89.2±0.8            | 83.9±0.9   | 87.1±0.7    |
| Reddit (ratio = 0.50%)     | 90.1±0.5     | 90.1±0.7            | 86.6±0.9   | 88.3±0.6    |