Paper: Mirage: Model-agnostic Graph Distillation for Graph Classification (ICLR-2024)

Reviewer comment (Experiments):
Would MIRAGE still be as effective if used on a very large data set?

Author response:
**Q1. Would MIRAGE still be as effective if used on a very large dataset?** 

*Response:* We do not expect graph size to have any effect on the performance of MIRAGE since the underlying principles are not dependent on graph dataset size.  This is already evident from Table 2. Specifically, we compute the pearson's correlation coefficient between dataset size and the generalization error (AUC in full dataset-AUC of MIRAGE). The correlation is 0.153 with $p$-value of 0.743 and thus rejecting any bias in AUC from dataset size.

Reviewer comment (Experiments):
Why is MIRAGE, which only uses the CPU, so much faster than DOSCOND and KIDD, which can use the GPU?

Author response:
**Q2. Why is MIRAGE, which only uses the CPU, so much faster than DOSCOND and KIDD, which can use the GPU?** 

*Response:* Existing algorithms, namely DosCOND and KiDD, seek to replicate the gradient trajectory of model parameters on the full dataset. Hence, they need to train on the full dataset for distillation (and as we comment in Sec 1, this compromizes the basic premise of data distillation). In contrast, MIRAGE does not require training on the full dataset. Rather, MIRAGE emulates the input data processed by message-passing GNNs. By shifting the computation task to the pre-learning phase, MIRAGE is not only faster, but also independent of the GNN architecture.

Reviewer comment (Experiments):
If a datasets is large, but its global computation tree is very similar, will such a datasets affect the effect of MIRAGE?

Author response:
**Q3. If a datasets is large, but its global computation tree is very similar, will such a datasets affect the effect of MIRAGE?** 

**Response:** As we discussed in our response to Q1, the size of the dataset does not affect the performance of MIRAGE. The performance depends primarily on the frequency distribution of computation trees in a dataset. If the distribution is skewed, MIRAGE is expected to perform well since the top-$k$ most frequent computation trees would effectively capture a substantial portion of the distribution mass. On the other hand, if the distribution is not skewed, MIRAGE is not the most suitable distillation technique for such scenarios. This aspect is explicitly acknowledged in our conclusion (Sec 5). The relevant text is reproduced verbatim below.

> Finally, MIRAGE relies on the assumption that the distribution of computation trees is skewed. Although we provide compelling evidence of its prevalence across a diverse range of datasets, this assumption may not hold universally, especially in the case of heterophilous datasets. The development of a model-agnostic distillation algorithm remains an open challenge in such scenarios.

Reviewer comment (Experiments):
How will MIRAGE perform in multi-classification tasks?

Author response:
**Q4. How will MIRAGE perform in multi-classification tasks?**

*Response:* We have added a new dataset, namely IMDB-M, to evaluate MIRAGE's performance on multi-class classification. As we see below, MIRAGE produces the best performance. These results, and the details of the IMDB-M dataset has been added to Table 2 and Table 1 respectively in the revised manuscript. 

Architecture|RANDOM(Mean)|RANDOM(sum)|HERDING|Kidd|DosCond|Mirage|\|Full Dataset|
-|-|-|-|-|-|-|-|
GCN|55.10$\pm$ 3.8|52.90 $\pm$ 2.52|61.0 $\pm$ 2.4|57.1 $\pm$ 1.11|55.9 $\pm$ 1.06|**63.20 $\pm$ 1.12**|\|64.10 $\pm$ 1.1|
GIN|60.1$\pm$ 2.67|56.30 $\pm$ 5.5$|58.47 $\pm$ 4.12|54.18 $\pm$ 0.90|58.30 $\pm$ 1.70|**61.8 $\pm$ 1.51**|\|64.8 $\pm$ 1.10 | 

The size of the dataset (in bytes) is as follows:

| $\frac{\text{ Method}\rightarrow}{\text{ Dataset}\downarrow}$ | Herding(GCN) | Herding(GIN) | KiDD |DosCond(GCN) | DosCond(GIN) | MIRAGE |\|Full Dataset|
|-------------------------------------------------------------------------|--------------|---------|---------|---------|------------|--------------|----------
|IMDB-M|1,156|1,256|936|720|824|**228**|\|645,160|

Reviewer comment (Experiments):
Unlike DOSCOND and KIDD, MIRAGE seems infeasible to be generalized to node classification tasks.

Author response:
While the primary focus of MIRAGE is on graph classification (as noted in the title itself), MIRAGE does generalize to node classification. We now empirically demonstrate this ability in App. E. (*Note:* KIDD also does not discuss node classification. Only, DosCOND does).

Generalization to node classification is simple since learning the node embeddings is a necessary step before aggregating them to form the graph embedding (Recollect Eqs. 4 and 5). Specifically, the pipeline follows in an identical manner except the omission of (Eq. 5) and the loss function being applied to node embeddings instead of graph embeddings.

We use ogbg-molbace to analyze performance in node classification. Here, each node is labeled as aromatic or non-aromatic depending on whether it is part of an aromatic ring substructure. The results are tabulated in Table H. Consistent with previous results, MIRAGE outperforms DosCOND in both AUC-ROC and compression (smaller size of the distilled dataset). DosCOND produces a dataset that is $\approx 4$ times the size of that produced by MIRAGE, yet performs more than $7\%$ lower in AUC-ROC. This coupled with model-agnostic-ness further solidifies the superiority of MIRAGE

Reviewer comment (Theory):
There is no theoretical analysis nor experimental proof to investigate the error when approximating the true graph representation by aggregating the root representations of only the highly frequent trees.

Author response:
The above question is indeed important. We have now included a new experiment to evaluate the *sufficiency* of the frequent computation tree patterns.
 
 **Sufficiency of frequent patterns:** In order to establish sufficiency of frequent tree patterns in capturing the dataset characteristic, we conduct the following experiment. We train the model on the full dataset and store its weights at each epoch. Then, we freeze the model at the weights after each epoch's training and pass both the distilled dataset consisting of just the frequent tree patterns, and the full dataset. We then compute the differences between the losses as shown in Figure I (GCN, GAT, and GIN). The rationale behind this is that the weights of the full model recognise the patterns that are important towards minimizing the loss. Now, if the same weights continue to be effective on the distilled train set, it indicates that the distilled dataset has retained the important information. In the figure, we can see that the difference quickly approaches $0$ for all the models for all the datasets, and only starts at a high value at the random initialization where the weights are not yet trained to recognize the important patterns. Furthermore, gradient descent will run more iterations on trees that it sees more often and hence infrequent trees have limited impact on the gradients. These results empirically establish the sufficiency of frequent tree patterns in capturing the majority of the dataset characteristic.
 
 **Additional information relevant to this question:** In Fig. 1, we show that the distribution of computational trees is long-tailed and hence by considering only the top-$k$ most frequent trees, majority of the input information processed by the GNN would be retained. Furthermore, in Fig. 5 (b), we show that the training loss against epochs is similar for the full dataset and distilled dataset.

Reviewer comment (Theory):
MIRAGE compresses the computation trees by mining the set of frequently co-occurring items. However, we can consider an extreme situation in which two computation trees are co-occurring in all graphs, but their frequencies in each graph are very low. So, these trees can’t approximate the true graph representation, which will destroy the synthetic dataset performance.

Author response:
The observation is correct that if the frequencies of the co-occurring pair within all graphs are consistently low (and no other frequent tree sets exist across these graphs), then the approximated graph embedding may be different from the true graph embedding. However, it is essential to clarify that the performance on the synthetic dataset is not contingent on the distance between the approximated and true graph embeddings. Instead, it hinges on whether the approximated embedding includes the necessary information for predicting the class label of the graphs.

Given this context, recollect that we mine frequent tree sets per class label (Section 3.4). Therefore, if the residual tree sets across these graphs exhibit diversity, with the co-occurring pair being the sole exception, it becomes challenging for the GNN to establish correlations from the residual portion not reflected in the approximated embedding. Consequently, the embedding constructed solely from the shared computation trees across graphs may still retain valuable information.

Nonetheless, we realize that there may be situations such as the one described here, or when the distribution of computation trees is not skewed, MIRAGE may not be the most suitable method. We now explicitly acknowledge this aspect in our conclusion. We present below the relevant text verbatim.

> MIRAGE relies on the assumption that the distribution of computation trees is skewed. Although we provide compelling evidence of its prevalence across a diverse range of datasets, this assumption may not hold universally, especially in the case of heterophilous datasets. The development of an unsupervised, model-agnostic distillation algorithm remains an open challenge in such scenarios.

Reviewer comment (Experiments):
The synthetic dataset can solely be used to train GNNs based on the message-passing framework, but can’t be used to train certain spectral GNNs, such as ChebyNet with multiple propagation steps.

Author response:
We agree and this aspect was already acknowledged in our Conclusion. We present below the exact text that attests to this limitation. We note that this limitation also exists for KIDD since it assumes the input GNN architecture can be equivalently expressed as a Graph Neural Tangent Kernel, which holds for MPNNs but not graph transformers or spectral GNNs.

> Additionally, there is a need to assess how these existing algorithms perform on contemporary architectures like graph transformers (e.g., (Ying et al., 2021; Ramp ́aˇsek et al., 2022)) or equivariant GNNs (e.g., (Satorras et al., 2022)). Our future work will be dedicated to exploring these avenues of research.

Reviewer comment (Theory):
Original dataset labels are still used during the distillation procedure. So is it a strictly unsupervised algorithm from this perspective?

Author response:
We thank the reviewer for highlighting this subtle point. We call our algorithm unsupervised from the perspective of not using the training gradients on the full dataset. To address this issue, we do not use the word "unsupervised" in isolation anywhere in the revised version. In all places, we mention "unsupervised to original training gradients".

Reviewer comment (Presentation):
There is no clear statement about how Figure 1 is plotted. And I can't get what the x-axis and y-axis represent, respectively.

Author response:
We apologize for not being clear in our description. We have now incorporated a detailed figure caption, which is reproduced verbatim below.

> The figure presents the frequency distribution of computation trees across datasets. The "frequency" of a computation tree denotes the number of occurrences of that specific tree across all graphs in a dataset. The *normalized* frequency of a tree is computed by dividing its frequency with the total number of trees in a dataset and thus falls in the range $[0,1]$. The $x$-axis of the plot depicts the normalized frequency counts observed in a dataset, while the $y$-axis represents the percentage of computation trees corresponding to each frequency count. Both $x$ and $y$ axes are in log scale. The distribution is highly skewed characterized by a dominance of trees with low frequency counts, while a small subset of trees exhibiting higher frequencies. For example, in ogbg-molhiv, the most frequent tree alone has normalized frequency of $0.32$.

Reviewer comment (Experiments):
It would be beneficial if the authors could delve deeper into the sensitivity of their approach with respect to distillation parameters \theta_1 and \theta_2. Undertaking further experiments to report the performance across varied distillation parameters might offer greater insights, given that these parameters significantly influence the size of the distilled dataset.

Author response:
**Q2. It would be beneficial if the authors could delve deeper into the sensitivity of their approach with respect to distillation parameters $\theta_1$ and $\theta_2$. Undertaking further experiments to report the performance across varied distillation parameters might offer greater insights, given that these parameters significantly influence the size of the distilled dataset.**

*Response:* We appreciate this constructive feedback. We have significantly expanded the studies on parameter sensitivity. Below we summarize the experiments conducted and the key insights derived from them.

**Impact of Frequency Threshold on distillation time, compression and accuracy (AUCROC):** In Section 4.5 (Fig. 6 presents the results), we measure the impact of frequency threshold on the distillation time. MIRAGE is significantly faster than training on full datasets unless the frequency thresholds are set to extreme values where the distilled dataset reaches equality with the full dataset.

In Fig. L in Appendix, we present the relationship between compression and AUCROC. The compression ratio is controlled by increasing the frequency threshold, i.e., the higher the threshold, the more the compression is. As noted, there is marginal improvement in AUCROC with lower thresholds (higher distillation size). This trend can be explained from the long-tailed distribution of computation trees. Even if we incorporate more computation trees, they have negligible impact on capturing more of the distribution mass and hence limited effect on AUCROC.

**Impact of number of hops (layers):**  The details are furnished in Appendix F and H of the appendix. We note the main observations here. First, even under high hop counts from the GNN perspective, the distillation process is more time-efficient than complete dataset training. Moreover, the distilled dataset's performance converges closely with that of the full dataset, as evident in Table 2. Second, the distillation process exceeds the time of full dataset training solely under extreme threshold values. This divergence occurs when distilled dataset reaches equality with the full dataset in size post-distillation. 

In Fig. K in Appendix, we present the impact of hops on the AUC. We see mild deterioration in AUC at higher hops. This is consistent with the literature since GNNs are known to suffer from oversmoothing and oversquashing at higher layers[1].
   
In summary, for pragmatic threshold values, the dataset distillation procedure consistently manifests as a significantly faster option to full dataset training.

[1] Hamed Shirzad, Ameya Velingker, Balaji Venkatachalam, Danica J. Sutherland, and Ali Kemal Sinop. Exphormer: Sparse transformers for graphs. In ICML, 2023

Reviewer comment (Evaluation):
On the ogbg-molhiv dataset, could the authors provide an explanation for the better performance observed achieved by distilled data compared to the original dataset.

Author response:
**Q3. On the ogbg-molhiv dataset, could the authors provide an explanation for the better performance observed achieved by distilled data compared to the original dataset.**

*Response:* As noted in our response to Q1, the performance on distilled dataset of ogbg-mohiv no longer outperforms the full dataset. Nonetheless, as we note in our manuscript, improved performance on the distilled dataset is possible and has been observed in the literature (Ex. In KiDD also this phenomenon has been reported). We reproduce the relevant text verbatim below.

> We observe instances, such as in DD, where the distilled dataset outperforms the full dataset, an outcome that might initially seem counter-intuitive. This phenomenon has been reported in the literature before~\citep{KIDD}. While pinpointing the exact cause behind this behavior is challenging, we hypothesize that the distillation process may tend to remove outliers from the training set, subsequently leading to improved accuracy. Additionally, given that distillation prioritizes the selection of graph components that are more informative to the task, it is likely to retain the most critical patterns, resulting in enhanced model generalizability.

Reviewer comment (Theory):
As we are removing the nodes that exist in the less-occurring computational trees, how can we be sure that we are not losing important information in the dataset (especially since this process does not consider the feature information)? Can authors justify it theoretically or/and empirically by running some experiments?

Author response:
**Q4. As we are removing the nodes that exist in the less-occurring computational trees, how can we be sure that we are not losing important information in the dataset (especially since this process does not consider the feature information)? Can authors justify it theoretically or/and empirically by running some experiments?**
 
 *Response:* The above question is indeed important. We have now included a new experiment to evaluate the *sufficiency* of the frequent computation tree patterns.
 
 **Sufficiency of frequent patterns:** In order to establish sufficiency of frequent tree patterns in capturing the dataset characteristic, we conduct the following experiment. We train the model on the full dataset and store its weights at each epoch. Then, we freeze the model at the weights after each epoch's training and pass both the distilled dataset consisting of just the frequent tree patterns, and the full dataset. We then compute the differences between the losses as shown in Figure I (GCN, GAT, and GIN). The rationale behind this is that the weights of the full model recognise the patterns that are important towards minimizing the loss. Now, if the same weights continue to be effective on the distilled train set, it indicates that the distilled dataset has retained the important information. In the figure, we can see that the difference quickly approaches $0$ for all the models for all the datasets, and only starts at a high value at the random initialization where the weights are not yet trained to recognize the important patterns. Furthermore, gradient descent will run more iterations on trees that it sees more often and hence infrequent trees have limited impact on the gradients. These results empirically establish the sufficiency of frequent tree patterns in capturing the majority of the dataset characteristic.
 
**Feature information:** Finally, we wanted to point out that we do consider feature information. In Def 2, we define graph (tree) isomorphism where feature correspondence is a required condition (Constraint (2)). Frequency is counted based on this definition (Eq. 6). 
 
 **Additional information relevant to this question:** In Fig. 1, we show that the distribution of computational trees is long-tailed and hence by considering only the top-$k$ most frequent trees, majority of the input information processed by the GNN would be retained. Furthermore, in Fig. 5 (b), we show that the training loss against epochs is similar for the full dataset and distilled dataset.

Reviewer comment (Experiments):
In the abstract, one of the identified limitations pertains to the reliance on training with the full dataset. To provide a clearer argument regarding the necessity of full dataset training for existing methods and the distinct advantages of MIRAGE, we should consider the following: What is the comparative performance of existing distillation methods (e.g., gradient matching) when applied to both randomly initialized and well-trained models? Is there a substantial performance gap between the two scenarios? Regarding computational costs in the distillation process, it's worth examining the resource requirements for calculating the computation tree frequency distribution, particularly in the context of deep GNNs. How does this computational overhead compare to training models on the full dataset?

Author response:
**1a(i). Problem motivation: What is the comparative performance of existing distillation methods (e.g., gradient matching) when applied to both randomly initialized and well-trained models? Is there a substantial performance gap between the two scenarios?**
   
*Response:* Below we present the AUCROC numbers of DosCOND on randomly initialized DosCOND model, warm start (100 epochs) and training DosCOND until convergence (typically 1000 epochs). As visible, there is a noticeable gap in the AUCROC numbers indicating training until convergence is necessary. This discussion has been added App. D.

|Dataset|Random|Warm start|Convergence|
|-|-|-|-|
|ogbg-molbace|$55.04\pm9.07$|$59.25\pm1.61$|$67.34\pm1.84$|
|NCI1 | $51.22\pm2.00$|$48.18\pm2.78$|$57.90\pm0.75$|
|ogbg-molbbbp|$52.64\pm1.98$|$50.72\pm3.48$|$59.19\pm0.95$|
|ogbg-molhiv|$48.21\pm5.95$|$34.99\pm7.25$|$73.16\pm0.69$|
|DD|$52.39\pm7.19$|$61.58\pm2.11$|$68.39\pm9.64$|

   
**1a(ii). Regarding computational costs in the distillation process, it's worth examining the resource requirements for calculating the computation tree frequency distribution, particularly in the context of deep GNNs. How does this computational overhead compare to training models on the full dataset?**

*Response:* The time taken to distill each dataset is provided in Fig. 4(a) in the form of a bar plot and the raw numbers are in Table E. MIRAGE is $\approx$ 500 times faster on average than KIDD and $\approx 150$ times faster than DOSCOND. To further expand on this aspect, we have added the following new experiments:
* **Comparison to full dataset training time:** We now report the time taken to train on the full dataset in Table I in Appendix. On average, the distillation time of MIRAGE is more than $30$ times faster than training on the full dataset. In contrast DosCOND and KiDD are slower than full dataset training due to reasons already highlighted in Sec 1 of our manuscript.
* **Impact of hops (layers):** In Fig. H (discussed in App F and referred to from Sec 4.4 in main paper), we report the growth in distillation time against the number of hops. We show that even at a larger values of hops, MIRAGE is 2 to 25 times faster, on average, than training on the full dataset. 
* **Impact of frequency threshold on distillation:** we subject the system to variations in threshold parameters, graphically showing the resulting time in Figure 6. Notably, the distillation process exceeds the time of full dataset training solely under extreme threshold values. This divergence occurs when distilled dataset reaches equality with the full dataset in size post-distillation. Conversely, for pragmatic threshold values, the dataset distillation procedure consistently manifests as a significantly faster option to full dataset training.

Reviewer comment (Theory):
While the concept of the computation tree is inherent to GNNs, we should address the sufficiency of the frequency distribution for the computation tree. Key points to explore include: Why is maintaining a high computation tree frequency distribution sufficient for effective learning? What is the distribution pattern of the computation tree, and does it exhibit a long-tail distribution?

Author response:
**1b. Technical Motivation: While the concept of the computation tree is inherent to GNNs, we should address the sufficiency of the frequency distribution for the computation tree. Key points to explore include: Why is maintaining a high computation tree frequency distribution sufficient for effective learning? What is the distribution pattern of the computation tree, and does it exhibit a long-tail distribution?**

 *Response*: We note that the frequency distribution of computation trees is already reported in our manuscript in Fig. 1 and it is indeed a long-tail distribution. To further establish the sufficiency of frequent tree patterns, we have now dedicated an explicit section (Sec 4.4). 
 
 **Sufficiency of frequent patterns:** In order to establish sufficiency of frequent tree patterns in capturing the dataset characteristic, we conduct the following experiment. We train the model on the full dataset and store its weights at each epoch. Then, we freeze the model at the weights after each epoch's training and pass both the distilled dataset consisting of just the frequent tree patterns, and the full dataset. We then compute the differences between the losses as shown in Figure 5(a) (GCN) and Figure I (GCN, GAT, and GIN). The rationale behind this is that the weights of the full model recognise the patterns that are important towards minimizing the loss. Now, if the same weights continue to be effective on the distilled train set, it indicates that the distilled dataset has retained the important information. In the figure, we can see that the difference quickly approaches $0$ for all the models for all the datasets, and only starts at a high value at the random initialization where the weights are not yet trained to recognize the important patterns. Furthermore, gradient descent will run more iterations on trees that it sees more often and hence infrequent trees have limited impact on the gradients. These results empirically establish the sufficiency of frequent tree patterns in capturing the majority of the dataset characteristics.

Reviewer comment (Novelty):
Ideally, the distillate data should mirror the distribution of the full dataset, implying that the gradients of the distillate data should align with those of the full dataset for any model (random or well-trained). It's essential to clarify that while distillation can reduce computation costs, the primary objective remains data distillation. Additionally, considering the architecture transferability for existing graph distillation methods, the advantage of model-agnostic techniques is not significant.

Author response:
**2(i). Objectives: Ideally, the distillate data should mirror the distribution of the full dataset, implying that the gradients of the distillate data should align with those of the full dataset for any model (random or well-trained). It's essential to clarify that while distillation can reduce computation costs, the primary objective remains data distillation.**  

*Response*: As mentioned in our response to 1b, the frequency distribution of computation trees is long tailed. With the newly added sufficiency experiments, we now also show that the frequent trees are the patterns that are being learned by the model parameters as well (Response to 1b.). Finally in Fig. 5b and J, we also show that the train losses on full dataset and distilled datasets, with their own respective training weights, are also aligned. 

**2(ii). Objectives: Additionally, considering the architecture transferability for existing graph distillation methods, the advantage of model-agnostic techniques is not significant.**

*Response:* Our manuscript includes experiments to evaluate transferability of model architectures in graph distillation. The results, reported in Table F of Appendix C, reveal there is a noticeable drop in AUCROC when the model architecture for distillation is different than the architecture used for training on the distilled dataset. The same observation has been reported in another recent work as well (See Table 3 in [1]). In KiDD [2], they theoretically establish why the distillation GNN and the training GNN needs to be same (Page 3 in KiDD). These results provide verification of why model-agnostic distillation is a problem of importance. 

To better highlight these results and motivate the proposed problem, we now point to this information from the introduction itself.

* [1] Beining Yang, Kai Wang, Qingyun Sun, Cheng Ji, Xingcheng Fu, Hao Tang, Yang You, and Jianxin Li. Does graph distillation see like vision dataset counterpart? NeurIPS, 2023*.(to appear)
* [2] Zhe Xu, Yuzhong Chen, Menghai Pan, Huiyuan Chen, Mahashweta Das, Hao Yang, and Hanghang Tong. Kernel ridge regression-based graph dataset distillation. ACM SIGKDD, 2023

Reviewer comment (Experiments):
In evaluating baselines, it would be beneficial to perform a comprehensive comparison across various model architectures. Specifically, assessing performance across different architecture pairs would enhance the evaluation's robustness and provide insights into transferability.

Author response:
**Experiments: 3a. Comprehensive Model Comparison: In evaluating baselines, it would be beneficial to perform a comprehensive comparison across various model architectures. Specifically, assessing **performance across different architecture pairs** would enhance the evaluation's robustness and provide insights into transferability.** 

*Response:* Our evaluation already includes performance assessment across three different GNN architectures namely- GCN, GAT, and GIN, in Table 2. The results establish the proposed technique to be superior on average and consistently ranked among the top-2 distillation algorithms across all dataset-architecture combinations.  

In addition, cross architecture transferability, i.e., the situation where the architecture used for distillation is different from the one used for training and inference is also included in Appendix C. As discussed in our response to 2(ii), there is a noticeable drop in AUCROC, which is consistent with existing literature [1]. This result motivates the need for model-agnostic distillation algorithms.

Reviewer comment (Experiments):
Investigate the significance of GNN model training prior to distillation for baselines. Explore performance outcomes for well-trained, randomly initialized, and warm-up trained models. Understanding the effectiveness of random initialization can help determine the necessity of training on the full data for baseline methods.

Author response:
**3b. Baselines and Model Initialization: Investigate the significance of GNN model training prior to distillation for baselines. Explore performance outcomes for well-trained, randomly initialized, and warm-up trained models. Understanding the effectiveness of random initialization can help determine the necessity of training on the full data for baseline methods.**
 
 *Response*: Below we present the AUCROC numbers of DosCOND on randomly initialized DosCOND model, warm start (100 epochs) and training DosCOND until convergence (typically around 1000 epochs). As visible, there is a noticeable gap in the AUCROC numbers indicating full training is necessary. This discussion has been added App. D.


|Dataset|Random|Warm start| Converged|
|-|-|-|-|
|ogbg-molbace|$55.04\pm9.07$|$59.25\pm1.61$|**67.34$\pm$ 1.84**|
|NCI1 | $51.22\pm2.00$|$48.18\pm2.78$|**57.90$\pm$ 0.75**|
|ogbg-molbbbp|$52.64\pm1.98$|$50.72\pm3.48$|**59.19$\pm$ 0.95**|
|ogbg-molhiv|$48.21\pm5.95$|$34.99\pm7.25$|**73.16$\pm$ 0.69**|
|DD|$52.39\pm7.19$|$61.58\pm2.11$|**68.39$\pm$ 9.64**|

Reviewer comment (Experiments):
Explore the impact of different hyperparameters, such as threshold values and the number of hops, on various metrics, including accuracy, efficiency, compression ratio, and transferability.

Author response:
**3c. Hyperparameter Investigation: Explore the impact of different hyperparameters, such as threshold values and the number of hops, on various metrics, including accuracy, efficiency, compression ratio, and transferability.**

*Response:* All of these experiments have been incorporated in the revised manuscript.

**Impact of Frequency Threshold on distillation time, compression and accuracy (AUCROC):** In Section 4.5 (Figure 6 presents the results), we measure the impact of frequency threshold on the distillation time. MIRAGE is significantly faster than training on full datasets unless the frequency thresholds are set to extreme values where the distilled dataset reaches equality with the full dataset.

In Fig. L in Appendix, we present the relationship between compression and AUCROC. The compression ratio is controlled by increasing the frequency threshold, i.e., the higher the threshold, the more is the compression. As noted, there is marginal improvement in AUCROC with lower thresholds (higher distillation size). This trend can be explained from the long-tailed distribution of computation trees. Even if we incorporate more computation trees, they have negligible impact on capturing more of the distribution mass and hence limited effect on AUCROC.

**Impact of number of hops (layers):**  The details are furnished in Appendix F and H of the appendix. We note the main observations here. First, even under high hop counts from the GNN perspective, the distillation process is more time-efficient than complete dataset training. Moreover, the distilled dataset's performance converges closely with that of the full dataset, as evident in Table 2. Second, the distillation process exceeds the time of full dataset training solely under extreme threshold values. This divergence occurs when distilled dataset reaches equality with the full dataset in size post-distillation. 

In Fig. K in Appendix, we present the impact of hops on the AUC. We see mild deterioration in AUC at higher hops. This is consistent with the literature since GNNs are known to suffer from oversmoothing and oversquashing at higher layers[1].

**Transferability:** Our algorithm is model-agnostic. Hence, transferability of knowledge from one model to another is not relevant for MIRAGE.

In summary, for pragmatic threshold values, the dataset distillation procedure consistently manifests as a significantly faster option to full dataset training.

[1] Hamed Shirzad, Ameya Velingker, Balaji Venkatachalam, Danica J. Sutherland, and Ali Kemal Sinop. Exphormer: Sparse transformers for graphs. In ICML, 2023

Reviewer comment (Evaluation):
Provide insights into the trade-off between accuracy, efficiency, and compression ratio. Understanding these trade-offs can offer valuable guidance for practical applications.

Author response:
**3d. Trade-off Analysis: Provide insights into the trade-off between accuracy, efficiency, and compression ratio. Understanding these trade-offs can offer valuable guidance for practical applications.**

*Response:* As mentioned in our response to Q3c above, the trade-off between compression ratio and AUCROC has been discussed in detail in App H (also references from Sec 4.5 in main paper). In our response to 3c., we also discuss the impact of frequency threshold on the distillation time. This is also discussed in detail in Sec 4.5.