Paper: CircuitNet 2.0: An Advanced Dataset for Promoting Machine Learning Innovations in Realistic Chip Design Environment (ICLR-2024)

Reviewer comment (Novelty):
Prediction tasks using the dataset are not the final task for chip design tasks. In other words, the most significant tasks in EDA are decision tasks. Thus, the impact of this data set still appears to be limited.

Author response:
The successful decision results in EDA are based on accurate estimation of target metrics, which is the common pursuit of classic methods and ML-based methods. CircuiNet 2.0 provides almost all data for decision tasks in EDA, including layout, netlist, power, and timing. For example, our dataset fully supports these recent works [1-5], which integrate ML models to help the decisions of classic optimization methods. In a word, CircuitNet 2.0 will help the community explore the boundary of decision tasks in EDA by combining classic methods and ML-based methods.

[1] Jiang et al. Accelerating Routability and Timing Optimization with Open-Source AI4EDA Dataset CircuitNet and Heterogeneous Platforms, ICCAD'23

[2] Liu et al., “Concurrent Sign-off Timing Optimization via Deep Steiner Points Refinement”, ACM/IEEE Design Automation Conference (DAC), San Francisco, Jul. 09–13, 2023.

[3] Zheng et al., “Mitigating Distribution Shift for Congestion Optimization in Global Placement”, ACM/IEEE Design Automation Conference (DAC), San Francisco, Jul. 09–13, 2023.

[4] Liu et al., “Global Placement with Deep Learning-Enabled Explicit Routability Optimization”, IEEE/ACM Proceedings Design, Automation and Test in Europe (DATE), Feb. 01–05, 2021.

[5] Mirhoseini et al., "A graph placement methodology for fast chip design", Nature, 594(7862) (2021), pp. 207-212.

Reviewer comment (Reproducibility):
Will the trained parameters of the timing, routing, and IR-drop prediction models be open-sourced to reproduce the results of Table 3, B1, and B2?

Author response:
Yes, we will open-source the trained parameters for the reproduction of our experiments.

Reviewer comment (Experiments):
The large-scale dataset which contains more than 10000 samples is originated from only eight open-source designs, especially three smaller CPU designs. It is likely that samples from the same design share similar features.

Author response:
CPU is the most typical hardware in the realistic design environment. We list the short descriptions of these CPU designs in Table A.1, which shows the variations among CPU-type designs are still significant. Besides, we demonstrate the t-SNE results of the features of CPU designs in Figure 5, which shows a reasonable distribution for ML tasks.

Reviewer comment (Experiments):
Note that most settings in the design flow are conducted in the floorplan and powerplan stages, some combination of parameters may be unreasonable compared to those realistic industrial designs.

Author response:
These parameter settings have been extensively verified by several experts in IC design who have been working for leading IC companies. We believe this dataset can provide a valuable foundation for ML for EDA research.

Reviewer comment (Presentation):
The t-SNE plot in section 4.2.2 is confusing. Where are the GPU and AI Chip?

Author response:
We find the missing points of GPU and AI Chip are caused by the incompatible vector format of Figure 5 under different PDF viewers. We have changed the format of Figure 5 from vector to PNG.

Reviewer comment (Experiments):
Could the authors verify the advantage of training with multi-modal data? It seems that only graph-based input is adpoted in section 4.1.

Author response:
Thank you for the comment. Recently, Wang et al. [1] have shown that fusion of multi-modal data can outperform purely graph-based or image-based methods. Our dataset provides all the data required to build such a model. As their model has not been open-source yet, we do not have enough time to conduct the experiments during the rebuttal period. We will reproduce their experiments on our dataset and release the sample code in the future. 
Besides, HybridNet [2] and Lay-Net [3] are multi-modal neural networks for congestion prediction, which also indicates the advantages of multi-modal learning in the EDA field.

[1] Wang et al. "Restructure-Tolerant Timing Prediction via Multimodal Fusion." 2023 60th ACM/IEEE Design Automation Conference (DAC). IEEE, 2023.

[2] Zhao et al. "HybridNet: Dual-Branch Fusion of Geometrical and Topological Views for VLSI Congestion Prediction" arXiv:2305.05374 

[3] Zheng et al. “Lay-Net: Grafting Netlist Knowledge on Layout-Based Congestion Prediction”, International Conference on Computer-Aided Design (ICCAD), 2023

Reviewer comment (Experiments):
If possible, can the author show several examples to demonstrate the benefit of the overall performance (e.g., P&R time) by combining ML model with the classical heuristic optimization methods?

Author response:
According to references in answers to Q1, Jiang et al. [1] integrate ML model with classical optimization methods to improve routability and timing optimization based on CircuitNet 1.0. Liu et al. [2] use GNN model to optimize the Steiner points of routing solution. Zheng et al. [3] and Liu et al. [4] use ML models to predict the routability for the optimization of placement.

Reviewer comment (Experiments):
Each sample in the same design has the same number of cells, nets, macros, pins, and IOs, so this dataset is not suitable, to some extent, for the majority of ML approaches since the training would easily lead to overfitting. The authors lack the discussion with regard to overfitting and latent negative impacts.

Author response:
The statistics listed in Table 1 are the average numbers of components in netlists after the synthesis stage in order to show the diversity among different hardware designs. All samples even originated from the same design will contain different numbers of components and nets after the synthesis and physical design stages due to various optimization steps to restructure the netlists. Note that even the physical design stage will change the circuit netlists.

To address the concern of overfitting, we perform K-fold experiments in Table 4 to show the generalization capability between samples of different designs. We can see that the model can still behave in a preferable generalization manner between different designs, which indicates the overfitting and latent negative impacts are limited.

Reviewer comment (Writing):
typo: line 12th in section 4.3, two `introduce’.

Author response:
We have fixed it in the revision.

Reviewer comment (Reproducibility):
I note that the samples took a very long time to generate. What is the total generation time? How many and what kind of machines are used to generate these samples?

Author response:
Our generation flow consists of data collection, SRAM generation, verilog code integration, logic synthesis and physical design using commercial tools, manual data check, feature extraction, feature check and model-based verification. We take about 1 year to generate and verify these samples. We use one 96-core Linux server with Intel(R) Xeon(R) Gold 6248R CPU @ 3.00GHz and 512 GB memory to generate these samples.

Reviewer comment (Presentation):
What does `details’ mean in Table A.2? What are the numbers below?

Author response:
The ‘details’ means the specific values for the settings of commercial tools. The numbers below are exactly the values for different settings. For 'Macro Setting' and 'PG Setting', different commands are used in tcl scripts, so we fill in '-'. We add '(MHz)' to the details of 'Frequency' and '(%)' to the details of 'Utilization (U)' to clarify the explanations.

Reviewer comment (Experiments):
Is the learning rate the same in transfer learning? (Figure 6)

Author response:
Yes, the learning rate in transfer learning is the same as that in training from scratch, which is 2e-4, because we use Adam optimizer which has adaptive learning rate, and we find the initial learning rate setting doesn't affect the training results. We also add the clarification in Section 4.3.

Reviewer comment (Experiments):
In the experiments of congestion prediction and IR-drop prediction, models using CircuitNet 2.0 as training data have more advanced performance than those using CircuitNet. In these two experiments, what are the `unseen designs in the test set’?

Author response:
The unseen designs are chosen from "ISPD 2015 Benchmarks with
Fence Regions and Routing Blockages for Detailed-Routing-Driven Placement." (https://ispd.cc/contests/15/web/benchmark.html). These designs are provided by the contest organizers and are completely different from the samples included in the CircuitNet 2.0.