Paper:
Neural architecture search (NAS) has achieved breakthrough success in a great number of applications in the past few years. It could be time to take a step back and analyze the good and bad aspects in the field of NAS. A variety of algorithms search architectures under different search space. These searched architectures are trained using different setups, e.g., hyper-parameters, data augmentation, regularization. This raises a comparability problem when comparing the performance of various NAS algorithms. NAS-Bench-101 has shown success to alleviate this problem. In this work, we propose an extension to NAS-Bench-101: NAS-Bench-102 with a different search space, results on multiple datasets, and more diagnostic information. NAS-Bench-102 has a fixed search space and provides a unified benchmark for almost any up-to-date NAS algorithms. The design of our search space is inspired from the one used in the most popular cell-based searching algorithms, where a cell is represented as a directed acyclic graph. Each edge here is associated with an operation selected from a predefined operation set. For it to be applicable for all NAS algorithms, the search space defined in NAS-Bench-102 includes all possible architectures generated by 4 nodes and 5 associated operation options, which results in 15,625 neural cell candidates in total. The training log using the same setup and the performance for each architecture candidate are provided for three datasets. This allows researchers to avoid unnecessary repetitive training for selected architecture and focus solely on the search algorithm itself. The training time saved for every architecture also largely improves the efficiency of most NAS algorithms and presents a more computational cost friendly NAS community for a broader range of researchers. We provide additional diagnostic information such as fine-grained loss and accuracy, which can give inspirations to new designs of NAS algorithms. In further support of the proposed NAS-Bench-102, we have analyzed it from many aspects and benchmarked 10 recent NAS algorithms, which verify its applicability.

Review:
--- Updated during response period ---
Authors successfully answers all my questions. I revise my rating to Accept.
-----
Summary:
This paper proposes another benchmark dataset for neural architecture search. The idea is following the NASBench-101 dataset, that in a given search space, densely sampled all existing architectures and train each of them on three tasks for multiple times, and using the obtained metrics as a tool to evaluate an arbitrary neural architecture search algorithm. The paper also presents comprehensive reports on the statistics, revealing a strong performance correlation between tasks, and evaluate some baseline NAS algorithms.
I think this paper will be valuable to the research community for these reasons: (1) the dataset contains a more geologically complex search space comparing to the original NASBench-101, whose search space is restrained in certain ways; (2) released metrics include more meaningful information rather than single point value in NASbench; (3) it uses 3 datasets rather than 1.
My major concerns, which I will detail later, is the phrasing "algorithm-agnostic" does not truly reflect the difference between their approach and NASBench-101, and about the architecture search space design details. 
Altogether, I think even the technical novelty is incremental, the work is not trivial considering the computational cost. I am willing to improve my score if my concerns are addressed during the rebuttal period. Nevertheless, this dataset is a strong subsidy of existing NASBench-101 and can benefit the research community and serves as an important baseline to evaluate a NAS algorithm. 
Strength
+ Clear motivation to use an operation-on-the-edge search space that is widely used in NAS domain.
+ Extensive experiments on evaluating 15K architectures over 3 datasets
+ Detailed statistics on the search space
+ Good baseline experiments comparison
Main concerns about this dataset:
- Comparing to NASBench-101 in terms of "Algorithm Agnostic", it is in a "more-or-less" game but not a "yes-or-no" one, so that AA-NAS-Bench does not seem appropriate. In my perspective, this dataset has not shown significant differences for the following reasons.
1. With proper adaptation, both NASBench-101 and the one in this paper are "algorithm agnostic". For example, original ENAS is training a reinforcement learning sampler that learns to predict a string with encoding [id1, op1, id2, op2] for each node, where id1, id2 is the IDs of the previous node to connect, op1, op2 is the operation choice for each edge. Since NASBench has operation on output node, one could simply make RL sampler to predict [id1, id2, op1], or another string encoding that suits the search space better. In my perspective, Ying et al. mentioned that many NAS algorithms cannot be directly evaluated on NASBench-101 are because the search space is different, but it does not mean using NASBench-101 is impossible. On the other hand, for some other state-of-the-art algorithms, like Proxyless-NAS on ImageNet, the search space is also different from the one proposed in this paper, but likewise, it does not indicate evaluating Proxyless-NAS on this dataset is impossible. 
2. NASBench-101 does impose constraints on maximum edge number equals to 9 with 7 nodes in their space and results in 423K architectures. However, this constraint is no longer applied if you reduce the number of node to 6 (i.e. all possible architectures can be sampled), yet it still contains around 64K architectures, which is more than 15K in the proposed dataset. In this perspective, NASBench is a larger dataset and "algorithm agnostic".
To summarize, I acknowledge the paper's contribution is using an operation-on-the-edge search space that is widely used in previous NAS algorithms while NASBench-101 is using operation-on-the-node space. However, it only makes the proposed dataset "more algorithm agnostic" with less effort, and it does not make the previous NASBench-101 "not" algorithm agnostic. If using the current name AA-NAS-Bench, I think it is not fair for the NASBench-101, specifically they are 4 times larger after removing the edge number constraints. 
- Questions about architecture space design
1. Why using average pooling instead of max pooling? 
2. How do you compute the total architecture number 15,625 in Table 3? In your setting, with the number of node V=4 densely connected DAG, it should have 6 edges as depicted in Figure 1, and each edge has 5 possible operations, i.e. total number = 6^5 = 7776. I am confused about this point, could author comment more on this number?
3. Is there any topologically equal architectures in this space? For example, let's name the node 1,2,3,4, and the following two architectures should be the same since input edges are summed before passed to the next node. I listed **non-zeroed** edge as, id1->id2: op
Architecture 1:
1->2: conv3x3
2->4: skip
1->3: conv1x1
3->4: skip
Architecture 2:
1->2: conv1x1
2->4: skip
1->3: conv3x3
3->4: skip
If the pruning is not effectively conducted, my worry is the actual number of architectures is smaller.
Minor comments:
1. DARTS results on the are quite poor as mentioned in this paper that, DARTS will eventually converge to an architecture with all skip connection. However, it could be a simple fix, by tracking the architecture evolution during the search and report the best like early-stopping. Will this improve DARTS results? 
2. Since ENAS is the first work using parameter sharing on the NAS problem, could the author add it to the baseline?
3. In table 4, what is the average (94.37 for CIFAR-10) mean in the "optimal" column? Is this the mean performance of all architectures? If so, it is quite strange to see all the baselines are selecting architectures worse than the average performance. Or it is the best architecture performance as indicated in the caption? This "average" column for "optimal" seems confusing.
4. The dynamic ranking of architecture in Figure 5 is very interesting. Architecture ranking seems stable after the 190th epoch. Could the author provide another visualization, showing when stabilization happens in between epoch number 150 and 190? 
5. Figure 4, correlation matrix for top 4743 architectures are significantly lower than the full and 1387 ones, is this possible because of repetitive architectures in the space are not pruned? And, what is the reason for number 4743 and 1387?
6. ResNet (star in Figure 2) seems to perform very well. Does this indicates the proposed search space is not much meaningful, considering there are only 1~2% for NAS to improve?