From: ICLR.cc/2023/Conference/Paper2781/Authors
title: Submission Withdrawn by the Authors

From: ICLR.cc/2023/Conference/Paper2781/Reviewer_4wGe
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: This paper collects a dataset of StarCraft game replays, alllowing the community to use this dataset via PyTorch for research purpose. 
strength_and_weaknesses: Strength:
1. a lot of efforts in prepraing and preprocessing the data. 
2. the largest publicly available source of StarCraft II esports data. 

Weakness: 
1. lack scienfic novelty. The paper is prefered to be a tutorial or GitHub repo. 
2. Do we really need the game of StarCraft to advance AI research?
clarity,_quality,_novelty_and_reproducibility: See above. 
summary_of_the_review: See above. 
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 1: The contributions are neither significant nor novel.
empirical_novelty_and_significance: 1: The contributions are neither significant nor novel.
flag_for_ethics_review: NO.
recommendation: 3: reject, not good enough

From: ICLR.cc/2023/Conference/Paper2781/Reviewer_2855
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: The paper describes a new dataset of StarCraft II games conducted by professional players, sourced from tournaments.
strength_and_weaknesses: To be frank, I don't think that ICLR is the appropriate venue for this paper. The main contribution here is a dataset, and there are already some RTS datasets out there, including STARDATA and the SCII replay packs (see below). There are no new tasks or challenges being proposed, and the analysis presented relies on baseline ML methods. I would suggest that AIIDE might be a better fit for this submission.
clarity,_quality,_novelty_and_reproducibility: I'm missing a detailed comparison to the StarCraft II replay pack released by Blizzard (https://github.com/Blizzard/s2client-proto#replay-packs).
summary_of_the_review: I think ICLR's the wrong venue for this paper.
correctness: 2: Several of the paper’s claims are incorrect or not well-supported.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 1: strong reject

From: ICLR.cc/2023/Conference/Paper2781/Reviewer_RMqu
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: The authors published a new dataset of StarCraft II game replays called “SC2EGSet”, which claimed to be the first to collect abundant esport replays ranging from 2016 to 2022, and is by far the largest public dataset containing esport replays. They included both slightly pre-processed raw replay data and fully processed dataset. Four tools programmed in pytorch, Golang and C++ separately are open-sourced to the public for everyone to process SC2 replay data. 

strength_and_weaknesses: **Strength:**

The purpose of the paper is straightforward: the authors hope to provide an abundant StarCraft II e-sport dataset for the community. Apart from that, there’s no significant contribution in this paper.

**Weaknesses:**

The major problem in the paper is that the contribution the authors claim is providing the largest e-sport dataset in the community, while all data they collected is from online public sources, thus the data is not first-handed and can be easily collected by others as well. As a result, the data collection in this paper doesn’t seem very valuable and unique.

Assume that the authors want to claim the contribution of preprocessing on the data. Then, they need to show significantly better data representation, achieving better information extraction efficiency compared to previous ones. Nevertheless, there is no evidence that shows their processed data has an advantage over data processed by currently available tools such as SpawningTool (at https://lotv.spawningtool.com/), or other available tools found on Github and other public sources. 

The authors perhaps need to include such comparisons to show the value of the dataset. For example in the experiment part in the paper, it’s better to include several sets of models trained on the same replay data but preprocessed by other available tools. And, probably the authors should do more analysis on their dataset, such as data distribution, to provide more concrete evidence that the data is truly useful and obviously better than current available datasets in some point of view.

Some other comments are listed as follows. 

- It’s better for the authors to have done more experiments for various usages as examples to support their claim that the dataset can be used for research in various fields.
- Some terms should be defined or explained, even though they may be trivial for someone familiar in related fields. For example, what are “Major” and “Premier” tournaments in StarCraft II ? I think the author should briefly introduce various tournaments of StarCraft II and list which ones are in the dataset.
- In the experiment part, Figure 5 puzzles me the most. Although this experiment is just an illustration for how data can be used, I will still point out what perhaps have to be improved. It is weird that according to Table 2, all models have achieved around 80% to 90% accuracy, however the Figure 5 contains accuracies ranging from 0.5 to 0.7, which seems contradictory to me. Furthermore, it’s not mentioned which set of models were shown on Figure 5, or is this another independent experiment?

Another minor point is that the reference is a little messy, some information such as doi or url link is redundant and do not need to be shown. 

clarity,_quality,_novelty_and_reproducibility: This paper is generally easy to follow but with some unclear and confusing messages. For example, in Figure 9, why is SC2EGSetDataset noted as “raw data” while it’s actually processed by 4 tools authors provided? Actually there are several possibilities when the words “raw data” were used. It could be SC2Replay data, preprocessed data and even fully-processed data (like that in Figure 9). It’s better to use different words when mentioning different data.

Furthermore, more information should be brought out but missing. For example comparing their data or toolsets with currently published ones. There seems no novelty in both dataset and tools.

The data is from public sources while the tools they used seem original. 

summary_of_the_review: In summary, this paper is not recommended since the dataset and the tools contribute little to the ICLR community. Although their collected and processed data might still help future researchers working on StarCraft II data when they need tournament replay data, the contribution does not reach the high standard of ICLR. 

correctness: 2: Several of the paper’s claims are incorrect or not well-supported.
technical_novelty_and_significance: 1: The contributions are neither significant nor novel.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 3: reject, not good enough