From: robot-learning.org/CoRL/2021/Conference/Blue_Sky/Program_Chairs
title: Paper Decision
decision: Accept
comment: The paper proposes a new way of looking at the robot pipeline -- from the module oriented to the data-centric point of view. While the paper is sketch of an idea, it provides a novel point of view, likely to spark a good discussion and potentially inspire different way of looking at the robotics stack.

From: robot-learning.org/CoRL/2021/Conference/Blue_Sky/Paper12/Reviewer_ShNX
summary: The article proposes RoboFlow, a cloud-based workflow management system that divides the robot software pipeline into four modules: data processing, algorithmic development, back testing and application adaptation.  Each of those modules interact with a centralized data engine with manages access to one or more datasets.  This architecture is motivated by the increased use of “data-centric” algorithms that are trained on large quantities of data.  The authors present a prototype variant of RoboFlow and demonstrate it in two domains.

This paper would be more suitable as a traditional conference or journal article than a Blue Sky submission.  Although the paper is forward thinking (it does propose a long term challenge and future architecture), so do many conference papers as well.  The main issue here is that there is not enough space in 4 pages to fully describe the proposed system and support the claims made by the authors.  Specifically:

* The claim that existing architectures do not support parallel development is unfounded and inaccurate.  I strongly encourage the authors to revisit and rephrase.  

* It is not clear what actual gains would be achieved from the proposed system. Is it computational efficiency?  Reduced data bandwidth?  Lower memory costs? 

* To address the above, the system should be benchmarked against similar systems and statistics should be reported

* Due to page limitations, the details of both the architecture and the performed case studies are insufficient 

The work would greatly benefit from being presented in a longer format at a different venue.
novelty: Fair: Incremental work, no significant advance.
impact: 2: Unsure, difficult to assess.
clarity_of_presentation: Good: Minor revision needed to improve clarity.
recommendation: Strong Reject: I recommend rejecting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
summary_of_recommendation: This paper would be more suitable as a traditional conference or journal article than a Blue Sky submission.  Although the paper is forward thinking (it does propose a long term challenge and future architecture), so do many conference papers as well.  The main issue here is that there is not enough space in 4 pages to fully describe the proposed software architecture and support the claims made by the authors.

From: robot-learning.org/CoRL/2021/Conference/Blue_Sky/Paper12/Reviewer_VRc6
summary: This is a unique software systems paper that describes a hypothetical robot development platform based on virtualization technologies currently under very active use in the systems community.  Two simple case studies implementing RL and motion planning algorithms are demonstrated.
novelty: Very Good:  Major advance in one or more aspects listed above.
impact: 4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.
clarity_of_presentation: Good: Minor revision needed to improve clarity.
recommendation: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
summary_of_recommendation: The paper is woefully light on details, but has good potential to spark discussion about future pipelines for data-driven robot development. In my opinion, the field needs to revise its outlook on middleware as being "just use ROS" which does a terrible job of integrating modern learning workflows, and this paper is a step in the right direction. The paper itself could benefit from describing more clearly their vision for a data-driven development framework, what a developer's workflow would look like, how such a system could aid in managing datasets, domain transfer, etc.