Systematic benchmarking for reproducibility of computer vision algorithms for realtime systems: The example of optic flow estimation Until now there have been few formalized methods for conducting systematic benchmarking aiming at reproducible results when it comes to computer vision algorithms. This is evident from lists of algorithms submitted to prominent datasets, authors of a novel method in many cases primarily state the performance of their algorithms in relation to a shallow description of the hardware system where it was evaluated. There are significant problems linked to this nonsystematic approach of reporting performance, especially when comparing different approaches and when it comes to the reproducibility of claimed results. Furthermore how to conduct retrospective performance analysis such as an algorithmxe2x80x99s suitability for embedded realtime systems over time with underlying hardware and software changes in place. This paper proposes and demonstrates a systematic way of addressing such challenges by adopting containerization of software aiming at formalization and reproducibility of benchmarks. Our results show maintainers of broadly accepted datasets in the computer vision community to strive for systematic comparison and reproducibility of submissions to increase the value and adoption of computer vision algorithms in the future.

Object Proposal Algorithms in the Wild: Are they Generalizable to Robot Perception? The recent emergence of object proposal algorithms in the computer vision community shows great promise to addressing difficult problems in robotic such as object discovery and salient object detection. However, it is difficult to determine how these algorithms actually perform for realworld robot vision applications, because the standard evaluation protocol uses datasets which do not adequately account for realworld noise (motion blur, occlusion, etc.). We evaluated several stateoftheart object proposal algorithms using naturalistic datasets from the robotics community, and found a substantial performance drop across all algorithms. This suggests that many object proposal algorithms are not as generalizable as the computer vision literature purports, which can have a significant impact on how they are applied to robotics. We also conducted a study on how each algorithm is influenced by specific kinds of realworld robot vision challenges, including variable brightness, gamma correction, Gaussian blur, and Gaussian noise. Our results provide insight into certain weaknesses of object proposal algorithms, which can be used to gauge how they might be suitable for different robotics applications. It is our intent that this work will motivate future research about how to design more flexible and robust object proposal algorithms for the robotics community.