  How do we build a general and broad object detection system? We use all
labels of all concepts ever annotated. These labels span diverse datasets with
potentially inconsistent taxonomies. In this paper, we present a simple method
for training a unified detector on multiple large-scale datasets. We use
dataset-specific training protocols and losses, but share a common detection
architecture with dataset-specific outputs. We show how to automatically
integrate these dataset-specific outputs into a common semantic taxonomy. In
contrast to prior work, our approach does not require manual taxonomy
reconciliation. Our multi-dataset detector performs as well as dataset-specific
models on each training domain, but generalizes much better to new unseen
domains. Entries based on the presented methodology ranked first in the object
detection and instance segmentation tracks of the ECCV 2020 Robust Vision
Challenge.
