  The recently proposed end-to-end transformer detectors, such as DETR and
Deformable DETR, have a cascade structure of stacking 6 decoder layers to
update object queries iteratively, without which their performance degrades
seriously. In this paper, we investigate that the random initialization of
object containers, which include object queries and reference points, is mainly
responsible for the requirement of multiple iterations. Based on our findings,
we propose Efficient DETR, a simple and efficient pipeline for end-to-end
object detection. By taking advantage of both dense detection and sparse set
detection, Efficient DETR leverages dense prior to initialize the object
containers and brings the gap of the 1-decoder structure and 6-decoder
structure. Experiments conducted on MS COCO show that our method, with only 3
encoder layers and 1 decoder layer, achieves competitive performance with
state-of-the-art object detection methods. Efficient DETR is also robust in
crowded scenes. It outperforms modern detectors on CrowdHuman dataset by a
large margin.
