  Although the recent image-based 3D object detection methods using
Pseudo-LiDAR representation have shown great capabilities, a notable gap in
efficiency and accuracy still exist compared with LiDAR-based methods. Besides,
over-reliance on the stand-alone depth estimator, requiring a large number of
pixel-wise annotations in the training stage and more computation in the
inferencing stage, limits the scaling application in the real world.
  In this paper, we propose an efficient and accurate 3D object detection
method from stereo images, named RTS3D. Different from the 3D occupancy space
in the Pseudo-LiDAR similar methods, we design a novel 4D feature-consistent
embedding (FCE) space as the intermediate representation of the 3D scene
without depth supervision. The FCE space encodes the object's structural and
semantic information by exploring the multi-scale feature consistency warped
from stereo pair. Furthermore, a semantic-guided RBF (Radial Basis Function)
and a structure-aware attention module are devised to reduce the influence of
FCE space noise without instance mask supervision. Experiments on the KITTI
benchmark show that RTS3D is the first true real-time system (FPS$>$24) for
stereo image 3D detection meanwhile achieves $10\%$ improvement in average
precision comparing with the previous state-of-the-art method. The code will be
available at https://github.com/Banconxuan/RTS3D
