  Cascaded architectures have brought significant performance improvement in
object detection and instance segmentation. However, there are lingering issues
regarding the disparity in the Intersection-over-Union (IoU) distribution of
the samples between training and inference. This disparity can potentially
exacerbate detection accuracy. This paper proposes an architecture referred to
as Sample Consistency Network (SCNet) to ensure that the IoU distribution of
the samples at training time is close to that at inference time. Furthermore,
SCNet incorporates feature relay and utilizes global contextual information to
further reinforce the reciprocal relationships among classifying, detecting,
and segmenting sub-tasks. Extensive experiments on the standard COCO dataset
reveal the effectiveness of the proposed method over multiple evaluation
metrics, including box AP, mask AP, and inference speed. In particular, while
running 38\% faster, the proposed SCNet improves the AP of the box and mask
predictions by respectively 1.3 and 2.3 points compared to the strong Cascade
Mask R-CNN baseline. Code is available at
\url{https://github.com/thangvubk/SCNet}.
