  Automatic colorectal polyp detection in colonoscopy video is a fundamental
task, which has received a lot of attention. Manually annotating polyp region
in a large scale video dataset is time-consuming and expensive, which limits
the development of deep learning techniques. A compromise is to train the
target model by using labeled images and infer on colonoscopy videos. However,
there are several issues between the image-based training and video-based
inference, including domain differences, lack of positive samples, and temporal
smoothness. To address these issues, we propose an Image-video-joint polyp
detection network (Ivy-Net) to address the domain gap between colonoscopy
images from historical medical reports and real-time videos. In our Ivy-Net, a
modified mixup is utilized to generate training data by combining the positive
images and negative video frames at the pixel level, which could learn the
domain adaptive representations and augment the positive samples.
Simultaneously, a temporal coherence regularization (TCR) is proposed to
introduce the smooth constraint on feature-level in adjacent frames and improve
polyp detection by unlabeled colonoscopy videos. For evaluation, a new large
colonoscopy polyp dataset is collected, which contains 3056 images from
historical medical reports of 889 positive patients and 7.5-hour videos of 69
patients (28 positive). The experiments on the collected dataset demonstrate
that our Ivy-Net achieves the state-of-the-art result on colonoscopy video.
