: The performance of deep networks trained for high-level computer vision tasks such as classification degrades under noise, blur, and other imperfections present in raw sensor data. (Left) An image of jelly beans corrupted by noise characteristic of low-light conditions is misclassified as a library by the Inception-v4 classification network. Cleaning up raw data using conventional low-level image processing does not necessarily improve performance. (Center) The image denoised with BM3D is still misclassified, now as a vending machine. We propose an end-to-end differentiable architecture for joint denoising, deblurring, and classification that makes classification robust to realistic noise and blur. The proposed architecture learns a denoising pipeline optimized for classification that enhances fine detail at the expense of more noise and artifacts. (Right) The learned denoiser recovers the jellybean texture, and the image is correctly classified in the "confection" category. The proposed architecture has a principled and modular design inspired by formal optimization methods that generalizes to other combinations of image formation models and high-level computer vision tasks. Real-world sensors suffer from noise, blur, and other imperfections that make high-level computer vision tasks like scene segmentation, tracking, and scene understanding difficult. Making highlevel computer vision networks robust is imperative for real-world applications like autonomous driving, robotics, and surveillance. We propose a novel end-to-end differentiable architecture for joint denoising, deblurring, and classification that makes classification robust to realistic noise and blur. The proposed architecture dramatically improves the accuracy of a classification network in low light and other challenging conditions, outperforming alternative approaches such as retraining the network on noisy and blurry images and preprocessing raw sensor inputs with conventional denoising and deblurring algorithms. The architecture learns denoising and deblurring pipelines optimized for classification whose outputs differ markedly from those of state-of-the-art denoising and deblurring methods, preserving fine detail at the cost of more noise and artifacts. Our results suggest that the best low-level image processing for computer vision is different from existing algorithms designed to produce visually pleasing images. The principles used to design the proposed architecture easily extend to other high-level computer vision tasks and image formation models, providing a general framework for integrating low-level and high-level image processing.
For improving the robustness of machine learning models to input perturbations, an end-to-end architecture is proposed in REF for joint denoising, deblurring, and classification.
Dirty Pixels: Optimizing Image Classification Architectures for Raw Sensor Data
ArXiv
ArXiv
Computer Science