  This paper introduces the large scale visual search algorithm and system
infrastructure at Alibaba. The following challenges are discussed under the
E-commercial circumstance at Alibaba (a) how to handle heterogeneous image data
and bridge the gap between real-shot images from user query and the online
images. (b) how to deal with large scale indexing for massive updating data.
(c) how to train deep models for effective feature representation without huge
human annotations. (d) how to improve the user engagement by considering the
quality of the content. We take advantage of large image collection of Alibaba
and state-of-the-art deep learning techniques to perform visual search at
scale. We present solutions and implementation details to overcome those
problems and also share our learnings from building such a large scale
commercial visual search engine. Specifically, model and search-based fusion
approach is introduced to effectively predict categories. Also, we propose a
deep CNN model for joint detection and feature learning by mining user click
behavior. The binary index engine is designed to scale up indexing without
compromising recall and precision. Finally, we apply all the stages into an
end-to-end system architecture, which can simultaneously achieve highly
efficient and scalable performance adapting to real-shot images. Extensive
experiments demonstrate the advancement of each module in our system. We hope
visual search at Alibaba becomes more widely incorporated into today's
commercial applications.
