  Modern deep learning techniques have enabled advances in image-based dietary
assessment such as food recognition and food portion size estimation. Valuable
information on the types of foods and the amount consumed are crucial for
prevention of many chronic diseases. However, existing methods for automated
image-based food analysis are neither end-to-end nor are capable of processing
multiple tasks (e.g., recognition and portion estimation) together, making it
difficult to apply to real life applications. In this paper, we propose an
image-based food analysis framework that integrates food localization,
classification and portion size estimation. Our proposed framework is
end-to-end, i.e., the input can be an arbitrary food image containing multiple
food items and our system can localize each single food item with its
corresponding predicted food type and portion size. We also improve the single
food portion estimation by consolidating localization results with a food
energy distribution map obtained by conditional GAN to generate a four-channel
RGB-Distribution image. Our end-to-end framework is evaluated on a real life
food image dataset collected from a nutrition feeding study.
