Abstract-Image-based dietary assessment is important for health monitoring and management because it can provide quantitative and objective information, such as food volume, nutrition type, and calorie intake. In this paper, a new framework, 3D/2D model-to-image registration, is presented for estimating food volume from a single-view 2D image containing a reference object (i.e., a circular dining plate). First, the food is segmented from the background image based on Otsu's thresholding and morphological operations. Next, the food volume is obtained from a user-selected, 3D shape model. The position, orientation and scale of the model are optimized by a model-to-image registration process. Then, the circular plate in the image is fitted and its spatial information is used as constraints for solving the registration problem. Our method takes the global contour information of the shape model into account to obtain a reliable food volume estimate. Experimental results using regularly shaped test objects and realistically shaped food models with known volumes both demonstrate the effectiveness of our method. I. INTRODUCTION Obesity has become a major health problem in many parts of the world because it is usually associated with an increased risk of cardiovascular disease, diabetes, and other chronic diseases. To study overweight and obesity, monitoring individual's food intake is important. Traditionally, dietary evaluation is conducted by self-report, which is subjective and inaccurate. Recently, image-based food volume estimation has been studied by a number of investigators. Chae et al. [1] and Zhu et al. [2] presented model-based methods, which achieved good results by an average error less than 15%. However, their methods only utilized 2D feature points to estimate the 3D model volume. In cases where a food has a complex shape, it is difficult to detect a sufficient number of feature points accurately from a single-view image in order to infer the volume using a 3D model. As a result, their methods are sensitive to feature detection errors, and are suitable only in limited cases. In this paper, we propose a new framework based on a model-to-image registration process for estimating food volume from a single-view image. Instead of relying on feature points, we use global food shape properties to obtain a more reliable estimate. Moreover, the spatial information of the circular plate obtained from the camera calibration process was adopted to reduce the solution range of the registration cost function. Our initial experiments indicate that this method is more robust in food volume estimation.  Given a single-view 2D food image (see Fig. 1(a) ), the first step is to segment the food item from the background region. Next, a 3D model is manually chosen to represent the food shape and is projected onto the image. Then, the position, orientation and scale of the model are optimized automatically by model-to-image registration. Finally, the food volume is estimated from the volume of the registered 3D model. In the segmentation step, Otsu's thresholding [3] is applied to the saturation channel resulting from the hue-saturation-intensity transformation (HIS). The procedure reduces the shadow effect in the image. In Otsu's thresholding, the optimal threshold k * , which separates two classes, is obtained by using optimization: where is the inter-class variance, and w 0 , w 1 , 0 , and 1 are the probabilities of class occurrences and the mean levels of the two classes, respectively. An example of segmentation result using Otsu's thresholding is shown in Fig.  1(b) . As the result may contain small holes, morphological closing is performed. A convex hull tightly enclosing the food regions is built by Andrew's monotone chain algorithm [4] . The final segmentation result is shown in Fig. 1(c) . In order to estimate the food volume, we propose a 3D/2D model-to-image registration scheme to fit a 3D virtual model onto the food item in the mage. Fig. 2 illustrates this concept. First, a 3D model is manually selected and is generated in the camera projection space. Let S be the set of surface points of the model with = (t x , t y , t z , x , y , z , s x , s y , s z ) as its pose parameters, including three displacements, three rotation angles and three scales with respect to the camera coordinates. We can project the model onto the image plane by using a perspective projection matrix M derived by camera calibration [5] . The projection points are obtained by S' = M T( ) S, where T is a 4 4 transformation matrix which
Chen et al. REF proposed a 3D/2D model-based image registration method for quantitative food intake assessment.
3D/2D model-to-image registration for quantitative dietary assessment
2012 38th Annual Northeast Bioengineering Conference (NEBEC)
2012 38th Annual Northeast Bioengineering Conference (NEBEC)
Engineering Medicine