We present a model and algorithm for segmentation of images with missing boundaries. In many situations, the human visual system fills in missing gaps in edges and boundaries, building and completing information that is not present. This presents a considerable challenge in computer vision, since most algorithms attempt to exploit existing data. Completion models, which postulate how to construct missing data, are popular but are often trained and specific to particular images. In this paper, we take the following perspective: We consider a reference point within an image as given and then develop an algorithm that tries to build missing information on the basis of the given point of view and the available information as boundary data to the algorithm. We test the algorithm on some standard images, including the classical triangle of Kanizsa and low signal͞noise ratio medical images. C onsider the images in Fig. 1 . It is apparent in all three images that internal boundaries exist. In the image on the left, a solid triangle in the center of the figure appears to have well defined contours even in completely homogeneous areas; in the center figure, a large rectangle is perceived; and, in the figure on the right, a white square partially occluded by a gray disk is perceived. These contours, which are defined without image gradient, are called ''apparent'' or ''subjective'' contours. Following convention, we distinguish, perhaps arbitrarily, between ''modal completion,'' in which the goal is to construct a perceived boundary, and (as in Fig.  1 Left and Center) and ''amodal completion,'' in which one reconstructs the shape of a partially occluded object (as in Fig. 1 Right) (see ref. 1). Our goal is to extract these internal objects, a process known as ''segmentation.'' Since some information is missing, algorithms typically complete the segmentation process by building models of what happens between the available data. In this paper, we propose a method for segmentation of images with missing boundaries. We take the following perspective: Y The observer is drawn to a reference point within the image; from that reference point, the completion process is constructed. Y Thus, starting from this given reference point, we devise an algorithm that takes advantage of the available boundary data to construct a complete segmentation. Thus, in our approach, the computed segmentation is a function of the reference point. We define a segmentation as a piecewise constant surface that varies rapidly across the boundary between different objects and stays flat within it. In ref. 2, the segmentation is a piecewise smooth͞constant approximation of the image. In our approach, the segmentation is a piecewise constant approximation of the pointof-view or reference surface. To obtain it, we define the following steps: (i) Select a fixation point and build the point-of-view surface. (ii) Detect local features in the image. (iii) Evolve the point of view surface with a flow that depends both on the geometry of the surface and on the image features. The flow evolves the initial condition towards a piecewise constant surface. (iv) (Automatically) Pick the level set that describes the desired object. During the evolution, the point-of-view surface is attracted by the existing boundaries and steepens. The surface evolves towards the piecewise constant solution by continuation and closing of the boundary fragments and the filling in the homogeneous regions. A solid object is delineated as a constant surface bounded by existing and recovered shape boundaries. With this method, the image completion process depends both on the point of view and on the geometric properties of the image. This observation is supported by experimental results on perceptual organization of the human visual system, as pointed out by Kanizsa: ''If you fix your gaze on an apparent contour, it disappears, yet if you direct your gaze to the entire figure, the contours appear to be real (ref. 1, p. 48). Both the mathematical and algorithmic approach in our method relies on a considerable body of recent work based on a partial differential equations approach to both front propagation and to image segmentation. Level set methods, introduced by Osher and Sethian (3), track the evolution of curves and surfaces, using the theory of curve and surface evolution and the link between front propagation and hyperbolic conservation laws discussed in refs. 4 and 5. These methods embed the desired interface as the zero level set of an implicit function and then employ finite differences to approximate the solution of the resulting initial value partial differential equation. Malladi et al. (6) and Caselles et al. (7) used this technology to segment images by growing trial shapes inside a region with a propagation velocity that depends on the image gradient and hence stops when the boundary is reached; thus, image segmentation is transformed into an initial value partial differential equation in which a propagating front stops at the desired edge. In ref. 8, Sochen et al. view image processing as the evolution of an image manifold embedded in a higher dimensional Riemannian space towards a minimal surface. This framework has been applied to processing both single-and vector-valued images defined in two and higher dimensions (see ref. 9). We follow some of these ideas in this paper and view segmentation as the evolution of an initial reference manifold under the influence of local image features. Our approach takes a more general view of the segmentation problem. Rather than follow a particular front or level curve that one attempts to steer to the desired edge (as in refs. 7 and 10), we begin with an initial surface, chosen on the basis of a user-supplied reference fixation point. We then flow this entire surface under speed law dependent on the image gradient, without regard to any particular level set. Suitably chosen, this flow sharpens the surface around the edges and connects segmented boundaries across the missing information. On the basis of this surface sharpening, we can identify a level set corresponding to an appropriate segmented edge. We test our algorithm on various figures, including those above, as well as medical images in which the signal-to-noise ratio is truly small and the most part of the edge is missed, as in the case of ultrasound images. The paper is organized as the following. In the next section, we review past work in segmentation of images with missing boundaries. In the subsequent section, we discuss the mathematical problem, and we present a numerical method to solve it. In the final section, we discuss results of the application of the method to different images, and we discuss both the modal and amodal completion scenarios.
In Ref. REF , a fixation point was selected inside the region bounded by the illusory contour, and then a surface on the image domain was constructed based on the fixation point.
Subjective surfaces: A method for completing missing boundaries
Proceedings of the National Academy of Sciences of the United States of America
Proceedings of the National Academy of Sciences of the United States of America
Computer Science Medicine