PATENT DOCUMENT

Publication Number: US-10043279-B1
Application Number: US-201615272455-A
Country: US
Kind Code: B1

Title: Robust detection and classification of body parts in a depth map

Abstract:
A method for processing data includes receiving a depth map of a scene containing at least a part of a body of a humanoid form. The depth map includes a matrix of pixels having respective pixel depth values. A digital processor extracts from the depth map a curvature map of the scene. The curvature map includes respective curvature values and curvature orientations of at least some of the pixels in the matrix. The depth map is segmented using the curvature values and curvature orientations in the curvature map so as to extract three-dimensional (3D) coordinates of one or more limbs of the humanoid form.

Claims:
The invention claimed is: 
     
       1. A method for processing data, comprising:
 receiving a depth map of a scene containing at least a part of a body of a humanoid form, the depth map comprising a matrix of pixels having respective pixel depth values; 
 using a digital processor, extracting from the depth map a curvature map of the scene, the curvature map comprising respective curvature values and curvature orientations of at least some of the pixels in the matrix; and 
 segmenting the depth map using the curvature values and curvature orientations in the curvature map so as to extract three-dimensional (3D) coordinates of one or more limbs of the humanoid form, 
 wherein segmenting the depth map comprises: 
 identifying blobs of the pixels in the depth map that correspond to the one or more limbs of the humanoid form; 
 computing, for each blob, a dominant curvature direction of the blob responsively to the curvature orientations; 
 defining a respective axis of each limb running perpendicular to the dominant curvature direction of a corresponding blob; 
 processing the depth map so as to locate ridges running along respective axes of the one or more limbs, each ridge comprising a chain of the pixels whose respective curvature values are local maxima; and 
 identifying locations and orientations of the one or more limbs based on the ridges. 
 
     
     
       2. The method according to  claim 1 , wherein extracting the curvature map comprises computing, for each of the at least some of the pixels in the curvature map, a dominant, signed value of the curvature and a corresponding curvature orientation. 
     
     
       3. The method according to  claim 1 , wherein computing the dominant curvature direction comprises, for each blob, constructing a histogram of curvature directions of the pixels in the blob, and identifying the dominant curvature direction with a mode of the histogram. 
     
     
       4. The method according to  claim 1 , wherein segmenting the depth map comprises identifying blobs in the curvature map over which the pixels have respective curvature values that are indicative of a convex surface. 
     
     
       5. The method according to  claim 4 , wherein identifying the blobs comprises finding edges of the blobs in the depth map at locations of sign changes in the curvature map. 
     
     
       6. Apparatus for processing data, comprising:
 an imaging assembly, which is configured to capture a depth map of a scene containing at least a part of a body of a humanoid form, the depth map comprising a matrix of pixels having respective pixel depth values; and 
 a processor, which is configured to extract from the depth map a curvature map of the scene, the curvature map comprising respective curvature values and curvature orientations of at least some of the pixels in the matrix, and to segment the depth map using the curvature values and curvature orientations in the curvature map so as to extract three-dimensional (3D) coordinates of one or more limbs of the humanoid form, 
 wherein the processor is configured to identify blobs of the pixels in the depth map that correspond to the one or more limbs of the humanoid form, to compute, for each blob, a dominant curvature direction of the blob responsively to the curvature orientations, to define a respective axis of each limb running perpendicular to the dominant curvature direction of a corresponding blob, to process the depth map so as to locate ridges running along respective axes of the one or more limbs, each ridge comprising a chain of the pixels whose respective curvature values are local maxima, and to identify locations and orientations of the one or more limbs based on the ridges. 
 
     
     
       7. The apparatus according to  claim 6 , wherein the curvature map comprises, for each of the at least some of pixels in the curvature map, a dominant, signed value of the curvature and a corresponding curvature orientation. 
     
     
       8. The apparatus according to  claim 6 , wherein the processor is configured to construct, for each blob, a histogram of curvature directions of the pixels in the blob, and to identify the dominant curvature direction with a mode of the histogram. 
     
     
       9. The apparatus according to  claim 6 , wherein the processor is configured to segment the depth map by identifying blobs in the curvature map over which the pixels have respective curvature values that are indicative of a convex surface. 
     
     
       10. The apparatus according to  claim 9 , wherein the processor is configured to find edges of the blobs in the depth map at locations of sign changes in the curvature map. 
     
     
       11. A computer software product, comprising a non-transitory computer-readable medium in which program instructions are stored, which instructions, when read by a computer, cause the computer to receive a depth map of a scene containing at least a part of a body of a humanoid form, the depth map comprising a matrix of pixels having respective pixel depth values, to extract from the depth map a curvature map of the scene, the curvature map comprising respective curvature values and curvature orientations of at least some of the pixels in the matrix, and to segment the depth map using the curvature values and curvature orientations in the curvature map so as to extract three-dimensional (3D) coordinates of one or more limbs of the humanoid form,
 wherein the instructions cause the computer to identify blobs of the pixels in the depth map that correspond to the one or more limbs of the humanoid form, to compute, for each blob, a dominant curvature direction of the blob responsively to the curvature orientations, to define a respective axis of each limb running perpendicular to the dominant curvature direction of a corresponding blob, to process the depth map so as to locate ridges running along respective axes of the one or more limbs, each ridge comprising a chain of the pixels whose respective curvature values are local maxima, and to identify locations and orientations of the one or more limbs based on the ridges. 
 
     
     
       12. The product according to  claim 11 , wherein the curvature map comprises, for each of the at least some of pixels in the curvature map, a dominant, signed value of the curvature and a corresponding curvature orientation. 
     
     
       13. The product according to  claim 11 , wherein the instructions cause the computer to construct, for each blob, a histogram of curvature directions of the pixels in the blob, and to identify the dominant curvature direction with a mode of the histogram. 
     
     
       14. The product according to  claim 11 , wherein the instructions cause the computer to segment the depth map by identifying blobs in the curvature map over which the pixels have respective curvature values that are indicative of a convex surface. 
     
     
       15. The product according to  claim 14 , wherein the instructions cause the computer to find edges of the blobs in the depth map at locations of sign changes in the curvature map.

Description:
CROSS-REFERENCE TO RELATED APPLICATION 
     This application is a continuation of U.S. patent application Ser. No. 14/960,465, filed Dec. 7, 2015. 
    
    
     FIELD OF THE INVENTION 
     The present invention relates generally to methods and systems for three-dimensional (3D) mapping, and specifically to processing of 3D map data. 
     BACKGROUND 
     A number of different methods and systems are known in the art for creating depth maps. In the present patent application and in the claims, the term “depth map” refers to a representation of a scene as a two-dimensional matrix of pixels, in which each pixel corresponds to a respective location in the scene and has a respective pixel depth value, indicative of the distance from a certain reference location to the respective scene location. In other words, the depth map has the form of an image in which the pixel values indicate topographical information, rather than brightness and/or color of the objects in the scene. Depth maps may be created, for example, by detection and processing of an image of an object onto which a pattern is projected, as described in U.S. Pat. No. 8,456,517, whose disclosure is incorporated herein by reference. The terms “depth map” and “3D map” are used herein interchangeably and have the same meaning. 
     Depth maps may be processed in order to segment and identify objects in the scene. Identification of humanoid forms (meaning 3D shapes whose structure resembles that of a human being) in a depth map, and changes in these forms from scene to scene, may be used as a means for controlling computer applications. For example, U.S. Pat. No. 8,249,334, whose disclosure is incorporated herein by reference, describes a computer-implemented method in which a depth map is segmented so as to find a contour of a humanoid body. The contour is processed in order to identify a torso and one or more limbs of the body. An input is generated to control an application program running on a computer by analyzing a disposition of at least one of the identified limbs in the depth map. 
     As another example, U.S. Pat. No. 8,565,479, whose disclosure is incorporated herein by reference, describes a method for processing a temporal sequence of depth maps of a scene containing a humanoid form. A digital processor processes at least one of the depth maps so as to find a location of the head of the humanoid form, and estimates dimensions of the humanoid form based on this location. The processor tracks movements of the humanoid form over the sequence using the estimated dimensions. 
     U.S. Pat. No. 9,047,507, whose disclosure is incorporated herein by reference, describes a method that includes receiving a depth map of a scene containing at least an upper body of a humanoid form. The depth map is processed so as to identify a head and at least one arm of the humanoid form in the depth map. Based on the identified head and at least one arm, and without reference to a lower body of the humanoid form, an upper-body pose, including at least three-dimensional (3D) coordinates of shoulder joints of the humanoid form, is extracted from the depth map. 
     SUMMARY 
     Embodiments of the present invention provide methods, devices and software for extracting information from depth maps. 
     There is therefore provided, in accordance with an embodiment of the invention, a method for processing data, which includes receiving a depth map of a scene containing at least a part of a body of a humanoid form, the depth map including a matrix of pixels having respective pixel depth values. Using a digital processor, a curvature map of the scene is extracted from the depth map, the curvature map including respective curvature values and curvature orientations of at least some of the pixels in the matrix. The depth map is segmented using the curvature values and curvature orientations in the curvature map so as to extract three-dimensional (3D) coordinates of one or more limbs of the humanoid form. 
     In a disclosed embodiment, extracting the curvature map includes computing, for each of the at the at least some of the pixels in the curvature map, a dominant, signed value of the curvature and a corresponding curvature orientation. 
     In some embodiments, segmenting the depth map includes identifying blobs of the pixels in the depth map that correspond to the one or more limbs of the humanoid form, computing, for each blob, a dominant curvature direction of the blob responsively to the curvature orientations, and defining a respective axis of each limb running perpendicular to the dominant curvature direction of a corresponding blob. In a disclosed embodiment, segmenting the depth map includes processing the depth map so as to locate ridges running along respective axes of the one or more limbs, each ridge including a chain of the pixels whose respective curvature values are local maxima, and identifying locations and orientations of the one or more limbs based on the ridges. Additionally or alternatively, computing the dominant curvature direction includes, for each blob, constructing a histogram of curvature directions of the pixels in the blob, and identifying the dominant curvature direction with a mode of the histogram. 
     Typically, segmenting the depth map includes identifying blobs in the curvature map over which the pixels have respective curvature values that are indicative of a convex surface, and finding edges of the blobs in the depth map at locations of sign changes in the curvature map. 
     There is also provided, in accordance with an embodiment of the invention, apparatus for processing data, including an imaging assembly, which is configured to capture a depth map of a scene containing at least a part of a body of a humanoid form. A processor is configured to extract from the depth map a curvature map of the scene, and to segment the depth map using the curvature values and curvature orientations in the curvature map so as to extract three-dimensional (3D) coordinates of one or more limbs of the humanoid form. 
     There is additionally provided, in accordance with an embodiment of the invention, a computer software product, including a non-transitory computer-readable medium in which program instructions are stored, which instructions, when read by a computer, cause the computer to receive a depth map of a scene containing at least a part of a body of a humanoid form, to extract from the depth map a curvature map of the scene, and to segment the depth map using the curvature values and curvature orientations in the curvature map so as to extract three-dimensional (3D) coordinates of one or more limbs of the humanoid form. 
     The present invention will be more fully understood from the following detailed description of the embodiments thereof, taken together with the drawings in which: 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         FIG. 1  is a schematic, pictorial illustration of a system for 3D mapping of humanoid forms, in accordance with an embodiment of the present invention; 
         FIG. 2  is a schematic representation of a depth map, in accordance with an embodiment of the present invention; 
         FIG. 3  is a flow chart that schematically illustrates a method for extracting a humanoid form from a depth map, in accordance with an embodiment of the present invention; 
         FIG. 4  is a schematic representation of a normal map extracted from the depth map of  FIG. 2 , in accordance with an embodiment of the present invention; 
         FIG. 5  is a schematic representation of a curvature map extracted from the normal map of  FIG. 4 , in accordance with an embodiment of the present invention; 
         FIG. 6  is a schematic representation of a map of curvature direction, extending over a portion of the curvature map of  FIG. 5 , in accordance with an embodiment of the invention; and 
         FIG. 7  is a schematic representation of a map of blobs and ridges extracted from the preceding maps, in accordance with an embodiment of the invention. 
     
    
    
     DETAILED DESCRIPTION OF EMBODIMENTS 
     Overview 
     Depth maps contain a wealth of data, but extracting high-level information from depth maps is often difficult. Finding and tracking the pose of a humanoid form in a depth map, for example, requires that a digital processor separate the limbs from the background and other objects in the depth map and correctly identify the positions and orientations of the limbs. (The term “limbs” is used in the context of the present description and in the claims to refer to the arms, legs, and head.) This task may become even more challenging when only a part of the humanoid form, such as the upper body, is captured in the depth map or when the humanoid form is seated or otherwise positioned directly against a background object. 
     Embodiments of the present invention that are described hereinbelow provide robust, efficient methods, systems and software for extracting humanoid forms from depth maps. In the disclosed methods, a digital processor extracts a curvature map from the depth map of a scene containing a humanoid form. The curvature map comprises respective oriented curvatures of at least some of the pixels in the matrix. In other words, at each of these pixels, the curvature map holds a scalar signed value indicating the dominant curvature value and the corresponding curvature orientation, i.e., the direction of the dominant curvature. The processor segments the depth map using both curvature values and orientations in the curvature map, and thus extracts 3D location and orientation coordinates of one or more limbs of the humanoid form. 
     In the disclosed embodiments, the processor segments the depth map by identifying blobs in the curvature map over which the pixels have a positive curvature, meaning that the surfaces of these blobs are convex (although this definition of “positive” curvature is arbitrary, and curvature could alternatively be defined so that convex surfaces have negative curvature). The edges of the blobs are identified in the depth map at locations of sign changes in the curvature map. This use of curvature enhances the reliability and robustness of segmentation, since it enables the processor to distinguish between different blobs and between blobs and the background even when there is no marked change in depth at this edges of a given blob, as may occur when one body part occludes another, or when a body part is resting against a background surface or other object. 
     In some embodiments, the processor computes the dominant curvature direction of the blob based on statistics of the pixel-wise curvature orientations—for example, by taking the mode of a histogram of the curvatures. The axis of each blob (and thus of the limb to which the blob corresponds) is defined as a perpendicular to the dominant curvature direction. The axis of each limb indicates its angle of orientation, and thus enables the processor to search over only a small range of rotations in detecting and classifying features such as the head and face of the humanoid subject. As part of the process of identifying the locations and orientations of the limbs, the processor may locate ridges running along the respective limb axes, wherein each ridge comprises a chain of the pixels whose respective curvature values are local maxima. 
     System Description 
       FIG. 1  is a schematic, pictorial illustration of a system  20  for depth mapping and imaging, in accordance with an embodiment of the present invention. In this example, an imaging assembly  24  is configured to capture and process depth maps and images of a scene, which in this case contains a humanoid subject  36 . An imaging assembly of this sort is described, for example, in the above-mentioned U.S. Pat. No. 8,456,517. The principles of the present invention are by no means limited to the sort of pattern-based mapping that is described in this patent, however, and may be applied in processing depth maps generated by substantially any suitable technique that is known in the art, such as depth mapping based on stereoscopic imaging or time-of-flight measurements. 
     In the example shown in  FIG. 1 , a projector  30  in imaging assembly  24  projects a pattern of optical radiation onto the scene, and a depth camera  32  captures an image of the pattern that appears on the scene (including subject  36 ). The optical radiation that is used for this purpose is typically in the infrared (IR) range. A processing device in assembly  24  processes the image of the pattern in order to generate a depth map of the body, i.e., an array of 3D coordinates, comprising a depth (Z) coordinate value of the objects in the scene at each point (X,Y) within a predefined area. (In the context of an array of image-related data, these (X,Y) points are also referred to as pixels.) 
     Optionally, a color camera  34  in imaging assembly  24  also captures color (2D) images of the scene. The imaging assembly registers and synchronizes the depth maps with the color images, and generates a data stream that includes the depth maps and image data for output to an image processor, such as a computer  26 . Although computer  26  is shown in  FIG. 1  as a separate unit from imaging assembly  24 , the functions of these two components may alternatively be combined in a single physical unit, and the depth mapping and image processing functions of system  20  may even be carried out by a single processor. 
     Computer  26  processes the data generated by assembly  24  in order to segment and identify the parts of the body of subject  36  and find the pose of the body, i.e., the locations and orientations of at least some of the limbs. Typically, computer  26  comprises a general-purpose computer processor, which is programmed in software to carry out the above functions. The software may be downloaded to the processor in electronic form, over a network, for example, or it may alternatively be provided on tangible, non-transitory media, such as optical, magnetic, or electronic memory media. Further alternatively or additionally, at least some of the functions of computer  26  may be carried out by hard-wired or programmable logic components. 
       FIG. 2  is a schematic representation of a depth map captured by assembly  24 , in accordance with an embodiment of the present invention. The depth map, as explained above, comprises a matrix of pixels having respective depth values. The depth values are represented in  FIG. 2  as gray-scale values, with darker shades of gray corresponding to smaller depth values, i.e., locations closer to assembly  24 . (Black areas correspond to pixels for which no depth values could be determined.) In this particular scene, the depth values of subject  36  blend into those of the chair on which the subject is sitting. Although a head  42  and a forearm  44  of the subject can be roughly discerned, the boundaries between the head and forearm and between the forearm and other parts of the body are difficult to delineate on the basis of the depth values by themselves. 
     Methods for Processing and Segmentation 
       FIG. 3  is a flow chart that schematically illustrates a method for extracting a humanoid form from a depth map, in accordance with an embodiment of the present invention. The method will be described hereinbelow, for the sake of clarify and concreteness, with specific reference to the elements of system  20  ( FIG. 1 ) and the features of depth map  40  ( FIG. 2 ). The principles of this method, however, may alternatively be applied in other system configurations and in processing of depth maps of humanoid subjects in other poses and surroundings. 
     Computer  26  initiates the method of  FIG. 3  upon receiving a depth map, such as map  40 , at a map acquisition step  50 . Taking u-v to be the surface parameterization grid, p=p(u,v) represents the surface points of map  40  in 3D. Based on the depth values in map  40 , computer  26  calculates the depth gradient at each point, at a normal mapping step  52 . The result of this computation is a normal map, in which N=N(u,v) is the surface normal at point p, so that each pixel holds a vector value corresponding to the direction of the normal to the surface defined by the depth map at the corresponding point is space. 
       FIG. 4  is a schematic representation of a normal map  54  extracted at step  52  from depth map  40 , in accordance with an embodiment of the present invention. The normal vectors are difficult to show in gray-scale representation, and map  54  is therefore presented only for the sake of general illustration. Pixels whose normals are close to the Z-direction (pointing out of the page) have light shades of gray in  FIG. 4 , while those angled toward the X-Y plane are darker. In this respect, the high curvature of head  42  and forearm  44  can be observed in terms of the marked gray-scale gradation in  FIG. 4 , and this feature will be used in the subsequent steps of the analysis. 
     Returning now to  FIG. 3 , computer  26  computes a curvature map, based on normal map  54 , at a curvature computation step  60 . The curvature computed for each pixel at this step can be represented in a 2×2 matrix form known in 3D geometry as the shape operator, S, which is defined as follows: 
     
       
         
           
             
               x 
               ⁢ 
               
                   
               
               ⁢ 
               1 
             
             = 
             
               
                 ∂ 
                 p 
               
               
                 ∂ 
                 u 
               
             
           
         
       
       
         
           
             
               x 
               ⁢ 
               
                   
               
               ⁢ 
               2 
             
             = 
             
               
                 ∂ 
                 p 
               
               
                 ∂ 
                 v 
               
             
           
         
       
       
         
           
             G 
             = 
             
               ( 
               
                 
                   
                     
                       x 
                       ⁢ 
                       
                           
                       
                       ⁢ 
                       
                         1 
                         · 
                         x 
                       
                       ⁢ 
                       
                           
                       
                       ⁢ 
                       1 
                     
                   
                   
                     
                       x 
                       ⁢ 
                       
                           
                       
                       ⁢ 
                       
                         1 
                         · 
                         x 
                       
                       ⁢ 
                       
                           
                       
                       ⁢ 
                       2 
                     
                   
                 
                 
                   
                     
                       x 
                       ⁢ 
                       
                           
                       
                       ⁢ 
                       
                         1 
                         · 
                         x 
                       
                       ⁢ 
                       
                           
                       
                       ⁢ 
                       2 
                     
                   
                   
                     
                       x 
                       ⁢ 
                       
                           
                       
                       ⁢ 
                       
                         2 
                         · 
                         x 
                       
                       ⁢ 
                       
                           
                       
                       ⁢ 
                       2 
                     
                   
                 
               
               ) 
             
           
         
       
       
         
           
             B 
             = 
             
               ( 
               
                 
                   
                     
                       
                         
                           
                             ∂ 
                             N 
                           
                           
                             ∂ 
                             u 
                           
                         
                         · 
                         x 
                       
                       ⁢ 
                       
                           
                       
                       ⁢ 
                       1 
                     
                   
                   
                     
                       
                         
                           
                             ∂ 
                             N 
                           
                           
                             ∂ 
                             u 
                           
                         
                         · 
                         x 
                       
                       ⁢ 
                       
                           
                       
                       ⁢ 
                       2 
                     
                   
                 
                 
                   
                     
                       
                         
                           
                             ∂ 
                             N 
                           
                           
                             ∂ 
                             v 
                           
                         
                         · 
                         x 
                       
                       ⁢ 
                       
                           
                       
                       ⁢ 
                       1 
                     
                   
                   
                     
                       
                         
                           
                             ∂ 
                             N 
                           
                           
                             ∂ 
                             v 
                           
                         
                         · 
                         x 
                       
                       ⁢ 
                       
                           
                       
                       ⁢ 
                       2 
                     
                   
                 
               
               ) 
             
           
         
       
       
         
           
             S 
             = 
             
               B 
               · 
               
                 G 
                 
                   - 
                   1 
                 
               
             
           
         
       
     
     Computer  26  extracts the shape operator eigenvectors, corresponding to the two main curvature orientation, and the shape operator eigenvalues, corresponding to the curvature values along these orientations. The curvature map comprises the dominant curvature per pixel, i.e., the eigenvalue with the larger absolute value and the corresponding curvature orientation. The raw curvature value can be either positive or negative, with positive curvature corresponding to convex surface patches, and negative curvature corresponding to concave surface patches. 
     Computer  26  uses the curvature map computed at step  60  in extracting blobs having positive curvature from depth map  40 , at a blob extraction step  62 . Since body parts, such as head  42  and forearm  44 , are inherently convex, positive curvature within a blob of pixels is a necessary condition for the blob to correspond to such a body part. Furthermore, as noted earlier, transitions from positive to negative curvature are good indicators of the edges of a body part, even when the body part is in contact with another object without a sharp depth gradation between the body part and the object. 
       FIG. 5  is a schematic representation of a curvature map  64  extracted from normal map  54  (and hence from depth map  40 ), in accordance with an embodiment of the present invention. Due to the limitations of gray-scale graphics, curvature map  64  shows only the magnitude of the curvature (i.e., the dominant eigenvalue of the curvature matrix, as explained above), whereas the curvature direction is shown in  FIG. 6 . Pixels with strongly positive curvature values have light shades of gray in map  64 , while pixels with negative curvature values are dark gray. Blobs  66  and  68  of strongly-positive curvature, corresponding respectively to head  42  and forearm  44 , can be clearly segmented from other objects and from one another based on the changes in sign of the curvature at their edges. 
     Referring again to  FIG. 3 , computer  26  uses the pixel-wise curvature orientations in curvature map  64  to find the axes of curvature of blobs in the curvature map, such as blobs  66  and  68 , at an axis extraction step  70 . The curvature vector direction, as explained above, is the direction of the major (dominant) eigenvector of the curvature matrix found at step  60 . The axis of each blob is a line in depth map  40  (or curvature map  64 ) that runs through the center of mass of the blob in a direction perpendicular to the dominant curvature direction over the blob. Typically, computer  26  identifies the dominant curvature direction as the statistical mode of the curvature directions of all the pixels. In other words, for each blob, the computer constructs a histogram of the curvature directions of the pixels in the blob, and identifies the dominant curvature direction as the mode of the histogram. If the histogram contains multi-modal behavior, each mode is analyzed independently, dividing the blob into multiple sub-blobs. Alternatively, other statistical averages, such as the mean or median, may be identified as the dominant curvature direction. 
     Curvature values are useful in segmenting blobs even in the absence of abrupt transition in depth. Adding curvature orientations serves as an additional refinement to detect multi-modalities and filter out pixels that are not aligned with the dominant blob orientation. The disclosed combination of depth, normals, curvature values, and curvature orientations boosts the robustness of the segmentation process. 
       FIG. 6  is a schematic representation of a map  72  of curvature direction, extending over a portion of curvature map  64 , in accordance with an embodiment of the invention. Axes  74  and  76  of blobs  66  and  68 , respectively, correspond to the orientation angles of head  42  and forearm  44 . The mode of the curvature thus provides a reliable indicator of the orientation of the corresponding body parts, which can be used to simplify and enhance the reliability of subsequent classification and pose detection. 
     Returning once again to  FIG. 3 , computer  26  uses the blobs extracted at step  62  and the axes found at step  70  in extracting ridges from depth map  40 , at a ridge extraction step  80 . Each ridge comprises a chain of pixels whose respective curvature values are local maxima. A ridge typically aligns with the dominant curvature direction of the blob to which it belongs. The ridges thus run approximately along the respective axes of the blobs. 
       FIG. 7  is a schematic representation of a map  82  of blobs  66 ,  68 , . . . , and ridges  84 ,  86 ,  88 , . . . extracted from the preceding maps, in accordance with an embodiment of the invention. Ridges  84  and  86  give a good indication of the location and orientation of head  42  and forearm  44 . Other ridges  88  correspond to the legs, right arm, and torso of the subject, which can be identified and classified by further analysis. 
     Based on ridges  84 ,  86  and the other blob features described above, computer  26  identifies the locations and orientations of at least some of the limbs in depth map  40 , at a body part identification step  90 . To complete the analysis, computer  26  runs one or more classification and/or identification algorithms over the data in depth map  40 , at a map analysis step  92 . For example, the directions and locations of axes  74 ,  76  and corresponding ridges  84 ,  86  may be used in finding the medial axes of the limbs of the humanoid form, and then assembling these medial axes into a skeleton indicative of the dimensions and pose of the limbs and torso. Techniques that can be applied for this latter purpose are described, for example, in the patents cited above in the Background section. 
     As another example, computer  26  may apply a face detection algorithm at step  92  in order to detect the subject&#39;s face of depth map  40 . In this regard, the localization and orientation estimation of blob  66 , axis and ridge  84 , enable the computer to search for a candidate face in a narrow range of scales and orientations. As a result, the computational resources and time required to complete the detection are reduced, while the reliability of the results is enhanced. 
     Although the embodiments described above relate particularly to segmenting 3D maps of humanoid subjects and finding the poses of the subjects in such maps, the principles of the present invention are similarly applicable, mutatis mutandis, in analysis of 3D maps of other sorts of objects and scenes. It will thus be appreciated that the embodiments described above are cited by way of example, and that the present invention is not limited to what has been particularly shown and described hereinabove. Rather, the scope of the present invention includes both combinations and subcombinations of the various features described hereinabove, as well as variations and modifications thereof which would occur to persons skilled in the art upon reading the foregoing description and which are not disclosed in the prior art.

Metadata:
Filing Date: 20160922
Publication Date: 20180807
Grant Date: 20180807
Priority Date: 20151207
Inventors: ESHET, Yaron
Assignee: APPLE INC
CPC Classifications: [{"code": "G06T2207/10028", "inventive": false, "first": false, "tree": "[]"}, {"code": "G06F3/0304", "inventive": true, "first": false, "tree": "[]"}, {"code": "G06T7/593", "inventive": true, "first": false, "tree": "[]"}, {"code": "G06T7/11", "inventive": true, "first": false, "tree": "[]"}, {"code": "G06T2207/10028", "inventive": false, "first": false, "tree": "[]"}, {"code": "G06T2207/10048", "inventive": false, "first": false, "tree": "[]"}, {"code": "G06T2207/10021", "inventive": false, "first": false, "tree": "[]"}, {"code": "G06T2207/10021", "inventive": false, "first": false, "tree": "[]"}, {"code": "G06T2207/10048", "inventive": false, "first": false, "tree": "[]"}, {"code": "G06T7/66", "inventive": true, "first": false, "tree": "[]"}, {"code": "G06T7/74", "inventive": true, "first": true, "tree": "[]"}, {"code": "G06F3/017", "inventive": true, "first": false, "tree": "[]"}, {"code": "G06T2207/10048", "inventive": false, "first": false, "tree": "[]"}, {"code": "G06T7/0067", "inventive": true, "first": false, "tree": "[]"}, {"code": "G06T2207/30196", "inventive": false, "first": false, "tree": "[]"}, {"code": "G06T7/606", "inventive": true, "first": false, "tree": "[]"}, {"code": "G06T7/0057", "inventive": true, "first": false, "tree": "[]"}, {"code": "G06T7/0085", "inventive": true, "first": false, "tree": "[]"}, {"code": "G06T2207/10028", "inventive": false, "first": false, "tree": "[]"}, {"code": "G06T7/0075", "inventive": true, "first": true, "tree": "[]"}, {"code": "G06T7/0044", "inventive": true, "first": false, "tree": "[]"}, {"code": "G06T2207/10021", "inventive": false, "first": false, "tree": "[]"}, {"code": "G06T7/66", "inventive": true, "first": false, "tree": "[]"}, {"code": "G06T7/11", "inventive": true, "first": false, "tree": "[]"}, {"code": "G06F3/0304", "inventive": true, "first": false, "tree": "[]"}, {"code": "G06F3/017", "inventive": true, "first": false, "tree": "[]"}, {"code": "G06T7/593", "inventive": true, "first": false, "tree": "[]"}, {"code": "G06T7/74", "inventive": true, "first": true, "tree": "[]"}]
Family ID: 63014102