PATENT DOCUMENT

Publication Number: US-12196860-B2
Application Number: US-202117189300-A
Country: US
Kind Code: B2

Title: Depth sensor calibration using internal reflections

Abstract:
Depth sensing apparatus includes a radiation source, which emits a first array of beams of light pulses through a window toward a target scene. Objective optics image the target scene onto a second array of sensing elements, which output signals indicative of respective times of incidence of photons. A first calibration, which associates the beams in the first array with respective first locations on the second array onto which the beams reflected from the target scene are imaged, is used in processing the signals in order to measure respective times of flight of the light pulses. A second calibration indicates second locations on which stray radiation is incident on the second array due to reflection of the beams from the window. Upon detecting a change in the second locations relative to the second calibration, the first calibration is corrected so as to compensate for the detected change.

Claims:
The invention claimed is: 
     
       1. Depth sensing apparatus, comprising:
 a transparent window; 
 a radiation source, which is configured to emit a first array of beams of light pulses through the window toward a target scene; 
 an imaging assembly, which comprises a second array of sensing elements, configured to output signals indicative of respective times of incidence of photons on the sensing elements, and objective optics configured to image the target scene onto the second array; and 
 processing and control circuitry, which is coupled to store a first calibration associating the beams in the first array with respective first locations on the second array onto which the beams reflected from the target scene are imaged, and to process the signals output by the sensing elements in accordance with the first calibration in order to measure respective times of flight of the light pulses, 
 wherein the processing and control circuitry is configured to store a second calibration indicating second locations on which stray radiation is incident on the second array due to reflection of the beams from the window, and to detect, responsively to the signals output by the sensing elements, a change in the second locations relative to the second calibration and to correct the first calibration so as to compensate for the detected change. 
 
     
     
       2. The apparatus according to  claim 1 , wherein the first array comprises a first number of the beams, and the second array comprises a second number of the sensing elements, wherein the second number is greater than the first number. 
     
     
       3. The apparatus according to  claim 2 , wherein the objective optics are configured to image the target scene sparsely onto the array of sensing elements so that only a subset of the sensing elements output the signals in response to the light pulses reflected from the target scene, and wherein the first calibration indicates the sensing elements that belong to the subset. 
     
     
       4. The apparatus according to  claim 3 , wherein the processing and control circuitry is configured to process the signals output by the sensing elements in the subset in order to measure the respective times of flight, without receiving the signals output by the sensing elements that are not in the subset, and to modify a membership of the subset responsively to the detected change. 
     
     
       5. The apparatus according to  claim 2 , wherein the processing and control circuitry is configured to group the sensing elements in each of the first locations together to define super-pixels, and to process together the signals from the sensing elements in each of the super-pixels in order to measure the respective times of flight. 
     
     
       6. The apparatus according to  claim 1 , wherein the processing and control circuitry is configured, in accordance with the first calibration, to actuate only the sensing elements at each of the first locations, while the remaining sensing elements in the array are inactive. 
     
     
       7. The apparatus according to  claim 1 , wherein the processing and control circuitry is configured to compute a coordinate transformation relating the second locations indicated by the second calibration to the second locations following the detected change, and to apply the coordinate transformation in correcting the first calibration. 
     
     
       8. The apparatus according to  claim 1 , wherein the radiation source comprises at least one vertical-cavity surface-emitting laser (VCSEL). 
     
     
       9. The apparatus according to  claim 8 , wherein the at least one VCSEL comprises an array of VCSELs. 
     
     
       10. The apparatus according to  claim 1 , wherein the sensing elements comprise single-photon avalanche diodes (SPADs). 
     
     
       11. The apparatus according to  claim 1 , and comprising a housing, which contains the radiation source and the imaging assembly and which comprises the window, wherein the beams of light exit the housing through the window. 
     
     
       12. A method for depth sensing, comprising:
 configuring a radiation source to emit a first array of beams of light pulses through a transparent window toward a target scene; 
 imaging the target scene onto a second array of sensing elements, which output signals indicative of respective times of incidence of photons on the sensing elements; 
 storing a first calibration associating the beams in the first array with respective first locations on the second array onto which the beams reflected from the target scene are imaged; 
 processing the signals output by the sensing elements in accordance with the first calibration in order to measure respective times of flight of the light pulses; 
 storing a second calibration indicating second locations on which stray radiation is incident on the second array due to reflection of the beams from the window; 
 detecting, responsively to the signals output by the sensing elements, a change in the second locations relative to the second calibration; and 
 correcting the first calibration so as to compensate for the detected change. 
 
     
     
       13. The method according to  claim 12 , wherein the first array comprises a first number of the beams, and the second array comprises a second number of the sensing elements, wherein the second number is greater than the first number. 
     
     
       14. The method according to  claim 13 , wherein the target scene is imaged sparsely onto the array of sensing elements so that only a subset of the sensing elements output the signals in response to the light pulses reflected from the target scene, and wherein the first calibration indicates the sensing elements that belong to the subset. 
     
     
       15. The method according to  claim 14 , wherein processing the signals comprises measuring the respective times of flight using the signals output by the sensing elements in the subset without receiving the signals output by the sensing elements that are not in the subset, and
 wherein correcting the first calibration comprises modifying a membership of the subset responsively to the detected change in the second locations. 
 
     
     
       16. The method according to  claim 13 , wherein processing the signals comprises grouping the sensing elements in each of the first locations together to define super-pixels, and processing together the signals from the sensing elements in each of the super-pixels in order to measure the respective times of flight. 
     
     
       17. The method according to  claim 12 , and comprising actuating, in accordance with the first calibration, only the sensing elements at each of the first locations, while the remaining sensing elements in the array are inactive. 
     
     
       18. The method according to  claim 12 , wherein correcting the first calibration comprises computing a coordinate transformation relating the second locations indicated by the second calibration to the second locations following the detected change, and applying the coordinate transformation in correcting the first calibration. 
     
     
       19. The method according to  claim 12 , wherein the radiation source comprises at least one vertical-cavity surface-emitting laser (VCSEL). 
     
     
       20. The method according to  claim 12 , wherein the sensing elements comprise single-photon avalanche diodes (SPADs).

Description:
FIELD OF THE INVENTION 
     The present invention relates generally to systems and methods for depth mapping, and particularly to alignment and calibration of beam sources and sensor arrays used in time-of-flight sensing. 
     BACKGROUND 
     Existing and emerging consumer applications have created an increasing need for real-time three-dimensional (3D) imagers. These imaging devices, also known as depth sensors, depth mappers, or light detection and ranging (LiDAR) sensors, enable the remote measurement of distance (and often intensity) to each point in a target scene—referred to as target scene depth—by illuminating the target scene with an optical beam and analyzing the reflected optical signal. A commonly-used technique to determine the distance to each point on the target scene involves transmitting one or more pulsed optical beams towards the target scene, followed by the measurement of the round-trip time, i.e. time-of-flight (ToF), taken by the optical beams as they travel from the source to the target scene and back to a detector array adjacent to the source. 
     Some ToF systems use single-photon avalanche diodes (SPADs), also known as Geiger-mode avalanche photodiodes (GAPDs), in measuring photon arrival time. For example, U.S. Pat. No. 9,997,551, whose disclosure is incorporated herein by reference, describes a sensing device that includes an array of SPAD sensing elements. Each sensing element includes a photodiode, including a p-n junction, and a local biasing circuit, which is coupled to reverse-bias the p-n junction at a bias voltage greater than the breakdown voltage of the p-n junction by a margin sufficient so that a single photon incident on the p-n junction triggers an avalanche pulse output from the sensing element. A bias control circuit is coupled to set the bias voltage in different ones of the sensing elements to different, respective values. 
     The pulsed optical beams that are used in ToF systems may be emitted by a single source or by an array of sources. Some systems use a “sparse” emitter array, in the sense that the number of pulsed beams of optical radiation that are emitted toward a target scene is substantially less than the number of pixels (i.e., SPADs or other sensing elements) in the array that receives the radiation reflected from the scene. 
     For example, U.S. Patent Application Publication 2020/0256669, whose disclosure is incorporated herein by reference, describes depth sensing apparatus that includes a radiation source, which emits a first plurality of beams of light pulses toward a target scene. An array of a second plurality of sensing elements outputs signals indicative of respective times of incidence of photons on the sensing element, wherein the second plurality exceeds the first plurality. Light collection optics image the target scene onto the array of sensing elements. Processing and control circuitry receives the signals from the array and searches over the sensing elements in order to identify, responsively to the signals, respective regions of the array on which the light pulses reflected from the target scene are incident, and to process the signals from the identified regions in order determine respective times of arrival of the light pulses. 
     The terms “optical radiation” and “light” are used in the context of the present description and in the claims to refer to electromagnetic radiation in any of the visible, infrared, and ultraviolet spectral ranges. 
     SUMMARY 
     Embodiments of the present invention that are described hereinbelow provide improved depth mapping systems and methods for operating such systems. 
     There is therefore provided, in accordance with an embodiment of the invention, depth sensing apparatus, which includes a transparent window. A radiation source is configured to emit a first array of beams of light pulses through the window toward a target scene. An imaging assembly includes a second array of sensing elements, configured to output signals indicative of respective times of incidence of photons on the sensing elements, and objective optics configured to image the target scene onto the second array. 
     Processing and control circuitry is coupled to store a first calibration associating the beams in the first array with respective first locations on the second array onto which the beams reflected from the target scene are imaged, and to process the signals in accordance with the first calibration in order to measure respective times of flight of the light pulses. The processing and control circuitry is configured to store a second calibration indicating second locations on which stray radiation is incident on the second array due to reflection of the beams from the window, and to detect, responsively to the signals, a change in the second locations relative to the second calibration and to correct the first calibration so as to compensate for the detected change. 
     In some embodiments, the first array includes a first number of the beams, and the second array includes a second number of the sensing elements, wherein the second number is greater than the first number. In the disclosed embodiments, the objective optics are configured to image the target scene sparsely onto the array of sensing elements so that only a subset of the sensing elements output the signals in response to the light pulses reflected from the target scene, and the first calibration indicates the sensing elements that belong to the subset. In one such embodiment, the processing and control circuitry is configured to process the signals output by the sensing elements in the subset in order to measure the respective times of flight, without receiving the signals output by the sensing elements that are not in the subset, and to modify a membership of the subset responsively to the detected change. 
     Additionally or alternatively, the processing and control circuitry is configured to group the sensing elements in each of the first locations together to define super-pixels, and to process together the signals from the sensing elements in each of the super-pixels in order to measure the respective times of flight. 
     Further additionally or alternatively, the processing and control circuitry is configured, in accordance with the first calibration, to actuate only the sensing elements at each of the first locations, while the remaining sensing elements in the array are inactive. 
     In one embodiment, the processing and control circuitry is configured to compute a coordinate transformation relating the second locations indicated by the second calibration to the second locations following the detected change, and to apply the coordinate transformation in correcting the first calibration. 
     In some embodiments, the radiation source includes at least one vertical-cavity surface-emitting laser (VCSEL) and possibly an array of VCSELs. Additionally or alternatively, the sensing elements include single-photon avalanche diodes (SPADs). 
     In a disclosed embodiment, the apparatus includes a housing, which contains the radiation source and the imaging assembly and which includes the window, wherein the beams of light exit the housing through the window. 
     There is also provided, in accordance with an embodiment of the invention, a method for depth sensing, which includes configuring a radiation source to emit a first array of beams of light pulses through a transparent window toward a target scene, and imaging the target scene onto a second array of sensing elements, which output signals indicative of respective times of incidence of photons on the sensing elements. A first calibration is stored, associating the beams in the first array with respective first locations on the second array onto which the beams reflected from the target scene are imaged. The signals are processed in accordance with the first calibration in order to measure respective times of flight of the light pulses. A second calibration is stored indicating second locations on which stray radiation is incident on the second array due to reflection of the beams from the window. Responsively to the signals, a change in the second locations relative to the second calibration is detected, and the first calibration is corrected so as to compensate for the detected change. 
     The present invention will be more fully understood from the following detailed description of the embodiments thereof, taken together with the drawings in which: 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         FIG.  1    is a schematic side view of a depth mapping system, in accordance with an embodiment of the invention; 
         FIG.  2    is a schematic detail view of a camera module used in a depth mapping system, in accordance with an embodiment of the invention; 
         FIG.  3 A  is a schematic representation of a pattern of spots projected onto a target scene, in accordance with an embodiment of the invention; 
         FIG.  3 B  is a schematic frontal view of a ToF sensing array, in accordance with an embodiment of the invention; 
         FIG.  3 C  is a schematic detail view of a part of the ToF sensing array of  FIG.  3 B , onto which images of the spots in a region of the target scene of  FIG.  3 A  are cast, in accordance with an embodiment of the invention; 
         FIG.  4    is a flow chart that schematically illustrates a method for depth mapping, in accordance with an embodiment of the invention; and 
         FIGS.  5 A- 5 C  are schematic side views of camera modules for use in a depth mapping system, in accordance with alternative embodiments of the invention. 
     
    
    
     DETAILED DESCRIPTION OF EMBODIMENTS 
     Overview 
     In some of the embodiments described in the above-mentioned U.S. Patent Application Publication 2020/0256669, SPADs are grouped together into “super-pixels,” wherein the term “super-pixel” refers to a group of mutually-adjacent pixels along with data processing elements that are coupled directly to these pixels. At any instant during operation of the system, only the sensing elements in the area or areas of the array that are to receive reflected illumination from a beam are actuated, for example by appropriate biasing of the SPADs in selected super-pixels, while the remaining sensing elements are inactive. The sensing elements are thus actuated only when their signals provide useful information. This approach reduces the background signal, thus enhancing the signal-to-background ratio, and lowers both the electrical power needs of the detector array and the number of data processing units that must be attached to the SPAD array. 
     One issue to be resolved in a depth mapping system of this sort is the sizes and locations of the super-pixels to be used. For accurate depth mapping, with high signal/background ratio, it is important that the super-pixels contain the detector elements onto which most of the energy of the reflected beams is imaged, while the sensing elements that do not receive reflected beams remain inactive. A mapping of SPAD pixels to processing units, i.e., the assignment of SPAD pixels to super-pixels, may be determined initially during a factory calibration. Temperature changes during operation, as well as mechanical shocks, however, may alter the mechanical parameters of the mapping, thus modifying the positions of the laser spots on the SPAD array and necessitating recalibration during operation in the field. 
     In response to this problem, U.S. Patent Application Publication 2020/0256669 describes methods for calibrating the locations of the reflected laser spots on the SPAD array. In these methods, processing and control circuitry receives timing signals from the SPAD array and searches over the sensing elements in order to identify the respective regions of the array on which the light pulses reflected from the target scene are incident. This search process, however, can be time-consuming, and the actual depth mapping operation cannot begin until the search is completed. 
     Embodiments of the present invention that are described herein provide improved methods for verifying and updating the calibration of the locations of the laser spots on an array of single-photon detectors, as well as devices implementing these improved methods. These methods make use of stray radiation that is reflected from an optical window, such as the window of a housing containing the radiation source and the imaging assembly. The term “stray radiation” refers to the small fraction of photons that are emitted by the radiation source and reflect back directly from the window to particular locations on the array of sensing elements. These stray photons can be identified readily, since their times of flight are much shorter than those of photons reflected from the target. 
     The locations at which the stray photons are incident on the array of sensing elements are fixed by the relative positions and geometry of the illumination and sensing assemblies in the depth sensing device. Consequently, any change in these locations is a reliable indicator that the internal alignment of the device has changed. Upon detecting such a change, processing and control circuitry in the device can immediately correct the calibration of the locations of the laser spots reflected from the target scene. In some cases, the circuitry is able to compute a coordinate transformation relating the previous locations of the stray reflections on the array of sensing elements, as indicated by the existing calibration, to the new locations following the detected change, and can then apply this coordinate transformation in correcting the calibration. Even when it is not possible to compute and apply such a transformation, the detection of the change in the stray reflections can prompt the processing and control circuitry to initiate a new search in order to update the calibration. 
     The disclosed embodiments thus provide depth sensing apparatus, which comprises a transparent window, a radiation source, and an imaging assembly. The radiation source emits a first array of beams of light pulses through the window toward a target scene. The imaging assembly comprises a second array of sensing elements, which output signals indicative of respective times of incidence of photons on the sensing elements, along with objective optics, which image the target scene onto the array of sensing elements. Processing and control circuitry in the apparatus stores and uses a dual calibration in processing these signals:
         A first calibration associates the beams in the first array with respective locations on the second array onto which the beams reflected from the target scene are imaged. The processing and control circuitry processes the signals output by the sensing elements, in accordance with this first calibration in order to measure respective times of flight of the light pulses and thus sense the depths of points in the target scene.   A second calibration indicates the locations on which stray radiation is incident on the array of sensing elements due to reflection of the beams from the window. In response to the signals output by the sensing elements, the processing and control circuitry is able to detect changes in the locations on which the stray radiation is incident relative to the second calibration. Upon detecting such a change, the processing and control circuitry corrects the first calibration, as explained above, in order to compensate for the detected change.       

     System Description 
       FIG.  1    is a schematic side view of a depth mapping system  20 , in accordance with an embodiment of the invention. Depth mapping system  20  comprises a camera module  48 , which comprises a radiation source  21  and an imaging assembly  23 . The elements of camera module  48  are contained in a housing  25 , which comprises a transparent window  29  through which optical radiation exits and enters the housing. 
     Radiation source  21  emits M individual beams  30  (for example, M may be on the order of 500). The radiation source typically comprises one or more banks of emitters arranged in a two-dimensional emitter array  22 , together with beam optics  37 . The emitters typically comprise solid-state devices, such as vertical-cavity surface-emitting lasers (VCSELs) or other sorts of lasers or light-emitting diodes (LEDs). Beam optics  37  typically comprise a collimating lens and may comprise a diffractive optical element (DOE, not shown), which replicates the actual beams emitted by array  22  to create the M beams that are projected onto a target scene  32 . For the sake of simplicity, these internal elements of beam optics  37  are not shown. 
     Imaging assembly  23  comprises a two-dimensional array  24  of sensing elements, for example single-photon detectors, such as SPADs. Imaging assembly  23  also comprises J processing units  28 , along with select and readout circuits (not shown) for coupling the processing units to the sensing elements and to a controller  26 . Array  24  comprises a number of sensing elements N that is much larger than M, for example, 100×100 pixels or 200×200 pixels. The number J of processing units  28  depends on the number of pixels of array  24  to which each processing unit is coupled. These features of imaging assembly  23  and their operation are described in greater detail in the above-mentioned U.S. Patent Application Publication 2020/0256669. 
     Radiation source  21  emits the M pulsed beams  30  of light through window  29  toward target scene  32 . Although beams  30  are depicted in  FIG.  1    as parallel beams of constant width, each beam diverges as dictated by diffraction. Furthermore, beams  30  diverge from each other so as to cover a required area of scene  32 . Scene  32  reflects or otherwise scatters those beams  30  that impinge on the scene. The reflected and scattered beams returning from scene  32  through window  29  are collected by objective optics  34 , represented by a lens in  FIG.  1   , which form an image of scene  32  on array  24 . Thus, for example, a small region  36  on scene  32 , on which a beam  30   a  has impinged, is imaged onto a small area  38  on array  24 . 
     For clarity, processing units  28  are shown as if separate from array  24 , but in practice they may be integrated with array  24  along with other processing and readout circuits. Processing units  28  comprise hardware amplification and logic circuits, which sense and record pulses output by the sensing elements in respective super-pixels, and thus measure the times of arrival of the photons that gave rise to the pulses, as well as the strengths of the optical pulses impinging on array  24 . The processing units and associated circuits may assemble histograms of the times of arrival of multiple pulses emitted by array  22 , and thus output signals that are indicative of the distance to respective points in scene  32 , as well as of signal strength. Circuitry that can be used for this purpose is described, for example, in the above-mentioned U.S. Patent Application Publication 2020/0256669. Alternatively or additionally, some or all of the components of processing units  28  and other processing circuitry may be separate from array  24  and may, for example, be integrated with controller  26 . For the sake of generality, controller  26 , processing units  28 , and the associated processing and readout circuitry are collectively referred to herein as “processing and control circuitry.” 
     Controller  26  is coupled to both radiation source  21  and imaging assembly  23 . Controller  26  actuates the emitters in array  22  to emit the pulsed beams. The controller also provides control signals to imaging assembly  23  and receives output signals from processing units  28 . The output signals may comprise histogram data, as noted earlier, and may be used by controller  26  to derive both times of incidence and signal strengths at the location of each laser spot that is imaged onto array  24 . 
     To make optimal use of the available sensing and processing resources, controller  26  identifies the respective locations on array  24  on which the pulses of optical radiation reflected from corresponding regions of target scene  32  are imaged by lens  34 , and groups the sensing elements into super-pixels that correspond to these locations. The signals output by sensing elements outside these areas are not used, and these sensing elements may thus be deactivated, for example by reducing or turning off the bias voltage to these sensing elements. Methods for choosing the super-pixels initially and for verifying and updating the selection of super-pixels are described, for example, in the above-mentioned U.S. Patent Application Publication 2020/0256669. 
     For clarity, the dimensions of arrays  22  and  24  have been exaggerated in  FIG.  1    relative to scene  32 . The lateral separation between arrays  22  and  24 , referred to as the “baseline,” is in reality much smaller than the distance from emitter array  22  to scene  32 . Consequently a chief ray  40  (a ray passing through the center of objective optics  34 ) from scene  32  to array  24  is nearly parallel to rays  30 , leading to only a small amount of parallax. 
     Controller  26  typically comprises a programmable processor, which is programmed in software and/or firmware to carry out the functions that are described herein. Alternatively or additionally, controller  26  comprises hard-wired and/or programmable hardware logic circuits, which carry out at least some of the functions of the controller. Although controller  26  is shown in the figure, for the sake of simplicity, as a single, monolithic functional block, in practice the controller may comprise a single chip or a set of two or more chips, with suitable interfaces for receiving and outputting the signals that are described herein. 
     One of the functional units of controller  26  is a depth processing unit (DPU)  27 , which processes signals output by processing units  28  in order to calculate the times of flight of the photons in each of beams  30 , and thus maps the corresponding distances to the points in target scene  32 . This mapping is based on the timing of the emission of beams  30  by emitter array  22  and from the times of arrival (i.e., times of incidence of reflected photons) measured by processing units  28 . Controller  26  typically stores the depth coordinates in a memory, and may output the corresponding depth map for display and/or further processing. 
       FIG.  2    is a schematic detail view of camera module  48 , in accordance with an embodiment of the invention. Emitter array  22  and sensing array  24  are mounted side by side on a substrate  50 , such as a printed circuit board. Beam optics  37  direct the beams emitted from array  22  through window  29  toward the target scene (not shown in this figure). Objective optics  34  image the target scene via a bandpass filter  52  onto sensing array  24 . 
     A number of the beams emitted from array  22 , however, are reflected from the outer surface of window  29  back into module  48  as stray beams  54 . These stray beams pass through objective optics  34  and filter  52  and are incident on certain sensing elements in array  24  (typically, although not necessarily, at the edge of array  24 ). The times of flight of the photons in these beams are very short, relative to the times of flight to and from target scene  32  (as shown in  FIG.  1   ). Controller  26  is thus able to distinguish stray beams  54  from the light that is actually returned from the target scene and can also identify readily the locations in array  24  on which the stray beams are incident. 
     As noted earlier, the locations on array  24  at which stray beams  54  are incident can be stored as part of a process of calibration of camera module  48 ; and controller  26  can use changes in these locations in correcting the calibration when necessary. A calibration and correction procedure of this sort is described in greater detail hereinbelow with reference to  FIG.  4   . 
     In the pictured example, window  29  is sufficiently thick and distant from optics  34  and  37  so that stray beams  54  fall within the field of view of objective optics  34 . When the outer surface of window  29  is closer to optics  34  and  37  (for example because a thinner window is used), the stray beams reflected in a single bounce from the outer surface of the window may fall outside the field of view of the objective optics. Even in this case, however, stray beams with sufficient intensity to be used in calibration may reach sensing array  24  after multiple bounces within window  29  or additional reflections from other surfaces within the camera module. 
     Super-Pixel Selection and Actuation 
       FIG.  3 A  is a schematic representation of a pattern of spots  70  of optical radiation that are projected onto target scene  32 , in accordance with an embodiment of the invention. Each spot  70  is cast by a corresponding beam  30  ( FIG.  1   ). 
       FIG.  3 B  is a schematic frontal view of sensing array  24  onto which target scene  32  is imaged, in accordance with an embodiment of the invention. The sensing elements, such as SPADs, in array  24  are too small to be seen in this figure. Rather,  FIG.  3 B  shows the locations of spots  72  that are reflected from target scene  32  and imaged onto array  24  by objective optics  34 . In other words, each spot  72  is the image on array  24  of a corresponding spot  70  that is projected onto scene  32  by emitter array  22 . Optics  34  image a region  74  of target scene  32  ( FIG.  3 A ), including spots  70  that the area contains, onto a corresponding area  76  on array  24 . 
       FIG.  3 C  is a schematic detail view of area  76  of array  24 , showing the locations of spots  72  that are imaged onto the array, in accordance with an embodiment of the invention. In this view, it can be seen that array  24  comprises a matrix of sensing elements  78 , such as SPADs. (As noted earlier, sensing elements  78  in an array are also referred to as “pixels.”) Controller  26  assigns each processing unit  28  to a super-pixel  80  comprising a 2×2 group of the sensing elements  78 . In this example, it is assumed that during an initial calibration stage, spots  72  were imaged onto array  24  at locations  72   a . Controller  26  thus selected the sensing elements  78  to assign to each super-pixel  80  so as to maximize the overlap between the corresponding spot  72  and the super-pixel, and thus maximize the signal received from each super-pixel. 
     At some later stage, however, spots  72  shifted to new locations  72   b  on array  24 . This shift may have occurred, for example, due to mechanical shock or thermal effects in camera module  48 , or due to other causes. Spots  72  at locations  72   b  no longer overlap with super-pixels  80  in area  76 , or overlap only minimally with the super-pixels. Sensing elements  78  on which the spots are now imaged, however, are inactive and are not connected to any of processing units  28 . To rectify this situation, controller  26  corrects the calibration of the locations of super-pixels  80 , as described below. 
       FIG.  4    is a flow chart that schematically illustrates a method for depth mapping, including calibration and correction when required, in accordance with an embodiment of the invention. This method is described here as one example of how direct reflections of stray beams  54  from a surface such as window  29  (as shown in  FIG.  2   ) can be used in detecting and correcting for certain changes in calibration. Other methods of calibration and correction based on these sorts of direct reflections will be apparent to those skilled in the art after reading the present description and are considered to be within the scope of the present invention. 
     As an initial step in this method, controller  26  stores an initial calibration, which associates each beam  30  that is emitted by radiation source  22  and forms a spot  70  on target scene  32  with a corresponding location on sensing array  24  onto which the spot is imaged. Procedures for performing this calibration are described in detail in the above-mentioned U.S. Patent Application Publication 2020/0256669. Typically, the calibration is performed initially in the factory, and it may be repeated as and when required in the field, for example when camera module  48  is found to have undergone a major shift in alignment. 
     As part of this calibration, controller  26  also registers and stores the locations on which stray beams  54  are incident on sensing array  24  following reflection of the stray beams from window  29 , at a registration step  90 . These stray beams are referred to as “direct reflections” (DR), and the locations at which they impinge on array  24  are referred to as “DR spots.” Controller  26  may also store other features of the DR spots, such as the magnitude of the corresponding signals output by the sensing array. 
     Subsequently, whenever camera module  48  is activated, for example by a user of system  20 , controller  26  detects and checks the signals output by sensing array  24  in response to the DR spots before it begins to track depth coordinates of points in target scene  32 , at a spot detection step  92 . As noted earlier, the DR spots can be identified readily on the basis of their short times of flight, as well as the calibrated locations at which they are incident on sensing array  24 . Controller  26  compares the current locations of the DR spots to the locations that were stored as part of the initial calibration, at a change assessment step  94 . Additionally or alternatively, at this step controller  26  may compare the magnitudes of the signals output by the sensing array in response to the DR spots. 
     If the locations and/or magnitudes are unchanged (to within a predetermined tolerance), controller  26  typically goes on to receive and process signals from the appropriate super-pixels  80  in sensing array  24  in order to track the depth coordinates of the target scene, at a tracking step  96 . Alternatively or additionally, controller  26  may consider other factors, such as prior failures of system  20  in acquiring a suitable depth map of target scene  32 , in deciding whether to proceed to tracking step  96  or to update the calibration of beams  30 , as described below. 
     If the DR spot locations have shifted relative to the stored calibration, controller  26  attempts to compute a coordinate transformation relating the current DR spot locations to the locations that were stored as part of the preceding calibration, at a transform evaluation step  98 . For example, the controller may compute a homographic transformation, which accounts for rotation and translation of the set of DR spots from the calibrated to the current DR spot locations, and possibly for changes in scale (magnification), as well. Controller  26  verifies that the transformation is valid, for example by applying the transformation to the calibrated DR spot locations and checking that the variance of the transformed locations relative to the actual, current DR spot locations is within a predefined tolerance. If so, controller  26  applies the coordinate transformation in correcting and updating the stored calibration of beams  30  relative to locations on sensing array  24 , at a calibration correction step  100 . Thus, for example, the translation and rotation operations of the transformation may be applied to replace locations  72   a  with locations  72   b  in the calibration, as illustrated in  FIG.  3 C . Controller  26  then goes on to receive and process signals from super-pixels  80  in sensing array  24  using the updated calibration in order to track the depth coordinates of the target scene at step  96 . 
     On the other hand, if the DR spots have shifted and it is not possible to compute a valid homographic transformation over the new DR spot locations, controller  26  returns to the initial calibration mode, at a recalibration step  102 . In this mode, controller  26  searches over the sensing elements in array  24  in order to find the locations on sensing array  24  onto which spots  70  are imaged. This calibration procedure may follow along the lines described in the above-mentioned U.S. Patent Application Publication 2020/0256669, possibly using the previous calibration as the starting point for the search. The new calibration will also include the updated locations of the DR spots for future reference. Once the calibration is completed, controller  26  continues to tracking mode at step  96 . 
     Alternative Embodiments 
       FIG.  5 A  is a schematic side views of a camera module  110  for use in a depth mapping system, in accordance with an embodiment of the invention. Module  110  may be used, for example, in place of camera module  48  in system  20 , and the operation of the system will remain substantially as described above. This embodiment, as well as the embodiment of  FIG.  5 B , is shown and described here in order to illustrate that the transparent window from which stray radiation reflects onto array  24  of sensing elements need not necessarily be a single, unitary window, but may rather be one of a number of windows. In this example, camera module  110  includes a receive window  112  and a transmit window  114 , which are joined together. The stray radiation from emitter array  22  may be reflected from either or both of windows  112  and  114 . 
       FIG.  5 B  is a schematic side views of a camera module  120  for use in a depth mapping system, in accordance with another embodiment of the invention. Module  120  is similar to module  110 , except that it comprises a separate receive window  122  and transmit window  124 . Typically (although not necessarily), the stray reflections will come from transmit window  124 . 
       FIG.  5 C  is a schematic side views of a camera module  130  for use in a depth mapping system, in accordance with yet another embodiment of the invention. Camera module  130  is shown and described here in order to illustrate that a transparent window  134  from which stray radiation reflects onto array  24  of sensing elements need not necessarily be a part of a housing  132  in which the elements of the camera module are contained. In this embodiment, window  134  is external to the camera module housing but still results in stray reflections of the transmitted beams reaching array  24  of sensing elements. 
     It will be appreciated that the embodiments described above are cited by way of example, and that the present invention is not limited to what has been particularly shown and described hereinabove. Rather, the scope of the present invention includes both combinations and subcombinations of the various features described hereinabove, as well as variations and modifications thereof which would occur to persons skilled in the art upon reading the foregoing description and which are not disclosed in the prior art.

Metadata:
Filing Date: 20210302
Publication Date: 20250114
Grant Date: 20250114
Priority Date: 20210302
Inventors: LAIFENFELD, MOSHE
ROTH, ZEEV
Assignee: APPLE INC
CPC Classifications: [{"code": "G01S17/10", "inventive": true, "first": false, "tree": "[]"}, {"code": "G01S17/04", "inventive": true, "first": false, "tree": "[]"}, {"code": "G01S17/10", "inventive": false, "first": false, "tree": "[]"}, {"code": "G01S17/894", "inventive": true, "first": false, "tree": "[]"}, {"code": "G01S17/89", "inventive": true, "first": true, "tree": "[]"}, {"code": "G01S7/497", "inventive": true, "first": true, "tree": "[]"}, {"code": "G01S17/10", "inventive": true, "first": false, "tree": "[]"}, {"code": "G01S17/04", "inventive": true, "first": false, "tree": "[]"}, {"code": "G01S17/89", "inventive": true, "first": true, "tree": "[]"}]
Family ID: 83116062