PATENT DOCUMENT

Publication Number: US-10038845-B2
Application Number: US-201715414866-A
Country: US
Kind Code: B2

Title: Modeless video and still frame capture

Abstract:
In an embodiment, an electronic device may be configured to capture still frames during video capture, but may capture the still frames in the 4×3 aspect ratio and at higher resolution than the 16×9 aspect ratio video frames. The device may interleave high resolution, 4×3 frames and lower resolution 16×9 frames in the video sequence, and may capture the nearest higher resolution, 4×3 frame when the user indicates the capture of a still frame. Alternatively, the device may display 16×9 frames in the video sequence, and then expand to 4×3 frames when a shutter button is pressed. The device may capture the still frame and return to the 16×9 video frames responsive to a release of the shutter button.

Claims:
What is claimed is: 
     
       1. A device comprising:
 at least one image sensor; 
 a memory; and 
 an integrated circuit coupled to the image sensor and the memory, wherein:
 the integrated circuit is configured to receive a first plurality of frames from the image sensor at a first aspect ratio and a first frame rate; 
 the integrated circuit is configured to output a first video sequence based on the first plurality of frames to the memory; 
 the first video sequence includes first frames having the first aspect ratio at a second frame rate and second frames having a second aspect ratio different from the first aspect ratio at a third frame rate; and 
 a sum of the second frame rate and the third frame rate is equal to the first frame rate. 
 
 
     
     
       2. The device as recited in  claim 1  wherein the first frames are interleaved with the second frames in the first video sequence. 
     
     
       3. The device as recited in  claim 1  wherein the first frames have a first resolution provided by the image sensor and the second frames have a second resolution less than the first resolution. 
     
     
       4. The device as recited in  claim 1  wherein the first aspect ratio is associated with still frames and the second aspect ratio is associated with video frames. 
     
     
       5. The device as recited in  claim 1  wherein the first aspect ratio is an aspect ratio of the image sensor and the second aspect ratio is associated with a display standard. 
     
     
       6. The device as recited in  claim 1  further comprising a display device coupled to the integrated circuit, wherein the integrated circuit is further configured to generate a preview video sequence based on the first plurality of frames for display on the display device, wherein the preview video sequence has a fourth frame rate that is less than the first frame rate. 
     
     
       7. The device as recited in  claim 6  wherein the preview video sequence has the first aspect ratio and a first resolution that is less than a second resolution of the image sensor. 
     
     
       8. In a device including an image sensor, a memory, and an integrated circuit coupled to the image sensor and the memory, a method comprising:
 receiving, in the integrated circuit, a first plurality of frames from the image sensor at a first aspect ratio and a first frame rate; and 
 outputting a first video sequence based on the first plurality of frames from the integrated circuit to the memory, wherein:
 the first video sequence includes first frames having the first aspect ratio at a second frame rate and second frames having a second aspect ratio different from the first aspect ratio at a third frame rate; and 
 a sum of the second frame rate and the third frame rate is equal to the first frame rate. 
 
 
     
     
       9. The method as recited in  claim 8  further comprising interleaving, by the integrated circuit, the first frames with the second frames in the first video sequence. 
     
     
       10. The method as recited in  claim 8  wherein the first frames have a first resolution provided by the image sensor and the second frames have a second resolution less than the first resolution. 
     
     
       11. The method as recited in  claim 8  wherein the first aspect ratio is associated with still frames and the second aspect ratio is associated with video frames. 
     
     
       12. The method as recited in  claim 8  wherein the first aspect ratio is an aspect ratio of the image sensor and the second aspect ratio is associated with a display standard. 
     
     
       13. The method as recited in  claim 8  wherein the device further comprises a display device coupled to the integrated circuit, and the method further comprises generating, by the integrated circuit, a preview video sequence based on the first plurality of frames for display on the display device, wherein the preview video sequence has a fourth frame rate that is less than the first frame rate. 
     
     
       14. The device as recited in  claim 13  wherein the preview video sequence has the first aspect ratio and a first resolution that is less than a second resolution of the image sensor. 
     
     
       15. An integrated circuit comprising:
 an image signal processor (ISP) have a sensor interface configured to receive a first plurality of frames from an image sensor at a first aspect ratio and a first frame rate and circuitry configured to output frames of a first video sequence responsive to the first plurality of frames, the first video sequence having first frames having the first aspect ratio at a second frame rate and second frames having a second aspect ratio different from the first aspect ratio at a third frame rate, and a sum of the second frame rate and the third frame rate is equal to the first frame rate; and 
 a memory controller coupled to the ISP and configured to write the first video sequence to memory. 
 
     
     
       16. The integrated circuit as recited in  claim 15  wherein the ISP is configured to interleave the first frames with the second frames in the first video sequence. 
     
     
       17. The integrated circuit as recited in  claim 15  wherein the first aspect ratio is associated with still frames and the second aspect ratio is associated with video frames. 
     
     
       18. The integrated circuit as recited in  claim 15  wherein the first aspect ratio is an aspect ratio of the image sensor and the second aspect ratio is associated with a display standard. 
     
     
       19. The integrated circuit as recited in  claim 15  further comprising a display controller coupled to the memory controller, wherein the ISP is configured to generate a preview video sequence based on the first plurality of frames for display on the display device, wherein the preview video sequence has a fourth frame rate that is less than the first frame rate, and wherein the ISP is configured to write the preview video sequence to memory and the display controller is configured to read the preview video sequence from memory to display. 
     
     
       20. The integrated circuit as recited in  claim 19  wherein the preview video sequence has the first aspect ratio and a first resolution that is less than a second resolution of the image sensor.

Description:
This application is a continuation of U.S. patent application Ser. No. 15/089,784, filed on Apr. 4, 2016, which is a divisional of U.S. patent application Ser. No. 14/082,390, filed on Nov. 18, 2013 and now U.S. Pat. No. 9,344,626, issued May 17, 2016. The above applications are incorporated herein by reference in their entireties. 
    
    
     BACKGROUND 
     Technical Field 
     Embodiments disclosed herein are related to the field of video and still frame capture in portable electronic devices. 
     Description of the Related Art 
     Various portable electronic devices are designed to capture video and/or still frames (pictures). For example, such portable electronic devices may include hand-held video cameras, digital cameras, personal digital assistants equipped with image sensors (“cameras”), cell phones/smart phones equipped with cameras, tablets equipped with cameras, laptops equipped with cameras, etc. 
     Devices that support both video capture and still frame capture are becoming common, including the above devices. Such devices often permit the user to capture a still frame during a video capture. For example, the devices may display the video being captured on a screen that is included in the device or attached to the device, and the device may include a button or other user input device that the user can depress to capture the still frame. The “button” may be a physical button on the device, or a virtual button on the screen if the screen is a touch screen. 
     There are several issues with capturing a still frame during a video capture. First, the video capture is often performed at a lower resolution than the camera supports and the higher resolution of the camera is typically used for the still frames captured when video is not being captured. Second, the aspect ratio of the video is typically 16×9 while still frames are typically captured with a 4×3 aspect ratio. Accordingly, when the user captures a still frame, the lower resolution and the different aspect ratio of the captured still frame can be surprising to the user and can be unsatisfying to the user. Generally, the camera sensor needs to be reconfigured when switching between high resolution still mode and lower resolution video mode, so one cannot simply switch modes to capture a higher resolution still image during video capture. 
     SUMMARY 
     In an embodiment, an electronic device may be configured to capture still frames during video capture, but may capture the still frames in the 4×3 aspect ratio and at higher resolution than the 16×9 aspect ratio video frames. In one implementation, the device may interleave high resolution, 4×3 frames and lower resolution 16×9 frames in the video sequence, and may capture the nearest higher resolution, 4×3 frame when the user indicates the capture of a still frame. The device may display the video being captured on a display screen, and may provide an indication of the 4×3 framing as well. For example, the video may be letterboxed with the remainder of the 4×3 framing shown in translucent form around the letterbox. In this manner, the user may be aware of the 4×3 framing and the video framing at the same time. In another implementation, the video being captured may be displayed on the display screen with the 16×9 aspect ratio. If user presses a shutter button to capture a still frame, the displayed video may be expanded to 4×3 aspect ratio (retaining the scale and placement of the video frame within the 4×3 frame). When the user releases the shutter button, the still may be captured and the display may return to the 16×9 aspect ratio. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
       The following detailed description makes reference to the accompanying drawings, which are now briefly described. 
         FIG. 1  is a block diagram of one embodiment of a system. 
         FIG. 2  is a block diagram illustrating frames captured over a period of time in one embodiment. 
         FIG. 3  is a block diagram of one embodiment of displaying frames on a display of the system shown in  FIG. 1 . 
         FIG. 4  is block diagram of one embodiment of an image signal processor (ISP) shown in  FIG. 1 . 
         FIG. 5  is a block diagram of another embodiment of the ISP shown in  FIG. 1 . 
         FIG. 6  is a flowchart illustrating video frame capture according to one embodiment of the system. 
         FIG. 7  is a flowchart illustrating still frame capture according to one embodiment of the system. 
         FIG. 8  is a flowchart illustrating still frame capture according to another embodiment of the system. 
         FIG. 9  is a block diagram of another embodiment of the system. 
     
    
    
     While the embodiments disclosed herein are susceptible to various modifications and alternative forms, specific embodiments thereof are shown by way of example in the drawings and will herein be described in detail. It should be understood, however, that the drawings and detailed description thereto are not intended to limit the embodiments to the particular form disclosed, but on the contrary, the intention is to cover all modifications, equivalents and alternatives falling within the spirit and scope of the appended claims. The headings used herein are for organizational purposes only and are not meant to be used to limit the scope of the description. As used throughout this application, the word “may” is used in a permissive sense (i.e., meaning having the potential to), rather than the mandatory sense (i.e., meaning must). Similarly, the words “include”, “including”, and “includes” mean including, but not limited to. 
     Various units, circuits, or other components may be described as “configured to” perform a task or tasks. In such contexts, “configured to” is a broad recitation of structure generally meaning “having circuitry that” performs the task or tasks during operation. As such, the unit/circuit/component can be configured to perform the task even when the unit/circuit/component is not currently on. In general, the circuitry that forms the structure corresponding to “configured to” may include hardware circuits and/or memory storing program instructions executable to implement the operation. The memory can include volatile memory such as static or dynamic random access memory and/or nonvolatile memory such as optical or magnetic disk storage, flash memory, programmable read-only memories, etc. Similarly, various units/circuits/components may be described as performing a task or tasks, for convenience in the description. Such descriptions should be interpreted as including the phrase “configured to.” Reciting a unit/circuit/component that is configured to perform one or more tasks is expressly intended not to invoke 35 U.S.C. §112, paragraph six interpretation for that unit/circuit/component. 
     This specification includes references to “one embodiment” or “an embodiment.” The appearances of the phrases “in one embodiment” or “in an embodiment” do not necessarily refer to the same embodiment, although embodiments that include any combination of the features are generally contemplated, unless expressly disclaimed herein. Particular features, structures, or characteristics may be combined in any suitable manner consistent with this disclosure. 
     DETAILED DESCRIPTION OF EMBODIMENTS 
     Turning now to  FIG. 1 , a block diagram of one embodiment of a system on a chip (SOC)  10  is shown coupled to a memory  12 , one or more image sensors  26 , and one or more displays  20 . As implied by the name, the components of the SOC  10  may be integrated onto a single semiconductor substrate as an integrated circuit “chip.” In some embodiments, the components may be implemented on two or more discrete chips in a system. Additionally, various components may be integrated on any integrated circuit (i.e. it need not be an SOC). However, the SOC  10  will be used as an example herein. In the illustrated embodiment, the components of the SOC  10  include a central processing unit (CPU) complex  14 , a display pipe  16 , peripheral components  18 A- 18 B (more briefly, “peripherals”), a memory controller  22 , an image signal processor (ISP)  24 , and a communication fabric  27 . The components  14 ,  16 ,  18 A- 18 B,  22 , and  24  may all be coupled to the communication fabric  27 . The memory controller  22  may be coupled to the memory  12  during use. Similarly, the ISP  24  may be coupled to the image sensors  26  during use and the display pipe  16  may be coupled to the displays  20  during use. Thus, the SOC  10 , the image sensors  26 , the memory  12 , and the displays  20  may all be components of a system such as a portable electronic device (examples of which were mentioned above) or any other computer system. In the illustrated embodiment, the CPU complex  14  includes one or more processors  28  and a level two (L2) cache  30 . 
     The ISP  24  may be configured to receive image sensor data from the image sensors  26  and may be configured to process the data to produce image frames that may be suitable, e.g., for display on a display  20  and/or other displays. The image frames may include still frames and video frames. The ISP  24  may be configured to write the image frames to the memory  12  (through the memory controller  22 ). 
     The image sensors  26  may generally include any device configured to sample light and provide an output representing the sampled light. The image sensors  26  may include cameras (e.g. charge coupled devices (CCDs), complementary metal-oxide-semiconductor (CMOS) sensors, etc.). The image sensors  26  may include various fixed or movable optical lenses as well, in some embodiments. 
     More particularly, in an embodiment, the ISP  24  may be configured to receive a series of frames over time from the image sensor  26  (e.g. at a specified frame rate, such as 60 frames per second (fps) although any frame rate may be used in various embodiments). The frames received from a given image sensor  26  may have a resolution and aspect ratio that is based on the maximum resolution of the given image sensor  26  and the aspect ratio of the given image sensor  26 . For example, in an embodiment, the frames may be received at the maximum resolution and the aspect ratio of the given image sensor  26 . In another embodiment, the frames may be received at a different aspect ratio than the given image sensor has, and the maximum resolution that may be supported at that different aspect ratio. For example, in an embodiment, the given image sensor  26  may have a 4×3 aspect ratio but the ISP  24  may receive frames in a 16×9 aspect ratio. The resolution of the received frames may be reduced from the maximum resolution to reflect the loss from 4×3 (16×12) to 16×9. Alternatively, the ISP  24  may be configured to reduce the aspect ratio on the received frames, in another embodiment. 
     In an embodiment, the ISP  24  may receive frames from more than one image sensor concurrently. For example, a video image sensor may be employed for lower resolution video capture along with a higher resolution still frame sensor. The ISP  24  may process data from the image sensors in an interleaved fashion. For example, the ISP  24  may capture and process a video image frame and may use the remainder of the time until the next video frame is captured to process the higher resolution still image frame (or a portion of the higher resolution frame). 
     The ISP  24  may be configured to process the received frames to produce a captured video sequence. The processing may include changing the resolution of at least some of the frames, as well as the aspect ratio, in an embodiment. In some embodiments, the frame rate may be changed as well. In one embodiment, the ISP  24  may interleave frames at the resolution and aspect ratio provided by the image sensor  26  with frames at a lower resolution and a different aspect ratio. For example, the lower resolution may be the resolution of a display standard to which the video sequence is being captured. The display standard may be any standard setting of resolution and aspect ratio which may be implemented by various displays. Since the resolution and aspect ratio is standard, it the video sequence may be suitable for display on many different types of display devices. For example, display standards may include 720p, 720i, 1080p, or 1080i. The 1080p standard is particularly popular presently, and is implemented by many video display devices such as televisions and computer monitors. These various display standards are also often referred to as high definition television (HDTV). The 1080p standard will be used as an example herein, and specifies a 16×9 aspect ratio and a resolution of 1920×1080, or 2 megapixels. On the other hand, the resolution of the image sensor  26  may be 8 megapixels, 10 megapixels, 12.5 megapixels, or more (or less, in an embodiment, but still higher than the 1080p resolution). The aspect ratio of the image sensor  26  may be 4×3 as well. Frames at the display standard resolution and aspect ratio may be referred to as display standard frames, or 1080p frames for a more specific example. Frames at the resolution and aspect ratio received from the image sensor  26  may be referred to as “full resolution” frames, even though in some cases the resolution may not be the maximum resolution of the image sensors  26 . 
     By interleaving the 1080p frames and the full resolution frames in the video sequence, the ISP  24  may reduce the amount of bandwidth and power consumed when outputting the video sequence (e.g. by writing it to the memory  12  over the communication fabric  27  and through the memory controller  22 ). Additionally, the existence of the full resolution frames in the captured video sequence may permit the capture of a still frame while the video is being captured. The still frame may have the full resolution and aspect ratio of the image sensor  26  (by choosing one of the full resolution frames from the video sequence when the user indicates that a still is to be captured), which may provide a more desirable still (e.g. similar to those captured in still mode) without requiring a change of mode. 
     Interleaving the high resolution frames and the lower resolution frames may produce image streams of high resolution frames and lower resolution frames, each at half of the frame rate at which frames are captured from the image sensor  26 . The frame rates need not be the same, however. For example, the high resolution frame rate may be ¼ of the frames in the stream, or ⅛ of the frames, or any other pattern of interleaving. The lower resolution frame rate may be increased to produce the desired total frame rate (e.g. the lower resolution frame may be ¾ of the frames if the high resolution frames are ¼ of the frames, or may be ⅞ of the frames if the high resolution frames are ⅛ of the frames). 
     In an embodiment, the ISP  24  may also generate a “preview” video sequence, which may be displayed by the display pipe  16  on the display  20 . The preview video sequence may be a lower resolution sequence (e.g. the resolution may be similar to the 1080p resolution) but the aspect ratio may be the aspect ratio of the image sensor  26 . That is, the aspect ratio may be the aspect ratio of still images. The preview sequence may be displayed for the user who is capturing the video, so the user may keep the video sequence framed as desired. Furthermore, displaying the still image aspect ratio may permit the user to view how the still image will be framed if a still image is captured. In an embodiment, the display standard aspect ratio may also be indicated in the preview sequence, so the user may see both the video frame and the still frame on the display  20  concurrently. In an embodiment, the frame rate of the preview video sequence may be lower than the frame rate of the captured video sequence as well (e.g. 30 fps as compared to 60 fbs). 
     Other embodiments may display the preview in other fashions. For example, the preview sequence may be generated at the display standard aspect ratio (e.g. 16×9) while video is being captured. The user may indicate a desire to capture a still (e.g. by depressing a shutter button on the device), and the preview may display the still aspect ratio around the video sequence (keeping the video sequence in its same position on the display  20 , but displaying the remainder of the still frame as well). The video capture may continue while the shutter button is depressed. The user may release the shutter button to capture the still image and return to video-only capture. The shutter button may be a physical button included on the housing of the system, or may be a virtual button on the display screen (e.g. if the display  20  is a touch screen display). 
     The display pipe  16  may include hardware to process one or more static frames and/or one or more video sequences for display on the displays  20 . Generally, for each source frame or video sequence, display pipe  16  may be configured to generate read memory operations to read the data representing the frame/video sequence from the memory  12  through the memory controller  22 . In particular in this embodiment, the display pipe  16  may be configured to read the preview sequence from the memory  12  through the memory controller  22 . Thus, the ISP  24  may write the preview sequence to the memory  12  as well as the captured sequence and any still frames. The display pipe  16  may be configured to perform any type of processing on the image data (static frames, video sequences, etc.). In one embodiment, the display pipe  16  may be configured to scale static frames and to dither, scale, and/or perform color space conversion on the frames of a video sequence. The display pipe  16  may be configured to blend the static frames and the video sequence frames to produce output frames for display. More generally, the display pipe  16  may be referred to as a display controller. 
     The displays  20  may be any sort of visual display devices. The displays may include, for example, touch screen style displays for mobile devices such as smart phones, tablets, etc. Various displays  20  may include liquid crystal display (LCD), light emitting diode (LED), plasma, cathode ray tube (CRT), etc. The displays may be integrated into a system including the SOC  10  (e.g. a smart phone or tablet) and/or may be a separately housed device such as a computer monitor, television, or other device. 
     Generally, the aspect ratio of the frame may refer to the ratio of pixels in the horizontal direction (as viewed by the user) to pixels in the vertical direction. The actual number of pixels in the frame may be referred to as the resolution of the frame. The more pixels in the frame, the finer grain the image may be and thus the more accurate the image may be. 
     The CPU complex  14  may include one or more CPU processors  28  that serve as the CPU of the SOC  10 . The CPU of the system includes the processor(s) that execute the main control software of the system, such as an operating system. Generally, software executed by the CPU during use may control the other components of the system to realize the desired functionality of the system. The CPU processors  28  may also execute other software, such as application programs. The application programs may provide user functionality, and may rely on the operating system for lower level device control. Accordingly, the CPU processors  28  may also be referred to as application processors. The CPU complex may further include other hardware such as the L2 cache  30  and/or an interface to the other components of the system (e.g. an interface to the communication fabric  27 ). 
     The peripherals  18 A- 18 B may be any set of additional hardware functionality included in the SOC  10 . For example, the peripherals  18 A- 18 B may include video peripherals such as video encoder/decoders, scalers, rotators, blenders, graphics processing units, etc. The peripherals may include audio peripherals such as microphones, speakers, interfaces to microphones and speakers, audio processors, digital signal processors, mixers, etc. The peripherals may include interface controllers for various interfaces external to the SOC  10  (e.g. the peripheral  18 B) including interfaces such as Universal Serial Bus (USB), peripheral component interconnect (PCI) including PCI Express (PCIe), serial and parallel ports, etc. The peripherals may include networking peripherals such as media access controllers (MACs). Any set of hardware may be included. 
     The memory controller  22  may generally include the circuitry for receiving memory requests from the other components of the SOC  10  and for accessing the memory  12  to complete the memory requests. The memory controller  22  may be configured to access any type of memory  12 . For example, the memory  12  may be static random access memory (SRAM), dynamic RAM (DRAM) such as synchronous DRAM (SDRAM) including double data rate (DDR, DDR2, DDR3, etc.) DRAM. Low power/mobile versions of the DDR DRAM may be supported (e.g. LPDDR, mDDR, etc.). 
     The communication fabric  27  may be any communication interconnect and protocol for communicating among the components of the SOC  10 . The communication fabric  27  may be bus-based, including shared bus configurations, cross bar configurations, and hierarchical buses with bridges. The communication fabric  27  may also be packet-based, and may be hierarchical with bridges, cross bar, point-to-point, or other interconnects. 
     It is noted that the number of components of the SOC  10  (and the number of subcomponents for those shown in  FIG. 1 , such as within the CPU complex  14 ) may vary from embodiment to embodiment. There may be more or fewer of each component/subcomponent than the number shown in  FIG. 1 . 
     Turning next to  FIG. 2 , a diagram illustrating consecutive frames of a video sequence according to one embodiment is shown. The video sequence includes frames  40 A- 40 D at the 4×3 aspect ratio and the full sensor resolution interleaved with frames  42 A- 42 D at the 16×9 aspect ratio and the 1080p resolution. That is, alternating frames are provided at the 4×3 aspect ratio and high resolution, and at the 16×9 aspect ratio and the lower resolution. The frame rate of the video sequence may, in an embodiment, be the same as the input frame rate from the image sensor  26  (e.g. 60 fps). Accordingly, the effective frame rate of each frame type (full resolution and 1080p resolution) may be ½ of the video sequence frame rate (e.g. 30 fps). 
     As mentioned above, the frame rate for the full resolution and 1080p resolution frames need not be the same. Depending on various factors such as the available bandwidth in the system and in the ISP itself, the effective frame rates may be varied. For example, the full resolution frames may be captured at 15 fps and the 1080p resolution frames may be captured at 45 fps to yield 60 fps. Any set of frame rates may be used. 
       FIG. 3  is a block diagram illustrating one embodiment of a frame  44  of a preview sequence as may be displayed on the display  20  during capture of the video sequence shown in  FIG. 2 . The frame  44  may include the video frame  46 , in 16×9 format as indicated by the braces  48  and  50 . Additionally, the portions of the 4×3 (16×12) frame which extend beyond the video frame may be shown in translucent letterbox form  52 A- 52 B, where the full 4×3 (16×12) frame is illustrated via braces  48  and  54 . 
     The translucent letterboxes  52 A- 52 B may be a visible indication of which part of the frame  44  is being captured in the video sequence (the video frame  46 ) and which portion is available for still image (the entire frame  44 ). The translucent letterboxes  52 A- 52 B may shade the image to provide a visual effect, but may allow the underlying pixels to be viewed as well so that the user may frame the desired still shot while video is still being captured. Other embodiments may use other visual indicators (e.g. lines separating the video frame and still frame portions, tick marks on the right and left of the frame  44  at the points where the video frame  46  begins and ends, etc.). 
     The letterboxing effect may be added by the ISP  24  when processing the video sequence, or may be added through the display pipe  16 , blending the transparent overlay as a static image onto the video sequence with a non-unitary alpha value that permits underlying pixel colors to be partially visible. 
     A virtual shutter button  56  is also displayed on the frame  44 . In the illustrated embodiment, the button  56  is displayed within the video frame  44  but it may also be displayed in the letterbox areas  52 A- 52 B. The button  56  may be used in embodiment in which the display  20  is a touch-enabled display. The user may depress the button  56  by making touch contact with the screen at a point at which the button is displayed to capture a still frame. In an embodiment, depressing button causes the still frame to be captured. In another embodiments, releasing the button after depressing it may cause the capture of the still frame. In the meantime, the video sequence may continued to be captured and displayed. In response to the user depressing (or releasing) the button  56 , the SOC  10  may be configured to capture the nearest full resolution frame from the captured video sequence. That is, the point in time at which the still frame capture occurs may be synchronized to the video sequence and the nearest full resolution frame to that point in time may be selected. 
     Turning now to  FIG. 4 , a block diagram of one embodiment of the ISP  24  is shown. In the illustrated embodiment, the ISP  24  includes a sensor interface  60 , a front end scaler  62 , a back end processing block  64 , and a back end scaler  66 . The sensor interface  60  is coupled to receive frames from the image sensor(s)  26 , and is coupled to the front end scaler  62 . The front end scaler  62  is coupled to provide scaled frames to the memory controller  22  for storage in the memory  12 . The back end processing block  64  is configured to read the scaled frames from memory  12  (illustrated by the dotted line  68  in  FIG. 4 ). The back end processing block  64  is coupled to the back end scaler  66 , which is coupled to provide a captured video sequence and a preview video sequence to the memory controller  22  for storage in the memory  12 . The preview video sequence may be read from the memory  12  by the display pipe  16  (illustrated by the dotted line  70 ). 
     The frames received from the image sensor  26  may be full sensor resolution frames at a desired frame rate (e.g. 60 fps, in this example). Additionally, the frames may have the sensor aspect ratio (e.g. 4×3 in this example). As mentioned previously, in some embodiments, multiple image sensors may be employed and frames from each image sensor may be received. The frame rates for receiving images from each sensor need not be the same, in various embodiments. The sensor interface  60  may be configured to receive the frame data and supply the data to the front end scaler  62 . The front end scaler  62  may output alternating frames at the full sensor resolution and aspect ratio (i.e. unmodified frames) and frames at a reduced resolution and a 16×9 aspect ratio. The reduced resolution may be the display standard resolution (e.g. 1080p) or may be an intermediate resolution. For example, the intermediate resolution may be based on the resolution of the preview video sequence. Specifically, the intermediate resolution may match the horizontal resolution of the preview video sequence, in an embodiment. In other embodiments, the relative rates at which high resolution frames and low resolution frames are generated may be varied, as discussed above. 
     The front end scaler  62  may be operating upon the raw sensor pixel data. The sensor pixel data may be provided in any format (e.g. Bayer format). The raw sensor pixel data may be converted to data that is used by the other components of the system (e.g. red-green-blue (RGB) pixels or Chrominance/luminance (YCrCb) pixels). The back end processing block  64  may be configured to perform the conversion, and the back end scaler  66  may be configured to scale the resulting frames. The back end scaler  66  may be configured to output two video streams to memory: the captured video sequence, which may include the interleaved 4×3, full sensor resolution frames interleaved with 1080p resolution, 16×9 frames; and the preview video sequence. 
     The preview video sequence need not be at the full frame rate of the captured video sequence. For example, in the illustrated embodiment, the preview video sequence may be 30 fps where the captured video sequence is 60 fps. Each preview frame may have a 4×3 aspect ratio, and may have a resolution suitable for the display  20 . The previous frame may include the indication of the 16×9 frame within the 4×3 frame (e.g. the letterboxing), in an embodiment. 
     The back end processing block  64  may be configured to implement any other desired image processing or transformation mechanisms in addition to the Bayer pixel conversion mentioned above. For example, the back end processing block  64  may include noise insertion, color enhancement, etc. 
       FIG. 5  is a block diagram of another embodiment of the ISP  24 . The embodiment of  FIG. 5  does not include the front end scaler  62 . Accordingly, the sensor interface  60  may be configured to receive the frames from the image sensor  26  and write the frames to memory. The received frames may have a 4×3 aspect ratio and the full sensor resolution, in the illustrated embodiment. Alternatively, the received frames may have the 16×9 aspect ratio, in another embodiment. The back end processing block  64  may read the received frames from memory (dotted line  72 ), and may generate the scaled video sequence interleaving the 4×3 aspect ratio, full sensor resolution frames and 16×9 aspect ratio, lower resolution frames that were generated by the front end scaler in the embodiment of  FIG. 4 . Furthermore, the back end processing block  64  may be configured to perform the Bayer to RGB/YCrCb processing and the like as described previously. The back end scaler  66  may be configured to generate the captured video sequence and the preview video sequence, as discussed previously with regard to  FIG. 4 . The display pipe  16  may read the preview video sequence from memory (dotted line  74 ) for display on the display  20 . 
       FIG. 6  is a flowchart illustrating operation of one embodiment of components of the SOC  10  for capturing the video sequence and the preview video sequence. While the blocks are shown in a particular order for ease of understanding, other orders may be used. Blocks may be performed in parallel in combinatorial logic in the components. Blocks may operate on different frames, or different portions of a frame, in parallel as well. Blocks, combinations of blocks, and/or the flowchart as a whole may be pipelined over multiple clock cycles. The components of the SOC  10  may be configured to implement the operation shown in  FIG. 6 . 
     The ISP  24  may be configured to receive the image sensor data at the desired frame rate (block  80 ). The ISP  24  may be configured to generate the sequence of interleaved frames (full sensor resolution, 4×3 aspect ratio frames and lower resolution, 16×9 aspect ratio frames). The sequences of frames of each type may effectively have half the frame rate of the received frames, and thus the interleaved sequence may have the same frame rate as the received frames (block  82 ). Alternatively, different frames rates for the low resolution frames and the high resolution frames may be supported, as discussed previously. The ISP  24  may be configured to generate the captured video sequence, which includes the full sensor, 4×3 aspect ratio frames interleaved with 1080p frames (block  84 ). Additionally, the ISP  24  may be configured to generate the preview resolution, 4×3 aspect ratio frames of the preview video sequence (block  86 ). In one embodiment, the frame rate of the preview video sequence is less than that of the captured video sequence. For example, the frame rate may be half of the captured video sequence. Frames of the preview video sequence may be generated, e.g., by merging a 4×3 frame (scaled to 1080p resolution) and an adjacent 16×9 frame from the captured video sequence. The display pipe  16  may display the preview video sequence with translucent letterboxing to indicate the video (16×9) portion of the frame (block  88 ). The translucent letterboxing effect may be part of the backend scaler  66  providing the preview video sequence, or may be applied via blending of the preview video sequence and a static letterbox image. The captured video sequence and the preview video sequence may be stored in separate memory regions in the memory  12 . Since the preview video sequence is provided for the user to view while the sequence is being captured, its region may be somewhat smaller if desired and older frames in the video sequence may be overwritten by newer frames. Conversely, the captured video sequence may be intended for retention (e.g. for later processing/viewing, perhaps offloading to another device, etc.). Accordingly, a larger region may be allocated for the captured video sequence. In some embodiments, the captured video sequence may be transferred to other storage (e.g. non-volatile memory in the system, not shown). 
     If different frame rates are used for the high resolution frames and the low resolution frames, the generation of high resolution frames may be performed in slices between generation of the lower resolution frames. For example, if low resolution frames are being captured at 30 fps, the ISP  24  may capture a low resolution frame in less than 1/30 of a second. The remaining time until the expiration of the 1/30 of a second may be used to process a high resolution slice. The slice may be a tile, or multiple tiles of the high resolution frame, for example. 
       FIG. 7  is a flowchart illustrating operation of one embodiment of components of the SOC  10  for capturing a still frame during video capture. While the blocks are shown in a particular order for ease of understanding, other orders may be used. Blocks may be performed in parallel in combinatorial logic in the components. Blocks, combinations of blocks, and/or the flowchart as a whole may be pipelined over multiple clock cycles. The components of the SOC  10  may be configured to implement the operation shown in  FIG. 7 . 
     The user may indicate to the system that a still frame is to be captured (decision block  90 ). For example, the user may press a physical shutter button on the system, or a virtual shutter button displayed on the display  20  (which may be a touch screen display in this embodiment). The detection of the button press or other indication may be performed by one of the processors  28 , e.g. in response to an interrupt from the device (display  20  or the physical button). If the user has indicated a still (decision block  90 , “yes” leg), the system may capture the nearest 4×3 aspect ratio, full sensor resolution frame from the captured video sequence (block  92 ). The system may determine the nearest frame by, for example, comparing a timestamp of the button press event or other indication event to timestamps of the captured video sequence frames. The still frame may be copied from its location in the captured video in the memory  12  to another location for storage as a still. 
     In some embodiments, the system may employ image quality filtering to select a still image to capture, rather than strictly capturing the nearest high resolution frame. For example, the frames may be checked for sharpness, and a sharper image that is farther away in time from the button press may be preferred over a nearer image. If the subject of the picture is a human, a frame in which the subject&#39;s eyes are open may be preferred over a nearer frame. Any sort of image quality filtering may be included in various embodiments. 
     In another embodiment, the preview video sequence may be displayed at the 16×9 resolution during video capture. If, during video capture, the user indicates that a still is to be captured, the preview video sequence may be expanded to the 4×3 aspect ratio so the user may observe the framing of the still. The video capture may continue during this time, capturing at the 16×9 aspect ratio. Once the still is captured, the preview video sequence may transition back to the 16×9 aspect ratio. Displaying only the 16×9 video frames when a still capture is not being performed may further reduce bandwidth and power consumption for the video data, while still providing the desired aspect ratio for stills, in an embodiment. 
       FIG. 8  is a flowchart illustrating operation of one embodiment of the system to implement the transition from 16×9 to 4×3 and back. While the blocks are shown in a particular order for ease of understanding, other orders may be used. Blocks may be performed in parallel in combinatorial logic in the components. Blocks, combinations of blocks, and/or the flowchart as a whole may be pipelined over multiple clock cycles. The components of the SOC  10  may be configured to implement the operation shown in  FIG. 8 . 
     The system may be in video capture mode, capturing video frames and displaying a 16×9 preview on the display  20 . The user may indicate that a still frame is desired, e.g., by pressing a physical or virtual shutter button (decision block  94 ). As discussed above, the user depressing the button may be detected, e.g., by a processor  28  receiving an interrupt. If the user does not depress the shutter button (decision block  94 , “no” leg), the system may continue displaying the 16×9 preview frames (block  96 ). On the other hand, if the user depresses the shutter button (decision block  94 , “yes” leg), the system may display a 4×3 preview (block  98 ) and may continue displaying the 4×3 preview until the user releases the shutter button (decision block  100 , “no” leg). The user may thus frame the still while the button is depressed, if desired. 
     The transition from 16×9 to 4×3 may be performed in a variety of fashions. For example, the 4×3 frame may fade in, retaining the 16×9 frame in its same position and scale within the 4×3 frame. The portion of the 4×3 frame outside of the 16×9 frame may be translucently letter boxed so that both the video framing and the still framing may be visible concurrently. Alternatively, the fade in may not reach full brightness, providing a visual distinction between the video framing and the still framing. 
     Once the user releases the shutter button (decision block  100 , “yes” leg), the system may capture the still in a memory location separate from the video sequence and the display may return to displaying the 16×9 video frame (block  102 ). For example, the 4×3 frame may fade out. During the framing and capture of the still frame (e.g. while the shutter button is pressed), the video capture may continue to occur. 
     It is noted that, while the discussion above uses 16×9 as the video aspect ratio and 4×3 as the still aspect ratio, other embodiments may employ other aspect ratios for one or both of the video frames and still frames. 
     Similar to the discussion above with regard to  FIG. 7 , the still frame captured in the embodiment of  FIG. 8  may be a frame that is near, in time, to the button release but may also employ image quality filtering. 
     Turning next to  FIG. 9 , a block diagram of one embodiment of a system  150  is shown. In the illustrated embodiment, the system  150  includes at least one instance of the SOC  10  coupled to one or more peripherals  154  and the external memory  12 . A power supply  156  is provided which supplies the supply voltages to the SOC  10  as well as one or more supply voltages to the memory  12  and/or the peripherals  154 . In some embodiments, more than one instance of the SOC  10  may be included (and more than one memory  12  may be included as well). 
     The peripherals  154  may include any desired circuitry, depending on the type of system  150 . For example, in one embodiment, the system  150  may be a mobile device (e.g. personal digital assistant (PDA), smart phone, etc.) and the peripherals  154  may include devices for various types of wireless communication, such as wifi, Bluetooth, cellular, global positioning system, etc. The peripherals  154  may also include additional storage, including RAM storage, solid state storage, or disk storage. The peripherals  154  may include user interface devices such as a display screen, including touch display screens or multitouch display screens, keyboard or other input devices, microphones, speakers, etc. In other embodiments, the system  150  may be any type of computing system (e.g. desktop personal computer, laptop, workstation, net top etc.). Specifically, the peripherals  154  may include the image sensor(s)  26  and the displays  20  shown in  FIG. 1 . 
     The external memory  12  may include any type of memory. For example, the external memory  12  may be SRAM, dynamic RAM (DRAM) such as synchronous DRAM (SDRAM), double data rate (DDR, DDR2, DDR3, etc.) SDRAM, RAMBUS DRAM, etc. The external memory  12  may include one or more memory modules to which the memory devices are mounted, such as single inline memory modules (SIMMs), dual inline memory modules (DIMMs), etc. Alternatively, the external memory  12  may include one or more memory devices that are mounted on the SOC  10  in a chip-on-chip or package-on-package implementation 
     Numerous variations and modifications will become apparent to those skilled in the art once the above disclosure is fully appreciated. It is intended that the following claims be interpreted to embrace all such variations and modifications.

Metadata:
Filing Date: 20170125
Publication Date: 20180731
Grant Date: 20180731
Priority Date: 20131118
Inventors: SILVERSTEIN, D. AMNON
GO, SHUN WAI
LIM, SUK HWAN
MILLET, TIMOTHY J.
CHEN, TING
NI, BIN
Assignee: APPLE INC
CPC Classifications: [{"code": "H04N23/667", "inventive": true, "first": true, "tree": "[]"}, {"code": "H04N23/63", "inventive": true, "first": false, "tree": "[]"}, {"code": "H04N23/667", "inventive": true, "first": false, "tree": "[]"}, {"code": "H04N23/951", "inventive": true, "first": false, "tree": "[]"}, {"code": "H04N23/62", "inventive": true, "first": false, "tree": "[]"}, {"code": "H04N7/0122", "inventive": true, "first": true, "tree": "[]"}, {"code": "H04N23/632", "inventive": true, "first": false, "tree": "[]"}, {"code": "H04N23/632", "inventive": true, "first": false, "tree": "[]"}, {"code": "H04N7/0122", "inventive": true, "first": false, "tree": "[]"}, {"code": "H04N1/212", "inventive": true, "first": false, "tree": "[]"}, {"code": "H04N9/87", "inventive": true, "first": false, "tree": "[]"}, {"code": "H04N1/212", "inventive": true, "first": false, "tree": "[]"}, {"code": "H04N9/87", "inventive": true, "first": false, "tree": "[]"}, {"code": "H04N5/23216", "inventive": true, "first": false, "tree": "[]"}, {"code": "H04N5/23232", "inventive": true, "first": false, "tree": "[]"}, {"code": "H04N7/0122", "inventive": true, "first": false, "tree": "[]"}, {"code": "H04N5/23293", "inventive": true, "first": false, "tree": "[]"}, {"code": "H04N5/23245", "inventive": true, "first": true, "tree": "[]"}, {"code": "H04N1/212", "inventive": true, "first": false, "tree": "[]"}, {"code": "H04N7/007", "inventive": true, "first": false, "tree": "[]"}, {"code": "H04N9/87", "inventive": true, "first": false, "tree": "[]"}, {"code": "H04N7/0122", "inventive": true, "first": false, "tree": "[]"}, {"code": "H04N1/212", "inventive": true, "first": false, "tree": "[]"}]
Family ID: 51730574