PATENT DOCUMENT

Publication Number: US-12086496-B2
Application Number: US-202318177653-A
Country: US
Kind Code: B2

Title: Systems with overlapped displays

Abstract:
A system may include electronic devices that communicate wirelessly. When positioned so that a pair of devices overlap or are near to one another, the devices may operate in a linked mode. During linked operations, devices may communicate wirelessly while input gathering and content displaying operations are shared among the devices. One or both of a pair of devices may have sensors. An orientation sensor, motion sensor, optical sensor, and/or other sensors may be used in identifying conditions in which to enter the linked mode and to identify a region where displays in the pair of devices overlap.

Claims:
What is claimed is: 
     
       1. A system comprising:
 a laptop computer with a touch-insensitive display and a keyboard; 
 a portable electronic device with a touch-sensitive display and an inertial measurement unit; and 
 control circuitry in at least one of the laptop computer and the portable electronic device that is configured to:
 operate the laptop computer and the portable electronic device in a linked mode based on information from the inertial measurement unit; and 
 in the linked mode, type text directly on the touch-sensitive display while receiving and in response to user input to the keyboard. 
 
 
     
     
       2. The system defined in  claim 1  wherein the portable electronic device is configured to wirelessly provide audio to a pair of headphones and wherein in the linked mode, the control circuitry is configured to adjust the audio in response to additional user input to the laptop computer. 
     
     
       3. The system defined in  claim 1  wherein the portable electronic device comprises a sensor and wherein in the linked mode, the control circuitry is configured to adjust an image on the touch-insensitive display based on sensor data gathered by the sensor. 
     
     
       4. The system defined in  claim 3  wherein the sensor comprises a camera and wherein the image comprises an image captured by the camera. 
     
     
       5. The system defined in  claim 1  wherein the laptop computer and the portable electronic device respectively include first and second speakers and wherein in the linked mode, the control circuitry is configured to provide stereo sound using the first and second speakers. 
     
     
       6. The system defined in  claim 1  wherein the laptop computer comprises a trackpad and wherein in the linked mode, the control circuitry is configured to adjust the display content on the touch-sensitive display in response to touch input to the trackpad. 
     
     
       7. The system defined in  claim 6  wherein the control circuitry is configured to move a cursor between the touch-insensitive display and the touch-sensitive display based on the touch input to the trackpad. 
     
     
       8. The system defined in  claim 1  wherein the inertial measurement unit comprises at least one of an accelerometer, a gyroscope, and a compass. 
     
     
       9. The system defined in  claim 8  wherein the inertial measurement unit is configured to measure an angular orientation of the portable electronic device relative to the laptop computer. 
     
     
       10. The system defined in  claim 1  wherein the portable electronic device runs an application and the control circuitry is configured to type the text directly into the application on the portable electronic device while receiving the user input to the keyboard. 
     
     
       11. A system, comprising:
 a laptop computer with a touch-insensitive display and a trackpad; 
 a portable electronic device with a touch-sensitive display and an optical sensor; and 
 control circuitry in at least one of the laptop computer and the portable electronic device that is configured to:
 operate the laptop computer and the portable electronic device in a linked mode based on information from the optical sensor; and 
 in the linked mode, drag-and-drop display content from the touch-insensitive display to the touch-sensitive display with a pointer and in response to user input to the trackpad, wherein the pointer moves from the touch-insensitive display onto the touch-sensitive display to drag-and-drop the display content from the touch-insensitive display to the touch-sensitive display. 
 
 
     
     
       12. The system defined in  claim 11  wherein the optical sensor comprises a camera. 
     
     
       13. The system defined in  claim 11  wherein the display content is selected from the group consisting of: a document, a photograph, and driving directions. 
     
     
       14. The system defined in  claim 11  wherein a first portion of the display content is displayed on the touch-sensitive display and a second portion of the display content is displayed on the touch-insensitive display while the display content is being dragged-and-dropped from the touch-insensitive display to the touch-sensitive display. 
     
     
       15. A system, comprising:
 a laptop computer with a touch-insensitive display and a trackpad; 
 a portable electronic device with a touch-sensitive display and an orientation sensor, wherein the orientation sensor is configured to detect an angle of orientation of the portable electronic device and to detect when the electronic device is at rest; and 
 control circuitry in at least one of the laptop computer and the portable electronic device that is configured to:
 operate the laptop computer and the portable electronic device in a linked mode in response to information from the orientation sensor indicating that the angle of orientation of the portable electronic device is within a predetermined range of angles and the portable electronic device is at rest; and 
 in the linked mode, drag-and-drop display content from the touch-sensitive display to the touch-insensitive display in response to touch input on the touch-sensitive display. 
 
 
     
     
       16. The system defined in  claim 15  wherein the orientation sensor indicates an amount of movement of the portable electronic device and information from the orientation sensor indicates that the portable electronic device is at rest when the amount of movement of the portable electronic device is less than a predetermined threshold amount. 
     
     
       17. The system defined in  claim 15  wherein a first portion of the display content is displayed on the touch-sensitive display and a second portion of the display content is displayed on the touch-insensitive display while the display content is being dragged-and-dropped from the touch-sensitive display to the touch-insensitive display. 
     
     
       18. The system defined in  claim 15  wherein the touch input comprises a swipe gesture. 
     
     
       19. The system defined in  claim 15  wherein the display content is selected from the group consisting of: a document, a photograph, and an email. 
     
     
       20. The system defined in  claim 11 , wherein the control circuitry is further configured to, in the linked mode, move the pointer on the touch-sensitive display in response to user input to the trackpad.

Description:
This application is a continuation of patent application Ser. No. 17/006,677, filed Aug. 28, 2020, which claims the benefit of provisional patent application No. 62/906,654, filed Sep. 26, 2019, both of which are hereby incorporated by reference herein in their entireties. 
    
    
     FIELD 
     This relates generally to electronic devices, and, more particularly, to systems with multiple electronic devices. 
     BACKGROUND 
     Electronic devices such as computers and cellular telephones are often used as stand-alone devices. Although it is possible to wirelessly share data between these devices, sharing can be complex and cumbersome. 
     SUMMARY 
     A system may include electronic devices that communicate wirelessly. The devices may include displays. In some arrangements, devices may be positioned so that the displays of the devices overlap. 
     When positioned so that a pair of devices overlap or are adjacent to one another, the devices may operate in a linked mode. During linked operations, devices may communicate wirelessly while input gathering and content displaying operations are shared among the devices. For example, a user may seamlessly move a pointer that is present on the display of a first device to the display of a second device. Using the pointer or other user input, content may be moved between devices (e.g., a file on one display may be dragged and dropped onto another display, thereby sharing the file between devices). 
     One or more devices in the system may have sensors. Sensor data such as motion and orientation data may be used in determining when devices should be linked. To determine which portion of a display in a first device is overlapped by a display in a second device, the system may adjust visual output on the display of the first device while gathering corresponding camera input or other optical measurements with the second device. A binary search or other position determination algorithm may be used by the system to identify the relative positions of the devices. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         FIG.  1    is a schematic diagram of an illustrative system with electronic devices in accordance with an embodiment. 
         FIG.  2    is a front view of an illustrative pair electronic devices in accordance with an embodiment. 
         FIG.  3    is a side view of an illustrative system with electronic devices in accordance with an embodiment. 
         FIG.  4    is a flow chart of illustrative operations involved in operating electronic devices in a linked mode in accordance with an embodiment. 
         FIG.  5    is a diagram of an illustrative system with a pair of electronic devices in which an item is being shared by dragging and dropping in accordance with an embodiment. 
         FIG.  6    is a diagram of an illustrative system with a pair of electronic devices in which an item is being shared by swiping in accordance with an embodiment. 
         FIG.  7    is a diagram of an illustrative system with a pair of electronic devices in which an item is being displayed partly on a first of the devices and partly on a second of the devices in accordance with an embodiment. 
         FIG.  8    is a diagram of an illustrative system with a pair of electronic devices in which an item is being shared using a virtual representation of a first of the devices that is being displayed on a second of the devices in accordance with an embodiment. 
         FIG.  9    is a diagram of an illustrative system with three interacting electronic devices in accordance with an embodiment. 
         FIG.  10    is a diagram of an illustrative shared hardware arrangement for a system with a pair of electronic devices in accordance with an embodiment. 
         FIG.  11    is a diagram of an illustrative system in which an item is being shared by transferring the item between a pair of devices using another device in accordance with an embodiment. 
         FIG.  12    is a perspective view of an illustrative system with a pair of devices in which one of the devices is revealing information about another of the devices in accordance with an embodiment. 
         FIG.  13    is a diagram of a system with a pair of devices each of which has on-screen areas for facilitating content sharing between the devices in accordance with an embodiment. 
     
    
    
     DETAILED DESCRIPTION 
     Electronic devices with displays may be linked. This allows a user to move content between devices and perform other operations involving the use of the linked devices. In some configurations, electronic devices are placed adjacent to one another or are positioned so that one device overlaps the other. 
     An illustrative system with electronic devices is shown in  FIG.  1   . As shown in  FIG.  1   , system  8  may include electronic devices  10 . There may be any suitable number of electronic devices  10  in system  8  (e.g., at least two, at least three, at least four, fewer than ten, fewer than five, etc.). 
     Each device  10  may be a laptop computer, a computer monitor containing an embedded computer, a tablet computer, a desktop computer, a cellular telephone, a media player, or other handheld or portable electronic device, a smaller device such as a wristwatch device, a pendant device, a headphone or earpiece device, a head-mounted device such as glasses, goggles, a helmet, or other equipment worn on a user&#39;s head, or other wearable or miniature device, a television, a computer display that does not contain an embedded computer, a gaming device, a navigation device, an embedded system such as a system in which equipment is mounted in a kiosk, in an automobile, airplane, or other vehicle, a removable external case for electronic equipment, an accessory such as a remote control, computer mouse, track pad, wireless or wired keyboard, or other accessory, and/or equipment that implements the functionality of two or more of these devices. 
     In the example of  FIG.  1   , devices  10  include a first device  10 A and a second device  10 B. Devices  10  may include control circuitry  12  (e.g., control circuitry  12 A in device  10 A and control circuitry  12 B in device  10 B). Control circuitry  12  may include storage and processing circuitry for supporting the operation of system  8 . The storage and processing circuitry may include storage such as nonvolatile memory (e.g., flash memory or other electrically-programmable-read-only memory configured to form a solid state drive), volatile memory (e.g., static or dynamic random-access-memory), etc. Processing circuitry in control circuitry  12  may be used to gather input from sensors and other input devices and may be used to control output devices. The processing circuitry may be based on one or more microprocessors, microcontrollers, digital signal processors, baseband processors and other wireless communications circuits, power management units, audio chips, application specific integrated circuits, etc. 
     To support communications between devices  10  and/or to support communications between equipment in system  8  and external electronic equipment, control circuitry  12  may communicate using communications circuitry  14  (e.g., communications circuitry  14 A in device  10 A and communications circuitry  14 B in device  10 B). Communications circuitry  14  may include antennas, radio-frequency transceiver circuitry, and other wireless communications circuitry and/or wired communications circuitry. 
     Circuitry  14 , which may sometimes be referred to as control circuitry and/or control and communications circuitry, may, for example, support bidirectional wireless communications between devices  10  over wireless link  6  (e.g., a wireless local area network link, a near-field communications link, or other suitable wired or wireless communications link (e.g., a Bluetooth® link, a WiFi® link, a simultaneous dual band WiFi link, a WiFi Direct link, a 60 GHz link or other millimeter wave link, etc.). Wired communications also be supported. 
     During operation of system  8 , devices  10  may communicate wirelessly or via wired paths to control the operation of system  8 . For example, user input and other input gathered using sensors and other circuitry in one or more devices  10 , output such as visual content to be displayed on displays in devices  10 , and other input and/or output information may be wirelessly transmitted or transmitted via wired connections to one or more devices  10  and thereby shared among devices  10 . For example, input can be gathered from a user on device  10 A and/or device  10 B and used in controlling device  10 A and/or device  10 B, output can be generated on device  10 A and/or device  10 B (e.g., using control circuitry  12 ) and subsequently presented on a display, speaker, or other output component(s) in device  10 A and/or  10 B, and/or other sharing operations may be performed. This allows a user to drag and drop content between devices  10 , to perform screen-sharing operations, and/or to perform other cooperative operations. When functionality is shared between devices  10 A and  10 B in this way, devices  10 A and  10 B may be referred to as operating in a linked mode. If desired, three or more electronic devices  10  may communicate with each other in system  8 . 
     As shown in  FIG.  1   , devices  10  may include input-output devices  16  (e.g., input-output devices  16 A on device  10 A and input-output devices  16 B on device  10 B). Input-output devices  16  may be used in gathering user input, in gathering information on the environment surrounding the user, and/or in providing a user with output. Devices  16  may include sensors  18 A and  18 B). Sensors  18  may include force sensors (e.g., strain gauges, capacitive force sensors, resistive force sensors, etc.), audio sensors such as microphones, touch and/or proximity sensors such as capacitive sensors, optical sensors such as optical sensors that emit and detect light, ultrasonic sensors, and/or other touch sensors and/or proximity sensors, monochromatic and color ambient light sensors, image sensors, sensors for detecting position, orientation, and/or motion (e.g., accelerometers, magnetic sensors such as compass sensors, gyroscopes, and/or inertial measurement units that contain some or all of these sensors), radio-frequency sensors, depth sensors (e.g., structured light sensors and/or depth sensors based on stereo imaging devices), optical sensors such as self-mixing sensors and light detection and ranging (lidar) sensors that gather time-of-flight measurements, optical sensors such as visual odometry sensors that gather position and/or orientation information using images gathered with digital image sensors in cameras, gaze tracking sensors, visible light and/or infrared cameras having digital image sensors, humidity sensors, moisture sensors, and/or other sensors. 
     Input-output devices  16  may also include displays  20  (e.g., one or more displays  20 A and/or one or more displays  20 B). Displays  20  may be organic light-emitting diode displays, displays based on arrays of crystalline semiconductor dies forming light-emitting diodes, liquid crystal displays, electrophoretic displays, and/or other displays. Displays  20  may be touch-insensitive displays (e.g., displays without touch sensor arrays that are insensitive to touch) or may, if desired, be overlapped by a two-dimensional capacitive touch sensor or other touch sensor (e.g., displays  20  may be touch screen displays). A touch display may have a two-dimensional capacitive touch sensor formed from a two-dimensional array of touch sensor electrodes (e.g., transparent conductive electrodes) overlapping an array of display pixels. A touch-insensitive display (sometimes referred to as a non-touch-sensor display) does not contain a two-dimensional array of touch sensor electrodes and does not gather user touch input. 
     If desired, input-output devices  16  may include other devices  22  (e.g., devices  22 A and/or  22 B). Devices  22  may include components such as status indicator lights (e.g., light-emitting diodes in devices  10  that serve as power indicators, and other light-based output devices), speakers and other audio output devices, electromagnets, permanent magnets, structures formed from magnetic material (e.g., iron bars or other ferromagnetic members that are attracted to magnets such as electromagnets and/or permanent magnets), batteries, etc. Devices  22  may also include power transmitting and/or receiving circuits configured to transmit and/or receive wired and/or wireless power signals. Devices  22  may include buttons, rotating buttons, push buttons, joysticks, keys such as alphanumeric keys in a keyboard or keypad, and/or other devices for gathering user input. 
     If desired, devices  22  may include haptic output devices. Haptic output devices can produce motion that is sensed by the user (e.g., through the user&#39;s fingertips, hands, arms, legs, face, or other body parts). Haptic output devices may include actuators such as electromagnetic actuators, motors, piezoelectric actuators, shape memory alloy actuators, electroactive polymer actuators, vibrators, linear actuators, rotational actuators, actuators that bend bendable members, actuator devices that create and/or control repulsive and/or attractive forces between devices  10  (e.g., components for creating electrostatic repulsion and/or attraction such as electrodes, components for producing ultrasonic output such as ultrasonic transducers, components for producing magnetic interactions such as electromagnets for producing direct-current and/or alternating-current magnetic fields, permanent magnets, magnetic materials such as iron or ferrite, and/or other circuitry for producing repulsive and/or attractive forces between devices  10 ). 
       FIG.  2    is a front view of system  8  showing how display  20 A of device  10 A may be overlapped by display  20 B of device  10 B (e.g., when a cellular telephone or other small portable device is placed in front of a computer or other equipment with a larger display). In some arrangements, device  10 B may be placed adjacent to device  10 A as shown by illustrative device  10 B′. The displays of devices  10 A and  10 B may share an adjacent edge or devices  10 A and  10 B may be separated from each other (e.g., by a distance of 1-100 cm, at least 10 cm, at least 100 cm, less than 10 m, less than 2 m, or other suitable distance). Configurations in which one display at least partially overlaps another display in system  8  are sometimes described herein as an example. Device  10 A may, if desired, be a laptop computer in which display  20 A is located in an upper housing (sometimes referred to as a display housing or upper housing portion) and in which input devices such as a keyboard with keys and trackpad (e.g., a two-dimensional touch sensor) are located in a lower housing (sometimes referred to as a base housing or lower housing portion). A hinge may be used to couple the upper and lower housing portions of device  10 A for rotational motion (e.g., so that the keyboard of the laptop computer may rotate relative to the display of the laptop when the laptop is being opened or closed). 
     When device displays are overlapped as shown in  FIG.  2   , device  10 A (e.g., the overlapped device) may display content in a region such as region  38  adjacent to one or more of the edges of device  10 B. This content may include a label (e.g., “Bob&#39;s phone” or other label corresponding to the identity of device  10 B) or instructions (e.g., “drag and drop files here to transfer”). Region  38  may serve as an indicator that devices  10 A and  10 B are linked and/or may operate as a transfer region to facilitate drag-and-drop sharing of content between devices  10 . When device  10 B is moved, region  38  may be moved accordingly. If desired, a shadow effect may be displayed around the periphery of device  10 B as device  10 B is being placed on device  10 A. A halo or other indicator (e.g., a flashing halo) may be displayed on display  20 A around at least some of the peripheral edge of display  20 B to indicate when devices  10 A and  10 B are placed adjacent to each other and/or are linked. Haptic output (e.g., haptic output provided with a haptic output device in device  10 B) may also be used to indicate when devices  10 A and  10 B are operating in linked mode and/or are preparing to operate in linked mode). 
     The placement of device  10 B overlapping device  10 A may also cause icons on display  20 A to be automatically repositioned to avoid obscuring these icons (see, e.g., illustrative icon  36  on display  20 A that is being moved to position  36 ′ automatically in response to detection that device  10 B is overlapping icon  36 ). 
     During linked operations, a user may move on-screen content between displays. For example, pointer  34  (and/or an icon or other content selected by pointer  34 ) may be moved seamlessly between devices  10  (e.g., to illustrative position  34 ′ on display  20 B and vice versa). This allows icon  32  and associated content on device  10 A to be shared with device  10 B (e.g., by dragging and dropping icon  32  to position  32 ′) and allows content on device  10 B to be shared with device  10 A (e.g., by dragging and dropping icon  30  to position  30 ′). During these operations, the content on display  20 B may seamlessly extend onto surrounding portions of display  20 A so that display  20 A and display  20 B operate as a single visual output space for the user of system  8  (e.g., a computer desktop). Icons that are moved or otherwise manipulated (e. g., by clicking or other gestures) may correspond to photographs, word processing documents, media files, email messages, software applications, files associated with other content, and/or other items. 
     Dragging and dropping operations may be performed using cursor  34  and/or touch input. For example, a user may use a track pad or other input component in device  10 A to move cursor  34  and thereby move an icon or other content between devices  10  and/or the user may perform a flick gesture (sometimes referred to as a swipe gesture) or drag-and-drop gesture using a touch sensor overlapping display  20 B (and/or display  20 A) to move content. In some configurations, a user may flick (swipe), drag and drop, or otherwise share content between devices  10  using region  38  (e.g., by placing an icon such as illustrative icon  40  of  FIG.  2    into region  38  or a predetermined portion of display  20 B). A user may also use pointer  34  on display  20 A and/or display  20 B to double click or otherwise select items (e.g., to click on an icon to launch an application, etc.). When an item is selected (e.g., when an email program is launched on device  10 B), a user may operate the selected item (e.g., the email program) using the shared input circuitry of device  10 A. For example, a user may type text into an email application on device  10 B using a keyboard in device  10 A. Both landscape and portrait orientations may be used for device  10 B. 
     Cooperative operations such as these may be performed using control circuitry  12 A and/or  12 B. In performing these operations, control circuitry  12  may gather sensor information indicative of the position of device  10 B (and display  20 A) relative to device  10 A (and display  20 A). For example, sensor measurements using sensors  18  (e.g., relative position information) may be used to determine the display pixel coordinates that correspond to the portion of display  20 A that is overlapped by display  20 B so that screen content can be shared accordingly (e.g., so that content can be seamlessly displayed across displays  20 A and  20 B, so that items can be moved between displays  20 A and  20 B, etc.). 
       FIG.  3    is a cross-sectional side view of system  8  in an illustrative configuration in which device  10 A is a laptop computer and device  10 B is a portable device such as a cellular telephone with a display that is smaller than display  20 A of device  10 A. In general, devices  10 A and  10 B may be any suitable electronic devices (e.g., device  10 A may be a tablet computer, television, desktop computer, cellular telephone, etc. and device  10 B may be a cellular telephone, wristwatch, tablet computer, etc.). 
     As shown in  FIG.  3   , device  10 A (a laptop computer) may have a first housing portion such as base housing portion  12 A- 1  that is coupled to a second housing portion such as display housing portion  12 A- 2  by hinge  50 . This allows portion  12 A- 2  to rotate about hinge axis  52  with respect to portion  12 A- 1 . Device  10 A may have a keyboard such as keyboard  60  with alphanumeric keys and may have a two-dimensional capacitive touch sensor (see, e.g., trackpad  70 ). 
     During normal operation, housing portion  12 A- 2  may be angled with respect to housing portion  12 A- 1 . For example, housing portion  12 A- 2  may be tilted at an angle A with respect to surface normal n of the surface of housing portion  12 A- 1 . Angle A may be, for example, 0-45°, 10-40°, at least 5°, at least 15°, at least 20°, less than 50°, less than 40°, less than 30°, or other suitable angle. As shown in  FIG.  3   , device  10 B may have a display such as display  20 B on front face F of device  10 B. Device  10 B may have one or more cameras such as front-facing camera  72  on front face F and opposing rear-facing camera  74  on an opposing rear face of device  10 B. In the orientation of  FIG.  3   , camera  74  faces display  20 A in display housing portion  12 A- 2  of device  10 A. 
     When device  10 B is placed in front of device  10 A, a viewer such as viewer  56  who is viewing display  20 A in direction  58  will observe that display  20 B of device  10 B is overlapping a portion of display  20 A such as the portion of display  20 A in region  54 . One or more sensors may be used to detect when device  10 B is resting against display  20 A of device  10 A in this way. For example, device  10 B may detect that device  10 B is resting against the display in the display housing of a laptop computer by analyzing angle A using a gyroscope or other orientation sensor (e.g., angular orientation sensor) and by sensing when device  10 B is at rest. If these conditions (and/or other suitable conditions) are met and if devices  10 A and  10 B are wirelessly paired (e.g., using local network pairing such as successful Bluetooth® pairing, near-field communications pairing, etc.), devices  10 A and  10 B may enter a linked mode of operation. The linked mode may be started automatically or may be started after a user manually confirms that linked mode should be entered. In general, linking of devices  10  may be performed based on user input (e.g., user input gathered by devices  16 ) and/or may be linked based on other criteria (e.g., devices  10  may be linked automatically and/or semiautomatically based on information from input-output devices  16  and/or communications circuitry  14  in addition to or instead of user input information). During linked mode, information on the position of device  10 B on device  10 A (e.g., information on the location of overlapped region  54  within display  20 A) can be obtained using optical sensing or other sensing arrangements. 
     Illustrative operations involved in using system  8  are shown in  FIG.  4   . 
     In the example of  FIG.  4   , control circuitry  12  determines whether devices  10  are coupled wirelessly during the operations of block  76 . For example, control circuitry  12  may use Bluetooth® or WiFi® circuitry or other communications circuitry  14  (e.g., wireless communications circuitry such as wireless local area network communications circuitry, near-field communications circuitry, etc.) to determine whether devices  10  are paired and/or are located on the same local area network or are otherwise in wireless communication over a wireless communications path (e.g., a localized path). 
     In response to determining that devices  10  are wirelessly communicating in this way, control circuitry  12  can conclude that devices  10  are in relatively close proximity to each other (e.g., within tens or hundreds of meters of each other). Operations may then proceed to block  78 . 
     During the operations of block  78 , control circuitry  12  may use one or more sensors in device  10 B (and/or device  10 A) to determine whether device  10 B is likely resting against display  20 A of device  10 A (e.g., a laptop display in this example). The sensor circuitry may include an inertial measurement unit (e.g., a gyroscope, compass, and/or accelerometer) and/or other position and/or orientation sensor to measure angle A of  FIG.  3    that the control circuitry of device  10 B (and/or the control circuitry of device  10 A) may use to determine whether angle A falls within a predetermined range of angles (e.g., 5-40°, 10-35°, etc.) that is indicative of scenarios in which device  10 B is resting against an open laptop computer screen. The control circuitry of device  10 B (and/or the control circuitry of device  10 A) may also determine whether device  10 B is at rest (e.g., whether accelerometer output for device  10 B indicates that device  10 B is moving less than a predetermined threshold amount). If the control circuitry determines that device  10 B is at rest at an angle within the predetermined angular range, devices  10 A and  10 B can conclude that devices  10 A and  10 B are ready for operation in linked mode. 
     If desired, system  8  may monitor for user input indicating that devices  10 A and  10 B are to be operated in the linked mode. This user input may be obtained instead of or in addition to automatically detecting readiness for operation in linked mode using motion and orientation sensor circuitry in device  10 B. The user input may be a particular gesture performed by moving device  10 B towards display  20 A, may be a shaking motion used to shake device  10 B, may be a touch screen input, voice input, and/or other input detected using one or more sensors  18  or other devices  16 . In response to detecting appropriate triggering input conditions (user input, a resting angle of device  10 B within a predetermined angular range, and/or other conditions), operations may proceed to block  80 . 
     During the operations of block  80 , system  8  may use optical sensing and/or other sensing techniques to determine the relative position between devices  10 A and  10 B (e.g., to determine display overlap coordinates). With one illustrative configuration, device  10 B may send a start command to device  10 A. Device  10 A may wirelessly receive the start command. In response to reception of the start command, device  10 A may sweep a visual element (e.g., a vertically extending black bar) across display  20 A (e.g., from left to right). The time of initiation of the sweep is known by system  8 . Device  10 B can monitor for the presence of the black bar using rear-facing camera  74 . When the black bar sweeps by camera  74 , device  10 B (e.g., the control circuitry of device  10 B) can measure the current time. From the measured time at which the black bar is detected, the known start time of the sweep operation, and a known predetermined sweep velocity for the sweeping black bar, system  8  can determine the horizontal position of device  10 B and display  20 B relative to display  20 A and device  10 A. The size of device  10 B and screen  20 B can be retrieved from a database by system  8  (e.g., using information on the type of device present), can be provided to device  10 A from device  10 B, can be measured by sweeping a horizontally extending black bar vertically to determine the height of camera  74 , etc. 
     With another illustrative configuration, the relative position between devices  10 A and  10 B and therefore the location of the portion of display  20 A that is overlapped by display  20 B can be determined by using a binary search. With this type of arrangement, a first half of display  20 A has a first visual attribute (e.g., the first half is black) and the other half of display  20 A has a second visual attribute (e.g., the second half is white). Camera  74  determines whether camera  74  is on the first or second half of display  20 A by measuring the output from display  20 A. After determining which half of display  20 A contains camera  74 , subsequent iterations of the binary search may be performed, each time dividing the current area into halves and providing the halves with different respective first and second attributes. In this way, system  8  can successively narrow the possible location of camera  74  until a precise measurement of the location of camera  74  is determined (and therefore the area of display  20 A that is overlapped by display  20 B is determined). In general, any suitable search algorithm may be used (e.g., a binary search algorithm or other iterative approach). Moreover, any type of visual attribute may be modified (e.g., pixel color, pixel intensity, pixel on/off patterns, etc.). If desired, the modulation of the visual appearance of the pixels of display  20 A may be subtle and therefore not detectable by an ordinary user (e.g., the adjustment of the visual appearance of display  20 A during binary searching operations may be unnoticeable to the unaided eye of a user). 
     After determining which region of display  20 A is overlapped by display  20 B, system  8  can operate in linked mode (block  82 ). During the operations of block  82 , tasks associated with input gathering and the displaying of content are shared by devices  10  in system  8  as described, for example, in connection with  FIG.  2    (e.g., input and may be shared across devices  10 , content may be seamlessly displayed on display  20 B overlapping display  20 A, etc.). 
     As shown in  FIG.  5   , a halo such as halo  90  or other feedback (e.g., haptic output from device  10 A and/or  10 B, audio output from device  10 A and/or  10 B, and/or visual output from device  10 A and/or  10 B) may be provided to the user in response to placement of device  10 B on device  10 A, in response to entering linked mode, and/or in response to other events. In the example, of  FIG.  5   , a user is using device  10 A and display  20 A to interact with a mapping application. The application displays map  86  and associated directions (e.g., driving directions)  84 . When a user links devices  10 A and  10 B, the user is able to drag and drop directions  84  onto display  20 B of device  10 B, as indicated by arrow  88 . Feedback (e.g. halo  90 , audio feedback, haptic feedback, etc.) may be provided to confirm that the drag and drop operation or other sharing operation has succeeded. 
     The dragging and dropping operation of  FIG.  5    causes device  10 A to wirelessly share directions  84  and/or other items associated with the current state of the mapping application on device  10 A with device  10 B. The user may initiate this action if, for example, the user has researched driving directions on device  10 A and is interested in having these directions available on device  10 B (e.g., because the user is embarking on a trip and is leaving device  10 A behind). 
     In the illustrative configuration of  FIG.  6   , a user is sharing an item between devices  10 B and  10 A using touch gestures. In particular, a user is presented with a list of items such as item  92  on display  20 B of device  10 B. Item  92  may an email message or other item (e.g., a document, song, picture, or other content). Display  20 B may be touch sensitive, so that the user may swipe item  92  to the side (e.g., to the right in  FIG.  6   ) to share item  92  between device  10 B and device  10 A. System  8  may, as an example, move item  92  or a copy of item  92  to display  20 A (as shown by shared item  92 M on display  20 A) and may copy associated email content, audio content, image content, etc. from device  10 B to device  10 A. If desired, force input, voice input, air gestures, and/or other input may be used to cause device  10 B to share information with device  10 A (or to cause device  10 A to share information with device  10 B). Sharing operations may be performed whenever devices  10 A and  10 B are linked. 
     If desired, linked mode operations can take place when devices  10 A and  10 B are in close proximity to each other without touching (e.g., within 20 m, within 2 m, at least 10 cm, or other suitable distance where devices  10 A and  10 B are not touching), as shown by the sharing (e.g., by swipe gesture or otherwise) of illustrative item  92 ′ on display  20 B′ of device  10 B′ with device  10 A. If desired, a user may place nearby devices in this type of linked mode of operation (e.g., a mode in which touch gestures such as swipes and/or other such inputs are used to initiate item sharing while devices  10 A and  10 B are in wireless communication but not touching each other) by use of a drop down menu (e.g., menu  94 ). The drop down menu may be presented when, for example, a user drags a finger downward from the upper edge of device  10 B′. 
       FIG.  7    shows how camera  72  of device  10 B may, if desired, be used to capture input such as video images (e.g., a moving image) for a video telephone call. During video telephone operations, an image of a user (e.g., the user&#39;s head captured with camera  72 ) may be displayed. A first portion of the image of the user may be displayed on display  20 A, as shown by portion  96 A and a second mating portion of the image of the user may be seamlessly displayed on overlapping display  20 B, as shown by portion  96 B. The video call may be initiated before or after devices  10 A and  10 B are in the linked mode. The use of camera  72  in device  10 B to capture video makes it possible to omit cameras from device  10 A, thereby helping to reduce the size of device  10 A while enhancing screen area. 
       FIG.  8    is a diagram of an illustrative system in which device  10 B is represented as a virtual device on display  20 A of device  10 A (e.g., an interactive image such as an interactive representation of device  10 B using graphics). Initially, a user may select option  98  on display  20 A of device  10 A. This causes device  10 A to display a visual representation of linked device  10 A (sometimes referred to as a virtual phone, etc.), as shown by icon  100 . Items such as email messages, photos, songs, videos, and/or other content present on device  10 B may be displayed on the display of the virtual phone icon, as shown by illustrative item  102  on icon  100  on display  20 A of device  10 A. Device  10 B may be linked to device  10 A and may be nearby to device  10 A without touching device  10 A. A user may therefore share content from device  10 B onto device  10 A by using device  10 A to drag and drop item  102  from the virtual phone represented by virtual device icon  100  on display  20 A to another portion of display  20 A (see, e.g., location  102 ′ of  FIG.  8   ). This operation causes device  10 B to share item  102  with device  10 A, even if device  10 B and display  20 B are not currently overlapping device  10 A and display  20 A. 
     In the example of  FIG.  9   , system  8  includes three linked electronic devices. Device  10 B may be a cellular telephone (as an example), device  10 A may be a laptop computer (as an example), and device  10 C may be a pair of wireless ear buds (as an example). Device  10 C may be paired with device  10 B, allowing a user to listen to audio from device  10 B on device  10 C. As shown in the upper portion of  FIG.  9   , device  10 A may initially be closed so that housing portions  12 A- 1  and  12 A- 2  rest flat against each other. When a user opens device  10 A as shown by open device  10 A′ in the lower portion of  FIG.  9   , device  10 A may display virtual phone icon  100  on the display of device  10 A. The user may then use on-screen options on icon  100  such as adjustable on-screen option  104  to adjust the playback of audio from device  10 B to device  10 C. For example, controls that are displayed as interactive elements on the display of device  10 A can be used to adjust playback volume and other playback attributes for device  10 B as device  10 B supplies audio to device  10 C. The presentation of other types of content (e.g., video, etc.) may be controlled in this way if desired (e.g., when device  10 C contains a display). 
       FIG.  10    shows how when device  10 B overlaps device  10 A, sensors in device  10 B (e.g., optical sensors, ultrasonic sensors, or other sensors that emit and detect signals such as light, ultrasonic sound, and/or other signals  108 ) can detect the presence of a user&#39;s finger and/or other external objects on the display of device  10 A (e.g., the finger sensing sensors of device  10 B may be used to provide touch sensitivity to the display of device  10 A, even when the display of device  10 A is insensitive to touch). When devices  10 A and  10 B contain duplicative (or complementary) hardware, the overall operation of system  8  may be enhanced by sharing this hardware. For example, if both device  10 A and device  10 B contain input devices such as cameras (see, e.g., front-facing camera  72  on device  10 B and camera  106  on device  10 A), system  8  may use these multiple cameras together to capture three-dimensional images. Similarly, an audio component or other component in device  10 B such as component  112  (e.g., a microphone or speaker) can be shared with an identical audio component or other component in device  10 A such as component  110  (e.g., a microphone or speaker). Shared microphones may be used to capture directional sound information. Shared speakers may provide stereo sound, surround sound effects, etc. In general, any input-output devices of the same type and/or different types may be shared between linked devices  10  in system  8 . 
       FIG.  11    is a diagram of system  8  in an illustrative configuration in which system  8  has three devices. Content can be shared between a first and third of these devices using a second of the devices as an intermediary. Initially, for example, content item  114  on display  20 B- 1  of the first device (e.g., a first cellular telephone belonging to a first user) can be shared (e.g., by dragging and dropping) with the second device (e.g., a laptop computer belonging to a second user), as shown by dragging and dropping item  114  of  FIG.  11    to the location of item  114 ′ on display  20 A of the second device. The second user may then share item  114 ′ from the second device to a third device (e.g., a cellular telephone belonging to the second user or other user). This may be done by using the second device to drag and drop item  114 ′ to display  20 B- 2  of the third device, as shown by shared icon  114 ″ on display  20 B- 2 . In this way, the second device may be used both to move items from the first device onto the second device and to move items from the second device to the third device. 
     In the illustrative configuration of  FIG.  12   , device  10 B is overlapping device  10 A while the housing of device  10 A is closed (e.g., so that upper housing portion  12 A- 2  is resting flat against lower housing portion  12 A- 1 ). In this scenario, the display of device  10 A is not visible. Nevertheless, after linking devices  10 A and  10 B, display  20 B of device  10 B may be used to allow a user to share items between devices (e.g. by swipe sharing or otherwise sharing items from device  20 B to device  10 A). As shown in  FIG.  12   , display  20 B may be used to display an item such as item  116 . Item  116  may be an item on device  10 B that can be shared with device  10 A while device  10 A is closed (e.g., by swiping or using other user input), may be a control that allows a user to adjust the operation of device  10 A while device  10 A is closed (e.g., by adjusting audio playback, etc.), or may represent content on device  10 A that a user can access through interactions with icon  116 . As an example, a user may have a document on device  10 A. When device  10 B is placed on the outer surface of the housing of device  10 A as shown in  FIG.  12   , item  116  (e.g., an icon corresponding to the document) may be displayed on display  20 B. The document file continues to reside exclusively on device  10 A in this example, but can be previewed by the user on display  20 B of device  10 B by selecting icon  116 . In this way, display  20 B of device  10 B may serve as a “window” into the content on device  10 A in situations in which the display of device  10 A is not visible to the user. 
       FIG.  13    is a diagram of system  8  in an illustrative configuration in which a first device  10 A-I is linked with a second device  10 A-II. Devices  10 A- 1  and  10 A-II may be, for example, immediately adjacent to each other with a shared physical edge or may be in close proximity to each other (e.g., within 20 m, less than 5 m, less than 2 m, or other suitable distance). When linked in this way, display  20 AI of device  10 A-I may present region  120  representing content on device  10 A-I such as item  128  and may present region  122  representing content on device  10 A-II. A user may share item  128  from device  10 A-I to device  10 A-II by dragging and dropping item  128  from region  120  to region  122 , as illustrated by shared item  128 ′. Items may be shared from device  10 A-II to device  10 A-I similarly. As shown on the right side of  FIG.  13   , display  20 AII of device  10 A-II may present region  124  representing content on device  10 A-II such as item  130  and may present region  126  representing content on device  10 A-I. A user may share item  130  from device  10 A-II to device  10 A-I by dragging and dropping item  130  from region  124  to region  126 , as illustrated by shared item  130 ′. 
     System  8  may gather and use personally identifiable information. It is well understood that the use of personally identifiable information should follow privacy policies and practices that are generally recognized as meeting or exceeding industry or governmental requirements for maintaining the privacy of users. In particular, personally identifiable information data should be managed and handled so as to minimize risks of unintentional or unauthorized access or use, and the nature of authorized use should be clearly indicated to users. 
     The foregoing is merely illustrative and various modifications can be made to the described embodiments. The foregoing embodiments may be implemented individually or in any combination.

Metadata:
Filing Date: 20230302
Publication Date: 20240910
Grant Date: 20240910
Priority Date: 20190926
Inventors: JOHNSON, PAUL V.
WANG, AARON
MATHEW, DINESH C.
WU, JIAYING
WANG, PAUL X.
Assignee: APPLE INC
CPC Classifications: [{"code": "G06V20/10", "inventive": true, "first": false, "tree": "[]"}, {"code": "G06F3/16", "inventive": true, "first": false, "tree": "[]"}, {"code": "G06F3/0346", "inventive": true, "first": false, "tree": "[]"}, {"code": "G06F3/041", "inventive": true, "first": false, "tree": "[]"}, {"code": "G06F3/016", "inventive": true, "first": false, "tree": "[]"}, {"code": "G06F3/0486", "inventive": true, "first": false, "tree": "[]"}, {"code": "G06F3/04883", "inventive": true, "first": false, "tree": "[]"}, {"code": "G06V20/52", "inventive": true, "first": false, "tree": "[]"}, {"code": "G09G2356/00", "inventive": false, "first": false, "tree": "[]"}, {"code": "G09G2340/0464", "inventive": false, "first": false, "tree": "[]"}, {"code": "G09G2354/00", "inventive": false, "first": false, "tree": "[]"}, {"code": "G06F3/1423", "inventive": true, "first": false, "tree": "[]"}, {"code": "G06F3/041", "inventive": true, "first": false, "tree": "[]"}, {"code": "G06F1/1698", "inventive": true, "first": false, "tree": "[]"}, {"code": "G06F3/04883", "inventive": true, "first": false, "tree": "[]"}, {"code": "G06F3/0486", "inventive": true, "first": false, "tree": "[]"}, {"code": "G06F1/1616", "inventive": true, "first": false, "tree": "[]"}, {"code": "G06F2203/0383", "inventive": false, "first": false, "tree": "[]"}, {"code": "G06F3/038", "inventive": true, "first": false, "tree": "[]"}, {"code": "G06F3/03547", "inventive": true, "first": false, "tree": "[]"}, {"code": "G06F3/1423", "inventive": true, "first": true, "tree": "[]"}, {"code": "G06F3/016", "inventive": true, "first": true, "tree": "[]"}, {"code": "G06V20/10", "inventive": true, "first": false, "tree": "[]"}, {"code": "G06F3/16", "inventive": true, "first": false, "tree": "[]"}, {"code": "G06F3/04883", "inventive": true, "first": false, "tree": "[]"}, {"code": "G06F3/0486", "inventive": true, "first": false, "tree": "[]"}, {"code": "G06F3/041", "inventive": true, "first": false, "tree": "[]"}, {"code": "G06F3/0346", "inventive": true, "first": false, "tree": "[]"}, {"code": "G06F3/016", "inventive": true, "first": false, "tree": "[]"}, {"code": "G06F3/1423", "inventive": true, "first": true, "tree": "[]"}]
Family ID: 85386933