Patent Document

FIELD OF THE INVENTION  
         [0001]    The present invention relates to a remote control data processing method in a remote control system including a client computer, a host computer, and a network for linking them, system thereof, and a computer program product thereof.  
         BACKGROUND OF THE INVENTION  
         [0002]    Recently, as the communication network is expanded and computer systems mounting easy-to-use graphical user interface are developed, remote control systems for controlling electronic appliances at distant places through the network are widely distributing.  
           [0003]    One of the problems of remote control by network is that it takes a certain time to transmit a remote control signal for a long distance. As a result, a time delay occurs between the local operation and operation at the distant place.  
           [0004]    The delay time in network becomes a serious problem in the case of composite operations composed of plural operations. In composite operations, the interval of one operation and other is an important factor for judging what operation has been done. For example, suppose to open a computer program by double-clicking an icon on the screen of a computer in a distant place.  
           [0005]    An example of a problem in remote control caused by delay time in the network is explained below.  
           [0006]    [0006]FIG. 8 is a configuration of a remote control system.  
           [0007]    In FIG. 8, the remote control system contains a client computer  2 , a host computer  1  controlled by the client computer  2  from a distant place, and a network  3  linking these computers.  
           [0008]    The host computer  1  comprises an information processing unit  11 , a display unit  12 , a communication unit  13 , and a pointing device  14 . The display unit  12  displays the image information processed in the information processing unit  11 . The communication unit  13  exchanges information with the network  3 . The pointing device  14  manipulates the host computer  1 .  
           [0009]    The client computer  2  comprises an information processing unit  21 , a display unit  22 , a communication unit  23 , and a pointing device  24 . The display unit  22  displays the image information processed in the information processing unit  21 . The communication unit  23  exchanges information with the network  3 . The pointing device  24  manipulates the client computer  2 .  
           [0010]    The pointing devices  14 ,  24  are a mouse, a digitizer, and other input devices.  
           [0011]    The pointing devices  14 ,  24  have on/off buttons for moving the pointer for indicating a position on the screen of the display units  12 ,  22 , and selecting the object indicated by the pointer by inputting coordinates data.  
           [0012]    The user can click the on/off button twice quickly without moving the pointing device (without updating the device coordinates data), which is called a double click. In this case, if the interval between the first click and second click is long, this operation is not recognized as a double click.  
           [0013]    The network  3  is a wired or wireless network such as a local area network (LAN) or a wide area network (WAN).  
           [0014]    The operation of the remote control system is explained.  
           [0015]    The host computer (hereinafter called host PC)  1  transmits the image information output in the display unit  12  to the communication unit  23  of the client computer (hereinafter called client PC)  2  from the communication unit  13  through the network  3 . At this time, the client computer  2  processes the image information sent from the host computer  1  in the information processing unit  21 , and outputs the image information to the display unit  22 .  
           [0016]    When the pointing device  24  is manipulated in the client PC  2 , this operation information is sent to the communication unit  13  from the communication unit  23 . The host PC  1  processes the operation information sent from the client PC  2  as the operation information by manipulation of the pointing device  14 .  
           [0017]    The host PC  1 , when the image information output in the display unit  12  is changed by manipulation of the pointing device  14 , sends the differential image information before and after the change from the first communication unit  13  to the communication unit  23  of the client PC  2 .  
           [0018]    The client PC  2  updates the image information in the display unit  22  according to the differential image information sent from the host PC  1 .  
           [0019]    In this manner, the host PC  1  located in a distant place can be operated from the client PC  2 , and the same image information as in the display unit  12  of the host PC  1  can be shown in the display unit  22  of the client PC  2 .  
           [0020]    In this remote control system, if the delay time in the network is large, the following phenomena may occur.  
           [0021]    i) If the double click entered in the client PC  2  is transmitted to the host PC  1  as operation information, it is not processed as a double click in the host PC  1 .  
           [0022]    ii) Two single clicks entered in the client PC  2  are processed as a double click at the host PC  1 .  
           [0023]    Such incorrect processes may occur if the time interval between packets is changed by an irregular delay characteristic of the network while the data of two packets showing two successive presses on the button of the pointing device  24  flow on the network. At this time, the host PC  1  recognizes and processes as if the button was pressed at different timing from the operation in transmission.  
           [0024]    Moreover, as compared with the case of the user manipulating the host PC  1  directly, in case that the host PC  1  remote-controled by the client PC  2 , it takes time until the input result at the client PC  2  is reflected in the first display unit  12  of the host side. Therefore, the user of the client computer  2  takes time until knowing that the double click is correctly entered at the host PC  1 . As a result, the working efficiency drops.  
         SUMMARY OF THE INVENTION  
         [0025]    It is hence an object of the invention to present a method, system and a computer program product for processing remote control data even if the delay time in the network is large. In the invention, the host computer terminal and client computer terminal are coupled through the network, and the remote control system is built up.  
           [0026]    The client computer terminal comprises:  
           [0027]    an information processing unit,  
           [0028]    a communication unit for exchanging information with the network, and  
           [0029]    a pointing device for controlling the client computer terminal.  
           [0030]    In this client computer terminal, the information processing unit transforms the operating procedure of the pointing device into the operation information including the button state data, coordinates data, and time interval data between the operations, and the communication unit sends the information to the network.  
           [0031]    On the other hand, the host computer terminal comprises an information processing unit and a communication unit coupled with the network. In the host computer terminal, the information processing unit decomposes the operation information received in its communication unit into the operating procedure at the client terminal, and reproduces the operating procedure.  
           [0032]    In this manner, the operating procedure of the pointing device at the client terminal can be reproduced at the host terminal without having effects of irregular delay of the network. 
       
    
    
     BRIEF DESCRIPTION OF THE DRAWINGS  
       [0033]    [0033]FIG. 1 shows a configuration of a remote control system in an embodiment of the invention.  
         [0034]    [0034]FIG. 2A is a functional block diagram of information processing unit of client PC in the embodiment of the invention.  
         [0035]    [0035]FIG. 2B is a functional block diagram of information processing unit of host PC in the embodiment of the invention.  
         [0036]    [0036]FIG. 3 is a flowchart showing the first button ON judging process in a double click input of client PC in the embodiment of the invention.  
         [0037]    [0037]FIG. 4 is a flowchart showing the first button OFF judging process after the first button ON in a double click input of the client PC in the embodiment of the invention.  
         [0038]    [0038]FIG. 5 is a flowchart showing the second button ON judging process after the first button OFF in a double click input of the client PC in the embodiment of the invention.  
         [0039]    [0039]FIG. 6 is a flowchart showing the process until advancing to the first button ON judging process in a double click input after the double click input or after going out of the double click input condition of the client PC in the embodiment of the invention.  
         [0040]    [0040]FIG. 7 is a flowchart showing reception process of operation packet related to a pointing device of the client PC in the host PC in the embodiment of the invention.  
         [0041]    [0041]FIG. 8 shows a configuration of a conventional remote control system. 
     
    
     DESCRIPTION OF THE PREFERRED EMBODIMENT  
       [0042]    An embodiment of the invention is explained by referring to FIG. 1 to FIG. 7.  
         [0043]    A configuration of remote control system in the embodiment is explained by referring to FIG. 1 and FIG. 2.  
         [0044]    [0044]FIG. 3 to FIG. 6 are flowcharts showing the transmission process of operation packet about a pointing device  44  of the client computer in the embodiment.  
         [0045]    In FIG. 1, information processing units  31 ,  41  monitor the operation of pointing devices  34 ,  44 , respectively.  
         [0046]    As the pointing device is manipulated, when the indicated coordinates are moved or the on/off of the button is changed, the processing units  31 ,  41  generate data, and display the coordinates data and the button status data on the screen of the display unit as a pointer. The processing units  31 ,  41  form data into packets, and send the data to the information processing unit of the partner of communication through communication units  33 ,  43  and network  70 .  
         [0047]    When the button of the pointing device is turned on, off and on within a maximum click interval T 1  without change in the coordinates indicated by the pointing device, the processing units  31 ,  41  judge an input of double click.  
         [0048]    The processing units  31 ,  41  have a timer for measuring the maximum click interval T 1 .  
         [0049]    The timer has a function like a watchdog timer, and measures the time from a first click to a second click on the pointing device. The timer setting time T 1  is counted from 0 second, and is the duration from the start at an arbitrary timing until the end.  
         [0050]    [0050]FIG. 2A is a functional block diagram showing the operation of the information processing unit  41  of the client PC  40 .  
         [0051]    [0051]FIG. 2B is a functional block diagram showing the operation of the information processing unit  31  of the host PC  30 .  
         [0052]    In FIG. 2A, a timer  60  counts the time as the reference for detecting the operation of the pointing device  44 . An operation determining unit  62  detects and judges the move of the pointing device  44  and on/off operation of its button. On the basis of result of this detection and judgement, the operation determining unit  62  further generates operation information. This operation information includes the coordinates data showing the move of the pointing device  44 , button status data showing the on/off operation of the button, and the time between the operations. An informing unit  64  outputs data for informing the user of the operation of the pointing device. A transmission data output unit  66  outputs the data about the operation of the pointing device including the operation information to the network  70  through the communication unit  43 .  
         [0053]    In FIG. 2B, a timer  50  counts the time for judging a double click of the pointing device  34  at the host PC  30 . A watching unit  52  monitors the reception of data about the operation of the pointing device including the operation information sent from the client PC. A data determining unit  54  judges the operation of the pointing device at the client PC  40  based on the data including the operation information sent from the client PC  40 . A reproducing unit  56  reproduces the operation of the pointing device at the client PC  40  depending on the data judged by the data determining unit  54  at the host PC  30 .  
         [0054]    The operation in the client PC  40  is explained by referring to FIG. 3 to FIG. 6.  
         [0055]    The remote control data processing in the client PC  40  includes the following processing in the input of double click.  
         [0056]    1) Judging the first button ON,  
         [0057]    2) Judging the first button OFF after the first button ON,  
         [0058]    3) Judging the second button ON after the first button OFF, and  
         [0059]    4) Processing before advancing to the first button ON judging process in a double click input, after the double click input or after going out of the double click input condition.  
         [0060]    The procedure is explained sequentially from the flowchart in FIG. 3.  
         [0061]    [0061]FIG. 3 is a flowchart showing the first button ON judging process in a double click input in the client PC  40 .  
         [0062]    In FIG. 3, at the click interval acquiring step S 101 , the timer  60  acquires the maximum click interval T 1  for judging a double click from the host PC  30 .  
         [0063]    At the first watching step S 102 , the operation determining unit  62  judges if the client side pointing device  44  has been operated or not. If not operated, the watching step continues until the operation is recognized. When the device  44  is operated, the process goes to the first move judging step S 103 .  
         [0064]    At step S 103 , the determining unit  62  judges if the operation of the pointing device  44  is accompanied by a move or not. If not moved, it is regarded that the button of the device  44  is turned on, and the process goes to the first timer start step S 104 . If moved, the process goes to the first operation packet transmit step S 105 .  
         [0065]    At step S 104 , the timer  60  starts, and the flowchart in FIG. 4 is started.  
         [0066]    At step S 105 , the data output unit  66  sends a coordinates data packet showing the move operation generated in the operation determining unit  62  to the host  30 . After transmission, the process goes to the first ON judging step S 106 .  
         [0067]    At step S 106 , the determining unit  62  judges if the button of the pointing device  44  has been turned on or not.  
         [0068]    If the button is turned on, the flowchart in FIG. 6 is started.  
         [0069]    If the button is not turned on, going back to step S 102 , the same process is repeated.  
         [0070]    The flowchart in FIG. 4 is explained.  
         [0071]    [0071]FIG. 4 is a flowchart showing the first button OFF judging process after first button ON in double click input in the client computer  40 .  
         [0072]    At the first lapse time measuring step S 201 , the timer  60  measures the lapse time, and judges if the time has exceeded T 1  or not. When exceeding T 1 , the process goes to the first timer stop step S 204 . Otherwise, the process goes to the second watching step S 202 .  
         [0073]    At step  202 , the determining unit  62  judges if the pointing device  44  is operated or not. If the device  44  is not operated, going back to step S 201 , the timer continues to watch until exceeding time T 1 .  
         [0074]    When operated, the process goes to the second move judging step S 203 .  
         [0075]    At step S 203 , it is judged if the operation of the pointing device  44  is accompanied by a move or not.  
         [0076]    If not moved, the determining unit  62  judges that the device  44  is turned off, and the flowchart in FIG. 5 is started.  
         [0077]    If moved, it is regarded that the device  44  is moved with the button remaining in ON position, and the process goes to the second timer stop step S 206 .  
         [0078]    At step S 206 , the timer  60  stops, and the output unit  66  transmits the packet data showing button ON to the host PC  30  through the communication unit  43  (the second ON packet transmit step S 207 ).  
         [0079]    In succession, the coordinates data packet of the device  44  is sent to the host PC  30  (the second operation packet send step S 208 ).  
         [0080]    After transmission, the process goes to the first OFF judging step S 209 .  
         [0081]    At step S 209 , the determining unit  62  judges if the ON state button is manipulated to the OFF state or not.  
         [0082]    If the button is manipulated to OFF state, going back to step S 102  in FIG. 1, the same process is repeated.  
         [0083]    When the button remains in ON position, the flowchart in FIG. 6 is started.  
         [0084]    At the first timer stop step S 204 , the timer  60  stops, and the process goes to first ON packet send step S 205 .  
         [0085]    At step S 205 , the output unit  66  transmits the button ON packet data to the host PC  30 , and the flowchart in FIG. 6 is started.  
         [0086]    The steps in the flowchart in FIG. 5 are explained.  
         [0087]    [0087]FIG. 5 is a flowchart showing the second button ON judging process after first button OFF in a double click input in the client PC  40 .  
         [0088]    At the second lapse time measuring step S 301 , the timer  60  measures the lapse time, and judges if the time has exceeded T 1  or not. When exceeding T 1 , the process goes to the fourth timer stop step S 307 . Otherwise, the process goes to the third watching step S 302 .  
         [0089]    At step  302 , the determining unit  62  judges if the second pointing device  44  is operated or not. If the device  44  is not operated, going back to step S 301 , the timer continues to watch until exceeding time T 1  When operated, the process goes to the third timer stop step S 303 .  
         [0090]    At step S 303 , the timer  60  stops, and the process goes to the third move judging step S 304 .  
         [0091]    At step S 304 , the determining unit  62  judges if the operation of the pointing device  44  of the client PC  40  is accompanied by a move or not.  
         [0092]    If not moved, the determining unit  62  judges that the device  44  is turned off. At this time, the output unit  66  transmits an operation command showing a double click to the host PC  30  (double click send step S 305 ).  
         [0093]    Consequently, the informing unit  64  transmits data to an audio output unit or a video output unit of the client PC  40  to inform the user of transmission of double click command by voice or display (informing step S 306 ).  
         [0094]    The next process goes to step S 401  in FIG. 6.  
         [0095]    If the pointing device  44  is moved by operation, on the other hand, the determining unit  62  judges that the button is turned on (step S 102 ), and then turned off (step S 202 ), and consequently that the device  44  is moved. Therefore, first, the output unit  66  transmits packet data showing button ON to the host PC  30  (fourth ON packet send step S 310 ).  
         [0096]    Next, the output unit  66  transmits packet data showing button OFF to the host computer  30  (fourth OFF packet send step S 311 ).  
         [0097]    Further, the output unit  66  transmits the coordinates data showing the coordinates of the moved position of the device  44  to the host computer  30  (third operation packet send step S 312 ). Then the process goes to the third ON judging step S 313 .  
         [0098]    At the second lapse time measuring step S 301 , when the lapse time of the timer  60  exceeds T 1 , the process goes to the fourth timer stop step S 307 .  
         [0099]    At step S 307 , the timer  60  stops, and the process goes to the third ON packet transmit step S 308 .  
         [0100]    At step S 308 , the output unit  66  transmits the packet data showing an button ON to the host PC  30 , and the process goes to third OFF packet send step S 309 .  
         [0101]    At step S 309 , the output unit  66  transmits the packet data showing an button OFF to the host PC  30 , and the process returns to step S 102  in FIG. 1.  
         [0102]    At the third ON judging step S 313 , the operation determining unit  62  judges if the operation judged at step S 302  is ON operation of button or not.  
         [0103]    In the case of ON operation of button, the process goes to the fourth watching step S 401  in FIG. 6.  
         [0104]    Otherwise, the process returns to step S 102  in FIG. 1.  
         [0105]    The flowchart in FIG. 6 is explained.  
         [0106]    [0106]FIG. 6 is a flowchart showing the process until advancing to the first button ON judging process in a double click input after the double click input or after going out of the double click input condition in the client PC  40 .  
         [0107]    At the fourth watching step S 401 , the determining unit  62  judges if the second pointing device  44  is operated or not. If not operated, watching continues until operation is recognized.  
         [0108]    When the device  44  is operated, the process goes to the fourth operation packet send step S 402 .  
         [0109]    At step S 402 , the output unit  66  transmits the operation packet showing the operation judged at step S 401  to the host computer  1 , and the process goes to the fourth OFF judging step S 403 .  
         [0110]    At step S 403 , the determining unit  62  judges if the operation judged at step S 401  is OFF operation of button or not.  
         [0111]    In the case of OFF operation of button, the process returns to step S 102  in FIG. 1.  
         [0112]    Otherwise, the process returns to step S 401 .  
         [0113]    Thus, in the information processing unit  41 , the operation determining unit  62  judges the operation of the pointing device  44 , and generates operation information as packet data. The data output unit  66  outputs the operation information as packet data to the network  70  through the communication unit  43 .  
         [0114]    Process of remote control data in the host PC  30  is explained below.  
         [0115]    [0115]FIG. 7 is a flowchart showing reception process of operation packet, that is, operation information about the pointing device  44  of the client PC  40  in the host PC  30  in the embodiment.  
         [0116]    Herein, T 2  is the maximum click interval for judging a double click by button ON-OFF-ON operation of the pointing device  34  in the host PC  30 . The timer  50  stops automatically when exceeding the time T 2 .  
         [0117]    In FIG. 7, at reception watching step S 501 , the watching unit  52  judges if the operation packet data transmitted from the client computer  40  is received or not.  
         [0118]    If not received, the watching unit  52  continues to monitor until the reception is confirmed.  
         [0119]    When received, the process goes to the fourth move judging step S 502 .  
         [0120]    At step S 502 , the data determining unit  54  judges if the operation packet data received at step S 501  shows the operation accompanied by move of the pointing device  44  of the coordinates or not.  
         [0121]    If accompanied by move of coordinates, the process goes to the fourth timer stop step S 503 . At this step, the timer  50  stops, and the process goes to the double click judging step S 504 .  
         [0122]    If not accompanied by move, the process goes to step S 504 .  
         [0123]    At step S 504 , the data determining unit  54  judges if the operation packet data received at step S 501  shows a double click or not.  
         [0124]    If the data shows a double click, the process goes to the first timer operation watching step S 505 .  
         [0125]    If the data does not show a double click, the process goes to the button operation judging step S 509 .  
         [0126]    At the first timer operation watching step S 505 , the watching unit  52  judges if the timer is in operation or not. If in operation, the unit waits until stopping automatically. Then, the reproducing unit  56 , after the first ON reproducing step S 506 , the OFF reproducing step S 507 , and the second ON reproducing step S 508 , reproduces the ON-OFF-ON operation of the button of the pointing device  14  so as to be recognized as a double click in the host PC  30 .  
         [0127]    If judged not to be a double click at step S 504 , the process goes to the button operation judging step S 509 .  
         [0128]    At step S 501 , the data determining unit  54  checks if the received operation packet shows the operation of changing the button of the pointing device from ON to OFF or not. If Yes, the process goes to the second timer operation watching step S 510 . Otherwise, the process goes to the operation packet reproducing step S 512  for reproducing the button operation.  
         [0129]    At the second timer watching step S 510 , the data determining unit  54  judges if the timer is in operation or not. During operation, waiting until stopping automatically, the process goes to the second timer start step S 511 .  
         [0130]    At step S 511 , the timer  50  starts, and the process goes to the operation packet reproducing step S 512 .  
         [0131]    At step S 512 , the reproducing unit  56  reproduces the operation indicated by the operation packet received at step S 501 .  
         [0132]    Thus, in the information processing unit  31 , the data determining unit  54  judges the operation of the pointing device in the client PC by the data including the operation information sent from the client PC  40 . The reproducing unit  56  reproduces the operation of the pointing device in the client PC at the host PC  30  depending on the data judged by the data determining unit  54 .  
         [0133]    According to the embodiment of the invention, the double click is transmitted from the client computer to the host computer, not as the plural operation packets showing on/off operation of the button, but as one operation packet showing the double click. Therefore, even if the delay time of the network is large, the host computer correctly recognizes the input of double click at the client computer.  
         [0134]    Moreover, the transmission of operation packet showing a double click input from the client computer to the host computer is noticed to the user of the client computer by voice or display. Thus, the user instantly understands the input of double click.  
         [0135]    As described herein, in the remote control data processing method and system of the invention, the information processing unit of the client computer converts the operating procedure of the pointing device into operation information including button status data, coordinates data, and interval data between operations, and transmits the operation information to the host computer. In the information processing unit at the host computer, the operating procedure is reproduced from the received operation information. Thus, even if the delay time of the network is large, the host computer correctly recognizes the operation entered in the client computer.  
         [0136]    In the foregoing embodiment, the specific method and device are explained. The invention may be also realized by using a computer program product containing a computer program for executing the method shown in the embodiment by a computer.

Technology Category: 5