PATENT DOCUMENT

Publication Number: US-11928604-B2
Application Number: US-201916379048-A
Country: US
Kind Code: B2

Title: Method and apparatus for building an intelligent automated assistant

Abstract:
A method and apparatus are provided for building an intelligent automated assistant. Embodiments of the present invention rely on the concept of “active ontologies” (e.g., execution environments constructed in an ontology-like manner) to build and run applications for use by intelligent automated assistants. In one specific embodiment, a method for building an automated assistant includes interfacing a service-oriented architecture that includes a plurality of remote services to an active ontology, where the active ontology includes at least one active processing element that models a domain. At least one of the remote services is then registered for use in the domain.

Claims:
What is claimed is: 
     
       1. A computing device, comprising:
 one or more processors; 
 memory; and 
 one or more programs stored in memory, the one or more programs including instructions for: 
 collecting at least one input corresponding to one or more events associated with an operating environment of the computing device; 
 obtaining priorities associated with a plurality of active processing elements included in an active ontology, of an intelligent automated assistant, each active processing element being configured to perform one or more actions responsive to receiving one or more inputs of the collected at least one input; 
 processing, based on the obtained priorities, the collected at least one input using the plurality of active processing elements; 
 identifying at least one action based on results of the plurality of active processing elements processing the collected at least one input, wherein identifying the at least one action comprises, identifying based on the processing results, one or more actions associated with valid conditions; and 
 executing the identified at least one action. 
 
     
     
       2. The computing device of  claim 1 , wherein collecting at least one input comprises:
 receiving the at least one input via one or more sensors distributed in the operating environment of the computing device. 
 
     
     
       3. The computing device of  claim 1 , wherein collecting at least one input comprises:
 receiving the at least one input via a multimodal user interface. 
 
     
     
       4. The computing device of  claim 1 , wherein the at least one action comprises at least one of: activity recognition, time recognition, natural language processing, reactive activity execution, rule-based planning, automated scheduling, task automation, temporal constraint management, and service brokering. 
     
     
       5. The computing device of  claim 1 , wherein the at least one input is tagged with an associated context and a current execution pass. 
     
     
       6. The computing device of  claim 1 , wherein processing, based on the obtained priorities, the collected at least one input using the plurality of active processing elements comprises:
 processing the collected at least one input in accordance with an order of priority associated with the plurality of active processing elements, wherein the order of priority is from highest priority to the lowest priority. 
 
     
     
       7. The computing device of  claim 6 , wherein processing the collected at least one input in accordance with an order of priority associated with the plurality of active processing elements comprises:
 for each active processing element of the plurality of active processing elements, processing the collected at least one input using a plurality of rules of the corresponding active processing element in accordance with an order of priority associated with the plurality of rules. 
 
     
     
       8. The computing device of  claim 7 , wherein each rule of the plurality of rules comprises a condition and an associated action. 
     
     
       9. The computing device of  claim 8 , wherein processing the collected at least one input using a plurality of rules comprises:
 determining whether the collected at least one input match with a condition associated with each rule of the plurality of rules. 
 
     
     
       10. The computing device of  claim 1 , wherein processing, based on the obtained priorities, the collected at least one input using the plurality of active processing elements comprises:
 sharing information among at least two of the plurality of active processing elements. 
 
     
     
       11. The computing device of  claim 10 , wherein sharing information among at least two of the plurality of active processing elements comprises:
 propagating information from a first active processing element to a second active processing element, wherein the information propagated is weighted or associated with a probability. 
 
     
     
       12. The computing device of  claim 1 , wherein the plurality of active processing elements is associated with a specific context. 
     
     
       13. The computing device of  claim 1 , wherein the plurality of active processing elements included in the active ontology are arranged in an ontology-like manner such that each active processing element of the plurality of active processing elements is communicatively coupled to at least one other active processing element. 
     
     
       14. The computing device of  claim 13 , wherein active processing elements that are communicatively coupled share processing results with one another. 
     
     
       15. The computing device of  claim 1 , wherein processing the collected at least one input using the plurality of active of active processing elements is further based on a predefined order of priority. 
     
     
       16. A method for processing inputs in accordance with an active ontology, comprising:
 at a computing device including one or more processors and a memory: 
 collecting at least one input corresponding to one or more events associated with an operating environment of the computing device; 
 obtaining priorities associated with a plurality of active processing elements included in the active ontology of an intelligent automated assistant, each active processing element being configured to perform one or more actions responsive to receiving one or more inputs of the collected at least one input; 
 processing, based on the obtained priorities, the collected at least one input using the plurality of active processing elements; 
 identifying at least one action based on results of the plurality of active processing elements processing the collected at least one input, wherein identifying the at least one action comprises, identifying based on the processing results, one or more actions associated with valid conditions; and 
 executing the identified at least one action. 
 
     
     
       17. The method of  claim 16 , wherein collecting at least one input comprises:
 receiving the at least one input via one or more sensors distributed in the operating environment of the computing device. 
 
     
     
       18. The method of  claim 16 , wherein collecting at least one input comprises:
 receiving the at least one input via a multimodal user interface. 
 
     
     
       19. The method of  claim 16 , wherein the at least one action comprises at least one of:
 activity recognition, time recognition, natural language processing, reactive activity execution, rule-based planning, automated scheduling, task automation, temporal constraint management, and service brokering. 
 
     
     
       20. The method of  claim 16 , wherein the at least one input is tagged with an associated context and a current execution pass. 
     
     
       21. The method of  claim 16 , wherein processing, based on the obtained priorities, the collected at least one input using the plurality of active processing elements comprises:
 processing the collected at least one input in accordance with an order of priority associated with the plurality of active processing elements, wherein the order of priority is from highest priority to the lowest priority. 
 
     
     
       22. The method of  claim 21 , wherein processing the collected at least one input in accordance with an order of priority associated with the plurality of active processing elements comprises:
 for each active processing element of the plurality of active processing elements, processing the collected at least one input using a plurality of rules of the corresponding active processing element in accordance with an order of priority associated with the plurality of rules. 
 
     
     
       23. The method of  claim 22 , wherein each rule of the plurality of rules comprises a condition and an associated action. 
     
     
       24. The method of  claim 23 , wherein processing the collected at least one input using a plurality of rules comprises:
 determining whether the collected at least one input match with a condition associated with each rule of the plurality of rules. 
 
     
     
       25. The method of  claim 16 , wherein processing, based on the obtained priorities, the collected at least one input using the plurality of active processing elements comprises:
 sharing information among at least two of the plurality of active processing elements. 
 
     
     
       26. The method of  claim 25 , wherein sharing information among at least two of the plurality of active processing elements comprises:
 propagating information from a first active processing element to a second active processing element, wherein the information propagated is weighted or associated with a probability. 
 
     
     
       27. The method of  claim 16 , wherein the plurality of active processing elements is associated with a specific context. 
     
     
       28. The method of  claim 16 , wherein the plurality of active processing elements included in the active ontology are arranged in an ontology-like manner such that each active processing element of the plurality of active processing elements is communicatively coupled to at least one other active processing element. 
     
     
       29. The method of  claim 28 , wherein active processing elements that are communicatively coupled share processing results with one another. 
     
     
       30. The method of  claim 16 , wherein processing the collected at least one input using the plurality of active of active processing elements is further based on a predefined order of priority. 
     
     
       31. A non-transitory computer-readable storage medium storing one or more programs, the one or more programs comprising instructions, which when executed by one or more processors of a computing device, cause the computing device to:
 collect at least one input corresponding to one or more events associated with an operating environment of the computing device; 
 obtain priorities associated with a plurality of active processing elements included in the active ontology of an intelligent automated assistant, each active processing element being configured to perform one or more actions responsive to receiving one or more inputs of the collected at least one input; 
 process, based on the obtained priorities, the collected at least one input using the plurality of active processing elements; 
 identify at least one action based on results of the plurality of active processing elements processing the collected at least one input, wherein identifying the at least one action comprises, identifying based on the processing results, one or more actions associated with valid conditions; and 
 execute the identified at least one action. 
 
     
     
       32. The non-transitory computer-readable storage medium of  claim 31 , wherein collecting at least one input comprises:
 receiving the at least one input via one or more sensors distributed in the operating environment of the computing device. 
 
     
     
       33. The non-transitory computer-readable storage medium of  claim 31 , wherein collecting at least one input comprises:
 receiving the at least one input via a multimodal user interface. 
 
     
     
       34. The non-transitory computer-readable storage medium of  claim 31 , wherein the at least one action comprises at least one of: activity recognition, time recognition, natural language processing, reactive activity execution, rule-based planning, automated scheduling, task automation, temporal constraint management, and service brokering. 
     
     
       35. The non-transitory computer-readable storage medium of  claim 31 , wherein the at least one input is tagged with an associated context and a current execution pass. 
     
     
       36. The non-transitory computer-readable storage medium of  claim 31 , wherein processing, based on the obtained priorities, the collected at least one input using the plurality of active processing elements comprises:
 processing the collected at least one input in accordance with an order of priority associated with the plurality of active processing elements, wherein the order of priority is from highest priority to the lowest priority. 
 
     
     
       37. The non-transitory computer-readable storage medium of  claim 36 , wherein processing the collected at least one input in accordance with an order of priority associated with the plurality of active processing elements comprises:
 for each active processing element of the plurality of active processing elements, processing the collected at least one input using a plurality of rules of the corresponding active processing element in accordance with an order of priority associated with the plurality of rules. 
 
     
     
       38. The non-transitory computer-readable storage medium of  claim 37 , wherein each rule of the plurality of rules comprises a condition and an associated action. 
     
     
       39. The non-transitory computer-readable storage medium of  claim 38 , wherein processing the collected at least one input using a plurality of rules comprises:
 determining whether the collected at least one input match with a condition associated with each rule of the plurality of rules. 
 
     
     
       40. The non-transitory computer-readable storage medium of  claim 31 , wherein processing, based on the obtained priorities, the collected at least one input using the plurality of active processing elements comprises:
 sharing information among at least two of the plurality of active processing elements. 
 
     
     
       41. The non-transitory computer-readable storage medium of  claim 40 , wherein sharing information among at least two of the plurality of active processing elements comprises:
 propagating information from a first active processing element to a second active processing element, wherein the information propagated is weighted or associated with a probability. 
 
     
     
       42. The non-transitory computer-readable storage medium of  claim 31 , wherein the plurality of active processing elements is associated with a specific context. 
     
     
       43. The non-transitory computer-readable storage medium of  claim 31 , wherein the plurality of active processing elements included in the active ontology are arranged in an ontology-like manner such that each active processing element of the plurality of active processing elements is communicatively coupled to at least one other active processing element. 
     
     
       44. The non-transitory computer-readable storage medium of  claim 43 , wherein active processing elements that are communicatively coupled share processing results with one another. 
     
     
       45. The non-transitory computer-readable storage medium of  claim 31 , wherein processing the collected at least one input using the plurality of active of active processing elements is further based on a predefined order of priority.

Description:
CROSS REFERENCE TO RELATED APPLICATIONS 
     This application is a continuation of U.S. application Ser. No. 15/299,264, filed Oct. 20, 2016, which is a continuation of U.S. application Ser. No. 14/141,261, filed on Dec. 26, 2013, which is a continuation of U.S. application Ser. No. 11/518,292, filed. Sep. 8, 2006, which claims the benefit of U.S. Provisional Patent Application Ser. No. 60/715,324, filed Sep. 8, 2005. The contents of all applications are herein incorporated by reference in their entirety. 
    
    
     FIELD OF THE INVENTION 
     The invention relates generally to intelligent systems and relates more specifically to tools for building classes of applications for intelligent automated assistants. 
     BACKGROUND OF THE DISCLOSURE 
     Intelligent systems, such as intelligent automated assistants, that are capable of interacting with humans (e.g., by observing user behavior, communicating with users, understanding observed situations, anticipating what a user may need and acting to produce useful behavior) are valuable in a variety of situations. For example, such systems may assist individuals who are impaired in some way (e.g., visually, auditorially, physically, cognitively, etc.), including elderly people (who may be stricken by one or more ailments), surgeons (whose eyes, hands and brains are constantly busy/when performing operations) and business executives (who may have numerous tasks to accomplish), among others. 
     To accomplish all of these objectives, intelligent automated assistants integrate a variety of capabilities provided by different software components (e.g., for supporting natural language recognition, multimodal input, managing distributed services, etc.). Development of a system incorporating these different software components typically requires knowledge of numerous different programming languages and artificial intelligence-based methods. Thus, the development of an intelligent automated assistant is a complex task that typically requires contribution from a plurality of highly skilled individuals each having expertise in different aspects of programming; it is nearly impossible for a lone software developer to build an intelligent automated assistant due to the breadth and variety of expertise that is required to build the system. The typical development process for building intelligent automated assistants is therefore relatively inefficient in terms of time, cost and manpower. 
     Thus, there is a need in the art for a method and apparatus for building an intelligent automated assistant. 
     SUMMARY OF THE INVENTION 
     A method and apparatus are provided for building an intelligent automated assistant. Embodiments of the present invention rely on the concept of “active ontologies” (e.g., execution environments constructed in an ontology-like manner) to build and run applications for use by intelligent automated assistants. In one specific embodiment, a method for building an automated assistant includes interfacing a service-oriented architecture that includes a plurality of remote services to an active ontology, where the active ontology includes at least one active processing element that models a domain. At least one of the remote services is then registered for use in the domain. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         FIG.  1    is a schematic diagram illustrating one embodiment of an active ontology execution environment according to the present invention; 
         FIG.  2    is a schematic diagram illustrating one embodiment of an exemplary active ontology that is configured as an autominder for reminding a user to take medicine after meals (e.g., configured for activity and/or time recognition); 
         FIG.  3    is a flow diagram illustrating one embodiment of a method for processing facts in accordance with an active ontology (e.g., configured in a manner similar to the active ontology of  FIG.  1   ) according to the present invention; 
         FIG.  4    is a schematic diagram illustrating one embodiment of an open standard-based system for developing and managing an intelligent system using active ontologies; 
         FIG.  5    is a schematic diagram illustrating one embodiment of a framework for dynamically registering and coordinating distributed services using active ontologies; and 
         FIG.  6    is a high level block diagram of the present intelligent system building method that is implemented using a general purpose computing device. 
     
    
    
     DETAILED DESCRIPTION 
     In one embodiment, the present invention is a method and apparatus for building an intelligent automated assistant. Embodiments of the present invention rely on a developer-friendly unified framework, referred to as an “active ontology”, which integrates multiple system-building capabilities in a single tool. An “ontology”, generally, is a passive data structure that represents domain knowledge, where distinct classes, attributes and relations among classes are defined. A separate engine may operate or reason on this data structure to produce certain results. Within the context of the present invention, an “active ontology” may be thought of as an execution environment in which distinct processing elements are arranged in an ontology-like manner (e.g., having distinct attributes and relations with other processing elements). These processing elements carry out at least some of the typical tasks of an intelligent automated assistant. Although described within the context of an intelligent automated assistant, it will be understood that the concepts of the present invention may be implemented in accordance with any application that involves interaction with software. 
       FIG.  1    is a schematic diagram illustrating one embodiment of an active ontology execution environment  100  according to the present invention. The execution environment comprises an active ontology  102  that is adapted to receive one or more input facts or events  104  and process these inputs  104  to produce useful actions  106 . In one embodiment, the active ontology  102  is tailored to a specific context. For example, the active ontology  102  may be adapted to remind a user to take medication after meals, where at least some of the input facts  104  relate to events collected from the user&#39;s surrounding environment (e.g., “the user is in the kitchen”, “the user is eating”, “the time is 8:00 AM”, etc.) and at least one of the actions  106  is produced in response to these facts (e.g., prompting the user to take the medication at the appropriate time). 
     To this end, the active ontology  102  comprises at least one active processing element or concept  108   1 - 108   n  (hereinafter collectively referred to as “concepts  108 ”). The concepts  108  may be thought of as the basic building block of the active ontology  102 ; concepts  108  both represent a sub-part of the execution environment  100  and perform computations or actions (e.g., natural language understanding, task execution, etc.) related to the sub-part. Concepts  108  are adapted to match to specific types of facts in order to perform these actions, and may be specifically configured to manage temporal constraints, service brokering strategies, fusion strategies, or other tasks. 
     In one embodiment, the concepts  108  are prioritized such that the input facts  104  are processed against the concepts  108  starting with a highest-ranked concept  108  and proceeding in descending order to a lowest-ranked concept  108 . In one embodiment, at least two of the concepts  108  are communicatively coupled by a channel  110   1 - 110   n  (hereinafter collectively referred to as “channels  110 ”) that allows the concepts  108  to communicate and to share information. For example, concepts  108  may share all or part of their respective computation results with other concepts  108  over these channels  110 . In this case, information propagated using the channels  110  may be weighted or associated with a probability such that joining concepts  108  may use these weights or probabilities when determining how further results derived from incoming information should be propagated. 
     In one embodiment, channels  110  possess properties  111   1 - 111   n  (hereinafter referred to as “properties  111 ”) that configure the relationship expressed between connected concepts. For example, a property “Mandatory” associated with a channel that connects a first concept and a second concept may express that the first concept cannot be valid if the second concept is not satisfied. Thus, for instance, if the first concept represents “MovieListing”, and the second concept represents “GeographicalArea”, the “Mandatory” property expresses that a user cannot request a MovieListing (e.g., “Find action movies starring actor X”) without also supplying a geographical location (“e.g., near San Francisco”). 
     Each concept  108  comprises a template including at least one rule  112   1 - 112   n  (hereinafter collectively referred to as “rules  112 ”) or rule set  114  (comprising multiple rules  112 ). Each rule  112  comprises a condition and an associated action. If an input fact matches a rule&#39;s condition, the associated action is triggered. In one embodiment, conditions may be evaluated as Java Script Boolean expressions (e.g., comprising slots), while associated actions are executed as Java Script programs. In further embodiment, rules  112  are prioritized within each concept  108  so that the rules are processed in an order of descending priority. Prioritization of rules allows for applications including activity recognition and action validation to share a common ontology structure. For example, a decision tree may be traversed in one direction for action execution, and then traversed in the opposite direction for action validation. 
     Among other advantages, active ontologies such as those described above can enable a simplified, efficient development scheme for intelligent automated assistants. Such a framework can be implemented to tie together multiple different programming applications, including applications capable of processing multimodal input, generating adaptable presentations, producing natural language dialogues, performing reactive planning, performing activity recognition and/or scheduling, and brokering, registering and coordinating distributed services, in a unified, visual framework. Thus, a single software developer may quickly and easily build an intelligent automated assistant using such a unified, developer-friendly development scheme. 
     As described in further detail with respect to  FIG.  3   , an active ontology such as the active ontology  100  may be built to process facts in order to achieve any one of a number of goals, including activity recognition (e.g., natural language processing, human activity recognition, etc.), reactive activity execution (e.g., pseudo-reactive planning, rule-based planning, etc.) and management of temporal constraints (e.g., for activity execution times), among others. A plurality of individual active ontologies may be built to execute a plurality of different goals, and these individual active ontologies may be integrated into a single processing unit such as an intelligent automated assistant. As described further below, these active ontologies may be managed in a manner that is significantly simpler than the traditional development process for intelligent automated assistants. 
       FIG.  2    is a schematic diagram illustrating one embodiment of an exemplary active ontology  200  that is configured as an assistant for reminding a user to take medicine after meals (e.g., configured for activity and/or time recognition). As illustrated, the active ontology  200  comprises at least one target task  202  (e.g., “take medicine after breakfast”) and a plurality of concepts  204   1 - 204   n . (hereinafter collectively referred to as “concepts  204 ”) representing potential user actions related to the target task  202  and to the modeled domain in which these actions are likely to occur (e.g., the user&#39;s home). The concepts  204  are communicatively coupled to other concepts  204  via a plurality of channels  210   1 - 210   n  (hereinafter collectively referred to as “channels  210 ”). 
     The active ontology  200  is also in communication with a plurality of sensors  206   1 - 206   n  (hereinafter collectively referred to as “sensors  206 ”) that are distributed throughout the modeled domain and are adapted to convey observed facts relating to the user&#39;s behavior (e.g., “the user is in the kitchen”; “the user is in the dining room”; “the user is in the bathroom”; etc.) to the higher-level concepts  204  for appropriate processing. Concepts  204  also communicate processing results (or at least partial processing results) to other concepts for further processing (e.g., if concept  204   8  verifies that the user is in the kitchen, concept  204   8  may convey this information to concept  204   3 , which is configured to determine whether the user is getting food). In addition, a clock  208  may also be in communication with the concepts  204  to enable the management of temporal constraints (e.g., the user is expected to sit at the table for approximately twenty-five minutes while eating breakfast; the user should take his/her medicine approximately thirty minutes after finishing breakfast; etc.). 
     Thus, if the target task  202  is not observed as being executed (e.g., the active ontology  200  does not determine that the user has taken the medicine after a meal or within a predefined period of time after finishing a meal), one or more of the processing elements (concepts  204 ) may generate a reminder to prompt the user to perform the target task  202 . Although the active ontology  200  illustrated in  FIG.  2    is configured to manage activity and time recognition tasks, those skilled in the art will appreciate that similarly constructed active ontologies may be built for natural language recognition, task automation and other useful tasks. In general, the same structure or rule or set of rules can be used both to observe and understand the surrounding world and to act on such observations. 
       FIG.  3    is a flow diagram illustrating one embodiment of a method  300  for processing facts in accordance with an active ontology (e.g., configured in a manner similar to the active ontology  100  of  FIG.  1   ) according to the present invention. The method  300  may be executed at, for example, an intelligent automated assistant that interacts with a user in accordance with an active ontology. 
     The method  300  is initialized at step  302  and proceeds to step  304 , where the method  300  collects at least one fact from the operating environment. In one embodiment, facts may be received via sensors (e.g., cameras, motion sensors, environmental and/or physical monitoring devices, etc.) distributed through the operating environment or from other multimodal user interfaces (e.g., a natural language interface, a graphical user interface, etc.). In one embodiment, collected facts are “tagged” upon receipt to identify an associated context and evaluation pass of the method  300 . 
     In step  306 , the method  300  processes the collected facts in accordance with at least one concept rule to identify valid conditions. In one embodiment, only facts that are associated (or “tagged”) with the current execution pass are processed in step  306 . In one embodiment, concepts are processed in accordance with a predefined order of priority, and rules for each concept are likewise processed in accordance with a predefined order of priority. Thus, step  306  begins by processing relevant facts against the highest priority rule of the highest priority concept, and then processes the facts against the second-highest priority rule of the highest priority concept, and so on until all rules of the highest priority concept have been processed. Step  306  then moves on to the next-highest priority concept and proceeds in a likewise manner. 
     In one embodiment, processing of facts in accordance with step  306  also includes the sharing of information among concepts (e.g., via channels as described above). The processing priority order described above ensures that higher priority concepts are evaluated first while lower priority concepts are still executable within a common execution pass of the method  300 . 
     In step  308 , the method  300  executes the corresponding action for each valid condition that is identified in accordance with  306 . The method  300  then proceeds to step  310  and waits for the next execution pass. In one embodiment, the execution passes of the method  300  are predefined such that the method  300  automatically executes at a given pace (e.g., once every x seconds). At the next execution pass, the method  300  returns to step  304  and proceeds as described above to evaluate a new set of facts. 
       FIG.  4    is a schematic diagram illustrating one embodiment of an open standard-based system  400  for developing and managing an intelligent system using active ontologics. In one embodiment, the system  400  includes an editor  402 , a console  404  and a server  406 . In one embodiment, the editor  402  and the console  404  communicate with the server  506  using SOAP-based communications. 
     The server  406  is adapted to store a plurality of individual active ontologies  408   1 - 408   n  (hereinafter collectively referred to as “active ontologies  408 ”) and to run selected (deployed) active ontologies  408  in an application domain. In one embodiment, the server  406  also stores observed facts for processing in accordance with an active ontology such as one of the stored or deployed active ontologies  408 . Specifically, these facts may be used to stimulate currently deployed active ontologies  408 . In one embodiment, the server  406  is a stand-alone application (e.g., a Java stand-alone application). 
     The editor  402  is interfaced to the server  406  and is adapted to retrieve active ontolgies  408  from the server  406  for editing. Specifically, the editor  402  is adapted to edit the concepts and/or rules embodied in the active ontologies  408  and to deploy or remove active ontolgies  408  from use. To this end, the editor  402  comprises a plurality of editing tools  410 , for example embodied in a graphical user interface such as a task bar. In further embodiments, the editor  402  is also adapted to receive feedback directly from deployed active ontologies  408 . In one embodiment, the editor  402  is a stand-alone application (e.g., a Java (Swing) stand-alone application). 
     The console  404  is also interfaced to the server  406  and is adapted to graphically model the application domain as an active ontology (e.g., by enabling the dragging and dropping and linking of objects and tasks) and to construct user queries (e.g., service requests) to be posed to the server  406 , for example using a graphical user interface. Active ontologies constructed using the console  404  are stored at the server  406 . The console  404  is further adapted to receive and display query results (e.g., specific stored active ontologies  408 ) sent by the server  406 . In further embodiments, the console  404  is also adapted to stimulate deployed active ontologies  408  by sending new observed facts to the server  406 . In one embodiment, the console  404  is a stand-alone application (e.g., a Java (Swing) stand-alone application). 
       FIG.  5    is a schematic diagram illustrating one embodiment of a framework  500  for dynamically registering and coordinating distributed services using active ontologies. Specifically, the framework  500  uses active ontologies to register remote services  508   1 - 508   n  (e.g., information services such as those for hotels, movies, etc., hereinafter collectively referred to as “services  508 ”) from a service-oriented architecture  502  (e.g., Web Services). These services  508  may then be implemented by an intelligent automated assistant, “on the fly”, to accomplish one or more tasks. Service-oriented architectures such as Web Services allow the standardized integration of web-based applications using the extensible markup language (XML), simple object access protocol (SOAP), web service definition language (WSDL) and universal description, discovery and integration (UDDI) open standards over an Internet Protocol (IP) backbone. 
     Thus, the framework  500  includes at least one active ontology  504  interfaced to a service-oriented architecture  502  via a service broker  506 . As described above, the active ontology  504  graphically models a relevant domain (e.g., the objects in the relevant environment and the associated tasks to be accomplished) using a plurality of concepts or active processing elements  510   1 - 510   n  (used, for example, for service registration, language recognition, activity and/or time recognition, task automation, etc.) that communicate through message passing. 
     The service broker  506  is adapted to communicate asynchronously with the remote service-oriented architecture  502  and with the active processing elements  510  to select specific services  508  (e.g., based on meta-data about the services  508 , including, for example, the service&#39;s call patterns or quality across various dimensions such as time, cost and completeness), to delegate tasks, queries and updates, and to monitor requests for services. 
     A service  508  registers with the active ontology  504  by specifying the active processing elements  510  that the service  508  can or cannot accept. Thus, the active ontology  504  is used to filter service requests to the appropriate services  508 . 
     The framework  500  enables the rapid development of a system having capabilities similar to an open agent architecture framework (e.g., the framework  500  can dynamically, broker, register and coordinate distributed services to resolve user queries and perform requested tasks). Moreover, the framework  500  enables better follow-ups and dialogues with users, flexible definition of timing constraints and task execution and recognition, and can adapt a user interface to the level of a specific user. 
       FIG.  6    is a high level block diagram of the present intelligent system building method that is implemented using a general purpose computing device  600 . In one embodiment, a general purpose computing device  600  comprises a processor  602 , a memory  604 , an intelligent system building module  605  and various input/output (I/O) devices  606  such as a display, a keyboard, a mouse, a modem, and the like. In one embodiment, at least one I/O device is a storage device (e.g., a disk drive, an optical disk drive, a floppy disk drive). It should be understood that the intelligent system building module  605  can be implemented as a physical device or subsystem that is coupled to a processor through a communication channel. 
     Alternatively, the intelligent system building module  605  can be represented by one or more software applications (or even a combination of software and hardware, e.g., using Application Specific Integrated Circuits (ASIC)), where the software is loaded from a storage medium (e.g., I/O devices  606 ) and operated by the processor  602  in the memory  604  of the general purpose computing device  600 . Thus, in one embodiment, the intelligent system building module  605  for building intelligent systems described herein with reference to the preceding Figures can be stored on a computer readable medium or carrier (e.g., RAM, magnetic or optical drive or diskette, and the like). 
     Thus, the present invention represents a significant advancement in the field of intelligent systems. The methods and apparatuses of the present invention enable intelligent systems, such as intelligent automated assistants, to be quickly and efficiently developed an “average” programmer (e.g., programmers who are not extensively versed in all necessary fields of technology). Capabilities including natural language interpretation and dialog, multimodal input, adaptable presentation (output), reactive planning, activity recognition/scheduling and coordination of distributed services, among others, can be integrated using a single, visual, unified framework. The manner in which software is built can thus be greatly simplified for many industries, including those which develop or rely on intelligent systems. 
     While foregoing is directed to the preferred embodiment of the present invention, other and further embodiments of the invention may be devised without departing from the basic scope thereof, and the scope thereof is determined by the claims that follow.

Metadata:
Filing Date: 20190409
Publication Date: 20240312
Grant Date: 20240312
Priority Date: 20050908
Inventors: CHEYER, ADAM
GUZZONI, DIDIER
Assignee: APPLE INC
CPC Classifications: [{"code": "G06N5/022", "inventive": true, "first": true, "tree": "[]"}, {"code": "G06F9/54", "inventive": true, "first": false, "tree": "[]"}, {"code": "G06F9/542", "inventive": true, "first": false, "tree": "[]"}, {"code": "G06N5/02", "inventive": true, "first": false, "tree": "[]"}, {"code": "G06N7/01", "inventive": true, "first": false, "tree": "[]"}, {"code": "G09B21/00", "inventive": true, "first": false, "tree": "[]"}, {"code": "G09B23/28", "inventive": true, "first": false, "tree": "[]"}, {"code": "G09B21/00", "inventive": true, "first": true, "tree": "[]"}, {"code": "G06N5/022", "inventive": true, "first": true, "tree": "[]"}, {"code": "G09B23/28", "inventive": true, "first": false, "tree": "[]"}, {"code": "G06F9/54", "inventive": true, "first": false, "tree": "[]"}, {"code": "G06F9/542", "inventive": true, "first": false, "tree": "[]"}, {"code": "G09B21/00", "inventive": true, "first": false, "tree": "[]"}, {"code": "G09B23/28", "inventive": true, "first": false, "tree": "[]"}, {"code": "G06N7/01", "inventive": true, "first": false, "tree": "[]"}, {"code": "G06N5/02", "inventive": true, "first": false, "tree": "[]"}]
Family ID: 37997762