Semantic Skill Grounding for Embodied Instruction-Following in Cross-Domain Environments (2024)

Abstract: In embodied instruction-following (EIF), the integration of pretrained language models (LMs) as task planners emerges as a significant branch, where tasks are planned at the skill level by prompting LMs with pretrained skills and user instructions. However, grounding these pretrained skills in different domains remains challenging due to their intricate entanglement with the domain-specific knowledge. To address this challenge, we present a semantic skill grounding (SemGro) framework that leverages the hierarchical nature of semantic skills. SemGro recognizes the broad spectrum of these skills, ranging from short-horizon low-semantic skills that are universally applicable across domains to long-horizon rich-semantic skills that are highly specialized and tailored for particular domains. The framework employs an iterative skill decomposition approach, starting from the higher levels of semantic skill hierarchy and then moving downwards, so as to ground each planned skill to an executable level within the target domain. To do so, we use the reasoning capabilities of LMs for composing and decomposing semantic skills, as well as their multi-modal extension for assessing the skill feasibility in the target domain. Our experiments in the VirtualHome benchmark show the efficacy of SemGro in 300 cross-domain EIF scenarios.

## introduction in the area of embodied instruction-following (eif), the quest to develop systems capable of understanding user instructions and achieving tasks in a physical environment has been a pivotal aspiration for general artificial intelligence. recent advancements in language models (lms) have demonstrated their extensive and impressive application in eif, where lms are employed as task planners, drawing on their commonsense knowledge of the * equally contributed to this work † honguk woo is the corresponding author physical world @xcite @xcite @xcite . the process of lm-based task planning in eif unfolds via a two-stage pipeline: (i) interpretable skills (i.e., semantic skills) are acquired, and (ii) given a user instruction, the lm is prompted with the instruction and the skill semantics to identify appropriate skills to perform. to ensure that such planned skill can be executed within the domain context the agent is placed in, several works have focused on incorporating environmental observations into task planning @xcite @xcite @xcite . yet, eif in cross-domain environments has not been fully explored. in cross-domain settings, the complexity goes beyond logical task planning by lms; it necessitates the adaptation of pretrained skills to align with the specific characteristics and limitations of the target physical environment. this adaptation challenge stems from the inherent domain specificity of semantic skill acquisition, calling for a strategy that disentangles these learned behaviors from their original domain contexts and reapplies them in new domains. in response to this challenge, we present a semantic skill grounding framework semgro. we observe a trade-off between semantic depth and domain-agnostic executability within the skill hierarchy. high-level skills, such as "prepare a meal", embody a richer semantic content and long-horizon behavior compared to low-level skills, such as "grab a cup". however, these high-level skills tend to be effective only within specific domains, as their execution heavily depends on a multitude of environmental factors, unlike their low-level counterparts which can be universally applicable across domains. semgro capitalizes on this insight by employing an iterative skill grounding scheme, designed to progressively decompose pretrained skills to fit in the cross-domain eif environment as well as to fully utilize the reasoning strengths of lms. to be specific, the iterative mechanism of semgro involves two procedures: (i) task planning with skills that align with a given instruction (in section 3.2) and (ii) assessing the executability of planned skills (in section 3.3). in semgro, a task planner employs in-context learning of lms to interpret a given instruction and generate a series of semantic skills for the instruction through iterative skill grounding and decomposition. concurrently, a skill critic employs the multi-modal extension of lms to evaluate the applicability and execution potential of these skills within target domains. when the critic deems a planned skill as unsuitable for execution, the planner responds by generating an alternative, detailed instruction. this instruction is intended to be carried out through a sequence of lower-level skills that align with the previously planned skill. through the collaborative interaction between the task planner and the skill critic, semgro navigates the semantic hierarchy of skills to establish the best trade-off between the logical relevance of skills to a given instruction and the skill feasibility in cross-domain environments. using the virtualhome benchmark @xcite to evaluate eif agents @xcite @xcite , we demonstrate the advantages of semgro for cross-domain eif scenarios. for instance, semgro outperforms the state-of-the-art task planning framework llm-planner @xcite by 23.97% in task completion performance. our contributions are three-fold: (i) we present a novel framework semgro to address the issue of cross-domain skill grounding in eif. (ii) we devise an iterative skill grounding algorithm, leveraging the reasoning capabilities of lms and their multimodal extension. (iii) we conduct an intensive evaluation of semgro in 300 cross-domain scenarios of the virtualhome benchmark, complemented by ablation studies to explore the efficacy of semgro. ## approaches we utilize the intrinsic semantic hierarchy of skills, a structure derived from their compositional nature; high-level semantic skills are established by chain-ing short-horizon low-level skills. high-level skills, characterized by rich semantics, closely align with the abstract nature of user instructions. this alignment in abstraction levels facilitates the task planning process, providing the lm with skills that are directly beneficial for interpreting and accomplishing the instructions. however, the execution of these long-horizon high-level skills is contingent upon the precise alignment of various environmental and domain-specific conditions. this precludes their adaptability in cross-domain settings. in contrast, low-level skills exhibit a robust capacity for execution that is not as tightly bound to specific domain contexts, offering greater adaptability across diverse domains. nonetheless, the sheer volume of low-level skills and their limited semantic depths pose additional challenges for task planning. the core of semgro lies in the hierarchical task planning scheme via iterative skill grounding. the framework initiates task planning with high-level rich-semantic skills, and then progressively decomposes each of these planned high-level skills into a series of low-level skills executable within the domain of the target environment. this can be seen as finding an optimal intermediate semantic representation for skills pertinent to a given instruction, exploring a balance between their logical relevance and actual feasibility in the target environment. such a balance ensures that the planned skills not only match the conceptual demands of the given instruction but are also effective within the limitations and resources of the target environment. ## conclusion in this paper, we presented a semantic skill grounding framework semgro, which supports the adaptation of a broad range of skills in cross-domain environments. semgro harnesses the reasoning capabilities of lms not only to compose semantic skills for establishing a hierarchical skill database but also to decompose semantic skills into executable low-level skills. this novel approach uniquely explores the trade-off between semantic richness and domain-agnostic executability of semantic skills. we hope our work provides valuable insights into the development and application of real-world eif agents.