Generating Instructions at Different Levels of Abstraction (2020)

Abstract: When generating technical instructions, it is often convenient to describe complex objects in the world at different levels of abstraction. A novice user might need an object explained piece by piece, while for an expert, talking about the complex object (e.g. a wall or railing) directly may be more succinct and efficient. We show how to generate building instructions at different levels of abstraction in Minecraft. We introduce the use of hierarchical planning to this end, a method from AI planning which can capture the structure of complex objects neatly. A crowdsourcing evaluation shows that the choice of abstraction level matters to users, and that an abstraction strategy which balances low-level and high-level object descriptions compares favorably to ones which don’t.

## introduction technical instructions in complex environments can often be stated at different levels of abstraction. for instance, a natural language generation (nlg) system for tech support might instruct a human instruction follower (if) to either plug "the broadband cable into the broadband filter" or "the thin white cable with grey ends into the small white box" @xcite . depending on how much the if knows about the domain, the first, high-level instruction may be difficult to understand, or the second, detailed instruction may be imprecise and annoyingly verbose. an effective instruction generation system will thus adapt the level of abstraction to the user. in this paper, we investigate the generation of instructions at different levels of abstraction in the context of the computer game "minecraft". minecraft offers a virtual 3d environment in which the player can mine materials, craft items, and construct complex objects, such as buildings and machines. the user constructs these objects bit by bit from atomic blocks, and it is always possible to generate natural-language instructions which describe the placement of each individual block. however, it can be more effective to generate more high-level instructions. in the bridge-building example shown in fig. 1 , it is probably better to simply say "build a railing on the other side" instead of explaining where to place the seven individual blocks -provided the if knows what a railing looks like. minecraft is the best-selling video game of all time (200 million users), which means that there is a large pool of potential experimental subjects for evaluating nlg systems. minecraft has been used previously as a platform for experimentation in ai and in particular for nlg @xcite . we present an instruction giving (ig) system which guides the user in constructing complex objects in minecraft, such as houses and bridges. the system consists of two parts: a hierarchical planning system based on hierarchical task networks (htn) @xcite , which computes a structured instruction plan for how to to explain the construction; and a chart-based generation system which inputs individual plan steps and generates the actual instruction sentences @xcite . planning systems generate plans based on expressive declarative models, making this approach easily applicable to a wide range of domains, and giving it the power to deal with large degrees of freedom in instruction generation. in particular, we leverage the hierarchical planning system to obtain three different strategies for describing complex objects as illustrated in fig. 1 : low-level, always instructing block-by-block (sentences (a) and ( d )); high-level, always instructing to build the next complex subobject (sentences (c) and ( e )); and a teaching strategy which first explains how to construct a complex subobject, and then uses high-level descriptions for that object in subsequent instructions (sentences (b) and (e)). we realize these strategies through designing the planner's action-cost function. the strategies constitute a first step towards a fully generic ig system which adapts its description of complex objects to the user's knowledge. the planner's cost function could also be used to incorporate additional criteria, e. g. that the generated sentences should be easy to understand in context. we evaluate our ig system by crowdsourcing, using the open-source mc-saar-instruct platform for ig experiments in the minecraft domain @xcite . each user receives instructions for building either a house or a bridge using one of the three abstraction strategies sketched above. the results show significant differences in completion times and user satisfaction across the three strategies. in the bridge scenario, the teaching strategy outperforms the high-level strategy in completion time and the low-level strategy in user satisfaction. in the house scenario, the high-level strategy outperforms the others in completion time for the walls, illustrating the importance of modeling user knowledge when choosing the level of abstraction. plan of the paper. after reviewing some related work (section 2), we overview the architecture of our ig system (section 3). we briefly introduce htn planning, and how we use it to define hierarchical minecraft construction planning models (section 4). we then explain how to get from construction planning models to instruction planning models, and how to generate instruction plans at different levels of abstraction (section 5). section 6 describes our evaluation; section 7 concludes. foot_0 2 related work generating natural-language instructions grounded in the mechanics of a real or virtual world is a wellestablished type of nlg task. for instance, the give challenge @xcite required nlg systems to guide human users through a maze while referring to locations and objects. we follow the give challenge in situating the task in a virtual environment and using crowdsourced task-based evaluation, but the minecraft world is much more complex than the give world, and contains complex objects. rookhuiszen et al. ( 2009 ) describe an ig system for the give challenge which dynamically adapts the level of detail of navigation instructions. they use a simple heuristic to switch between abstraction levels. @xcite generate referring expressions at different levels of abstraction in an electronics repair scenario, and adapt to novice vs. expert users. they assume a finite set of possible descriptions for each object, and do not exploit the internal structure of complex objects. the use of ai planning for elaborating and organizing the things that need to be said in an nlg system (i. e. discourse planning) dates back into the early days of nlg @xcite . @xcite use non-hierarchical planning to compute communicative plans. they use planning operators which encode communicative actions, and allow them to have effects both on the communicative state and on the state of the world; we disentangle these effects in our hierarchical planning model. @xcite use a hierarchical planner to generate technical instructions at different levels of abstraction, but their system can only utter sentences which were stored with the planning operators as canned text. their evaluation does not show that users prefer their system over a baseline, illustrating the difficulty of generating instructions at the right abstraction level. the minecraft domain has been used extensively for various tasks in ai @xcite , including planning @xcite and natural language understanding @xcite . regarding nlg specifically, @xcite trained a neural model to generate building instructions in minecraft; in the absence of symbolic domain knowledge, their model struggles to generate correct instructions. @xcite show how individual instructions can be generated in minecraft. their focus is on generating indefinite referring expressions to objects which do not exist yet because the user is supposed to build them. here we do not address how to generate the individual utterances, but how to determine the semantic content of these utterances. ## nlg system architecture our overall ig system consists of two separate modules: an instruction planner and a sentence generator. the role of the instruction planner is to compute an instruction plan, i. e. a sequence of instruction actions. an instruction action is an abstract semantic representation of a sentence; the sentence generator then translates it into a natural-language utterance, such as "place a block on top of the yellow block", "build a floor from the black block to the yellow block", or "now i will teach you how to build a railing". the technical focus of this paper is on the instruction planner. given a state of the minecraft world and a specification for the complex object the if is supposed to build (e. g. a bridge or a house), the instruction planner will compute a sequence of instruction actions as explained above. one technical contribution of this paper is that the instruction actions can be at different levels of abstraction; for example, the instruction plan can simply say "build a railing on the other side" in the situation of fig. 1 , or it can explain how to build the railing block by block. to ensure the correctness of the instruction plan, i. e. to ensure that the instructions, followed correctly, actually do result in the intended complex object, the instruction planner performs construction planning as part of its planning process: it internally refines its high-level actions into block-by-block plans and checks that these work. the sentence generator takes instruction actions as input and produces natural-language utterances. we use the chart generation system of @xcite to generate sentences. this system generates sentences while simultaneously generating definite and indefinite referring expressions (res); definite res are used to refer to objects which already exist in the minecraft environment, and indefinites are generated to refer to objects which do not exist yet because the if is supposed to build them. thus for instance, the sentence generator might translate the instruction action ins-railing(0,1,4,5,south) to "build a railing from the top of the blue block to the top of the red block" or "build a railing on the other side of the floor", depending on the state of the dialogue and the minecraft world. ## hierarchical construction planning as indicated, we distinguish between construction planning, which determines how a complex object can be built block-by-block; and instruction planning, which determines how, and in particular at which level of abstraction, the building of a complex object can be explained to the if. instruction planning encompasses construction planning as a sub-problem. we employ hierarchical planning for both tasks, with hierarchies of complex subobjects and the associated building activities. these are not required for construction planning per se (which can always proceed on a block-by-block basis). but they speed up the construction planning process, and they are key to instruction planning as proposed here. previous work on planning in minecraft @xcite has considered construction planning only, and has not considered hierarchical planning. we now introduce our construction planning models, which we will extend to instruction planning models in section 5. ## from construction planning to instruction planning having discussed how to compute a construction plan with an htn planner, we will now explain how to compute instruction plans. the actions in an instruction plan represent communicative actions, in which the ig system sends a sentence to the if. these actions can describe the intended construction steps at different levels of abstraction. we capture this in the htn model by defining a task for instructing the if to build each complex subobject, say a railing. the planner can then choose to either achieve this task with a single primitive instruction action (which might send "build a railing" to the if), or to further decompose it into smaller tasks, which will instruct the if to build two blocks and connect them with a row. one key challenge is that while the instruction actions represent natural-language instructions, they still need to be grounded in activities in the minecraft world: we must guarantee that, if the if follows the instructions correctly, then the correct complex object results. to this end, we reason about the construction and its explanation simultaneously. for every instruction action, the plan also contains the corresponding block-by-block construction actions, thus validating the instruction plan. ## conclusion we have shown how to generate building instructions in minecraft at different levels of abstraction through htn planning. our evaluation shows that the level of abstraction matters to human users, and that our "teaching" strategy strikes an effective balance of high-level and low-level instructions. we view our system as the first step towards a more general framework. we used a very basic model of the user's knowledge; in future work, we will train statistical models on user interactions to estimate what a user knows initially and what they learn during the experiment. furthermore, we used hand-designed action costs. perhaps the most interesting avenue of future research is to let the sentence generator specify the action costs, based on how easy it is to express a given instruction action in language in the current world and dialogue state. this will allow the sentence generator to influence the search of the discourse planner, offering a new method for closing the "generation gap" @xcite .