Draw Me a Flower: Processing and Grounding Abstraction in Natural Language (2022)

Abstract: Abstraction is a core tenet of human cognition and communication. When composing natural language instructions, humans naturally evoke abstraction to convey complex procedures in an efficient and concise way. Yet, interpreting and grounding abstraction expressed in NL has not yet been systematically studied in NLP, with no accepted benchmarks specifically eliciting abstraction in NL. In this work, we set the foundation for a systematic study of processing and grounding abstraction in NLP. First, we deliver a novel abstraction elicitation method and present Hexagons, a 2D instruction-following game. Using Hexagons we collected over 4k naturally occurring visually-grounded instructions rich with diverse types of abstractions. From these data, we derive an instruction-to-execution task and assess different types of neural models. Our results show that contemporary models and modeling practices are substantially inferior to human performance, and that model performance is inversely correlated with the level of abstraction, showing less satisfying performance on higher levels of abstraction. These findings are consistent across models and setups, confirming that abstraction is a challenging phenomenon deserving further attention and study in NLP/AI research.

## introduction as human-computer interaction in natural language (nl) becomes more and more pervasive (e.g., via smart devices and chatbots), a cognitive phenomenon known as abstraction, which is prevalent in human communication and cognition, is taking a central role in the way users communicate their intentions and needs to artificial agents. when communicating in nl with a human or an artificial agent, a human may issue a request for a single action such as ''send an email'' or ''set my alarm''. however, when engaging with more complex tasks that require multiple actions, humans often evoke abstraction in order to communicate their intentions in an economic-yet-precise way. examples for evoking abstraction when issuing a complex request that consists of multiple actions may be: ''schedule a group meeting every other wednesday until the end of the year, unless there are holidays.'' for an autonomous car, an envisioned request might be ''circle the block looking for a shady parking spot, park at the first spot you see. try this four times, and one more time allowing for non-shady parking. in no luck, try the adjacent block.'' in fact, even the individual request ''send an email'' is in itself an abstraction over a sequence of multiple individual actions such as: ''open your inbox, click new mail, select a recipient,'' and so forth. abstraction is defined by @xcite as ''[letting] one object stand for many. it is used to capture essential properties common to a set of objects while hiding irrelevant distinctions among them''. in the calendar example, a single utterance references multiple meetings in multiple days. likewise, in the autonomous car example, the speaker evokes some sort of control structure in order to iterate a process several times. abstraction so construed is both critical and pervasive in nl. referring to multiple instances at once may be done by means of the shape they form, via a process that iterates them, or via an action/ condition applied to select or manipulate them. to illustrate this, figure 1 showcases an example of a natural language procedure for drawing a target image (the bottom-right image) on an empty board. the instructions start with the construction of an object, a red flower, covering six tiles. then, the instructor prescribes multiple flower patterns via repeat actions that realize a nested 'loop'. finally, the instructor states the color of the flower centers via a 'condition' (green for red, blue for yellow). this example goes to show both the essence and power of abstraction in nl; here, merely four nl utterances suffice to prescribe a complex image over a 180-tiles board. despite its importance and widespread use, detecting and grounding abstraction in nl has not yet been systematically studied in nlp. previous studies on grounding instructions target linguistic phenomena as anaphora and ellipsis @xcite , spatial relations @xcite , and referring expressions @xcite but do not specifically elicit abstraction. in studies on navigation @xcite @xcite eliciting abstract statements is also sparse. instructions often refer to specifics of the environment rather than abstract phenomena. in this work we aim to add a new facet to the study of natural language understanding, that of interpreting abstraction. we set out to provide a foundation for systematically studying the phenomenon of processing and grounding diverse levels of abstraction found in naturally occurring nl utterances. achieving this goal is far from trivial. as is standard in nlp, we would first need to establish an appropriate dataset for studying this phenomenon. specifically, we'd like to collect naturally occurring data that manifest abstraction. but how can we purposefully request for the presence of abstraction in naturallyoccurring data? to overcome this challenge, we develop an abstraction elicitation protocol by adopting practices from stem education, specifically from computational thinking (ct) research @xcite . the idea, in a nutshell, is to develop visual stimuli that evoke, and thus cultivate (and elicit), higher-order thinking, which is then narrated in nl. we implement the proposed protocol in a novel hexagons game, a situated collaborative game where an instructor provides instructions that should be grounded and executed in a virtual world @xcite @xcite . in contrast to previous studies, we use practices from ct research to design visual triggers of abstraction. hence, on the one hand, we implicitly call for the presence of abstraction in the instructions, but on the other hand, we provide naturally elicited abstract instructions from workers not possessing formal knowledge of what abstraction is. using the hexagons game and the task stimuli, we collected over 4k human instructions manifesting a variety of formal abstractions (objects, control structures, and functions) expressed naturally and intuitively in nl and grounded on the hexagons board. to showcase how this data may be used for studying abstraction processing in nl, we derive an instruction-to-execution task, where the model needs to ground and execute nl instructions on the hexagons board. we propose a naïve rule-based baseline as well as two neural modeling alternatives-one based on classification, one on generation-and assess their performance on the elicited abstraction data. our experiments show that, while our models perform better than the naïve rule-based baseline, they are substantially inferior to human performance. moreover, we show that model performance is inversely correlated with the level of abstraction, that is, the models execute concrete instructions quite well, but perform poorly on higher-level abstractions. this holds across different models, setups, task conditions, amount and type of training data, and board contexts. we further observe that the instruction's history is another important factor in models' performance; the longer the history, the better the performance. the contribution of this paper is thus manifold. first, we bring to the fore of nlp research a critical aspect of human-computer communication, namely, the ability to detect, process, and ground abstraction in natural language. next, we devise a novel abstraction elicitation methodology and deliver the hexagons dataset as a novel benchmark to explore the automatic processing of different levels of abstraction. this dataset may also serve broader communities such as ai researchers, linguists, cognitive psychologists, and stem educators in the study of human processing of abstraction. finally, for the instruction-to-execution task we derive from the hexagons data, we show experimental evidence that unequivocally confirms that abstract instructions in nl are indeed more challenging for current systems to process, and we expose abstraction as an important and challenging dimension for further study in nlp. foot_0 ## the challenge: eliciting and processing abstraction in nl abstraction is a cognitive phenomenon related to diverse human activities such as learning, decision making, and behavior regulation (cf. @xcite . in the context of human-computer interaction, and in general in human problemsolving, abstraction is said to be one of a set of cognitive skills known as computational thinking (ct) skills, defined by @xcite as ''the thought processes involved in formulating problems and their solutions so that the solutions are represented in a form that can be effectively carried out by an information-processing agent.'' abstraction in this context refers to a process of information reduction @xcite , where multiple instances are conceived as arising from a single object, ''consisting of their shared properties while discarding irrelevant distinctions'' @xcite . abstraction is considered by many as the most important ct skill, allowing a human to think in terms of objects and concentrating on their essential features, while ignoring irrelevant details @xcite @xcite @xcite . thus, abstraction leads to a speaker's capacity for being more precise and less error-prone (dijkstra, accessed 1 may 2021 @xcite and to designing more concise, elegant and efficient solutions @xcite . to illustrate how humans may exhibit different levels of abstraction, consider the simple example in figure 2 , where a human is requested to describe a pattern on a 2d hexagons board. the first (top) nl procedure expresses low-level abstraction; it refers to each occurrence of a half-column as a unique event. this is a repetitive and lengthy procedure. in contrast, the second (bottom) nl procedure refers to all the occurrences of this halfcolumn at once (via 'repeat but alternate'), discarding distinctions related to, for example, tiles' positions and colors. the result of this abstraction is thus concise, clear and far more efficient. in a broader sense, in order to express abstraction speakers employ so-called abstraction mechanisms-such as objects, functions and control flow @xcite -and expertise in using them is considered an important part of humans' ct skills @xcite . such mechanisms are invoked in figure 1 . this kind of communication is not limited to the simple hexagons board used in figures 1 2 . it is relevant in countless many other domains as in the aforementioned calendar and car examples. due to the prevalence of abstraction in human communication, models would need to process varied levels of abstraction in order to correctly interpret nl instructions. but in order to successfully develop such models, we have to collect data that systematically reflect such phenomena, and to the best of our knowledge, this has not yet been done in nlp. the challenge of eliciting abstraction is genuine, as we aspire to elicit natural language that reflects authentic human communication of any speaker. but we cannot simply request crowdworkers to employ abstraction mechanisms as they are not familiar with these formal concepts. on the other hand, explicitly teaching them to employ abstraction undermines the naturalness of expression. so, how do we break out of this loop? ## the proposed methodology in this work we are interested in creating situations where humans express abstraction in nl spontaneously and naturally, towards learning models that can interpret such abstractions. to achieve this goal, we turn to the vast research in human learning and stem education, on cultivating (and thus eliciting) higher-order ct skills in humans @xcite . eliciting such higher-order thinking requires a careful task design, drawing on literature on the development of instruments that probe and assess humans' ct skills @xcite @xcite . our designed task stimuli are carefully crafted to evoke abstraction without explicitly requesting workers to do so. our elicitation methodology extends a recent trend in grounded semantic parsing, where players engage in a referential game (situated collaborative scenarios in terms of @xcite ) where an instructor provides instructions that should be grounded and executed in a (simulated) world @xcite @xcite . the remainder of this section elaborates on the virtual environment we devise, and the task stimuli we design for elicitation. ## data collection and curation for the data collection we employed englishspeaking workers from amazon mechanical turk (mturk) and adopted the methodology of controlled crowdsourcing @xcite to ensure a high quality corpus. specifically, the process includes four stages: pilot, recruitment, annotation, and consolidation. we collect drawing procedures for the task stimuli in a process that comprises of two steps: this two-phase process is intended to reveal faulty instructions and to ensure the quality and executability of the collected procedures, by making the instructor's intentions explicit (step 1), and by exposing disagreements with verifiers (step 2). pilot and recruitment we checked the flow, clarity, and feasibility of the data collection in a pilot study, followed by two separate rounds of recruiting instructors and verifiers. to recruit instructors, we screened workers by examining understanding and engagement in the instructor task. appropriate candidates had to complete three instructor tasks, that is, repeat the collection phase with three randomly selected images from our pool. in no stage did we formally teach workers what abstraction is. instead we engage the workers in several tasks and encourage them to write their instructions efficiently and to avoid tiresome repetitions of the primitive paint. out of the 34 candidates, we assembled a group of 28 instructors exhibiting diverse levels of abstraction, out of which 24 took an active role during the annotation procedure. we follow a separate process in recruiting verifiers using the verification phase, instructing candidates to be as accurate as possible while executing drawing procedures. out of 27 candidates, we recruited 16 workers exhibiting the most precise work. the groups of instructors and verifiers are disjoint, so drawing procedures are verified by workers other than those who generate them. annotation procedure the annotation procedure is based on a generation-validation cycle, which is similar to previous protocols for constructing large-scale corpora by untrained crowdworkers @xcite . specifically, based on images from our crafted stimuli, drawing procedures are first generated and verified by the instructors in the collection phase, and then each procedure is given to two additional verifiers, that work through the verification phase to check the understandability and executability of the procedures. the annotation process itself consisted of two rounds. in the first round we gave each image from our pool to three instructors in order to generate three different drawing procedures for each image. each of the generated procedures was verified as usual. in the second round, we presented instructors with the opportunity to draw new images on a blank hexagons board. the goal is to scale-up the extension of the image-pool with interesting compositions using crowdsourcing. indeed, the collected images in this round reflect similar rationale to our own set of images (e.g., figure 4(a-b )) yet demonstrate more complex interactions between structures and patterns (figure 4(c-d )), with both abstract and figurative images (figure 4(e-f )). this new pool of images then passed through the collection and verification phases as usual. consolidation having collected the raw dataset, we manually inspected all drawing procedures that had at least one disagreement between an instructor and each of the verifiers. then, we developed a protocol to (i) detect instructors' errors, (ii) classify the types of errors, and (iii) fix the instructor execution. the protocol was applied to the data by the two first authors. the reported agreement on error classification was 0.95 and 0.98 cohen's kappa for the first two tasks and 95% agreement for the last one. following this protocol, we detected cases where the instructor's execution is not properly aligned with the instruction, and manually corrected the execution to match the instruction. types of errors include: over-/under-execution where instructors executes more/less than the instructions require; miscounting of positions on the board; error propagation from previous steps, and others such as using wrong colors. all in all we inspected 1461 drawing steps out of which 20.8% were identified as having instructors' error and subsequently were manually corrected. this process results in data with fully-aligned instruction-execution pairs for each of the instructions in each of the drawing procedures. annotation costs we used amazon mechanical turk to recruit english-speaking workers for this study. participants in the data collection rounds were paid higher rates than in pilot and recruitment. the payments for collection and verification were @xmath00.50, respectively. ## the hexagons dataset our finalized dataset is the collection of all drawing procedures, composed of instructions and their aligned executions collected in both annotation rounds and some from the recruitment stage, after having passed our quality assurance and consolidation process. in total, we collected 620 drawing procedures yielding 4177 drawing steps, that is, 4177 instructions with aligned executions, circa 100k tokens. ## experiments the task given the hexagons dataset, we aim to devise models that interpret nl instructions and mimic an executor's role, in order to assess how standard pre-trained language models (plm) interpret these utterances. to this end, we define a computational task as follows. input configurations for each of the models, we experiment with several input settings that differ in the type and extent of context provided: (i) no-history. the input is only the instruction to be executed (current drawing step). (ii) 1-previous. the input contains the instruction to be executed (current step) and one previous instruction. (iii) full-history. the input contains the instruction to be executed and all previous instructions in that drawing procedure. (iv) oracle board-state. the input contains the instruction to be executed and the gold board-state that is obtained prior to the current drawing step. no previous instruction history is included. (v) predicted board-state. the input contains the instruction to be executed and the predicted board-state, predicted for all steps so far. no previous instruction history is included. (vi) full-history + oracle/predicted board-state, a combination of (iv) and (v) with full history (iii). ## results and analysis our ablated experiments on input configurations are designed to empirically assess the contribution of two kinds of contexts, textual and board-state contexts. we observe that textual context (previous instructions) is an important factor in model performance; the longer the context is, the better the model performs. performance is lowest when predicting executions on the hexagons board with only the current instruction as input (no-history). adding more context proves to be beneficial, with the full-history condition having the best realistic (non-oracle) performance. this result corroborates previous findings which show that models benefit from textual history @xcite . a different way of providing context for the execution of an instruction is via the state of previous executions on the board. here, we experiment with either providing an oracle board-state at each step, or iteratively feeding the predicted board-states from the previous step to the current step. while providing the oracle board-state improves performance upon the no-history condition, our results show that it is not as informative as including the full instruction history. a possible reason may be that textual instructions often refer back to previously introduced (or decomposed) objects, while board states do not explicitly name these decomposed concepts. adding both the oracle board-state and all previous 1350 instructions as input results in the best performance, however this is not a realistic setup. the more realistic context setting is that of a predicted board and all previous instructions, but it performs worse than only providing full history, due to error propagation in the predicted states. abstraction levels figures 6 7 present the models' performance by abstraction levels (section 5) across increasing train set sizes. the results on the full train set show that models' performance is inversely correlated with the abstraction-level of the instructions; models' performance on executions of concrete primitive-like instructions exceeds those of mid-to-high level of abstraction. this result is significant across models, metrics, and input configurations. comparing these results with baseline and skyline by the levels of abstraction ( the results on the gradually increasing train set size show that although in general all levels of abstraction benefit from larger train sets, still model performance on non-abstract instructions is consistently better than instructions exhibiting mid-to-high-level of abstraction, keeping a mean gap of circa 38 mean f1. noticeably, the increase for the highest abstraction level is very mild. especially for em. this hints that no substantial learning is happening at the highest abstraction level, and a different architecture or training regime, geared towards abstraction, is needed. we manually inspected executions of our models with respect to the levels of abstraction of the instruction. looking at the successful executions of high-level instructions, we observe that those instructions are mainly instances where no actions should be performed, for example, goal/result declarations, or instances where very common objects (e.g., flowers) are defined and drawn. more complex functions, such as repetitions with conditions, are harder for the models to interpret. an example of how executions differ between different abstraction levels is displayed in figure 8 . the model correctly executes the first instruction which contains no abstraction. the next instruction is of high abstraction including replication of the triangle object. the model does not manage to identify the spots where to attach the new triangles, or to generate appropriate triangles. and deberta), scores (f1 and em) and input configurations (full-history and no-history). for brevity, figures 6 7 show only the input configurations of full-history models. these findings are all consistent with the claim that abstract instructions pose a challenge for current nlp technology, orthogonally to data size and various other factors. ## related work studying abstraction in collaborative communication is related to previous studies on collaborative games that focus on how interlocutors generate referring expressions accepted and understood by both the speaker and hearer @xcite @xcite . in such situations a speaker attempts to generate the shortest refererring expression that will sufficiently communicate their intention. this phenomenon of minimizing speakers' effort is inline with grice's (1975) maxim of quantity, stating that speakers will give as much information as needed and not more. the settingsofcollaborativegamesareverycommon in creating datasets for grounded semantic parsing as navigation tasks @xcite @xcite @xcite @xcite , the 2-d/3-d blocks world @xcite @xcite , and other instruction-following scenarios @xcite . some of these studies observe abstraction as a phenomenon that indeed occurs in nl instructions (e.g., @xcite , implying that abstraction is a cross-domain and hence critical phenomenon for natural language understanding. however, eliciting naturally-occurring nl instructions that reflect a variety of abstraction levels in a systematic way is novel to the hexagons data. to confirm this, we inspected the 2-d blocks dataset @xcite , which most resembles our setting. sampling 594 instructions (5% of the train set), we found that almost all the instructions (96.5%) map to actions of shifting a single block to some location, with some spatial expressions (e.g., ''place box 17 three spaces above box 20''). such instructions are labeled ''no abstraction'' in our protocol (see section 5). notably, a fraction of the sample does express some low-level (2.7%) and mid-level (0.8%) abstraction. two other studies that are particularly related to our work are by @xcite and @xcite . in the voxelurn study @xcite , a community of users is interacting with a computerized agent to deliver constructions in a 3-d blocks world. the community gradually and collaboratively builds increasingly complex and more abstract language from a core programming language via a process called ''naturalization''. @xcite exhibits similar constructions but on an individual rather than a community effort. both studies indeed address abstraction, but from an opposite direction to ours; while these works assume a strict narrow and synthetic language and build abstractions bottom-up, our work aims to tackle the opposite direction, uncovering abstractions that are expressed in unrestricted informal nl and grounding them in an executable 'backend'. thus, these studies and ours exhibit orthogonal ways to address abstraction. ## conclusion we bring to the fore of nlp a novel and critical aspect of human-computer communication, namely, the ability to automatically detect, interpret, and ground abstraction in nl. we devise an abstraction elicitation methodology and deliver a novel benchmark, hexagons, manifesting the denotation of instructions rich and diverse in their levels of abstraction. our results on the instruction-to-execution task derived from these data show that the models' performance is significantly inversely correlated with the level of abstraction, and this holds across models, contexts, and data sizes. this work opens a manifold of directions for future research such as generating human-like abstractions or detecting the level of abstraction, as well as studying abstraction in adjacent fields as linguistics, cognitive science and nl programming.