Pento- DIAR ef: A Diagnostic Dataset for Learning the Incremental Algorithm for Referring Expression Generation from Examples (2023)

Abstract: NLP tasks are typically defined extensionally through datasets containing example instantiations (e.g., pairs of image _i_ and text _t_), but motivated intensionally through capabilities invoked in verbal descriptions of the task (e.g., “_t_ is a description of _i_, for which the content of _i_ needs to be recognised and understood”).We present Pento-DIARef, a diagnostic dataset in a visual domain of puzzle pieces where referring expressions are generated by a well-known symbolic algorithm (the “Incremental Algorithm”),which itself is motivated by appeal to a hypothesised capability (eliminating distractors through application of Gricean maxims). Our question then is whether the extensional description (the dataset) is sufficient for a neural model to pick up the underlying regularity and exhibit this capability given the simple task definition of producing expressions from visual inputs. We find that a model supported by a vision detection step and a targeted data generation scheme achieves an almost perfect BLEU@1 score and sentence accuracy, whereas simpler baselines do not.

## introduction being able to effectively and efficiently refer to objects is a central component of human language competence @xcite . the computational task of referring expression generation (reg) goes beyond the production of image descriptions (as in image captioning), in that it is a uniquely identifying description that needs to be produced, given a specific situation. in the formulation of @xcite , the reg task involves reasoning over all relevant objects in a scene, in order to determine what would make a description uniquely identifying. additionally, maxims of efficiency @xcite predict that it is a minimal natural language expression that should be preferred. the incremental algorithm (ia) @xcite is a well-known classic symbolic algorithm that tries to realise these desiderata. for example given a reference target and various distractors as in figure 1 (an example of the domain chosen in this paper (pentomino, @xcite ; @xcite ; @xcite )), then the incremental algorithm (ia) produces "take the x", achieving the desired uniquely identifying reference by mentioning only the shape and not also color and position. can such a reference strategy be learned by neural generation models from visual inputs alone? this is a question that is typically not systematically challenged in language generation from images @xcite @xcite @xcite , as in natural scenes (such as in the refcoco dataset @xcite ), it has been shown that descriptions can be produced diagnostic dataset task input condition output generalizability testing wu et al. (2021) nav. symb. state text (command) text (actions) words, phrases, action length liu et al. (2019) rec image text (re) bbox color-shapes pento-diaref (ours) reg image bbox text (ia-re) color-shapes, positions, ia-res in this work we study how a intensionally defined task (in the distinction of @xcite ) for which a verbal and theoretically motivated description is given (through a symbolic algorithm) can be learned from its extensional exemplification. our contention is that the use of synthetic data @xcite @xcite @xcite offers the opportunity to strengthen the link, insofar as guarantees can be given on the exemplification relation. more specifically, we choose the incremental algorithm @xcite for the data generation process, which itself comes with a motivation through recourse to underlying fundamental conversational capabilities (appeal to gricean maxims, grice (1967)). our contributions are as follows: 1 compositional reasoning. lake and baroni (2018) introduced a systematic benchmark to test 1 the source code and datasets are made publicly available at https://github.com/clp-research/pento-diaref . the generalization capabilities of recurrent neural networks through the use of compositional splits and found that these models fail "spectacularly". @xcite extended the task of mapping text commands to actions (navigation) by conditioning the learner additionally on a symbolic world state. later @xcite provided a curated dataset along with new dimensions for generalizability testing. our work follows the idea of generalization testing through compositional datasets in language and vision settings where training examples are composed in such a way that the models are exposed towards all property values of objects, but not to all the possible combinations of them, so that they can be tested on unseen combinations. in contrast to their work we use images instead of symbolic world states as the input. ## pento-diaref task and dataset we present a diagnostic dataset of ia references in a pentomino domain (pento-diaref) that ties extensional and intensional definitions more closely together, insofar as the latter is the generative process creating the former @xcite . in this chapter we describe the task ( §3.1) and how it is tied to the incremental algorithm ( §3.2) via the generation process ( §3.3) and present our compositional splits ( §3.6) for generalization testing. ## learning the incremental algorithm our goal in producing the collection of scenes was to ensure that a model f must indeed be based on features of the x i that we care about (that is, which figure in the desired capability), namely the need to indeed compare the perceivable target piece and in the following, we present our neural models ( §5.1) and the conducted experiments ( §5.3) to test if neural language generators are indeed able to acquire such a "programatic" capability by the simple task definition of producing expressions from visual inputs. the generation models f will be trained on the basis of (x i , y i ) pairs only. we train two common network architectures for this task of which one is an lstm-based approach to reg proposed by @xcite for natural scenes and the other is a transformer @xcite . in addition, we propose a variant for processing the inputs along with a simple classifier-based baseline. 5.1 models lstm. @xcite , who present a model of reg in natural scene images, embed the scenes and the referent within them with a pre-trained vgg @xcite . we follow their procedure but use the 512 dimensional embeddings after global average pooling of a resnet-34 @xcite and fine-tune all of its layers because our images look very different to the ones from the pre-training on imagenet @xcite . we cut out the target piece using the bounding box information. then the piece snippet is dilated with 5 context pixels and up-scaled to the size of the board image. we additionally randomly shift the snippet by 0-5% of the pixels in either direction horizontally or vertically (fill-color is white). the target piece and board image embeddings are then concatenated together with five location and size features of the target. the resulting 1029-dimensional feature vector is fed to an lstm at each time step to condition the language production (using greedy decoding). we reduce the word embedding dims to 512 because our vocabulary is very small and apply an adam optimizer (kingma and ba, 2015). transformer. for comparison with @xcite we resize, augment and encode the target piece and visual board with a resnet-34 in the same way as described before. then the image embeddings are fed into the transformer @xcite individually (not concatenated) as "visual words" together with the target piece attributes embedding as shown in figure 6 to compute an intermediate representation of the inputs altogether. this "memory" embedding is then fed into the decoder to generate the re using masked selfattention as in other machine translation tasks. for the variable length expressions we use a padding symbol and ignore prediction at padded positions during loss computation. we reduced the original capacity of the model to avoid overfitting and applied a learning rate scheduling strategy as described by @xcite , using an adamw optimizer @xcite . transformer+vse. we assume that the transformer should be particularly capable of generating ia-like expressions because self-attention might allow it to learn the required piece-wise comparison operation. the self-attention mechanism has already been proven powerful for other image-related tasks @xcite @xcite . therefore we follow @xcite and implement a visual sequence encoding (vse) mechanism. for this we cut out each piece on the board to produce a sequence of piece snippets as shown in figure 7 and project the visual features onto the models' input dimensions fj and add region embeddings pj to them that contain the snippets size and location information: to let a model distinguish between target and distractor "words" in the input sequence we add a type embedding tj , similar to word embeddings, and normalize. furthermore, we have a variable amount of pieces on the board (between 4 and 10), but a transformer model assumes a fixed-size input sequence (per batch, during training). thus we indicate "padding" pieces with a padding index in the sequence as implemented in pytorch @xcite and use images with all zeros for them. classifier+vse. the representations of the vse might already capture enough information to perform the task. therefore we test this assumption by training a simple linear sentence classifier just on top of the concatenated embeddings. the classifier has to predict the correct sentence out of the 1,689 possible ones. this framing is similar to that often used in visual question answering @xcite , where the possible answers are framed as classes in a classification task. ## conclusion and future work in future work we want to evaluate more models on our diagnostic dataset to find potential weaknesses. an interesting question is whether a plm (brown et al., 2020) might have picked up such gricean constraints and would be able to recognise their desirability from being prompted with only a few examples. we also plan to explore to what extent our best model is applicable to more realistic settings following sim-to-real approaches @xcite .