Explicitly Representing Syntax Improves Sentence-to-Layout Prediction of Unexpected Situations (2024)

Abstract: Recognizing visual entities in a natural language sentence and arranging them in a 2D spatial layout require a compositional understanding of language and space. This task of layout prediction is valuable in text-to-image synthesis as it allows localized and controlled in-painting of the image. In this comparative study it is shown that we can predict layouts from language representations that implicitly or explicitly encode sentence syntax, if the sentences mention similar entity-relationships to the ones seen during training. To test compositional understanding, we collect a test set of grammatically correct sentences and layouts describing compositions of entities and relations that unlikely have been seen during training. Performance on this test set substantially drops, showing that current models rely on correlations in the training data and have difficulties in understanding the structure of the input sentences. We propose a novel structural loss function that better enforces the syntactic structure of the input sentence and show large performance gains in the task of 2D spatial layout prediction conditioned on text. The loss has the potential to be used in other generation tasks where a tree-like structure underlies the conditioning modality. Code, trained models, and the USCOCO evaluation set are available via Github.1

## introduction current neural networks and especially transformer architectures pretrained on large amounts of data build powerful representations of content. however, unlike humans, they fail when confronted with unexpected situations and content which is out of context @xcite . compositionality is considered a powerful tool in human cognition as it enables humans to understand and generate a potentially infinite number of novel situations by viewing the situation as a novel composition of familiar simpler parts @xcite @xcite . in this paper we hypothesize that representations that better encode the syntactical structure of a sentence-in our case a constituency tree-are less sensitive to a decline in performance when confronted with unexpected situations. we test this hypothesis with the task of 2d visual object layout prediction given a natural language input sentence (figure 2 gives an overview of the task and of our models). we collect a test set of grammatically correct sentences and layouts (visual ''imagined'' situations), called unexpected situations of common objects in context (uscoco) describing compositions of entities and relations that unlikely have been seen during training. most importantly, we propose a novel structural loss function that better retains the syntactic structure of the sentence in its representation by enforcing the alignment between the syntax tree embeddings and the output of the downstream task, in our case the visual embeddings. this loss function is evaluated both with models that explicitly integrate syntax (i.e., linearized constituent trees using brackets and tags as tokens) and with models that implicitly encode syntax (i.e., language models trained with a transformer architecture). models that explicitly integrate syntax show large performance gains in the task of 2d spatial layout prediction conditioned on text when using the proposed structural loss. the task of layout prediction is valuable in text-to-image synthesis as it allows localized and controlled in-painting of the image. apart from measuring the understanding of natural language by the machine @xcite , text-to-image synthesis is popular because of the large application potential (e.g., when creating games and virtual worlds). current generative diffusion models create a naturalistic image from a description in natural language (e.g., dall-e 2, @xcite , but lack local control in the image triggered by the role a word has in the interpretation of the sentence @xcite , and fail to adhere to specified relations between objects @xcite . additionally, if you change the description, for instance, by changing an object name or its attribute, a new image is generated from scratch, instead of a locally changed version of the current image, which is a concern in research (e.g., @xcite . @xcite and @xcite show that using scene layouts as additional input greatly improves the spatial reasoning of text-to-image models, substantiating the importance of the text-to-layout synthesis task. we restrict the visual scene to the spatial 2d arrangements of objects mentioned in the natural language sentence, taking into account the size and positions of the objects (figure 1 ). from these layouts images can be generated that accurately adhere to the spatial restrictions encoded by the layouts @xcite , but this is not within the scope of this paper. we emphasize that while we argue that good layout predictions are valuable, the question this study aims to answer is not how to build the best possible layout predictors, but whether and if so how explicitly representing syntax can improve such predictors, especially with respect to their robustness to unseen and unexpected inputs. the contributions of our research are the following: (1) we introduce a new test set called uscoco for the task of 2d visual layout prediction based on a natural language sentence containing unusual interactions between known objects; (2) we compare multiple sentence encoding models that implicitly and explicitly integrate syntactical structure in the neural encoding, and evaluate them with the downstream task of layout prediction; (3) we introduce a novel parallel decoder for layout prediction based on transformers; (4) we propose a novel contrastive structural loss that enforces the encoding of syntax in the representation of a visual scene and show that it increases generalization to unexpected compositions; and (5) we conduct a comprehensive set of experiments, using quantitative metrics, human evaluation and probing to evaluate the encoding of structure, more specifically syntax. ## limitations & future work one limitation of layout decoding with explicitly structured language models is the reliance upon a syntax parsing model to obtain the constituency trees for input captions. while syntax parsing models have shown very high performance (the parser of @xcite obtains an f1 score of 95.13 on the penn treebank @xcite , which contains longer and more syntactically complex sentences than typical coco captions), grammatical errors in the used prompts might result in incorrect parses and hence in worse layout generations, compared to language models without explicit syntax (that do not need a parser). we leave an investigation into this phenomenon for further research. however, we do note an increased performance of layout generation even with only trivial right-branching trees over implicit syntax models (visible in furthermore, while we show that explicitly modeling syntax improves layout prediction for absurd situations, this out-of-distribution generation task still remains difficult even for the best layout predictor models: there is a 35%-37% drop in f1 score and 17%-26% drop in f1 dpw on uscoco compared to the in-domain test set. ## conclusion we evaluated models that implicitly and explicitly capture the syntax of a sentence and assessed how well they retain this syntax in the representations when performing the downstream task of layout prediction of objects on a 2d canvas. to test compositional understanding, we collected a test set of grammatically correct sentences and layouts describing compositions of entities and relations that unlikely have been seen during training. we introduced a novel parallel decoder for layout prediction based on a transformer architecture, but most importantly we proposed a novel contrastive structural loss that enforces the encoding of syntax structure in the representation of a visual scene and show that it increases generalization to unexpected compositions resulting in large performance gains in the task of 2d spatial layout prediction conditioned on text. the loss has the potential to be used in other generation tasks that condition on structured input, which could be investigated in future work. our research is a step forward in retaining structured knowledge in neural models.