Cross-modal Attention Congruence Regularization for Vision-Language Relation Alignment (2023)

Abstract: Despite recent progress towards scaling up multimodal vision-language models, these models are still known to struggle on compositional generalization benchmarks such as Winoground. We find that a critical component lacking from current vision-language models is relation-level alignment: the ability to match directional semantic relations in text (e.g., ‘mug in grass’) with spatial relationships in the image (e.g., the position of the mug relative to the grass). To tackle this problem, we show that relation alignment can be enforced by encouraging the language attention from ‘mug’ to ‘grass’ (capturing the semantic relation ‘in’) to match the visual attention from the mug to the grass (capturing the corresponding physical relation). Tokens and their corresponding objects are softly identified using a weighted mean of cross-modal attention. We prove that this notion of soft cross-modal equivalence is equivalent to enforcing congruence between vision and language attention matrices under a ‘change of basis’ provided by the cross-modal attention matrix. Intuitively, our approach projects visual attention into the language attention space to calculate its divergence from the actual language attention, and vice versa. We apply our Cross-modal Attention Congruence Regularization (CACR) loss to fine-tune UNITER and improve its Winoground Group score by 5.75 points.

## introduction compositionality is the ability to combine meanings of constituents according to structured rules. recent work shows that vision-language models (vlms) fail to construct compositional representations and generally ignore syntactic & structural information @xcite @xcite . winoground @xcite is a vision-language compositionality task that tests a vlm's ability to match syntactic permutations of text with their visual interpreta- figure 1 : global alignment (ga) only aligns the entire image with the corresponding caption. entity alignment (ea) extracts entities from the image and caption for finer-grained alignment. relation alignment (ra) cross-modally aligns the intra-modal relations between entities in both the image and the text. we show ra is vital to improve compositional performance. since models must determine whether the compositional structure of an image matches that of the caption, it's important for the model to learn to cross-modally align intra-modal relations. that is, if the relation from 'mug' to 'grass' is 'in-ness', the model should recognize when the equivalent physical relation holds between a mug and grass in the image, and representationally align these relations such that an image-text matching head may more easily determine whether the relations are cross-modally equivalent. in simpler terms, the compositional structure of input for each modality should be represented such that they can be crossmodally matched even for difficult examples like winoground. unfortunately, there has been less highly influential work on relation alignment between vision & language, and thrush et al. @xcite did not benchmark any such models. in this work, we begin exploration of these relation alignment approaches by tentatively grouping them into 3 categories: in this work, we propose a structural training approach for relation alignment that uses the crossmodal attention matrix as a change of basis 1 to the opposite modality, which we then compare to the original modality to calculate a divergence loss, effectively measuring cross-modal congruence between intra-modal attentions. we show how our approach, cross-modal attention congruence regularization (cacr), generalizes previous structural training work on crossmodal attention regularization @xcite ) by taking into account all possible entity alignments and computationally simplifying relation alignment. the cacr regularization term can easily be dropped into most transformerbased vision-language model objectives with no 1 not defined in a strict linear algebraic sense added data and minimal computational overhead, to encourage relation alignment during training. ## cross-modal attention congruence regularization to attempt a solution to this question, we begin by noting that attention activations encode some degree of relational information. attention values in transformers may be seen as an informational gating mechanism that implicitly encode how representations are composed @xcite . for example, past work in language has shown how syntax trees may be extracted @xcite from attention across layers and used to guide attention @xcite for improved compositionality. in this section, we extend this intuition to the multimodal domain by proposing to use the cross-modal attentions, which as a change-of-basis matrix encode a transformation from one modality's compositional structure to the opposite modality's, to encourage cross-modal relation alignment. ## analysis why does cacr's soft cross-modal equivalence approach outperform hard cross-modal equivalence? ## conclusion in this work, we identified that a key factor holding back models from vision-language representational compositionality is cross we then identified a potential key weakness in iais, its hard argmax assumption, and developed a soft cross-modal equivalence approach to address it. having linear algebraically simplified this approach, we arrived at cacr, an auxiliary loss that encourages cross-modal congruence of intra-modal attention. cacr improves on iais' performance on winoground, and even outperforms a uniter model nearly twice as large. as computational scaling becomes more widespread, it's necessary to develop compositional inductive biases that do not require complex annotated data or exotic model architectures. our work illustrates how taking advantage of the transformer's own attentional structure can improve the quality of fine-grained vision-language representations, opening the avenue for large scale approaches to visually-grounded compositionality. not free from this flaw, but we see our work as teaching vlms to learn better structured representations rather than memorizing spurious correlations in data. we remain far from solving the vision-language compositionality problem, so biases must continue to be actively mitigated.