Rethinking Multimodal Entity and Relation Extraction from a Translation Point of View (2023)

Abstract: We revisit the multimodal entity and relation extraction from a translation point of view. Special attention is paid on the misalignment issue in text-image datasets which may mislead the learning. We are motivated by the fact that the cross-modal misalignment is a similar problem of cross-lingual divergence issue in machine translation. The problem can then be transformed and existing solutions can be borrowed by treating a text and its paired image as the translation to each other. We implement a multimodal back-translation using diffusion-based generative models for pseudo-paralleled pairs and a divergence estimator by constructing a high-resource corpora as a bridge for low-resource learners. Fine-grained confidence scores are generated to indicate both types and degrees of alignments with which better representations are obtained. The method has been validated in the experiments by outperforming 14 state-of-the-art methods in both entity and relation extraction tasks. The source code is available at https://github.com/thecharm/TMR .

## introduction multimodal language understanding has received intensive attention recently for its advantage of mining semantics by collaborating the cross-modal inference @xcite . examples include methods for multimodal name entity recognition (mner) @xcite and multimodal relation extraction (mre) @xcite . both benefit from the collaborative reasoning based on the alignment of textual and visual content. however, statistics on commonly adopted text-image relation benchmarks (e.g., trc @xcite and twitter100k @xcite ) shows that the misalignment rate between images and texts is as high as 60%. noise introduced by the misalignment can mislead the learning and degrade the performance of resulting models. taylor swift sets new amas record, urges people to vote via @reuterstv. as shown in fig. 1 , the misalignment can be categorized into partial and irrelevant alignment. in case of incomplete alignment, textual entities (e.g., nato) might be mismatched to the visual evidence (e.g., person) which results in incorrect labels (e.g., per). this further leads to underline relations between entities (e.g., <trump, president of, usa), <usa, member of, @xmath0. in case of irrelevant alignment, the textual entities might be randomly matched to visual evidence (e.g., misc) resulting in dirty data for inference. while the misalignment with the ambiguity/distraction it brings to the learning has long been noticed, it has been rarely studied and addressed @xcite . the challenge is that it is nearly impossible to know the degree of misalignment prior to the inference. otherwise, the inference may has already been done. in this paper, we conduct a pilot study to address this problem. the motivation is that the misalignment of cross-modal pairs is a similar problem to the divergence of cross-lingual machine translations @xcite . the problem can thus be transformed by treating the text-image pairs in mner/mre as translations to each other. the divergence problem is more widely studied and existing solutions such as back-translation @xcite ) can be borrowed. while this sounds appealing, it introduces new challenges as follows. modality gap: the cross-lingual divergence is defined in a monomodal setting. the divergence can be measured explicitly by using features such as difference of sentence lengths, ratio of aligned words, and number of unaligned contiguous sequences @xcite . however, those features are not available in a cross-modal setting. we address it in an implicitly way in which disalignment of cross-lingual words (e.g., textual words and visual patches) is indicated by the divergence of their representations in the embedded space. parallelism: the detection/assessment of crosslingual divergence relies on large-scale parallel corpora, in which the sentences are aligned into wordlevel. the alignment is symmetric which makes high quality back-translation possible. however, in the cross-modal setting, mner/mre benchmark datasets are with a small scale due to the high cost of name entities labeling. the datasets are not well paralleled and there is no word-level alignment. we address those problems by taking advantage of the latest development of diffusion-based generative models @xcite . those models are trained on large-scale and better paralleled datasets, with which the back-translation can be conducted in a generate-to-translate way, in a sense that, for each text sentence, we can generate an image as its visual language "translation". visual grounding @xcite can then be employed to make the alignment into word-level. more details will be given in section 3.3. ## conclusion we have revisited the misalignment issue in multimodal benchmarks. by borrowing the ideas from translation methods, we have implemented multimodal versions of back-translation and highresource bridging, which provide a multi-view to the misalignment between modalities. the method has been validated in the experiments and outperforms 14 sota methods. changmeng zheng, junhao feng, ze fu, yi cai, qing li, and tao wang. 2021a. multimodal relation extraction with efficient graph alignment. in proceedings of the 29th acm international conference on multimedia, pages 5298-5306. changmeng zheng, zhiwei wu, junhao feng, ze fu, and yi cai. 2021b. mnre: a challenge multimodal dataset for neural relation extraction with visual evidence in social media posts. in 2021 ieee international conference on multimedia and expo (icme), pages 1-6. ieee. changmeng zheng, zhiwei wu, tao wang, yi cai, and qing li. 2020. object-aware multimodal named entity recognition in social media posts with adversarial learning. ieee transactions on multimedia, 23:2520-2532. this section details the training procedures and hyperparameters for named entity recognition. we use the bert-base-uncased model from hugging face library . we follow umgf @xcite to revise some wrong annotations in the twitter-2015 dataset. we utilize pytorch to conduct experiments with 1 nvidia 3090 gpus. all optimizations are performed with the adamw optimizer with a linear warmup of learning rate 3e-5 over the first 10% of gradient updates to a maximum value, then linear decay over the remainder of the training. and weight decay on all non-bias parameters is set to 0.01. we set the number of grounding regions and image patches to 3 and 49, respectively. max length of noun phrases is set to 4 and max length for sentence is set to 80.