PV 2 TEA : Patching Visual Modality to Textual-Established Information Extraction (2023)

Abstract: Information extraction, e.g., attribute value extraction, has been extensively studied and formulated based only on text. However, many attributes can benefit from image-based extraction, like color, shape, pattern, among others. The visual modality has long been underutilized, mainly due to multimodal annotation difficulty. In this paper, we aim to patch the visual modality to the textual-established attribute in- formation extractor. The cross-modality integration faces several unique challenges: (C1) images and textual descriptions are loosely paired intra-sample and inter-samples; (C2) images usually contain rich backgrounds that can mislead the prediction; (C3) weakly supervised labels from textual-established ex- tractors are biased for multimodal training. We present PV2TEA, an encoder-decoder architecture equipped with three bias reduction schemes: (S1) Augmented label-smoothed contrast to improve the cross-modality alignment for loosely-paired image and text; (S2) Attention-pruning that adaptively distinguishes the visual foreground; (S3) Two-level neighborhood regularization that mitigates the label textual bias via reliability estimation. Empirical results on real-world e-Commerce datasets1 demonstrate up to 11.74% absolute (20.97% relatively) F1 increase over unimodal baselines.

## introduction information extraction, e.g., attribute value extraction, aims to extract structured knowledge triples, i.e., @xcite , from the unstructured information. as shown in figure 1 , the inputs include text descriptions and images (optional) along with the queried attribute, and the output is the extracted value. in practice, textual description has played as the main or only input in mainstream approaches for automatic attribute value extraction @xcite @xcite @xcite . such models perform well when the prediction targets are inferrable from the text. as the datasets evolve, interest in incorporating visual modality naturally arises, especially for image-driven attributes, e.g., color, pattern, item shape. such extraction tasks rely heavily on visual information to obtain the correct attribute values. the complementary information contained in the images can improve recall in cases where the target values are not mentioned in the texts. in the meantime, the cross-modality information can help with ambiguous cases and improve precision. however, extending a single-modality task to multi-modality can be very challenging, especially due to the lack of annotations in the new modality. performing accurate labeling based on multiple modalities requires the annotator to refer to multiple information resources, leading to a high cost of human labor. although there are some initial explorations on multimodal attribute value extraction @xcite @xcite , all of them are fully supervised and overlook the resource-constrained setting of building a multimodal attribute extraction framework based on the previous textual-established models. in this paper, we aim to patch the visual modality to attribute value extraction by leveraging textual-based models for weak supervision, thus reducing the manual labeling effort. challenges. several unique challenges exist in visual modality patching: c1. images and their textual descriptions are usually loosely aligned in two aspects: from the intra-sample aspect, they are usually weakly related considering the rich characteristics, making it difficult to ground the language fragments to the corresponding image regions; from the inter-samples aspect, it is commonly observed that the text description of one sample may also partially match the image of another. as illustrated in figure 1 , the textual description of the mattress product is fragmented and can also correspond to other images in the training data. therefore, traditional training objectives for multimodal learning such as binary matching @xcite or contrastive loss @xcite that only treat the text and image of the same sample as positive pairs may not be appropriate. c2. bias can be brought by the visual input from the noisy contextual background. the images usually not only contain the interested object itself but also demonstrate a complex background scene. although the backgrounds are helpful for scene understanding, they may also introduce spurious correlation in a fine-grained task such as attribute value extraction, which leads to imprecise prediction @xcite . c3. bias also exists in language perspective regarding the biased weak labels from textual-based models. as illustrated in figure 1 , the color label of mattress is misled by 'green tea infused' in the text. these noisy labels can be more catastrophic for a multimodal model due to their incorrect grounding in images. directly training the model with these biased labels can lead to gaps between the stronger language modality and the weaker vision modality @xcite . solutions. we propose pv2tea, a sequence-tosequence backbone composed of three modules: visual encoding, cross-modality fusion and grounding, and attribute value generation, each with a bias-reduction scheme dedicated to the above challenges: s1. to better integrate the loosely-aligned texts and images, we design an augmented labelsmoothed contrast schema for cross-modality fusion and grounding, which considers both the intrasample weak correlation and the inter-sample potential alignment, encouraging knowledge transfer from the strong textual modality to the weak visual one. s2. during the visual encoding, we equip pv2tea with an attention-pruning mechanism that adaptively distinguishes the distracting background and attends to the most relevant regions given the entire input image, aiming to improve precision in the fine-grained task of attribute extraction. s3. to mitigate the bias from textual-biased weak labels, a two-level neighborhood regularization based on visual features and previous predictions, is designed to emphasize trustworthy training samples while mitigating the influence of textual-biased labels. in this way, the model learns to generate more balanced results rather than being dominated by one modality of information. in summary, the main contributions of pv2tea are three-fold: • we propose pv2tea, an encoder-decoder framework effectively patching up visual modality to textual-established attribute value extraction. ## pv2tea we present the backbone architecture and three bias reduction designs of pv2tea, shown in figure 3 . the backbone is formulated based on visual question answering (vqa) composed of three modules: ## conclusion in this work, we propose pv2tea, a bias-mitigated visual modality patching-up model for multimodal information extraction. specifically, we take attribution value extraction as an example for illustration. results on our released sourceaware benchmarks demonstrate remarkable improvements: the augmented label-smoothed contrast promotes a more accurate and complete alignment for loosely related images and texts; the visual attention pruning improves precision by masking out task-irrelevant regions; and the neighborhoodregularized sample weight adjustment reduces textual bias by lowering the influence of noisy samples. we anticipate the investigated challenges and proposed solutions will inspire future scenarios where the task is first established on the text and then expanded to multiple modalities.