pwc-archive/papers-with-abstracts huggingface dataset:
Bi-ViT: Pushing the Limit of Vision Transformer Quantization
Authors: Yanjing Li, Sheng Xu, Mingbao Lin, Xianbin Cao, Chuanjian Liu, Xiao Sun, Baochang Zhang
Date: 2023-05-21 00:00:00
Tasks: Binarization, Quantization

Vision transformers (ViTs) quantization offers a promising prospect to facilitate deploying large pre-trained networks on resource-limited devices. Fully-binarized ViTs (Bi-ViT) that pushes the quantization of ViTs to its limit remain largely unexplored and a very challenging task yet, due to their unacceptable performance. Through extensive empirical analyses, we identify the severe drop in ViT binarization is caused by attention distortion in self-attention, which technically stems from the gradient vanishing and ranking disorder. To address these issues, we first introduce a learnable scaling factor to reactivate the vanished gradients and illustrate its effectiveness through theoretical and experimental analyses. We then propose a ranking-aware distillation method to rectify the disordered ranking in a teacher-student framework. Bi-ViT achieves significant improvements over popular DeiT and Swin backbones in terms of Top-1 accuracy and FLOPs. For example, with DeiT-Tiny and Swin-Tiny, our method significantly outperforms baselines by 22.1% and 21.4% respectively, while 61.5x and 56.1x theoretical acceleration in terms of FLOPs compared with real-valued counterparts on ImageNet.
YOLOv3 with Spatial Pyramid Pooling for Object Detection with Unmanned Aerial Vehicles
Authors: Wahyu Pebrianto, Panca Mudjirahardjo, Sholeh Hadi Pramono, Rahmadwati, Raden Arief Setyawan
Date: 2023-05-21 00:00:00
Tasks: Object, object-detection, Object Detection

Object detection with Unmanned Aerial Vehicles (UAVs) has attracted much attention in the research field of computer vision. However, not easy to accurately detect objects with data obtained from UAVs, which capture images from very high altitudes, making the image dominated by small object sizes, that difficult to detect. Motivated by that challenge, we aim to improve the performance of the one-stage detector YOLOv3 by adding a Spatial Pyramid Pooling (SPP) layer on the end of the backbone darknet-53 to obtain more efficient feature extraction process in object detection tasks with UAVs. We also conducted an evaluation study on different versions of YOLOv3 methods. Includes YOLOv3 with SPP, YOLOv3, and YOLOv3-tiny, which we analyzed with the VisDrone2019-Det dataset. Here we show that YOLOv3 with SPP can get results mAP 0.6% higher than YOLOv3 and 26.6% than YOLOv3-Tiny at 640x640 input scale and is even able to maintain accuracy at different input image scales than other versions of the YOLOv3 method. Those results prove that the addition of SPP layers to YOLOv3 can be an efficient solution for improving the performance of the object detection method with data obtained from UAVs.
Coronary Artery Semantic Labeling using Edge Attention Graph Matching Network
Authors: Chen Zhao, Zhihui Xu, Guang-Uei Hung, Weihua Zhou
Date: 2023-05-21 00:00:00
Tasks: Graph Matching, Semantic Segmentation

Coronary artery disease (CAD) is one of the primary causes leading deaths worldwide. The presence of atherosclerotic lesions in coronary arteries is the underlying pathophysiological basis of CAD, and accurate extraction of individual arterial branches using invasive coronary angiography (ICA) is crucial for stenosis detection and CAD diagnosis. We propose an innovative approach called the Edge Attention Graph Matching Network (EAGMN) for coronary artery semantic labeling. By converting the coronary artery semantic segmentation task into a graph node similarity comparison task, identifying the node-to-node correspondence would assign semantic labels for each arterial branch. More specifically, The EAGMN utilizes the association graph constructed from the two individual graphs as input. Experimental results indicate the EAGMN achieved a weighted accuracy of 0.8653, a weighted precision of 0.8656, a weighted recall of 0.8653 and a weighted F1-score of 0.8643. Furthermore, we employ ZORRO to provide interpretability and explainability of the graph matching for artery semantic labeling. These findings highlight the potential of the EAGMN for accurate and efficient coronary artery semantic labeling using ICAs. By leveraging the inherent characteristics of ICAs and incorporating graph matching techniques, our proposed model provides a promising solution for improving CAD diagnosis and treatment
Low-Earth Satellite Orbit Determination Using Deep Convolutional Networks with Satellite Imagery
Authors: Rohit Khorana
Date: 2023-05-20 00:00:00
Tasks: Satellite Orbit Determination

Given the critical roles that satellites play in national defense, public safety, and worldwide communications, finding ways to determine satellite trajectories is a crucially important task for improved space situational awareness. However, it is increasingly common for satellites to lose connection to the ground stations with which they communicate due to signal interruptions from the Earth's ionosphere and magnetosphere, among other interferences. In this work, we propose utilizing a computer vision based approach that relies on images of the Earth taken by the satellite in real-time to predict its orbit upon losing contact with ground stations. In contrast with other works, we train neural networks on an image-based dataset and show that the neural networks outperform the de facto standard in orbit determination (the Kalman filter) in the scenario where the satellite has lost connection with its ground-based station. Moreover, our approach does not require $\textit{a priori}$ knowledge of the satellite's state and it takes into account the external factors influencing the satellite's motion using images taken in real-time.
Cross2StrA: Unpaired Cross-lingual Image Captioning with Cross-lingual Cross-modal Structure-pivoted Alignment
Authors: Shengqiong Wu, Hao Fei, Wei Ji, Tat-Seng Chua
Date: 2023-05-20 00:00:00
Tasks: Image Captioning, Translation

Unpaired cross-lingual image captioning has long suffered from irrelevancy and disfluency issues, due to the inconsistencies of the semantic scene and syntax attributes during transfer. In this work, we propose to address the above problems by incorporating the scene graph (SG) structures and the syntactic constituency (SC) trees. Our captioner contains the semantic structure-guided image-to-pivot captioning and the syntactic structure-guided pivot-to-target translation, two of which are joined via pivot language. We then take the SG and SC structures as pivoting, performing cross-modal semantic structure alignment and cross-lingual syntactic structure alignment learning. We further introduce cross-lingual&cross-modal back-translation training to fully align the captioning and translation stages. Experiments on English-Chinese transfers show that our model shows great superiority in improving captioning relevancy and fluency.
A request for clarity over the End of Sequence token in the Self-Critical Sequence Training
Authors: Jia Cheng Hu, Roberto Cavicchioli, Alessandro Capotondi
Date: 2023-05-20 00:00:00
Tasks: Image Captioning, Sentence

The Image Captioning research field is currently compromised by the lack of transparency and awareness over the End-of-Sequence token (<Eos>) in the Self-Critical Sequence Training. If the <Eos> token is omitted, a model can boost its performance up to +4.1 CIDEr-D using trivial sentence fragments. While this phenomenon poses an obstacle to a fair evaluation and comparison of established works, people involved in new projects are given the arduous choice between lower scores and unsatisfactory descriptions due to the competitive nature of the research. This work proposes to solve the problem by spreading awareness of the issue itself. In particular, we invite future works to share a simple and informative signature with the help of a library called SacreEOS. Code available at \emph{\href{https://github.com/jchenghu/sacreeos}{https://github.com/jchenghu/sacreeos}}
Boosting Human-Object Interaction Detection with Text-to-Image Diffusion Model
Authors: Jie Yang, Bingliang Li, Fengyu Yang, Ailing Zeng, Lei Zhang, Ruimao Zhang
Date: 2023-05-20 00:00:00
Tasks: Diversity, Human-Object Interaction Detection, Triplet, Zero-Shot Human-Object Interaction Detection

This paper investigates the problem of the current HOI detection methods and introduces DiffHOI, a novel HOI detection scheme grounded on a pre-trained text-image diffusion model, which enhances the detector's performance via improved data diversity and HOI representation. We demonstrate that the internal representation space of a frozen text-to-image diffusion model is highly relevant to verb concepts and their corresponding context. Accordingly, we propose an adapter-style tuning method to extract the various semantic associated representation from a frozen diffusion model and CLIP model to enhance the human and object representations from the pre-trained detector, further reducing the ambiguity in interaction prediction. Moreover, to fill in the gaps of HOI datasets, we propose SynHOI, a class-balance, large-scale, and high-diversity synthetic dataset containing over 140K HOI images with fully triplet annotations. It is built using an automatic and scalable pipeline designed to scale up the generation of diverse and high-precision HOI-annotated data. SynHOI could effectively relieve the long-tail issue in existing datasets and facilitate learning interaction representations. Extensive experiments demonstrate that DiffHOI significantly outperforms the state-of-the-art in regular detection (i.e., 41.50 mAP) and zero-shot detection. Furthermore, SynHOI can improve the performance of model-agnostic and backbone-agnostic HOI detection, particularly exhibiting an outstanding 11.55% mAP improvement in rare classes.
DAC: Detector-Agnostic Spatial Covariances for Deep Local Features
Authors: Javier Tirado-Garín, Frederik Warburg, Javier Civera
Date: 2023-05-20 00:00:00

Current deep visual local feature detectors do not model the spatial uncertainty of detected features, producing suboptimal results in downstream applications. In this work, we propose two post-hoc covariance estimates that can be plugged into any pretrained deep feature detector: a simple, isotropic covariance estimate that uses the predicted score at a given pixel location, and a full covariance estimate via the local structure tensor of the learned score maps. Both methods are easy to implement and can be applied to any deep feature detector. We show that these covariances are directly related to errors in feature matching, leading to improvements in downstream tasks, including solving the perspective-n-point problem and motion-only bundle adjustment. Code is available at https://github.com/javrtg/DAC
Brain encoding models based on multimodal transformers can transfer across language and vision
Date: 2023-05-20 00:00:00
Proceeding: NeurIPS 2023 11

Encoding models have been used to assess how the human brain represents concepts in language and vision. While language and vision rely on similar concept representations, current encoding models are typically trained and tested on brain responses to each modality in isolation. Recent advances in multimodal pretraining have produced transformers that can extract aligned representations of concepts in language and vision. In this work, we used representations from multimodal transformers to train encoding models that can transfer across fMRI responses to stories and movies. We found that encoding models trained on brain responses to one modality can successfully predict brain responses to the other modality, particularly in cortical regions that represent conceptual meaning. Further analysis of these encoding models revealed shared semantic dimensions that underlie concept representations in language and vision. Comparing encoding models trained using representations from multimodal and unimodal transformers, we found that multimodal transformers learn more aligned representations of concepts in language and vision. Our results demonstrate how multimodal transformers can provide insights into the brain's capacity for multimodal processing.
Comparative Analysis of Deep Learning Models for Brand Logo Classification in Real-World Scenarios
Authors: Qimao Yang, Huili Chen, Qiwei Dong
Date: 2023-05-20 00:00:00
Tasks: Classification, Deep Learning

This report presents a comprehensive study on deep learning models for brand logo classification in real-world scenarios. The dataset contains 3,717 labeled images of logos from ten prominent brands. Two types of models, Convolutional Neural Networks (CNN) and Vision Transformer (ViT), were evaluated for their performance. The ViT model, DaViT small, achieved the highest accuracy of 99.60%, while the DenseNet29 achieved the fastest inference speed of 366.62 FPS. The findings suggest that the DaViT model is a suitable choice for offline applications due to its superior accuracy. This study demonstrates the practical application of deep learning in brand logo classification tasks.
Searching a Compact Architecture for Robust Multi-Exposure Image Fusion
Authors: Zhu Liu, JinYuan Liu, Guanyao Wu, Zihang Chen, Xin Fan, Risheng Liu
Date: 2023-05-20 00:00:00
Tasks: Multi-Exposure Image Fusion, Neural Architecture Search

In recent years, learning-based methods have achieved significant advancements in multi-exposure image fusion. However, two major stumbling blocks hinder the development, including pixel misalignment and inefficient inference. Reliance on aligned image pairs in existing methods causes susceptibility to artifacts due to device motion. Additionally, existing techniques often rely on handcrafted architectures with huge network engineering, resulting in redundant parameters, adversely impacting inference efficiency and flexibility. To mitigate these limitations, this study introduces an architecture search-based paradigm incorporating self-alignment and detail repletion modules for robust multi-exposure image fusion. Specifically, targeting the extreme discrepancy of exposure, we propose the self-alignment module, leveraging scene relighting to constrain the illumination degree for following alignment and feature extraction. Detail repletion is proposed to enhance the texture details of scenes. Additionally, incorporating a hardware-sensitive constraint, we present the fusion-oriented architecture search to explore compact and efficient networks for fusion. The proposed method outperforms various competitive schemes, achieving a noteworthy 3.19\% improvement in PSNR for general scenarios and an impressive 23.5\% enhancement in misaligned scenarios. Moreover, it significantly reduces inference time by 69.1\%. The code will be available at https://github.com/LiuZhu-CV/CRMEF.
Bi-VLGM : Bi-Level Class-Severity-Aware Vision-Language Graph Matching for Text Guided Medical Image Segmentation
Authors: Chen Wenting, Liu Jie, Yuan Yixuan
Date: 2023-05-20 00:00:00
Tasks: Graph Matching, Image Segmentation, Medical Image Segmentation, Prompt Learning, Relation, Segmentation, Semantic Segmentation, Sentence

Medical reports with substantial information can be naturally complementary to medical images for computer vision tasks, and the modality gap between vision and language can be solved by vision-language matching (VLM). However, current vision-language models distort the intra-model relation and mainly include class information in prompt learning that is insufficient for segmentation task. In this paper, we introduce a Bi-level class-severity-aware Vision-Language Graph Matching (Bi-VLGM) for text guided medical image segmentation, composed of a word-level VLGM module and a sentence-level VLGM module, to exploit the class-severity-aware relation among visual-textual features. In word-level VLGM, to mitigate the distorted intra-modal relation during VLM, we reformulate VLM as graph matching problem and introduce a vision-language graph matching (VLGM) to exploit the high-order relation among visual-textual features. Then, we perform VLGM between the local features for each class region and class-aware prompts to bridge their gap. In sentence-level VLGM, to provide disease severity information for segmentation task, we introduce a severity-aware prompting to quantify the severity level of retinal lesion, and perform VLGM between the global features and the severity-aware prompts. By exploiting the relation between the local (global) and class (severity) features, the segmentation model can selectively learn the class-aware and severity-aware information to promote performance. Extensive experiments prove the effectiveness of our method and its superiority to existing methods. Source code is to be released.
What Makes for Good Visual Tokenizers for Large Language Models?
Authors: Guangzhi Wang, Yixiao Ge, Xiaohan Ding, Mohan Kankanhalli, Ying Shan
Date: 2023-05-20 00:00:00
Tasks: Image Captioning, Object Counting, Question Answering, Visual Question Answering

We empirically investigate proper pre-training methods to build good visual tokenizers, making Large Language Models (LLMs) powerful Multimodal Large Language Models (MLLMs). In our benchmark, which is curated to evaluate MLLMs visual semantic understanding and fine-grained perception capabilities, we discussed different visual tokenizers pre-trained with dominant methods (i.e., DeiT, CLIP, MAE, DINO), and observe that: i) Fully/weakly supervised models capture more semantics than self-supervised models, but the gap is narrowed by scaling up the pre-training dataset. ii) Self-supervised models are better at fine-grained perception, where patch-level supervision is particularly effective. iii) Tuning the visual tokenizer leads to the loss of semantics obtained from large-scale pretraining, which is unfavorable with relatively small-scale instruction-tuning dataset. Given the findings, we reviewed methods that attempted to unify semantics and fine-grained visual understanding, e.g., patch-level feature distillation with semantically-rich targets. We obtain an intriguing insight mask-based strategies that were once all the rage may not be applicable for obtaining good visual tokenizers. Based on this critical observation, we obtain a new MLLM equipped with a tailored Good Visual Tokenizer (GVT), which exhibits strong visual comprehension capability at multiple scales. In particular, without introducing extra parameters and task-specific fine-tuning, GVT achieves superior performance on visual question answering, image captioning, and other fine-grained visual understanding tasks such as object counting and multi-class identification.
Dual-Diffusion: Dual Conditional Denoising Diffusion Probabilistic Models for Blind Super-Resolution Reconstruction in RSIs
Authors: Mengze Xu, Jie Ma, Yuanyuan Zhu
Date: 2023-05-20 00:00:00
Tasks: Blind Super-Resolution, Denoising, Image Generation, Super-Resolution

Previous super-resolution reconstruction (SR) works are always designed on the assumption that the degradation operation is fixed, such as bicubic downsampling. However, as for remote sensing images, some unexpected factors can cause the blurred visual performance, like weather factors, orbit altitude, etc. Blind SR methods are proposed to deal with various degradations. There are two main challenges of blind SR in RSIs: 1) the accu-rate estimation of degradation kernels; 2) the realistic image generation in the ill-posed problem. To rise to the challenge, we propose a novel blind SR framework based on dual conditional denoising diffusion probabilistic models (DDSR). In our work, we introduce conditional denoising diffusion probabilistic models (DDPM) from two aspects: kernel estimation progress and re-construction progress, named as the dual-diffusion. As for kernel estimation progress, conditioned on low-resolution (LR) images, a new DDPM-based kernel predictor is constructed by studying the invertible mapping between the kernel distribution and the latent distribution. As for reconstruction progress, regarding the predicted degradation kernels and LR images as conditional information, we construct a DDPM-based reconstructor to learning the mapping from the LR images to HR images. Com-prehensive experiments show the priority of our proposal com-pared with SOTA blind SR methods. Source Code is available at https://github.com/Lincoln20030413/DDSR
Movie101: A New Movie Understanding Benchmark
Authors: Zihao Yue, Qi Zhang, Anwen Hu, Liang Zhang, Ziheng Wang, Qin Jin
Date: 2023-05-20 00:00:00
Tasks: Video Captioning

To help the visually impaired enjoy movies, automatic movie narrating systems are expected to narrate accurate, coherent, and role-aware plots when there are no speaking lines of actors. Existing works benchmark this challenge as a normal video captioning task via some simplifications, such as removing role names and evaluating narrations with ngram-based metrics, which makes it difficult for automatic systems to meet the needs of real application scenarios. To narrow this gap, we construct a large-scale Chinese movie benchmark, named Movie101. Closer to real scenarios, the Movie Clip Narrating (MCN) task in our benchmark asks models to generate role-aware narration paragraphs for complete movie clips where no actors are speaking. External knowledge, such as role information and movie genres, is also provided for better movie understanding. Besides, we propose a new metric called Movie Narration Score (MNScore) for movie narrating evaluation, which achieves the best correlation with human evaluation. Our benchmark also supports the Temporal Narration Grounding (TNG) task to investigate clip localization given text descriptions. For both two tasks, our proposed methods well leverage external knowledge and outperform carefully designed baselines. The dataset and codes are released at https://github.com/yuezih/Movie101.
Chest X-ray Image Classification: A Causal Perspective
Authors: Weizhi Nie, Chen Zhang, Dan Song, Lina Zhao, Yunpeng Bai, Keliang Xie, AnAn Liu
Date: 2023-05-20 00:00:00
Tasks: Classification, image-classification, Image Classification

The chest X-ray (CXR) is one of the most common and easy-to-get medical tests used to diagnose common diseases of the chest. Recently, many deep learning-based methods have been proposed that are capable of effectively classifying CXRs. Even though these techniques have worked quite well, it is difficult to establish whether what these algorithms actually learn is the cause-and-effect link between diseases and their causes or just how to map labels to photos.In this paper, we propose a causal approach to address the CXR classification problem, which constructs a structural causal model (SCM) and uses the backdoor adjustment to select effective visual information for CXR classification. Specially, we design different probability optimization functions to eliminate the influence of confounders on the learning of real causality. Experimental results demonstrate that our proposed method outperforms the open-source NIH ChestX-ray14 in terms of classification performance.