pwc-archive/papers-with-abstracts huggingface dataset:
Optimizing ViViT Training: Time and Memory Reduction for Action Recognition
Authors: Shreyank N Gowda, Anurag Arnab, Jonathan Huang
Date: 2023-06-07 00:00:00
Tasks: Action Recognition

In this paper, we address the challenges posed by the substantial training time and memory consumption associated with video transformers, focusing on the ViViT (Video Vision Transformer) model, in particular the Factorised Encoder version, as our baseline for action recognition tasks. The factorised encoder variant follows the late-fusion approach that is adopted by many state of the art approaches. Despite standing out for its favorable speed/accuracy tradeoffs among the different variants of ViViT, its considerable training time and memory requirements still pose a significant barrier to entry. Our method is designed to lower this barrier and is based on the idea of freezing the spatial transformer during training. This leads to a low accuracy model if naively done. But we show that by (1) appropriately initializing the temporal transformer (a module responsible for processing temporal information) (2) introducing a compact adapter model connecting frozen spatial representations ((a module that selectively focuses on regions of the input image) to the temporal transformer, we can enjoy the benefits of freezing the spatial transformer without sacrificing accuracy. Through extensive experimentation over 6 benchmarks, we demonstrate that our proposed training strategy significantly reduces training costs (by $\sim 50\%$) and memory consumption while maintaining or slightly improving performance by up to 1.79\% compared to the baseline model. Our approach additionally unlocks the capability to utilize larger image transformer models as our spatial transformer and access more frames with the same memory consumption.
RefineVIS: Video Instance Segmentation with Temporal Attention Refinement
Authors: Andre Abrantes, Jiang Wang, Peng Chu, Quanzeng You, Zicheng Liu
Date: 2023-06-07 00:00:00
Tasks: Contrastive Learning, Denoising, Instance Segmentation, Segmentation, TAR, Video Instance Segmentation

We introduce a novel framework called RefineVIS for Video Instance Segmentation (VIS) that achieves good object association between frames and accurate segmentation masks by iteratively refining the representations using sequence context. RefineVIS learns two separate representations on top of an off-the-shelf frame-level image instance segmentation model: an association representation responsible for associating objects across frames and a segmentation representation that produces accurate segmentation masks. Contrastive learning is utilized to learn temporally stable association representations. A Temporal Attention Refinement (TAR) module learns discriminative segmentation representations by exploiting temporal relationships and a novel temporal contrastive denoising technique. Our method supports both online and offline inference. It achieves state-of-the-art video instance segmentation accuracy on YouTube-VIS 2019 (64.4 AP), Youtube-VIS 2021 (61.4 AP), and OVIS (46.1 AP) datasets. The visualization shows that the TAR module can generate more accurate instance segmentation masks, particularly for challenging cases such as highly occluded objects.
Context-Aware Self-Supervised Learning of Whole Slide Images
Authors: Milan Aryal, Nasim Yahyasoltani
Date: 2023-06-07 00:00:00
Tasks: Self-Supervised Learning, whole slide images

Presenting whole slide images (WSIs) as graph will enable a more efficient and accurate learning framework for cancer diagnosis. Due to the fact that a single WSI consists of billions of pixels and there is a lack of vast annotated datasets required for computational pathology, the problem of learning from WSIs using typical deep learning approaches such as convolutional neural network (CNN) is challenging. Additionally, WSIs down-sampling may lead to the loss of data that is essential for cancer detection. A novel two-stage learning technique is presented in this work. Since context, such as topological features in the tumor surroundings, may hold important information for cancer grading and diagnosis, a graph representation capturing all dependencies among regions in the WSI is very intuitive. Graph convolutional network (GCN) is deployed to include context from the tumor and adjacent tissues, and self-supervised learning is used to enhance training through unlabeled data. More specifically, the entire slide is presented as a graph, where the nodes correspond to the patches from the WSI. The proposed framework is then tested using WSIs from prostate and kidney cancers. To assess the performance improvement through self-supervised mechanism, the proposed context-aware model is tested with and without use of pre-trained self-supervised layer. The overall model is also compared with multi-instance learning (MIL) based and other existing approaches.
Computational Modeling of Deep Multiresolution-Fractal Texture and Its Application to Abnormal Brain Tissue Segmentation
Authors: A. Temtam, L. Pei, K. Iftekharuddin
Date: 2023-06-07 00:00:00
Tasks: Brain Tumor Segmentation, Segmentation, Tumor Segmentation

Computational modeling of Multiresolution- Fractional Brownian motion (fBm) has been effective in stochastic multiscale fractal texture feature extraction and machine learning of abnormal brain tissue segmentation. Further, deep multiresolution methods have been used for pixel-wise brain tissue segmentation. Robust tissue segmentation and volumetric measurement may provide more objective quantification of disease burden and offer improved tracking of treatment response for the disease. However, we posit that computational modeling of deep multiresolution fractal texture features may offer elegant feature learning. Consequently, this work proposes novel modeling of Multiresolution Fractal Deep Neural Network (MFDNN) and its computational implementation that mathematically combines a multiresolution fBm model and deep multiresolution analysis. The proposed full 3D MFDNN model offers the desirable properties of estimating multiresolution stochastic texture features by analyzing large amount of raw MRI image data for brain tumor segmentation. We apply the proposed MFDNN to estimate stochastic deep multiresolution fractal texture features for tumor tissues in brain MRI images. The MFDNN model is evaluated using 1251 patient cases for brain tumor segmentation using the most recent BRATS 2021 Challenges dataset. The evaluation of the proposed model using Dice overlap score, Husdorff distance and associated uncertainty estimation offers either better or comparable performances in abnormal brain tissue segmentation when compared to the state-of-the-art methods in the literature. Index Terms: Computational Modeling, Multiresolution Fractional Brownian Motion (fBm), Deep Multiresolution Analysis, Fractal Dimension (FD), Texture Features, Brain tumor segmentation, Deep Learning.
WOUAF: Weight Modulation for User Attribution and Fingerprinting in Text-to-Image Diffusion Models
Authors: Changhoon Kim, Kyle Min, Maitreya Patel, Sheng Cheng, Yezhou Yang
Date: 2023-06-07 00:00:00
Proceeding: CVPR 2024 1
Tasks: Misinformation

The rapid advancement of generative models, facilitating the creation of hyper-realistic images from textual descriptions, has concurrently escalated critical societal concerns such as misinformation. Although providing some mitigation, traditional fingerprinting mechanisms fall short in attributing responsibility for the malicious use of synthetic images. This paper introduces a novel approach to model fingerprinting that assigns responsibility for the generated images, thereby serving as a potential countermeasure to model misuse. Our method modifies generative models based on each user's unique digital fingerprint, imprinting a unique identifier onto the resultant content that can be traced back to the user. This approach, incorporating fine-tuning into Text-to-Image (T2I) tasks using the Stable Diffusion Model, demonstrates near-perfect attribution accuracy with a minimal impact on output quality. Through extensive evaluation, we show that our method outperforms baseline methods with an average improvement of 11\% in handling image post-processes. Our method presents a promising and novel avenue for accountable model distribution and responsible use. Our code is available in \url{https://github.com/kylemin/WOUAF}.
BU-CVKit: Extendable Computer Vision Framework for Species Independent Tracking and Analysis
Authors: Mahir Patel, Lucas Carstensen, Yiwen Gu, Michael E. Hasselmo, Margrit Betke
Date: 2023-06-07 00:00:00
Tasks: 3D Reconstruction, Camera Calibration, Pose Estimation

A major bottleneck of interdisciplinary computer vision (CV) research is the lack of a framework that eases the reuse and abstraction of state-of-the-art CV models by CV and non-CV researchers alike. We present here BU-CVKit, a computer vision framework that allows the creation of research pipelines with chainable Processors. The community can create plugins of their work for the framework, hence improving the re-usability, accessibility, and exposure of their work with minimal overhead. Furthermore, we provide MuSeqPose Kit, a user interface for the pose estimation package of BU-CVKit, which automatically scans for installed plugins and programmatically generates an interface for them based on the metadata provided by the user. It also provides software support for standard pose estimation features such as annotations, 3D reconstruction, reprojection, and camera calibration. Finally, we show examples of behavioral neuroscience pipelines created through the sample plugins created for our framework.
UniBoost: Unsupervised Unimodal Pre-training for Boosting Zero-shot Vision-Language Tasks
Authors: Yanan sun, Zihan Zhong, Qi Fan, Chi-Keung Tang, Yu-Wing Tai
Date: 2023-06-07 00:00:00
Tasks: Semantic Segmentation

Large-scale joint training of multimodal models, e.g., CLIP, have demonstrated great performance in many vision-language tasks. However, image-text pairs for pre-training are restricted to the intersection of images and texts, limiting their ability to cover a large distribution of real-world data, where noise can also be introduced as misaligned pairs during pre-processing. Conversely, unimodal models trained on text or image data alone through unsupervised techniques can achieve broader coverage of diverse real-world data and are not constrained by the requirement of simultaneous presence of image and text. In this paper, we demonstrate that using large-scale unsupervised unimodal models as pre-training can enhance the zero-shot performance of image-text pair models. Our thorough studies validate that models pre-trained as such can learn rich representations of both modalities, improving their ability to understand how images and text relate to each other. Our experiments show that unimodal pre-training outperforms state-of-the-art CLIP-based models by 6.5% (52.3% $\rightarrow$ 58.8%) on PASCAL-5$^i$ and 6.2% (27.2% $\rightarrow$ 33.4%) on COCO-20$^i$ semantic segmentation under zero-shot setting respectively. By learning representations of both modalities, unimodal pre-training offers broader coverage, reduced misalignment errors, and the ability to capture more complex features and patterns in the real-world data resulting in better performance especially for zero-shot vision-language tasks.
Improved statistical benchmarking of digital pathology models using pairwise frames evaluation
Authors: Ylaine Gerardin, John Shamshoian, Judy Shen, Nhat Le, Jamie Prezioso, John Abel, Isaac Finberg, Daniel Borders, Raymond Biju, Michael Nercessian, Vaed Prasad, Joseph Lee, Spencer Wyman, Sid Gupta, Abigail Emerson, Bahar Rahsepar, Darpan Sanghavi, Ryan Leung, Limin Yu, Archit Khosla, Amaro Taylor-Weiner
Date: 2023-06-07 00:00:00
Tasks: Benchmarking, Classification

Nested pairwise frames is a method for relative benchmarking of cell or tissue digital pathology models against manual pathologist annotations on a set of sampled patches. At a high level, the method compares agreement between a candidate model and pathologist annotations with agreement among pathologists' annotations. This evaluation framework addresses fundamental issues of data size and annotator variability in using manual pathologist annotations as a source of ground truth for model validation. We implemented nested pairwise frames evaluation for tissue classification, cell classification, and cell count prediction tasks and show results for cell and tissue models deployed on an H&E-stained melanoma dataset.
Robust-DefReg: A Robust Deformable Point Cloud Registration Method based on Graph Convolutional Neural Networks
Authors: Sara Monji-Azad, Marvin Kinz, Jürgen Hesser
Date: 2023-06-07 00:00:00
Tasks: Computational Efficiency, Point Cloud Registration

Point cloud registration is a fundamental problem in computer vision that aims to estimate the transformation between corresponding sets of points. Non-rigid registration, in particular, involves addressing challenges including various levels of deformation, noise, outliers, and data incompleteness. This paper introduces Robust-DefReg, a robust non-rigid point cloud registration method based on graph convolutional networks (GCNNs). Robust-DefReg is a coarse-to-fine registration approach within an end-to-end pipeline, leveraging the advantages of both coarse and fine methods. The method learns global features to find correspondences between source and target point clouds, to enable appropriate initial alignment, and subsequently fine registration. The simultaneous achievement of high accuracy and robustness across all challenges is reported less frequently in existing studies, making it a key objective of the Robust-DefReg method. The proposed method achieves high accuracy in large deformations while maintaining computational efficiency. This method possesses three primary attributes: high accuracy, robustness to different challenges, and computational efficiency. The experimental results show that the proposed Robust-DefReg holds significant potential as a foundational architecture for future investigations in non-rigid point cloud registration. The source code of Robust-DefReg is available.
DiViNeT: 3D Reconstruction from Disparate Views via Neural Template Regularization
Authors: Aditya Vora, Akshay Gadi Patil, Hao Zhang
Date: 2023-06-07 00:00:00
Tasks: 3D Reconstruction, Surface Reconstruction

We present a volume rendering-based neural surface reconstruction method that takes as few as three disparate RGB images as input. Our key idea is to regularize the reconstruction, which is severely ill-posed and leaving significant gaps between the sparse views, by learning a set of neural templates to act as surface priors. Our method, coined DiViNet, operates in two stages. It first learns the templates, in the form of 3D Gaussian functions, across different scenes, without 3D supervision. In the reconstruction stage, our predicted templates serve as anchors to help "stitch'' the surfaces over sparse regions. We demonstrate that our approach is not only able to complete the surface geometry but also reconstructs surface details to a reasonable extent from a few disparate input views. On the DTU and BlendedMVS datasets, our approach achieves the best reconstruction quality among existing methods in the presence of such sparse views and performs on par, if not better, with competing methods when dense views are employed as inputs.
ConceptBed: Evaluating Concept Learning Abilities of Text-to-Image Diffusion Models
Authors: Maitreya Patel, Tejas Gokhale, Chitta Baral, Yezhou Yang
Date: 2023-06-07 00:00:00
Tasks: Concept Alignment

The ability to understand visual concepts and replicate and compose these concepts from images is a central goal for computer vision. Recent advances in text-to-image (T2I) models have lead to high definition and realistic image quality generation by learning from large databases of images and their descriptions. However, the evaluation of T2I models has focused on photorealism and limited qualitative measures of visual understanding. To quantify the ability of T2I models in learning and synthesizing novel visual concepts (a.k.a. personalized T2I), we introduce ConceptBed, a large-scale dataset that consists of 284 unique visual concepts, and 33K composite text prompts. Along with the dataset, we propose an evaluation metric, Concept Confidence Deviation (CCD), that uses the confidence of oracle concept classifiers to measure the alignment between concepts generated by T2I generators and concepts contained in target images. We evaluate visual concepts that are either objects, attributes, or styles, and also evaluate four dimensions of compositionality: counting, attributes, relations, and actions. Our human study shows that CCD is highly correlated with human understanding of concepts. Our results point to a trade-off between learning the concepts and preserving the compositionality which existing approaches struggle to overcome. The data, code, and interactive demo is available at: https://conceptbed.github.io/
Object Detection with Transformers: A Review
Authors: Tahira Shehzadi, Khurram Azeem Hashmi, Didier Stricker, Muhammad Zeshan Afzal
Date: 2023-06-07 00:00:00
Tasks: 2D Object Detection, Object, object-detection, Object Detection

The astounding performance of transformers in natural language processing (NLP) has motivated researchers to explore their applications in computer vision tasks. DEtection TRansformer (DETR) introduces transformers to object detection tasks by reframing detection as a set prediction problem. Consequently, eliminating the need for proposal generation and post-processing steps. Initially, despite competitive performance, DETR suffered from slow training convergence and ineffective detection of smaller objects. However, numerous improvements are proposed to address these issues, leading to substantial improvements in DETR and enabling it to exhibit state-of-the-art performance. To our knowledge, this is the first paper to provide a comprehensive review of 21 recently proposed advancements in the original DETR model. We dive into both the foundational modules of DETR and its recent enhancements, such as modifications to the backbone structure, query design strategies, and refinements to attention mechanisms. Moreover, we conduct a comparative analysis across various detection transformers, evaluating their performance and network architectures. We hope that this study will ignite further interest among researchers in addressing the existing challenges and exploring the application of transformers in the object detection domain. Readers interested in the ongoing developments in detection transformers can refer to our website at: https://github.com/mindgarage-shan/trans_object_detection_survey
SMRVIS: Point cloud extraction from 3-D ultrasound for non-destructive testing
Authors: Lisa Y. W. Tang
Date: 2023-06-07 00:00:00
Tasks: Image Segmentation, Semantic Segmentation

We propose to formulate point cloud extraction from ultrasound volumes as an image segmentation problem. Through this convenient formulation, a quick prototype exploring various variants of the Residual Network, U-Net, and the Squeeze and Excitation Network was developed and evaluated. This report documents the experimental results compiled using a training dataset of five labeled ultrasound volumes and 84 unlabeled volumes that got completed in a two-week period as part of a submission to the open challenge "3D Surface Mesh Estimation for CVPR workshop on Deep Learning in Ultrasound Image Analysis". Based on external evaluation performed by the challenge's organizers, the framework came first place on the challenge's \href{https://www.cvpr2023-dl-ultrasound.com/}{Leaderboard}. Source code is shared with the research community at a \href{https://github.com/lisatwyw/smrvis}{public repository}.
Estimating Uncertainty in PET Image Reconstruction via Deep Posterior Sampling
Authors: Tin Vlašić, Tomislav Matulić, Damir Seršić
Date: 2023-06-07 00:00:00
Tasks: Generative Adversarial Network, Image Reconstruction, Uncertainty Quantification

Positron emission tomography (PET) is an important functional medical imaging technique often used in the evaluation of certain brain disorders, whose reconstruction problem is ill-posed. The vast majority of reconstruction methods in PET imaging, both iterative and deep learning, return a single estimate without quantifying the associated uncertainty. Due to ill-posedness and noise, a single solution can be misleading or inaccurate. Thus, providing a measure of uncertainty in PET image reconstruction can help medical practitioners in making critical decisions. This paper proposes a deep learning-based method for uncertainty quantification in PET image reconstruction via posterior sampling. The method is based on training a conditional generative adversarial network whose generator approximates sampling from the posterior in Bayesian inversion. The generator is conditioned on reconstruction from a low-dose PET scan obtained by a conventional reconstruction method and a high-quality magnetic resonance image and learned to estimate a corresponding standard-dose PET scan reconstruction. We show that the proposed model generates high-quality posterior samples and yields physically-meaningful uncertainty estimates.
DenseDINO: Boosting Dense Self-Supervised Learning with Token-Based Point-Level Consistency
Authors: Yike Yuan, Xinghe Fu, Yunlong Yu, Xi Li
Date: 2023-06-06 00:00:00
Tasks: Position, Segmentation, Self-Supervised Learning, Semantic Segmentation

In this paper, we propose a simple yet effective transformer framework for self-supervised learning called DenseDINO to learn dense visual representations. To exploit the spatial information that the dense prediction tasks require but neglected by the existing self-supervised transformers, we introduce point-level supervision across views in a novel token-based way. Specifically, DenseDINO introduces some extra input tokens called reference tokens to match the point-level features with the position prior. With the reference token, the model could maintain spatial consistency and deal with multi-object complex scene images, thus generalizing better on dense prediction tasks. Compared with the vanilla DINO, our approach obtains competitive performance when evaluated on classification in ImageNet and achieves a large margin (+7.2% mIoU) improvement in semantic segmentation on PascalVOC under the linear probing protocol for segmentation.
From Data to Action: Exploring AI and IoT-driven Solutions for Smarter Cities
Authors: Tiago Dias, Tiago Fonseca, João Vitorino, Andreia Martins, Sofia Malpique, Isabel Praça
Date: 2023-06-06 00:00:00
Tasks: Management

The emergence of smart cities demands harnessing advanced technologies like the Internet of Things (IoT) and Artificial Intelligence (AI) and promises to unlock cities' potential to become more sustainable, efficient, and ultimately livable for their inhabitants. This work introduces an intelligent city management system that provides a data-driven approach to three use cases: (i) analyze traffic information to reduce the risk of traffic collisions and improve driver and pedestrian safety, (ii) identify when and where energy consumption can be reduced to improve cost savings, and (iii) detect maintenance issues like potholes in the city's roads and sidewalks, as well as the beginning of hazards like floods and fires. A case study in Aveiro City demonstrates the system's effectiveness in generating actionable insights that enhance security, energy efficiency, and sustainability, while highlighting the potential of AI and IoT-driven solutions for smart city development.