pwc-archive/papers-with-abstracts huggingface dataset:
DPA-Net: Structured 3D Abstraction from Sparse Views via Differentiable Primitive Assembly
Authors: Fenggen Yu, Yiming Qian, Xu Zhang, Francisca Gil-Ureta, Brian Jackson, Eric Bennett, Hao Zhang
Date: 2024-04-01 00:00:00
Tasks: NeRF, Test-time Adaptation

We present a differentiable rendering framework to learn structured 3D abstractions in the form of primitive assemblies from sparse RGB images capturing a 3D object. By leveraging differentiable volume rendering, our method does not require 3D supervision. Architecturally, our network follows the general pipeline of an image-conditioned neural radiance field (NeRF) exemplified by pixelNeRF for color prediction. As our core contribution, we introduce differential primitive assembly (DPA) into NeRF to output a 3D occupancy field in place of density prediction, where the predicted occupancies serve as opacity values for volume rendering. Our network, coined DPA-Net, produces a union of convexes, each as an intersection of convex quadric primitives, to approximate the target 3D object, subject to an abstraction loss and a masking loss, both defined in the image space upon volume rendering. With test-time adaptation and additional sampling and loss designs aimed at improving the accuracy and compactness of the obtained assemblies, our method demonstrates superior performance over state-of-the-art alternatives for 3D primitive abstraction from sparse views.
A Comprehensive Review of Knowledge Distillation in Computer Vision
Authors: Gousia Habib, Tausifa Jan Saleem, Sheikh Musa Kaleem, Tufail Rouf, Brejesh lall
Date: 2024-04-01 00:00:00
Tasks: Deep Learning, Knowledge Distillation

Deep learning techniques have been demonstrated to surpass preceding cutting-edge machine learning techniques in recent years, with computer vision being one of the most prominent examples. However, deep learning models suffer from significant drawbacks when deployed in resource-constrained environments due to their large model size and high complexity. Knowledge Distillation is one of the prominent solutions to overcome this challenge. This review paper examines the current state of research on knowledge distillation, a technique for compressing complex models into smaller and simpler ones. The paper provides an overview of the major principles and techniques associated with knowledge distillation and reviews the applications of knowledge distillation in the domain of computer vision. The review focuses on the benefits of knowledge distillation, as well as the problems that must be overcome to improve its effectiveness.
PosterLlama: Bridging Design Ability of Langauge Model to Contents-Aware Layout Generation
Authors: Jaejung Seol, Seojun Kim, Jaejun Yoo
Date: 2024-04-01 00:00:00
Tasks: Layout Design, Layout Generation

Visual layout plays a critical role in graphic design fields such as advertising, posters, and web UI design. The recent trend towards content-aware layout generation through generative models has shown promise, yet it often overlooks the semantic intricacies of layout design by treating it as a simple numerical optimization. To bridge this gap, we introduce PosterLlama, a network designed for generating visually and textually coherent layouts by reformatting layout elements into HTML code and leveraging the rich design knowledge embedded within language models. Furthermore, we enhance the robustness of our model with a unique depth-based poster augmentation strategy. This ensures our generated layouts remain semantically rich but also visually appealing, even with limited data. Our extensive evaluations across several benchmarks demonstrate that PosterLlama outperforms existing methods in producing authentic and content-aware layouts. It supports an unparalleled range of conditions, including but not limited to unconditional layout generation, element conditional layout generation, layout completion, among others, serving as a highly versatile user manipulation tool.
SurMo: Surface-based 4D Motion Modeling for Dynamic Human Rendering
Authors: Tao Hu, Fangzhou Hong, Ziwei Liu
Date: 2024-04-01 00:00:00
Proceeding: CVPR 2024 1
Tasks: Generalizable Novel View Synthesis, Novel View Synthesis

Dynamic human rendering from video sequences has achieved remarkable progress by formulating the rendering as a mapping from static poses to human images. However, existing methods focus on the human appearance reconstruction of every single frame while the temporal motion relations are not fully explored. In this paper, we propose a new 4D motion modeling paradigm, SurMo, that jointly models the temporal dynamics and human appearances in a unified framework with three key designs: 1) Surface-based motion encoding that models 4D human motions with an efficient compact surface-based triplane. It encodes both spatial and temporal motion relations on the dense surface manifold of a statistical body template, which inherits body topology priors for generalizable novel view synthesis with sparse training observations. 2) Physical motion decoding that is designed to encourage physical motion learning by decoding the motion triplane features at timestep t to predict both spatial derivatives and temporal derivatives at the next timestep t+1 in the training stage. 3) 4D appearance decoding that renders the motion triplanes into images by an efficient volumetric surface-conditioned renderer that focuses on the rendering of body surfaces with motion learning conditioning. Extensive experiments validate the state-of-the-art performance of our new paradigm and illustrate the expressiveness of surface-based motion triplanes for rendering high-fidelity view-consistent humans with fast motions and even motion-dependent shadows. Our project page is at: https://taohuumd.github.io/projects/SurMo/
StructLDM: Structured Latent Diffusion for 3D Human Generation
Authors: Tao Hu, Fangzhou Hong, Ziwei Liu
Date: 2024-04-01 00:00:00
Tasks: Virtual Try-on

Recent 3D human generative models have achieved remarkable progress by learning 3D-aware GANs from 2D images. However, existing 3D human generative methods model humans in a compact 1D latent space, ignoring the articulated structure and semantics of human body topology. In this paper, we explore more expressive and higher-dimensional latent space for 3D human modeling and propose StructLDM, a diffusion-based unconditional 3D human generative model, which is learned from 2D images. StructLDM solves the challenges imposed due to the high-dimensional growth of latent space with three key designs: 1) A semantic structured latent space defined on the dense surface manifold of a statistical human body template. 2) A structured 3D-aware auto-decoder that factorizes the global latent space into several semantic body parts parameterized by a set of conditional structured local NeRFs anchored to the body template, which embeds the properties learned from the 2D training data and can be decoded to render view-consistent humans under different poses and clothing styles. 3) A structured latent diffusion model for generative human appearance sampling. Extensive experiments validate StructLDM's state-of-the-art generation performance and illustrate the expressiveness of the structured latent space over the well-adopted 1D latent space. Notably, StructLDM enables different levels of controllable 3D human generation and editing, including pose/view/shape control, and high-level tasks including compositional generations, part-aware clothing editing, 3D virtual try-on, etc. Our project page is at: https://taohuumd.github.io/projects/StructLDM/.
CPPO: Continual Learning for Reinforcement Learning with Human Feedback
Authors: Han Zhang, Yu Lei, Lin Gui, Min Yang, Yulan He, Hui Wang, Ruifeng Xu
Date: 2024-01-16 00:00:00
Proceeding: Conference 2024 1
Tasks: Continual Learning, reinforcement-learning, Reinforcement Learning

The approach of Reinforcement Learning from Human Feedback (RLHF) is widely
used for enhancing pre-trained Language Models (LM), enabling them to better
align with human preferences. Existing RLHF-based LMs however require
complete retraining whenever new queries or feedback are introduced, as human
preferences may differ across different domains or topics. LM retraining is often
impracticable in most real-world scenarios, due to the substantial time and
computational costs involved, as well as data privacy concerns. To address this
limitation, we propose Continual Proximal Policy Optimization (CPPO), a novel
method that is able to continually align LM with dynamic human preferences.
Specifically, CPPO adopts a weighting strategy to decide which samples should be
utilized for enhancing policy learning and which should be used for solidifying past
experiences. This seeks a good trade-off between policy learning and knowledge
retention. Our experimental results show that CPPO outperforms strong Continuous
learning (CL) baselines when it comes to consistently aligning with human
preferences. Furthermore, compared to PPO, CPPO offers more efficient and stable
learning in non-continual scenarios.
Uncovering Bias in Large Vision-Language Models with Counterfactuals
Authors: Phillip Howard, Anahita Bhiwandiwalla, Kathleen C. Fraser, Svetlana Kiritchenko
Date: 2024-03-29 00:00:00
Tasks: counterfactual, Question Answering, Visual Question Answering

With the advent of Large Language Models (LLMs) possessing increasingly impressive capabilities, a number of Large Vision-Language Models (LVLMs) have been proposed to augment LLMs with visual inputs. Such models condition generated text on both an input image and a text prompt, enabling a variety of use cases such as visual question answering and multimodal chat. While prior studies have examined the social biases contained in text generated by LLMs, this topic has been relatively unexplored in LVLMs. Examining social biases in LVLMs is particularly challenging due to the confounding contributions of bias induced by information contained across the text and visual modalities. To address this challenging problem, we conduct a large-scale study of text generated by different LVLMs under counterfactual changes to input images. Specifically, we present LVLMs with identical open-ended text prompts while conditioning on images from different counterfactual sets, where each set contains images which are largely identical in their depiction of a common subject (e.g., a doctor), but vary only in terms of intersectional social attributes (e.g., race and gender). We comprehensively evaluate the text produced by different LVLMs under this counterfactual generation setting and find that social attributes such as race, gender, and physical characteristics depicted in input images can significantly influence toxicity and the generation of competency-associated words.
MSTCGAN: Multiscale time conditional generative adversarial network for long-term satellite image sequence prediction
Authors: Kuai Dai, Xutao Li, Yunming Ye, Shanshan Feng, Danyu Qin, Rui Ye
Date: 2022-06-01 00:00:00
Proceeding: IEEE Transactions on Geoscience and Remote Sensing 2022 6
Tasks: Generative Adversarial Network, Optical Flow Estimation, Prediction

Satellite image sequence prediction is a crucial and challenging task. Previous studies leverage optical flow methods or existing deep learning methods on spatial–temporal sequence models for the task. However, they suffer from either oversimplified model assumptions or blurry predictions and sequential error accumulation issue, for a long-term forecast requirement. In this article, we propose a novel multiscale time conditional generative adversarial network (MSTCGAN). To address the sequential error accumulation issue, MSTCGAN adopts a parallel prediction framework to produce the future image sequences by a one-hot time condition input. In addition, a powerful multiscale generator is designed with the multihead axial attention, which helps to carefully preserve the fine-grained details for appearance consistency. Moreover, we develop a temporal discriminator to address the blurry issue and maintain the motion consistency in prediction. Extensive experiments have been conducted on the FengYun-4A satellite dataset, and the results demonstrate the effectiveness and superiority of the proposed method over state-of-the-art approaches.
Wrist-Based Fall Detection: Towards Generalization across Datasets
Authors: Vanilson Fula, Plinio Moreno
Date: 2024-03-05 00:00:00
Proceeding: Sensors 2024 3
Tasks: Specificity

Increasing age is related to a decrease in independence of movement and with this decrease comes falls, millions of falls occur every year and the most affected people are the older adults. These falls usually have a big impact on health and independence of the older adults, as well as financial impact on the health systems. Thus, many studies have developed fall detectors from several types of sensors. Previous studies related to the creation of fall detection systems models use only one dataset that usually has a small number of samples. Training and testing machine learning models in this small scope: (i) yield overoptimistic classification rates, (ii) do not generalize to real-life situations and (iii) have very high rate of false positives. Given this, the proposal of this research work is the creation of a new dataset that encompasses data from three different datasets, with more than 1300 fall samples and 28 K negative samples. Our new dataset includes a standard way of adding samples, which allow the future addition of other data sources. We evaluate our dataset by using classic cost-sensitive Machine Leaning methods that deal with class imbalance. For the training and validation of this model, a set of temporal and frequency features were extracted from the raw data of an accelerometer and a gyroscope using a sliding window of 2 s with an overlap of 50%. We study the generalization properties of each dataset, by testing on the other datasets and also the performance of our new dataset. The model showed a good ability to distinguish between activities of daily living and falls, achieving a recall of 90.57%, a specificity of 96.91% and an Area Under the Receiver Operating Characteristic curve (AUC-ROC) value of 98.85% against the combination of three datasets.
FRED: Towards a Full Rotation-Equivariance in Aerial Image Object Detection
Authors: Chanho Lee, Jinsu Son, Hyounguk Shon, Yunho Jeon, Junmo Kim
Date: 2023-12-22 00:00:00
Tasks: Data Augmentation, Object, object-detection, Object Detection, Object Localization, Oriented Object Detection

Rotation-equivariance is an essential yet challenging property in oriented object detection. While general object detectors naturally leverage robustness to spatial shifts due to the translation-equivariance of the conventional CNNs, achieving rotation-equivariance remains an elusive goal. Current detectors deploy various alignment techniques to derive rotation-invariant features, but still rely on high capacity models and heavy data augmentation with all possible rotations. In this paper, we introduce a Fully Rotation-Equivariant Oriented Object Detector (FRED), whose entire process from the image to the bounding box prediction is strictly equivariant. Specifically, we decouple the invariant task (object classification) and the equivariant task (object localization) to achieve end-to-end equivariance. We represent the bounding box as a set of rotation-equivariant vectors to implement rotation-equivariant localization. Moreover, we utilized these rotation-equivariant vectors as offsets in the deformable convolution, thereby enhancing the existing advantages of spatial adaptation. Leveraging full rotation-equivariance, our FRED demonstrates higher robustness to image-level rotation compared to existing methods. Furthermore, we show that FRED is one step closer to non-axis aligned learning through our experiments. Compared to state-of-the-art methods, our proposed method delivers comparable performance on DOTA-v1.0 and outperforms by 1.5 mAP on DOTA-v1.5, all while significantly reducing the model parameters to 16%.
Evaluating deep learning methods applied to Landsat time series subsequences to detect and classify boreal forest disturbances events: The challenge of partial and progressive disturbances
Authors: Pauline Perbet, Luc Guindon, Jean-François Côté, Martin Béland
Date: 2024-05-25 00:00:00
Proceeding: Remote Sensing of Environment 2024 5
Tasks: Deep Learning, Time Series

The monitoring of forest ecosystems is significantly affected by the lack of consistent historical data of low-severity (forest partially disturbed) or gradual disturbance (e.g. eastern spruce budworm epidemic). The goal of this paper is to explore the use of a subset of Landsat time series and deep learning models to identify both the type and the year of disturbances, including low-severity and gradual disturbances, in the boreal forest of eastern Canada at the pixel level. Remote sensing data such as the spectral information from Landsat time series are the best available option for large scale observations of disturbances that go back decades. Traditional modeling approaches, like LandTrendr, require substantial handcrafted pre-processing to remove noise and to extract temporal features from the image sequences before using them as input to a classical machine-learning model. Deep-learning models can autonomously discern which features are relevant within the coarse temporal and spectral information from the Landsat annual dense time series.

We evaluated the performance of TempCNN and Transformer model in detecting and classifying the type and the year of the forest disturbance using Landsat time series subsequences. Our findings resulted in the generation of four disturbance maps outlining the forest history from 1986 to 2021 within the eastern Canadian boreal forest. Our experimental outcomes demonstrate several significant benefits of employing deep learning models. Firstly, using noisy Landsat time series they achieve comparable accuracy for classifying fire and total harvesting than existing publicly available disturbance maps. Secondly, the use of shorter time series subsequence with deep learning models enables to map adequately different overlapping disturbances occurring in the complete time series. Finally, they increase the number of distinguishable disturbance classes by adding partial harvesting, gradual disturbances, and forest recovery from older events, making them useful approaches for obtaining the first remote sensing-based map for areas affected by the eastern spruce budworm.
Equilibrium in Style: A Modeling Framework on the Cash Flow and the Life Cycle of a Consumer Store
Authors: Shanyu Han, Jian Lei, Yang Liu
Date: 2024-04-03 00:00:00

The consumer store is ubiquitous and plays an important role in our everyday lives. It is an open question why stores usually have such short life cycles (typically around 3 years in China). This paper proposes a theoretical framework based on an equilibrium in style supply of stores and style demand of consumers to characterize store cash flow (revenue), leading to a strong explanation of this puzzle. In our model, we derive that the preference shifting of consumers is the main reason for the cash flow decreasing to its break-even line over time, while the visibility broadening leads to initial growth, resulting in rainbow-shaped cash flow and its life cycle. Moreover, the intensified spatial competition will lead to an unexpected decrease in the store's cash flow, or even closure. We calibrate our model with proprietary data of three Chinese stores from three representative industries and study the relationship between customers' preference shifting and cash flow. To our knowledge, there have been no prior attempts to quantitatively model the life cycle of the store.
Enhanced Precision in Rainfall Forecasting for Mumbai: Utilizing Physics Informed ConvLSTM2D Models for Finer Spatial and Temporal Resolution
Authors: Ajay Devda, Akshay Sunil, Murthy R, B Deepthi
Date: 2024-04-01 00:00:00
Tasks: Deep Learning, Prediction

Forecasting rainfall in tropical areas is challenging due to complex atmospheric behaviour, elevated humidity levels, and the common presence of convective rain events. In the Indian context, the difficulty is further exacerbated because of the monsoon intra seasonal oscillations, which introduce significant variability in rainfall patterns over short periods. Earlier investigations into rainfall prediction leveraged numerical weather prediction methods, along with statistical and deep learning approaches. This study introduces deep learning spatial model aimed at enhancing rainfall prediction accuracy on a finer scale. In this study, we hypothesize that integrating physical understanding improves the precipitation prediction skill of deep learning models with high precision for finer spatial scales, such as cities. To test this hypothesis, we introduce a physics informed ConvLSTM2D model to predict precipitation 6hr and 12hr ahead for Mumbai, India. We utilize ERA5 reanalysis data select predictor variables, across various geopotential levels. The ConvLSTM2D model was trained on the target variable precipitation for 4 different grids representing different spatial grid locations of Mumbai. Thus, the use of the ConvLSTM2D model for rainfall prediction, utilizing physics informed data from specific grids with limited spatial information, reflects current advancements in meteorological research that emphasize both efficiency and localized precision.
Energy-based Model for Accurate Shapley Value Estimation in Interpretable Deep Learning Predictive Modeling
Authors: Cheng Lu, Jiusun Zeng, Yu Xia, Jinhui Cai, Shihua Luo
Date: 2024-04-01 00:00:00
Tasks: Explainable artificial intelligence, Explainable Artificial Intelligence (XAI)

As a favorable tool for explainable artificial intelligence (XAI), Shapley value has been widely used to interpret deep learning based predictive models. However, accurate and efficient estimation of Shapley value is difficult since the computation load grows exponentially with the increase of input features. Most existing accelerated estimation methods have to compromise on estimation accuracy with efficiency. In this article, we present EmSHAP(Energy-based model for Shapley value estimation) to estimate the expectation of Shapley contribution function under arbitrary subset of features given the rest. The energy-based model estimates the conditional density in the Shapley contribution function, which involves an energy network for approximating the unnormalized conditional density and a GRU (Gated Recurrent Unit) network for approximating the partition function. The GRU network maps the input features onto a hidden space to eliminate the impact of input orderings. In order to theoretically evaluate the performance of different Shapley value estimation methods, Theorems 1, 2 and 3 analyzed the error bounds of EmSHAP as well as two state-of-the-art methods, namely KernelSHAP and VAEAC. It is proved that EmSHAP has tighter error bound than KernelSHAP and VAEAC. Finally, case studies on two application examples show the enhanced estimation accuracy of EmSHAP.