pwc-archive/papers-with-abstracts huggingface dataset:
Differentiable Dynamics for Articulated 3d Human Motion Reconstruction
Authors: Erik Gärtner, Mykhaylo Andriluka, Erwin Coumans, Cristian Sminchisescu
Date: 2022-05-24 00:00:00
Proceeding: CVPR 2022 1
Tasks: 3D Human Pose Estimation

We introduce DiffPhy, a differentiable physics-based model for articulated 3d human motion reconstruction from video. Applications of physics-based reasoning in human motion analysis have so far been limited, both by the complexity of constructing adequate physical models of articulated human motion, and by the formidable challenges of performing stable and efficient inference with physics in the loop. We jointly address such modeling and inference challenges by proposing an approach that combines a physically plausible body representation with anatomical joint limits, a differentiable physics simulator, and optimization techniques that ensure good performance and robustness to suboptimal local optima. In contrast to several recent methods, our approach readily supports full-body contact including interactions with objects in the scene. Most importantly, our model connects end-to-end with images, thus supporting direct gradient-based physics optimization by means of image-based loss functions. We validate the model by demonstrating that it can accurately reconstruct physically plausible 3d human motion from monocular video, both on public benchmarks with available 3d ground-truth, and on videos from the internet.
ASSET: Autoregressive Semantic Scene Editing with Transformers at High Resolutions
Authors: Difan Liu, Sandesh Shetty, Tobias Hinz, Matthew Fisher, Richard Zhang, Taesung Park, Evangelos Kalogerakis
Date: 2022-05-24 00:00:00
Tasks: Semantic Segmentation, Vocal Bursts Intensity Prediction

We present ASSET, a neural architecture for automatically modifying an input high-resolution image according to a user's edits on its semantic segmentation map. Our architecture is based on a transformer with a novel attention mechanism. Our key idea is to sparsify the transformer's attention matrix at high resolutions, guided by dense attention extracted at lower image resolutions. While previous attention mechanisms are computationally too expensive for handling high-resolution images or are overly constrained within specific image regions hampering long-range interactions, our novel attention mechanism is both computationally efficient and effective. Our sparsified attention mechanism is able to capture long-range interactions and context, leading to synthesizing interesting phenomena in scenes, such as reflections of landscapes onto water or flora consistent with the rest of the landscape, that were not possible to generate reliably with previous convnets and transformer approaches. We present qualitative and quantitative results, along with user studies, demonstrating the effectiveness of our method.
GLObal Building heights for Urban Studies (UT-GLOBUS) for city- and street- scale urban simulations: Development and first applications
Authors: Harsh G. Kamath, Manmeet Singh, Neetiraj Malviya, Alberto Martilli, Liu He, Daniel Aliaga, Cenlin He, Fei Chen, Lori A. Magruder, Zong-Liang Yang, Dev Niyogi
Date: 2022-05-24 00:00:00

We introduce University of Texas - Global Building heights for Urban Studies (UT-GLOBUS), a dataset providing building heights and urban canopy parameters (UCPs) for more than 1200 cities or locales worldwide. UT-GLOBUS combines open-source spaceborne altimetry (ICESat-2 and GEDI) and coarse-resolution urban canopy elevation data with a machine-learning model to estimate building-level information. Validation using LiDAR data from six US cities showed UT-GLOBUS-derived building heights had a root mean squared error (RMSE) of 9.1 meters. Validation of mean building heights within 1-km^2 grid cells, including data from Hamburg and Sydney, resulted in an RMSE of 7.8 meters. Testing the UCPs in the urban Weather Research and Forecasting (WRF-Urban) model resulted in a significant improvement (55% in RMSE) in intra-urban air temperature representation compared to the existing table-based local climate zone approach in Houston, TX. Additionally, we demonstrated the dataset's utility for simulating heat mitigation strategies and building energy consumption using WRF-Urban, with test cases in Chicago, IL, and Austin, TX. Street-scale mean radiant temperature simulations using the Solar and LongWave Environmental Irradiance Geometry (SOLWEIG) model, incorporating UT-GLOBUS and LiDAR-derived building heights, confirmed the dataset's effectiveness in modeling human thermal comfort in Baltimore, MD (daytime RMSE = 2.85 C). Thus, UT-GLOBUS can be used for modeling urban hazards with significant socioeconomic and biometeorological risks, enabling finer scale urban climate simulations and overcoming previous limitations due to the lack of building information.
Absolute Triangulation Algorithms for Space Exploration
Authors: Sebastien Henry, John A. Christian
Date: 2022-05-24 00:00:00

Images are an important source of information for spacecraft navigation and for three-dimensional reconstruction of observed space objects. Both of these applications take the form of a triangulation problem when the camera has a known attitude and the measurements extracted from the image are line of sight (LOS) directions. This work provides a comprehensive review of the history and theoretical foundations of triangulation. A variety of classical triangulation algorithms are reviewed, including a number of suboptimal linear methods (many LOS measurements) and the optimal method of Hartley and Sturm (only two LOS measurements). It is shown that the optimal many-measurement case may be solved without iteration as a linear system using the new Linear Optimal Sine Triangulation (LOST) method. Both LOST and the polynomial method of Hartley and Sturm provide the same result in the case of only two measurements. The various triangulation algorithms are assessed with a few numerical examples, including planetary terrain relative navigation, angles-only optical navigation at Uranus, 3-D reconstruction of Notre-Dame de Paris, and angles-only relative navigation.
StylizedNeRF: Consistent 3D Scene Stylization as Stylized NeRF via 2D-3D Mutual Learning
Authors: Yi-Hua Huang, Yue He, Yu-Jie Yuan, Yu-Kun Lai, Lin Gao
Date: 2022-05-24 00:00:00
Proceeding: CVPR 2022 1
Tasks: Image Stylization, NeRF

3D scene stylization aims at generating stylized images of the scene from arbitrary novel views following a given set of style examples, while ensuring consistency when rendered from different views. Directly applying methods for image or video stylization to 3D scenes cannot achieve such consistency. Thanks to recently proposed neural radiance fields (NeRF), we are able to represent a 3D scene in a consistent way. Consistent 3D scene stylization can be effectively achieved by stylizing the corresponding NeRF. However, there is a significant domain gap between style examples which are 2D images and NeRF which is an implicit volumetric representation. To address this problem, we propose a novel mutual learning framework for 3D scene stylization that combines a 2D image stylization network and NeRF to fuse the stylization ability of 2D stylization network with the 3D consistency of NeRF. We first pre-train a standard NeRF of the 3D scene to be stylized and replace its color prediction module with a style network to obtain a stylized NeRF. It is followed by distilling the prior knowledge of spatial consistency from NeRF to the 2D stylization network through an introduced consistency loss. We also introduce a mimic loss to supervise the mutual learning of the NeRF style module and fine-tune the 2D stylization decoder. In order to further make our model handle ambiguities of 2D stylization results, we introduce learnable latent codes that obey the probability distributions conditioned on the style. They are attached to training samples as conditional inputs to better learn the style module in our novel stylized NeRF. Experimental results demonstrate that our method is superior to existing approaches in both visual quality and long-range consistency.
Improving the Latent Space of Image Style Transfer
Authors: Yunpeng Bai, Cairong Wang, Chun Yuan, Yanbo Fan, Jue Wang
Date: 2022-05-24 00:00:00
Tasks: Style Transfer

Existing neural style transfer researches have studied to match statistical information between the deep features of content and style images, which were extracted by a pre-trained VGG, and achieved significant improvement in synthesizing artistic images. However, in some cases, the feature statistics from the pre-trained encoder may not be consistent with the visual style we perceived. For example, the style distance between images of different styles is less than that of the same style. In such an inappropriate latent space, the objective function of the existing methods will be optimized in the wrong direction, resulting in bad stylization results. In addition, the lack of content details in the features extracted by the pre-trained encoder also leads to the content leak problem. In order to solve these issues in the latent space used by style transfer, we propose two contrastive training schemes to get a refined encoder that is more suitable for this task. The style contrastive loss pulls the stylized result closer to the same visual style image and pushes it away from the content image. The content contrastive loss enables the encoder to retain more available details. We can directly add our training scheme to some existing style transfer methods and significantly improve their results. Extensive experimental results demonstrate the effectiveness and superiority of our methods.
Detecting Deforestation from Sentinel-1 Data in the Absence of Reliable Reference Data
Authors: Johannes N. Hansen, Edward T. A. Mitchard, Stuart King
Date: 2022-05-24 00:00:00
Tasks: Change Detection

Forests are vital for the wellbeing of our planet. Large and small scale deforestation across the globe is threatening the stability of our climate, forest biodiversity, and therefore the preservation of fragile ecosystems and our natural habitat as a whole. With increasing public interest in climate change issues and forest preservation, a large demand for carbon offsetting, carbon footprint ratings, and environmental impact assessments is emerging. Most often, deforestation maps are created from optical data such as Landsat and MODIS. These maps are not typically available at less than annual intervals due to persistent cloud cover in many parts of the world, especially the tropics where most of the world's forest biomass is concentrated. Synthetic Aperture Radar (SAR) can fill this gap as it penetrates clouds. We propose and evaluate a novel method for deforestation detection in the absence of reliable reference data which often constitutes the largest practical hurdle. This method achieves a change detection sensitivity (producer's accuracy) of 96.5% in the study area, although false positives lead to a lower user's accuracy of about 75.7%, with a total balanced accuracy of 90.4%. The change detection accuracy is maintained when adding up to 20% noise to the reference labels. While further work is required to reduce the false positive rate, improve detection delay, and validate this method in additional circumstances, the results show that Sentinel-1 data have the potential to advance the timeliness of global deforestation monitoring.
Full-Reference Calibration-Free Image Quality Assessment
Authors: Elio D. Di Claudio, Paolo Giannitrapani, Giovanni Jacovitti
Date: 2022-05-24 00:00:00
Tasks: Image Quality Assessment

One major problem of objective Image Quality Assessment (IQA) methods is the lack of linearity of their quality estimates with respect to scores expressed by human subjects. For this reason, usually IQA metrics undergo a calibration process based on subjective quality examples. However, example-based training makes generalization problematic, hampering result comparison across different applications and operative conditions. In this paper, new Full Reference (FR) techniques, providing estimates linearly correlated with human scores without using calibration are introduced. To reach this objective, these techniques are deeply rooted on principles and theoretical constraints. Restricting the interest on the IQA of the set of natural images, it is first recognized that application of estimation theory and psycho physical principles to images degraded by Gaussian blur leads to a so-called canonical IQA method, whose estimates are not only highly linearly correlated to subjective scores, but are also straightforwardly related to the Viewing Distance (VD). Then, it is shown that mainstream IQA methods can be reconducted to the canonical method applying a preliminary metric conversion based on a unique specimen image. The application of this scheme is then extended to a significant class of degraded images other than Gaussian blur, including noisy and compressed images. The resulting calibration-free FR IQA methods are suited for applications where comparability and interoperability across different imaging systems and on different VDs is a major requirement. A comparison of their statistical performance with respect to some conventional calibration prone methods is finally provided.
Learning to Drive Using Sparse Imitation Reinforcement Learning
Authors: Yuci Han, Alper Yilmaz
Date: 2022-05-24 00:00:00
Tasks: Autonomous Driving, reinforcement-learning, Reinforcement Learning, Reinforcement Learning (RL), Safe Exploration

In this paper, we propose Sparse Imitation Reinforcement Learning (SIRL), a hybrid end-to-end control policy that combines the sparse expert driving knowledge with reinforcement learning (RL) policy for autonomous driving (AD) task in CARLA simulation environment. The sparse expert is designed based on hand-crafted rules which is suboptimal but provides a risk-averse strategy by enforcing experience for critical scenarios such as pedestrian and vehicle avoidance, and traffic light detection. As it has been demonstrated, training a RL agent from scratch is data-inefficient and time consuming particularly for the urban driving task, due to the complexity of situations stemming from the vast size of state space. Our SIRL strategy provides a solution to solve these problems by fusing the output distribution of the sparse expert policy and the RL policy to generate a composite driving policy. With the guidance of the sparse expert during the early training stage, SIRL strategy accelerates the training process and keeps the RL exploration from causing a catastrophe outcome, and ensures safe exploration. To some extent, the SIRL agent is imitating the driving expert's behavior. At the same time, it continuously gains knowledge during training therefore it keeps making improvement beyond the sparse expert, and can surpass both the sparse expert and a traditional RL agent. We experimentally validate the efficacy of proposed SIRL approach in a complex urban scenario within the CARLA simulator. Besides, we compare the SIRL agent's performance for risk-averse exploration and high learning efficiency with the traditional RL approach. We additionally demonstrate the SIRL agent's generalization ability to transfer the driving skill to unseen environment.
HiVLP: Hierarchical Vision-Language Pre-Training for Fast Image-Text Retrieval
Authors: Feilong Chen, Xiuyi Chen, Jiaxin Shi, Duzhen Zhang, Jianlong Chang, Qi Tian
Date: 2022-05-24 00:00:00
Tasks: Cross-Modal Retrieval, Image-text Retrieval, Retrieval, Text Retrieval

In the past few years, the emergence of vision-language pre-training (VLP) has brought cross-modal retrieval to a new era. However, due to the latency and computation demand, it is commonly challenging to apply VLP in a real-time online retrieval system. To alleviate the defect, this paper proposes a \textbf{Hi}erarchical \textbf{V}ision-\textbf{}Language \textbf{P}re-Training (\textbf{HiVLP}) for fast Image-Text Retrieval (ITR). Specifically, we design a novel hierarchical retrieval objective, which uses the representation of different dimensions for coarse-to-fine ITR, i.e., using low-dimensional representation for large-scale coarse retrieval and high-dimensional representation for small-scale fine retrieval. We evaluate our proposed HiVLP on two popular image-text retrieval benchmarks, i.e., Flickr30k and COCO. Extensive experiments demonstrate that our HiVLP not only has fast inference speed but also can be easily scaled to large-scale ITR scenarios. The detailed results show that HiVLP is $1,427$$\sim$$120,649\times$ faster than the fusion-based model UNITER and 2$\sim$5 faster than the fastest embedding-based model LightingDot in different candidate scenarios. It also achieves about +4.9 AR on COCO and +3.8 AR on Flickr30K than LightingDot and achieves comparable performance with the state-of-the-art (SOTA) fusion-based model METER.
Classification of Phonological Parameters in Sign Languages
Authors: Boris Mocialov, Graham Turner, Helen Hastie
Date: 2022-05-24 00:00:00
Tasks: Classification, Pose Estimation

Signers compose sign language phonemes that enable communication by combining phonological parameters such as handshape, orientation, location, movement, and non-manual features. Linguistic research often breaks down signs into their constituent parts to study sign languages and often a lot of effort is invested into the annotation of the videos. In this work we show how a single model can be used to recognise the individual phonological parameters within sign languages with the aim of either to assist linguistic annotations or to describe the signs for the sign recognition models. We use Danish Sign Language data set `Ordbog over Dansk Tegnsprog' to generate multiple data sets using pose estimation model, which are then used for training the multi-label Fast R-CNN model to support multi-label modelling. Moreover, we show that there is a significant co-dependence between the orientation and location phonological parameters in the generated data and we incorporate this co-dependence in the model to achieve better performance.
Context Attention Network for Skeleton Extraction
Authors: Zixuan Huang, Yunfeng Wang, Zhiwen Chen, Xin Gao, Ruili Feng, Xiaobo Li
Date: 2022-05-24 00:00:00

Skeleton extraction is a task focused on providing a simple representation of an object by extracting the skeleton from the given binary or RGB image. In recent years many attractive works in skeleton extraction have been made. But as far as we know, there is little research on how to utilize the context information in the binary shape of objects. In this paper, we propose an attention-based model called Context Attention Network (CANet), which integrates the context extraction module in a UNet architecture and can effectively improve the ability of network to extract the skeleton pixels. Meanwhile, we also use some novel techniques including distance transform, weight focal loss to achieve good results on the given dataset. Finally, without model ensemble and with only 80% of the training images, our method achieves 0.822 F1 score during the development phase and 0.8507 F1 score during the final phase of the Pixel SkelNetOn Competition, ranking 1st place on the leaderboard.
Privacy-Preserving Image Classification Using Vision Transformer
Authors: Zheng Qi, AprilPyone MaungMaung, Yuma Kinoshita, Hitoshi Kiya
Date: 2022-05-24 00:00:00
Tasks: Classification, image-classification, Image Classification, Position, Privacy Preserving

In this paper, we propose a privacy-preserving image classification method that is based on the combined use of encrypted images and the vision transformer (ViT). The proposed method allows us not only to apply images without visual information to ViT models for both training and testing but to also maintain a high classification accuracy. ViT utilizes patch embedding and position embedding for image patches, so this architecture is shown to reduce the influence of block-wise image transformation. In an experiment, the proposed method for privacy-preserving image classification is demonstrated to outperform state-of-the-art methods in terms of classification accuracy and robustness against various attacks.
VLCDoC: Vision-Language Contrastive Pre-Training Model for Cross-Modal Document Classification
Authors: Souhail Bakkali, Zuheng Ming, Mickael Coustaty, Marçal Rusiñol, Oriol Ramos Terrades
Date: 2022-05-24 00:00:00
Tasks: Document Classification, Document Image Classification

Multimodal learning from document data has achieved great success lately as it allows to pre-train semantically meaningful features as a prior into a learnable downstream task. In this paper, we approach the document classification problem by learning cross-modal representations through language and vision cues, considering intra- and inter-modality relationships. Instead of merging features from different modalities into a joint representation space, the proposed method exploits high-level interactions and learns relevant semantic information from effective attention flows within and across modalities. The proposed learning objective is devised between intra- and inter-modality alignment tasks, where the similarity distribution per task is computed by contracting positive sample pairs while simultaneously contrasting negative ones in the joint representation space}. Extensive experiments on public document classification datasets demonstrate the effectiveness and the generality of our model on low-scale and large-scale datasets.
Effect of Gender, Pose and Camera Distance on Human Body Dimensions Estimation
Authors: Yansel Gónzalez Tejeda, Helmut A. Mayer
Date: 2022-05-24 00:00:00

Human Body Dimensions Estimation (HBDE) is a task that an intelligent agent can perform to attempt to determine human body information from images (2D) or point clouds or meshes (3D). More specifically, if we define the HBDE problem as inferring human body measurements from images, then HBDE is a difficult, inverse, multi-task regression problem that can be tackled with machine learning techniques, particularly convolutional neural networks (CNN). Despite the community's tremendous effort to advance human shape analysis, there is a lack of systematic experiments to assess CNNs estimation of human body dimensions from images. Our contribution lies in assessing a CNN estimation performance in a series of controlled experiments. To that end, we augment our recently published neural anthropometer dataset by rendering images with different camera distance. We evaluate the network inference absolute and relative mean error between the estimated and actual HBDs. We train and evaluate the CNN in four scenarios: (1) training with subjects of a specific gender, (2) in a specific pose, (3) sparse camera distance and (4) dense camera distance. Not only our experiments demonstrate that the network can perform the task successfully, but also reveal a number of relevant facts that contribute to better understand the task of HBDE.