pwc-archive/papers-with-abstracts huggingface dataset:
Interactive Semantic Map Representation for Skill-based Visual Object Navigation
Authors: Tatiana Zemskova, Aleksei Staroverov, Kirill Muravyev, Dmitry Yudin, Aleksandr Panov
Date: 2023-11-07 00:00:00

Visual object navigation using learning methods is one of the key tasks in mobile robotics. This paper introduces a new representation of a scene semantic map formed during the embodied agent interaction with the indoor environment. It is based on a neural network method that adjusts the weights of the segmentation model with backpropagation of the predicted fusion loss values during inference on a regular (backward) or delayed (forward) image sequence. We have implemented this representation into a full-fledged navigation approach called SkillTron, which can select robot skills from end-to-end policies based on reinforcement learning and classic map-based planning methods. The proposed approach makes it possible to form both intermediate goals for robot exploration and the final goal for object navigation. We conducted intensive experiments with the proposed approach in the Habitat environment, which showed a significant superiority in navigation quality metrics compared to state-of-the-art approaches. The developed code and used custom datasets are publicly available at github.com/AIRI-Institute/skill-fusion.
DeepPatent2: A Large-Scale Benchmarking Corpus for Technical Drawing Understanding
Authors: Kehinde Ajayi, Xin Wei, Martin Gryder, Winston Shields, Jian Wu, Shawn M. Jones, Michal Kucer, Diane Oyen
Date: 2023-11-07 00:00:00
Tasks: 3D Reconstruction, Benchmarking, Diversity, Image Captioning, Image Reconstruction, Image Retrieval, Retrieval

Recent advances in computer vision (CV) and natural language processing have been driven by exploiting big data on practical applications. However, these research fields are still limited by the sheer volume, versatility, and diversity of the available datasets. CV tasks, such as image captioning, which has primarily been carried out on natural images, still struggle to produce accurate and meaningful captions on sketched images often included in scientific and technical documents. The advancement of other tasks such as 3D reconstruction from 2D images requires larger datasets with multiple viewpoints. We introduce DeepPatent2, a large-scale dataset, providing more than 2.7 million technical drawings with 132,890 object names and 22,394 viewpoints extracted from 14 years of US design patent documents. We demonstrate the usefulness of DeepPatent2 with conceptual captioning. We further provide the potential usefulness of our dataset to facilitate other research areas such as 3D image reconstruction and image retrieval.
Image-Pointcloud Fusion based Anomaly Detection using PD-REAL Dataset
Authors: Jianjian Qin, Chunzhi Gu, Jun Yu, Chao Zhang
Date: 2023-11-07 00:00:00
Tasks: Anomaly Detection, Unsupervised Anomaly Detection

We present PD-REAL, a novel large-scale dataset for unsupervised anomaly detection (AD) in the 3D domain. It is motivated by the fact that 2D-only representations in the AD task may fail to capture the geometric structures of anomalies due to uncertainty in lighting conditions or shooting angles. PD-REAL consists entirely of Play-Doh models for 15 object categories and focuses on the analysis of potential benefits from 3D information in a controlled environment. Specifically, objects are first created with six types of anomalies, such as dent, crack, or perforation, and then photographed under different lighting conditions to mimic real-world inspection scenarios. To demonstrate the usefulness of 3D information, we use a commercially available RealSense camera to capture RGB and depth images. Compared to the existing 3D dataset for AD tasks, the data acquisition of PD-REAL is significantly cheaper, easily scalable and easier to control variables. Extensive evaluations with state-of-the-art AD algorithms on our dataset demonstrate the benefits as well as challenges of using 3D information. Our dataset can be downloaded from https://github.com/Andy-cs008/PD-REAL
Proceedings of the 5th International Workshop on Reading Music Systems
Authors: Jorge Calvo-Zaragoza, Alexander Pacha, Elona Shatri
Date: 2023-11-07 00:00:00
Tasks: Information Retrieval, Music Information Retrieval, Retrieval

The International Workshop on Reading Music Systems (WoRMS) is a workshop that tries to connect researchers who develop systems for reading music, such as in the field of Optical Music Recognition, with other researchers and practitioners that could benefit from such systems, like librarians or musicologists. The relevant topics of interest for the workshop include, but are not limited to: Music reading systems; Optical music recognition; Datasets and performance evaluation; Image processing on music scores; Writer identification; Authoring, editing, storing and presentation systems for music scores; Multi-modal systems; Novel input-methods for music to produce written music; Web-based Music Information Retrieval services; Applications and projects; Use-cases related to written music. These are the proceedings of the 5th International Workshop on Reading Music Systems, held in Milan, Italy on Nov. 4th 2023.
Augmenting Lane Perception and Topology Understanding with Standard Definition Navigation Maps
Authors: Katie Z Luo, Xinshuo Weng, Yan Wang, Shuang Wu, Jie Li, Kilian Q Weinberger, Yue Wang, Marco Pavone
Date: 2023-11-07 00:00:00
Tasks: Autonomous Driving, Lane Detection, Prediction

Autonomous driving has traditionally relied heavily on costly and labor-intensive High Definition (HD) maps, hindering scalability. In contrast, Standard Definition (SD) maps are more affordable and have worldwide coverage, offering a scalable alternative. In this work, we systematically explore the effect of SD maps for real-time lane-topology understanding. We propose a novel framework to integrate SD maps into online map prediction and propose a Transformer-based encoder, SD Map Encoder Representations from transFormers, to leverage priors in SD maps for the lane-topology prediction task. This enhancement consistently and significantly boosts (by up to 60%) lane detection and topology prediction on current state-of-the-art online map prediction methods without bells and whistles and can be immediately incorporated into any Transformer-based lane-topology method. Code is available at https://github.com/NVlabs/SMERF.
Energy-Calibrated VAE with Test Time Free Lunch
Authors: Yihong Luo, Siya Qiu, Xingjian Tao, Yujun Cai, Jing Tang
Date: 2023-11-07 00:00:00
Tasks: Image Generation, Image Restoration

In this paper, we propose a novel generative model that utilizes a conditional Energy-Based Model (EBM) for enhancing Variational Autoencoder (VAE), termed Energy-Calibrated VAE (EC-VAE). Specifically, VAEs often suffer from blurry generated samples due to the lack of a tailored training on the samples generated in the generative direction. On the other hand, EBMs can generate high-quality samples but require expensive Markov Chain Monte Carlo (MCMC) sampling. To address these issues, we introduce a conditional EBM for calibrating the generative direction of VAE during training, without requiring it for the generation at test time. In particular, we train EC-VAE upon both the input data and the calibrated samples with adaptive weight to enhance efficacy while avoiding MCMC sampling at test time. Furthermore, we extend the calibration idea of EC-VAE to variational learning and normalizing flows, and apply EC-VAE to an additional application of zero-shot image restoration via neural transport prior and range-null theory. We evaluate the proposed method with two applications, including image generation and zero-shot image restoration, and the experimental results show that our method achieves competitive performance over single-step non-adversarial generation. Our code is available at https://github.com/DJ-LYH/EC-VAE.
Debiased Fixed Effects Estimation of Binary Logit Models with Three-Dimensional Panel Data
Authors: Amrei Stammann
Date: 2023-11-07 00:00:00

Naive maximum likelihood estimation of binary logit models with fixed effects leads to unreliable inference due to the incidental parameter problem. We study the case of three-dimensional panel data, where the model includes three sets of additive and overlapping unobserved effects. This encompasses models for network panel data, where senders and receivers maintain bilateral relationships over time, and fixed effects account for unobserved heterogeneity at the sender-time, receiver-time, and sender-receiver levels. In an asymptotic framework, where all three panel dimensions grow large at constant relative rates, we characterize the leading bias of the naive estimator. The inference problem we identify is particularly severe, as it is not possible to balance the order of the bias and the standard deviation. As a consequence, the naive estimator has a degenerating asymptotic distribution, which exacerbates the inference problem relative to other fixed effects estimators studied in the literature. To resolve the inference problem, we derive explicit expressions to debias the fixed effects estimator.
Estimation and Inference for a Class of Generalized Hierarchical Models
Authors: Chaohua Dong, Jiti Gao, Bin Peng, Yayi Yan
Date: 2023-11-05 00:00:00

In this paper, we consider estimation and inference for the unknown parameters and function involved in a class of generalized hierarchical models. Such models are of great interest in the literature of neural networks (such as Bauer and Kohler, 2019). We propose a rectified linear unit (ReLU) based deep neural network (DNN) approach, and contribute to the design of DNN by i) providing more transparency for practical implementation, ii) defining different types of sparsity, iii) showing the differentiability, iv) pointing out the set of effective parameters, and v) offering a new variant of rectified linear activation function (ReLU), etc. Asymptotic properties are established accordingly, and a feasible procedure for the purpose of inference is also proposed. We conduct extensive numerical studies to examine the finite-sample performance of the estimation methods, and we also evaluate the empirical relevance and applicability of the proposed models and estimation methods to real data.
LISBET: a machine learning model for the automatic segmentation of social behavior motifs
Authors: Giuseppe Chindemi, Benoit Girard, Camilla Bellone
Date: 2023-11-07 00:00:00
Tasks: feature selection, Self-Supervised Learning

Social behavior is crucial for survival in many animal species, and a heavily investigated research subject. Current analysis methods generally rely on measuring animal interaction time or annotating predefined behaviors. However, these approaches are time consuming, human biased, and can fail to capture subtle behaviors. Here we introduce LISBET (LISBET Is a Social BEhavior Transformer), a machine learning model for detecting and segmenting social interactions. Using self-supervised learning on body tracking data, our model eliminates the need for extensive human annotation. We tested LISBET in three scenarios across multiple datasets in mice: supervised behavior classification, unsupervised motifs segmentation, and unsupervised animal phenotyping. Additionally, in vivo electrophysiology revealed distinct neural signatures in the Ventral Tegmental Area corresponding to motifs identified by our model. In summary, LISBET automates data annotation and reduces human bias in social behavior research, offering a promising approach to enhance our understanding of behavior and its neural correlates.
Optimal Estimation Methodologies for Panel Data Regression Models
Authors: Christis Katsouris
Date: 2023-11-06 00:00:00
Tasks: Econometrics, regression, Survey, Time Series

This survey study discusses main aspects to optimal estimation methodologies for panel data regression models. In particular, we present current methodological developments for modeling stationary panel data as well as robust methods for estimation and inference in nonstationary panel data regression models. Some applications from the network econometrics and high dimensional statistics literature are also discussed within a stationary time series environment.
The Fragility of Sparsity
Authors: Michal Kolesár, Ulrich K. Müller, Sebastian T. Roelsgaard
Date: 2023-11-04 00:00:00

We show, using three empirical applications, that linear regression estimates which rely on the assumption of sparsity are fragile in two ways. First, we document that different choices of the regressor matrix that do not impact ordinary least squares (OLS) estimates, such as the choice of baseline category with categorical controls, can move sparsity-based estimates by two standard errors or more. Second, we develop two tests of the sparsity assumption based on comparing sparsity-based estimators with OLS. The tests tend to reject the sparsity assumption in all three applications. Unless the number of regressors is comparable to or exceeds the sample size, OLS yields more robust inference at little efficiency cost.
mmFUSION: Multimodal Fusion for 3D Objects Detection
Authors: Javed Ahmad, Alessio Del Bue
Date: 2023-11-07 00:00:00
Tasks: 3D Object Detection, object-detection, Object Detection, Sensor Fusion

Multi-sensor fusion is essential for accurate 3D object detection in self-driving systems. Camera and LiDAR are the most commonly used sensors, and usually, their fusion happens at the early or late stages of 3D detectors with the help of regions of interest (RoIs). On the other hand, fusion at the intermediate level is more adaptive because it does not need RoIs from modalities but is complex as the features of both modalities are presented from different points of view. In this paper, we propose a new intermediate-level multi-modal fusion (mmFUSION) approach to overcome these challenges. First, the mmFUSION uses separate encoders for each modality to compute features at a desired lower space volume. Second, these features are fused through cross-modality and multi-modality attention mechanisms proposed in mmFUSION. The mmFUSION framework preserves multi-modal information and learns to complement modalities' deficiencies through attention weights. The strong multi-modal features from the mmFUSION framework are fed to a simple 3D detection head for 3D predictions. We evaluate mmFUSION on the KITTI and NuScenes dataset where it performs better than available early, intermediate, late, and even two-stage based fusion schemes. The code with the mmdetection3D project plugin will be publicly available soon.
Generative AIBIM: An automatic and intelligent structural design pipeline integrating BIM and generative AI
Authors: Zhili He, Yu-Hsing Wang, Jian Zhang
Date: 2023-11-07 00:00:00
Tasks: Generative Adversarial Network

AI-based structural design represents a transformative approach that addresses the inefficiencies inherent in traditional structural design practices. This paper innovates the existing AI-based design frameworks from four aspects and proposes Generative AIBIM: an intelligent design pipeline that integrates BIM and generative AI. First, the proposed pipeline not only broadens the application scope of BIM, which aligns with BIM's growing relevance in civil engineering, but also marks a significant supplement to previous methods that relied on CAD drawings. Second, a two-stage generation framework incorporating generative AI (TGAI), inspired by the human drawing process, is designed to simplify the complexity of the design problem. Third, for the AI model in TGAI, this paper pioneers to fuse physical conditions into diffusion models (DMs) to build a novel physics-based conditional diffusion model (PCDM). In contrast to conventional DMs, on the one hand, PCDM directly predicts shear wall drawings to focus on similarity, and on the other hand, PCDM effectively fuses cross-domain information, i.e., design drawings, timesteps, and physical conditions, by integrating well-designed attention modules. Additionally, a new evaluation system including objective and subjective measures is designed to evaluate models' performance, complementing the evaluation system in the traditional methods. The quantitative results demonstrate that PCDM significantly surpasses recent SOTA techniques across both measures. The qualitative results highlight PCDM's superior capabilities in generating high-perceptual-quality drawings adhering to essential design criteria. In addition, benefiting from the fusion of physical conditions, PCDM effectively supports diverse and creative designs tailored to building heights and seismic precautionary intensities, showcasing its unique generation and generalization abilities.
Pooled Bewley Estimator of Long Run Relationships in Dynamic Heterogenous Panels
Authors: Alexander Chudik, M. Hashem Pesaran, Ron P. Smith
Date: 2023-11-03 00:00:00

Using a transformation of the autoregressive distributed lag model due to Bewley, a novel pooled Bewley (PB) estimator of long-run coefficients for dynamic panels with heterogeneous short-run dynamics is proposed. The PB estimator is directly comparable to the widely used Pooled Mean Group (PMG) estimator, and is shown to be consistent and asymptotically normal. Monte Carlo simulations show good small sample performance of PB compared to the existing estimators in the literature, namely PMG, panel dynamic OLS (PDOLS), and panel fully-modified OLS (FMOLS). Application of two bias-correction methods and a bootstrapping of critical values to conduct inference robust to cross-sectional dependence of errors are also considered. The utility of the PB estimator is illustrated in an empirical application to the aggregate consumption function.
3D EAGAN: 3D edge-aware attention generative adversarial network for prostate segmentation in transrectal ultrasound images
Authors: Mengqing Liu, Xiao Shao, Liping Jiang, Kaizhi Wu
Date: 2023-11-07 00:00:00
Tasks: Decoder, Generative Adversarial Network, Segmentation

Automatic prostate segmentation in TRUS images has always been a challenging problem, since prostates in TRUS images have ambiguous boundaries and inhomogeneous intensity distribution. Although many prostate segmentation methods have been proposed, they still need to be improved due to the lack of sensibility to edge information. Consequently, the objective of this study is to devise a highly effective prostate segmentation method that overcomes these limitations and achieves accurate segmentation of prostates in TRUS images. A 3D edge-aware attention generative adversarial network (3D EAGAN)-based prostate segmentation method is proposed in this paper, which consists of an edge-aware segmentation network (EASNet) that performs the prostate segmentation and a discriminator network that distinguishes predicted prostates from real prostates. The proposed EASNet is composed of an encoder-decoder-based U-Net backbone network, a detail compensation module, four 3D spatial and channel attention modules, an edge enhance module, and a global feature extractor. The detail compensation module is proposed to compensate for the loss of detailed information caused by the down-sampling process of the encoder. The features of the detail compensation module are selectively enhanced by the 3D spatial and channel attention module. Furthermore, an edge enhance module is proposed to guide shallow layers in the EASNet to focus on contour and edge information in prostates. Finally, features from shallow layers and hierarchical features from the decoder module are fused through the global feature extractor to predict the segmentation prostates.
Analyzing Near-Infrared Hyperspectral Imaging for Protein Content Regression and Grain Variety Classification Using Bulk References and Varying Grain-to-Background Ratios
Authors: Ole-Christian Galbo Engstrøm, Erik Schou Dreier, Birthe Møller Jespersen, Kim Steenstrup Pedersen
Date: 2023-11-07 00:00:00

Based on previous work, we assess the use of NIR-HSI images for calibrating models on two datasets, focusing on protein content regression and grain variety classification. Limited reference data for protein content is expanded by subsampling and associating it with the bulk sample. However, this method introduces significant biases due to skewed leptokurtic prediction distributions, affecting both PLS-R and deep CNN models. We propose adjustments to mitigate these biases, improving mean protein reference predictions. Additionally, we investigate the impact of grain-to-background ratios on both tasks. Higher ratios yield more accurate predictions, but including lower-ratio images in calibration enhances model robustness for such scenarios.
Data exploitation: multi-task learning of object detection and semantic segmentation on partially annotated data
Authors: Hoàng-Ân Lê, Minh-Tan Pham
Date: 2023-11-07 00:00:00
Tasks: Knowledge Distillation, Multi-Task Learning, object-detection, Object Detection, Semantic Segmentation

Multi-task partially annotated data where each data point is annotated for only a single task are potentially helpful for data scarcity if a network can leverage the inter-task relationship. In this paper, we study the joint learning of object detection and semantic segmentation, the two most popular vision problems, from multi-task data with partial annotations. Extensive experiments are performed to evaluate each task performance and explore their complementarity when a multi-task network cannot optimize both tasks simultaneously. We propose employing knowledge distillation to leverage joint-task optimization. The experimental results show favorable results for multi-task learning and knowledge distillation over single-task learning and even full supervision scenario. All code and data splits are available at https://github.com/lhoangan/multas
Exploring Dataset-Scale Indicators of Data Quality
Authors: Benjamin Feuer, Chinmay Hegde
Date: 2023-11-07 00:00:00

Modern computer vision foundation models are trained on massive amounts of data, incurring large economic and environmental costs. Recent research has suggested that improving data quality can significantly reduce the need for data quantity. But what constitutes data quality in computer vision? We posit that the quality of a given dataset can be decomposed into distinct sample-level and dataset-level constituents, and that the former have been more extensively studied than the latter. We ablate the effects of two important dataset-level constituents: label set design, and class balance. By monitoring these constituents using key indicators we provide, researchers and practitioners can better anticipate model performance, measured in terms of its accuracy and robustness to distribution shifts.
AGNES: Abstraction-guided Framework for Deep Neural Networks Security
Authors: Akshay Dhonthi, Marcello Eiermann, Ernst Moritz Hahn, Vahid Hashemi
Date: 2023-11-07 00:00:00
Tasks: Autonomous Driving

Deep Neural Networks (DNNs) are becoming widespread, particularly in safety-critical areas. One prominent application is image recognition in autonomous driving, where the correct classification of objects, such as traffic signs, is essential for safe driving. Unfortunately, DNNs are prone to backdoors, meaning that they concentrate on attributes of the image that should be irrelevant for their correct classification. Backdoors are integrated into a DNN during training, either with malicious intent (such as a manipulated training process, because of which a yellow sticker always leads to a traffic sign being recognised as a stop sign) or unintentional (such as a rural background leading to any traffic sign being recognised as animal crossing, because of biased training data). In this paper, we introduce AGNES, a tool to detect backdoors in DNNs for image recognition. We discuss the principle approach on which AGNES is based. Afterwards, we show that our tool performs better than many state-of-the-art methods for multiple relevant case studies.