pwc-archive/papers-with-abstracts huggingface dataset:
Learning Fair Representations
Authors: Rich Zemel, Yu Wu, Kevin Swersky, Toni Pitassi, Cynthia Dwork
Date: 2013-06-17 00:00:00
Proceeding: International Conference on Machine Learning 2013 6
Tasks: Classification, Fairness, Transfer Learning

We propose a learning algorithm for fair classification that achieves both group fairness (the proportion of members in a protected group receiving positive classification is identical to the proportion in the population as a whole), and individual fairness (similar individuals should be treated similarly). We formulate fairness as an optimization problem of finding a good representation of the data with two competing goals: to encode the data as well as possible, while simultaneously obfuscating any information about membership in the protected group. We show positive results of our algorithm relative to other known techniques, on three datasets. Moreover, we demonstrate several advantages to our approach. First, our intermediate representation can be used for other classification tasks (i.e., transfer learning is possible); secondly, we take a step toward learning a distance metric which can find important dimensions of the data for classification.
Deep Feature Selection for Anomaly Detection Based on Pretrained Network and Gaussian Discriminative Analysis
Authors: Jie Lin, Song Chen, Enping Lin, Yu Yang
Date: 2022-09-12 00:00:00
Proceeding: IEEE Open Journal of Instrumentation and Measurement (Volume: 1) 2022 9
Tasks: Anomaly Detection, Fault Diagnosis, feature selection

Deep learning neural network serves as a powerful tool for visual anomaly detection (AD) and fault diagnosis, attributed to its strong abstractive interpretation ability in the representation domain. The deep features from neural networks that are pretrained on the ImageNet classification task have been proved to be useful for AD based on Gaussian discriminant analysis. However, with the ever-increasing complexity of deep learning neural networks, the set of deep features becomes massive where redundancy appears to be inevitable. The redundant features increase computational cost and degrade the performance of the AD method. In this article, we discuss the deep feature selection for the AD task and show how to reduce the redundancy in the representation domain. We propose a horizontal selection (dimensional reduction) method of features with subspace decomposition and a vertical selection to identify the most effective network layer for AD and fault diagnosis. We test the proposed method on two public datasets, one for AD task and the other for fault diagnosis of bearings. We show the significance of different network layers and feature subspaces on AD tasks and prove the effectiveness of the feature selection strategy.
Prior land surface reflectance-based sandstorm detection from space using deep learning
Authors: Yu Qu, Lin Sun, Qing hua Su, Nan Ma, Zhi hui Wang, Xi rong Liu
Date: 2022-09-13 00:00:00
Proceeding: Frontiers in Earth Science 2022 9

Traditional sandstorm detection methods use radiation differences among dust, underlying surface, and cloud to distinguish them by setting appropriate thresholds. Owing to the complex structure of the underlying surface, dust, and cloud, it is difficult to set a uniform threshold to achieve high-precision separation. Deep learning (DL) has powerful information mining capabilities and can fully use spectral differences between dust, land surface, and clouds. However, under the limited band information provided by satellite sensors, DL cannot easily distinguish highly heterogeneous land surfaces from multi-modal dust and cloud. This study proposes a sandstorm detection algorithm with DL supported by a land surface reflectance (LSR) dataset. The clear sky LSR dataset was obtained based on the MOD09A1 product. Based on the dataset, the difference between the reflectance observed by the satellite and the corresponding LSR is generated, which is used as a characteristic parameter of sandstorm detection with the deep learning method. The sandstorm detection of MODIS data is realized using multi-band radiation and radiation difference with DL. Results showed that the sandstorm detection algorithm used in this study was consistent with the OMI AI product with a detection accuracy of 84.6%. Compared with the detection results without the LSR dataset, this method effectively improves the accuracy of sandstorm identification.
Exploring Discrete Diffusion Models for Image Captioning
Authors: Zixin Zhu, Yixuan Wei, JianFeng Wang, Zhe Gan, Zheng Zhang, Le Wang, Gang Hua, Lijuan Wang, Zicheng Liu, Han Hu
Date: 2022-11-21 00:00:00
Tasks: Image Captioning, Image Generation

The image captioning task is typically realized by an auto-regressive method that decodes the text tokens one by one. We present a diffusion-based captioning model, dubbed the name DDCap, to allow more decoding flexibility. Unlike image generation, where the output is continuous and redundant with a fixed length, texts in image captions are categorical and short with varied lengths. Therefore, naively applying the discrete diffusion model to text decoding does not work well, as shown in our experiments. To address the performance gap, we propose several key techniques including best-first inference, concentrated attention mask, text length prediction, and image-free training. On COCO without additional caption pre-training, it achieves a CIDEr score of 117.8, which is +5.0 higher than the auto-regressive baseline with the same architecture in the controlled setting. It also performs +26.8 higher CIDEr score than the auto-regressive baseline (230.3 v.s.203.5) on a caption infilling task. With 4M vision-language pre-training images and the base-sized model, we reach a CIDEr score of 125.1 on COCO, which is competitive to the best well-developed auto-regressive frameworks. The code is available at https://github.com/buxiangzhiren/DDCap.
Simultaneous Multiple Object Detection and Pose Estimation using 3D Model Infusion with Monocular Vision
Authors: Congliang Li, ShiJie Sun, XiangYu Song, HuanSheng Song, Naveed Akhtar, Ajmal Saeed Mian
Date: 2022-11-21 00:00:00
Tasks: Autonomous Driving, Object, object-detection, Object Detection, Pose Estimation

Multiple object detection and pose estimation are vital computer vision tasks. The latter relates to the former as a downstream problem in applications such as robotics and autonomous driving. However, due to the high complexity of both tasks, existing methods generally treat them independently, which is sub-optimal. We propose simultaneous neural modeling of both using monocular vision and 3D model infusion. Our Simultaneous Multiple Object detection and Pose Estimation network (SMOPE-Net) is an end-to-end trainable multitasking network with a composite loss that also provides the advantages of anchor-free detections for efficient downstream pose estimation. To enable the annotation of training data for our learning objective, we develop a Twin-Space object labeling method and demonstrate its correctness analytically and empirically. Using the labeling method, we provide the KITTI-6DoF dataset with $\sim7.5$K annotated frames. Extensive experiments on KITTI-6DoF and the popular LineMod datasets show a consistent performance gain with SMOPE-Net over existing pose estimation methods. Here are links to our proposed SMOPE-Net, KITTI-6DoF dataset, and LabelImg3D labeling tool.
A Transformer Framework for Data Fusion and Multi-Task Learning in Smart Cities
Authors: Alexander C. DeRieux, Walid Saad, Wangda Zuo, Rachmawan Budiarto, Mochamad Donny Koerniawan, Dwi Novitasari
Date: 2022-11-18 00:00:00
Tasks: Computational Efficiency, Humanitarian, Image Classification, Multi-Task Learning, Multivariate Time Series Forecasting, regression, Time Series, Time Series Analysis, Time Series Regression

Rapid global urbanization is a double-edged sword, heralding promises of economical prosperity and public health while also posing unique environmental and humanitarian challenges. Smart and connected communities (S&CCs) apply data-centric solutions to these problems by integrating artificial intelligence (AI) and the Internet of Things (IoT). This coupling of intelligent technologies also poses interesting system design challenges regarding heterogeneous data fusion and task diversity. Transformers are of particular interest to address these problems, given their success across diverse fields of natural language processing (NLP), computer vision, time-series regression, and multi-modal data fusion. This begs the question whether Transformers can be further diversified to leverage fusions of IoT data sources for heterogeneous multi-task learning in S&CC trade spaces. In this paper, a Transformer-based AI system for emerging smart cities is proposed. Designed using a pure encoder backbone, and further customized through interchangeable input embedding and output task heads, the system supports virtually any input data and output task types present S&CCs. This generalizability is demonstrated through learning diverse task sets representative of S&CC environments, including multivariate time-series regression, visual plant disease classification, and image-time-series fusion tasks using a combination of Beijing PM2.5 and Plant Village datasets. Simulation results show that the proposed Transformer-based system can handle various input data types via custom sequence embedding techniques, and are naturally suited to learning a diverse set of tasks. The results also show that multi-task learners increase both memory and computational efficiency while maintaining comparable performance to both single-task variants, and non-Transformer baselines.
Exploring the Efficacy of Pre-trained Checkpoints in Text-to-Music Generation Task
Authors: Shangda Wu, Maosong Sun
Date: 2022-11-21 00:00:00
Tasks: Music Generation, Text-to-Music Generation

Benefiting from large-scale datasets and pre-trained models, the field of generative models has recently gained significant momentum. However, most datasets for symbolic music are very small, which potentially limits the performance of data-driven multimodal models. An intuitive solution to this problem is to leverage pre-trained models from other modalities (e.g., natural language) to improve the performance of symbolic music-related multimodal tasks. In this paper, we carry out the first study of generating complete and semantically consistent symbolic music scores from text descriptions, and explore the efficacy of using publicly available checkpoints (i.e., BERT, GPT-2, and BART) for natural language processing in the task of text-to-music generation. Our experimental results show that the improvement from using pre-trained checkpoints is statistically significant in terms of BLEU score and edit distance similarity. We analyse the capabilities and limitations of our model to better understand the potential of language-music models.
Aircraft engine remaining useful life estimation via a double attention-based data-driven architecture
Authors: Lu liu;Xiao Song;Zhetao Zhou;
Date: 2022-05-01 00:00:00
Proceeding: RESS 2022 5
Tasks: Management

Remaining useful life (RUL) estimation has been intensively studied, given its important role in prognostics and health management (PHM) of industry. Recently, data-driven structures such as convolutional neural networks (CNNs), have achieved outstanding RUL prediction performance. However, conventional CNNs do not include an adequate mechanism for adaptively weighing input features. In this paper, we propose a double attention-based data-driven framework for aircraft engine RUL prognostics. Specifically, a channel attention-based CNN was utilized to apply greater weights to more significant features. Next, a Transformer was used to focus attention on these features at critical time steps. We validated the effectiveness of the proposed framework on benchmark datasets for aircraft engine RUL estimation. The experimental results indicate that the proposed double attention-based architecture outperformed the existing state-of-the-art (SOTA) algorithms. The double attention-based RUL prediction method can detect the risk of equipment failure and reduce loss.
Diffusion Denoising Process for Perceptron Bias in Out-of-distribution Detection
Authors: Luping Liu, Yi Ren, Xize Cheng, Rongjie Huang, Chongxuan Li, Zhou Zhao
Date: 2022-11-21 00:00:00
Tasks: Denoising, Out-of-Distribution Detection, Out of Distribution (OOD) Detection

Out-of-distribution (OOD) detection is a crucial task for ensuring the reliability and safety of deep learning. Currently, discriminator models outperform other methods in this regard. However, the feature extraction process used by discriminator models suffers from the loss of critical information, leaving room for bad cases and malicious attacks. In this paper, we introduce a new perceptron bias assumption that suggests discriminator models are more sensitive to certain features of the input, leading to the overconfidence problem. To address this issue, we propose a novel framework that combines discriminator and generation models and integrates diffusion models (DMs) into OOD detection. We demonstrate that the diffusion denoising process (DDP) of DMs serves as a novel form of asymmetric interpolation, which is well-suited to enhance the input and mitigate the overconfidence problem. The discriminator model features of OOD data exhibit sharp changes under DDP, and we utilize the norm of this change as the indicator score. Our experiments on CIFAR10, CIFAR100, and ImageNet show that our method outperforms SOTA approaches. Notably, for the challenging InD ImageNet and OOD species datasets, our method achieves an AUROC of 85.7, surpassing the previous SOTA method's score of 77.4. Our implementation is available at \url{https://github.com/luping-liu/DiffOOD}.
VER: Unifying Verbalizing Entities and Relations
Authors: Jie Huang, Kevin Chen-Chuan Chang
Date: 2022-11-20 00:00:00
Tasks: Sentence

Entities and relationships between entities are vital in the real world. Essentially, we understand the world by understanding entities and relations. For instance, to understand a field, e.g., computer science, we need to understand the relevant concepts, e.g., machine learning, and the relationships between concepts, e.g., machine learning and artificial intelligence. To understand a person, we should first know who he/she is and how he/she is related to others. To understand entities and relations, humans may refer to natural language descriptions. For instance, when learning a new scientific term, people usually start by reading its definition in dictionaries or encyclopedias. To know the relationship between two entities, humans tend to create a sentence to connect them. In this paper, we propose VER: a unified model for Verbalizing Entities and Relations. Specifically, we attempt to build a system that takes any entity or entity set as input and generates a sentence to represent entities and relations. Extensive experiments demonstrate that our model can generate high-quality sentences describing entities and entity relationships and facilitate various tasks on entities and relations, including definition modeling, relation modeling, and generative commonsense reasoning.
LibSignal: An Open Library for Traffic Signal Control
Authors: Hao Mei, Xiaoliang Lei, Longchao Da, Bin Shi, Hua Wei
Date: 2022-11-19 00:00:00
Tasks: reinforcement-learning, Reinforcement Learning, Reinforcement Learning (RL), Traffic Signal Control

This paper introduces a library for cross-simulator comparison of reinforcement learning models in traffic signal control tasks. This library is developed to implement recent state-of-the-art reinforcement learning models with extensible interfaces and unified cross-simulator evaluation metrics. It supports commonly-used simulators in traffic signal control tasks, including Simulation of Urban MObility(SUMO) and CityFlow, and multiple benchmark datasets for fair comparisons. We conducted experiments to validate our implementation of the models and to calibrate the simulators so that the experiments from one simulator could be referential to the other. Based on the validated models and calibrated environments, this paper compares and reports the performance of current state-of-the-art RL algorithms across different datasets and simulators. This is the first time that these methods have been compared fairly under the same datasets with different simulators.
Multi-Level Knowledge Distillation for Out-of-Distribution Detection in Text
Authors: Qianhui Wu, Huiqiang Jiang, Haonan Yin, Börje F. Karlsson, Chin-Yew Lin
Date: 2022-11-21 00:00:00
Tasks: Knowledge Distillation, Language Modeling, Language Modelling, Out-of-Distribution Detection, Out of Distribution (OOD) Detection, Representation Learning

Self-supervised representation learning has proved to be a valuable component for out-of-distribution (OoD) detection with only the texts of in-distribution (ID) examples. These approaches either train a language model from scratch or fine-tune a pre-trained language model using ID examples, and then take the perplexity output by the language model as OoD scores. In this paper, we analyze the complementary characteristics of both OoD detection methods and propose a multi-level knowledge distillation approach that integrates their strengths while mitigating their limitations. Specifically, we use a fine-tuned model as the teacher to teach a randomly initialized student model on the ID examples. Besides the prediction layer distillation, we present a similarity-based intermediate layer distillation method to thoroughly explore the representation space of the teacher model. In this way, the learned student can better represent the ID data manifold while gaining a stronger ability to map OoD examples outside the ID data manifold with the regularization inherited from pre-training. Besides, the student model sees only ID examples during parameter learning, further promoting more distinguishable features for OoD detection. We conduct extensive experiments over multiple benchmark datasets, i.e., CLINC150, SST, ROSTD, 20 NewsGroups, and AG News; showing that the proposed method yields new state-of-the-art performance. We also explore its application as an AIGC detector to distinguish between answers generated by ChatGPT and human experts. It is observed that our model exceeds human evaluators in the pair-expert task on the Human ChatGPT Comparison Corpus.
DS-1000: A Natural and Reliable Benchmark for Data Science Code Generation
Authors: Yuhang Lai, Chengxi Li, Yiming Wang, Tianyi Zhang, Ruiqi Zhong, Luke Zettlemoyer, Scott Wen-tau Yih, Daniel Fried, Sida Wang, Tao Yu
Date: 2022-11-18 00:00:00
Tasks: Code Generation, Memorization

We introduce DS-1000, a code generation benchmark with a thousand data science problems spanning seven Python libraries, such as NumPy and Pandas. Compared to prior works, DS-1000 incorporates three core features. First, our problems reflect diverse, realistic, and practical use cases since we collected them from StackOverflow. Second, our automatic evaluation is highly specific (reliable) -- across all Codex-002-predicted solutions that our evaluation accept, only 1.8% of them are incorrect; we achieve this with multi-criteria metrics, checking both functional correctness by running test cases and surface-form constraints by restricting API usages or keywords. Finally, we proactively defend against memorization by slightly modifying our problems to be different from the original StackOverflow source; consequently, models cannot answer them correctly by memorizing the solutions from pre-training. The current best public system (Codex-002) achieves 43.3% accuracy, leaving ample room for improvement. We release our benchmark at https://ds1000-code-gen.github.io.
First Steps Toward Understanding the Extrapolation of Nonlinear Models to Unseen Domains
Authors: Kefan Dong, Tengyu Ma
Date: 2022-11-21 00:00:00

Real-world machine learning applications often involve deploying neural networks to domains that are not seen in the training time. Hence, we need to understand the extrapolation of nonlinear models -- under what conditions on the distributions and function class, models can be guaranteed to extrapolate to new test distributions. The question is very challenging because even two-layer neural networks cannot be guaranteed to extrapolate outside the support of the training distribution without further assumptions on the domain shift. This paper makes some initial steps toward analyzing the extrapolation of nonlinear models for structured domain shift. We primarily consider settings where the marginal distribution of each coordinate of the data (or subset of coordinates) does not shift significantly across the training and test distributions, but the joint distribution may have a much bigger shift. We prove that the family of nonlinear models of the form $f(x)=\sum f_i(x_i)$, where $f_i$ is an arbitrary function on the subset of features $x_i$, can extrapolate to unseen distributions, if the covariance of the features is well-conditioned. To the best of our knowledge, this is the first result that goes beyond linear models and the bounded density ratio assumption, even though the assumptions on the distribution shift and function class are stylized.
Neural tangent kernel analysis of PINN for advection-diffusion equation
Authors: M. H. Saadat, B. Gjorgiev, L. Das, G. Sansavini
Date: 2022-11-21 00:00:00

Physics-informed neural networks (PINNs) numerically approximate the solution of a partial differential equation (PDE) by incorporating the residual of the PDE along with its initial/boundary conditions into the loss function. In spite of their partial success, PINNs are known to struggle even in simple cases where the closed-form analytical solution is available. In order to better understand the learning mechanism of PINNs, this work focuses on a systematic analysis of PINNs for the linear advection-diffusion equation (LAD) using the Neural Tangent Kernel (NTK) theory. Thanks to the NTK analysis, the effects of the advection speed/diffusion parameter on the training dynamics of PINNs are studied and clarified. We show that the training difficulty of PINNs is a result of 1) the so-called spectral bias, which leads to difficulty in learning high-frequency behaviours; and 2) convergence rate disparity between different loss components that results in training failure. The latter occurs even in the cases where the solution of the underlying PDE does not exhibit high-frequency behaviour. Furthermore, we observe that this training difficulty manifests itself, to some extent, differently in advection-dominated and diffusion-dominated regimes. Different strategies to address these issues are also discussed. In particular, it is demonstrated that periodic activation functions can be used to partly resolve the spectral bias issue.