pwc-archive/papers-with-abstracts huggingface dataset:
A path algorithm for the Fused Lasso Signal Approximator
Authors: Holger Hoefling
Date: 2009-10-03 00:00:00
Tasks: regression

The Lasso is a very well known penalized regression model, which adds an $L_{1}$ penalty with parameter $\lambda_{1}$ on the coefficients to the squared error loss function. The Fused Lasso extends this model by also putting an $L_{1}$ penalty with parameter $\lambda_{2}$ on the difference of neighboring coefficients, assuming there is a natural ordering. In this paper, we develop a fast path algorithm for solving the Fused Lasso Signal Approximator that computes the solutions for all values of $\lambda_1$ and $\lambda_2$. In the supplement, we also give an algorithm for the general Fused Lasso for the case with predictor matrix $\bX \in \mathds{R}^{n \times p}$ with $\text{rank}(\bX)=p$.
MBGDT:Robust Mini-Batch Gradient Descent
Authors: Hanming Wang, Haozheng Luo, Yue Wang
Date: 2022-06-14 00:00:00
Tasks: regression

In high dimensions, most machine learning method perform fragile even there are a little outliers. To address this, we hope to introduce a new method with the base learner, such as Bayesian regression or stochastic gradient descent to solve the problem of the vulnerability in the model. Because the mini-batch gradient descent allows for a more robust convergence than the batch gradient descent, we work a method with the mini-batch gradient descent, called Mini-Batch Gradient Descent with Trimming (MBGDT). Our method show state-of-art performance and have greater robustness than several baselines when we apply our method in designed dataset.
Lazy Queries Can Reduce Variance in Zeroth-order Optimization
Authors: Quan Xiao, Qing Ling, Tianyi Chen
Date: 2022-06-14 00:00:00

A major challenge of applying zeroth-order (ZO) methods is the high query complexity, especially when queries are costly. We propose a novel gradient estimation technique for ZO methods based on adaptive lazy queries that we term as LAZO. Different from the classic one-point or two-point gradient estimation methods, LAZO develops two alternative ways to check the usefulness of old queries from previous iterations, and then adaptively reuses them to construct the low-variance gradient estimates. We rigorously establish that through judiciously reusing the old queries, LAZO can reduce the variance of stochastic gradient estimates so that it not only saves queries per iteration but also achieves the regret bound for the symmetric two-point method. We evaluate the numerical performance of LAZO, and demonstrate the low-variance property and the performance gain of LAZO in both regret and query complexity relative to several existing ZO methods. The idea of LAZO is general, and can be applied to other variants of ZO methods.
Proximal Splitting Adversarial Attacks for Semantic Segmentation
Authors: Jérôme Rony, Jean-Christophe Pesquet, Ismail Ben Ayed
Date: 2022-06-14 00:00:00
Tasks: Adversarial Attack, Segmentation, Semantic Segmentation

Classification has been the focal point of research on adversarial attacks, but only a few works investigate methods suited to denser prediction tasks, such as semantic segmentation. The methods proposed in these works do not accurately solve the adversarial segmentation problem and, therefore, overestimate the size of the perturbations required to fool models. Here, we propose a white-box attack for these models based on a proximal splitting to produce adversarial perturbations with much smaller $\ell_\infty$ norms. Our attack can handle large numbers of constraints within a nonconvex minimization framework via an Augmented Lagrangian approach, coupled with adaptive constraint scaling and masking strategies. We demonstrate that our attack significantly outperforms previously proposed ones, as well as classification attacks that we adapted for segmentation, providing a first comprehensive benchmark for this dense task.
FleetX
Authors: EW
Date: 2021-12-06 00:00:00
Proceeding: arXiv 2021 12
Tasks: CPU, Distributed Computing, GPU, Language Modelling, Recommendation Systems, Scheduling

Distributed training has become a pervasive and effective approach for training a large neural network
(NN) model with processing massive data. However, it is very challenging to satisfy requirements
from various NN models, diverse computing resources, and their dynamic changes during a training
job. In this study, we design our distributed training framework in a systematic end-to-end view to
provide the built-in adaptive ability for different scenarios, especially for industrial applications and
production environments, by fully considering resource allocation, model partition, task placement,
and distributed execution. Based on the unified distributed graph and the unified cluster object,
our adaptive framework is equipped with a global cost model and a global planner, which can
enable arbitrary parallelism, resource-aware placement, multi-mode execution, fault-tolerant, and
elastic distributed training. The experiments demonstrate that our framework can satisfy various
requirements from the diversity of applications and the heterogeneity of resources with highly
competitive performance. The ERNIE language model with 260 billion parameters is efficiently
trained on thousands of AI processors with 91.7% weak scalability. The throughput of the model
from the recommender system by employing the heterogeneous pipeline asynchronous execution can
be increased up to 2.1 times and 3.3 times that of the GPU-only and CPU-only training respectively.
Moreover, the fault-tolerant and elastic distributed training have been successfully applied to the
online industrial applications, which give a reduction of 34.49% in the number of failed long-term
training jobs and an increase of 33.91% for the global scheduling efficiency in the production
environment.
PolyU-BPCoMa: A Dataset and Benchmark Towards Mobile Colorized Mapping Using a Backpack Multisensorial System
Authors: Wenzhong Shi, Pengxin Chen, Muyang Wang, Sheng Bao, Haodong Xiang, Yue Yu, Daping Yang
Date: 2022-06-15 00:00:00
Tasks: Colorization

Constructing colorized point clouds from mobile laser scanning and images is a fundamental work in surveying and mapping. It is also an essential prerequisite for building digital twins for smart cities. However, existing public datasets are either in relatively small scales or lack accurate geometrical and color ground truth. This paper documents a multisensorial dataset named PolyU-BPCoMA which is distinctively positioned towards mobile colorized mapping. The dataset incorporates resources of 3D LiDAR, spherical imaging, GNSS and IMU on a backpack platform. Color checker boards are pasted in each surveyed area as targets and ground truth data are collected by an advanced terrestrial laser scanner (TLS). 3D geometrical and color information can be recovered in the colorized point clouds produced by the backpack system and the TLS, respectively. Accordingly, we provide an opportunity to benchmark the mapping and colorization accuracy simultaneously for a mobile multisensorial system. The dataset is approximately 800 GB in size covering both indoor and outdoor environments. The dataset and development kits are available at https://github.com/chenpengxin/PolyU-BPCoMa.git.
Benefits of Additive Noise in Composing Classes with Bounded Capacity
Authors: Alireza Fathollah Pour, Hassan Ashtiani
Date: 2022-06-14 00:00:00

We observe that given two (compatible) classes of functions $\mathcal{F}$ and $\mathcal{H}$ with small capacity as measured by their uniform covering numbers, the capacity of the composition class $\mathcal{H} \circ \mathcal{F}$ can become prohibitively large or even unbounded. We then show that adding a small amount of Gaussian noise to the output of $\mathcal{F}$ before composing it with $\mathcal{H}$ can effectively control the capacity of $\mathcal{H} \circ \mathcal{F}$, offering a general recipe for modular design. To prove our results, we define new notions of uniform covering number of random functions with respect to the total variation and Wasserstein distances. We instantiate our results for the case of multi-layer sigmoid neural networks. Preliminary empirical results on MNIST dataset indicate that the amount of noise required to improve over existing uniform bounds can be numerically negligible (i.e., element-wise i.i.d. Gaussian noise with standard deviation $10^{-240}$). The source codes are available at https://github.com/fathollahpour/composition_noise.
TA-SBERT: Token Attention Sentence-BERT for Improving Sentence Representation
Authors: Wonik Choi
Date: 2022-03-16 00:00:00
Proceeding: IEEE Access 2022 3
Tasks: Language Modeling, Language Modelling, Semantic Textual Similarity, Sentence, Sentence Embedding, Sentence-Embedding, STS

ABSTRACT A sentence embedding vector can be obtained by connecting a global average pooling (GAP)
to a pre-trained language model. The problem of such a sentence embedding vector using a GAP is that
it is generated with the same weight for all words appearing in the sentence. We propose a novel sentence
embedding-method-based model Token Attention-SentenceBERT (TA-SBERT) to address this problem. The
rationale of TA-SBERT is to enhance the performance of sentence embedding by introducing three strategies.
First, we convert the base form while preprocessing the input sentence to reduce misunderstanding. Second,
we propose a novel Token Attention (TA) technique that distinguishes important words to produce more
informative sentence vectors. Third, we increase stability of fine-tuning to avoid catastrophic forgetting
by adding a reconstruction loss to the word embedding vector. Extensive ablation studies demonstrate that
our TA-SBERT outperforms the original SentenceBERT (SBERT) in the sentence vector evaluation using
semantic textual similarity (STS) tasks and the SentEval toolkit.
On the Eve of True Explainability for OWL Ontologies: Description Logic Proofs with Evee and Evonne (Extended Version)
Authors: Christian Alrabbaa, Stefan Borgwardt, Tom Friese, Patrick Koopmann, Julián Méndez, Alexej Popovič
Date: 2022-06-15 00:00:00

When working with description logic ontologies, understanding entailments derived by a description logic reasoner is not always straightforward. So far, the standard ontology editor Prot\'eg\'e offers two services to help: (black-box) justifications for OWL 2 DL ontologies, and (glass-box) proofs for lightweight OWL EL ontologies, where the latter exploits the proof facilities of reasoner ELK. Since justifications are often insufficient in explaining inferences, there is thus only little tool support for explaining inferences in more expressive DLs. In this paper, we introduce EVEE-LIBS, a Java library for computing proofs for DLs up to ALCH, and EVEE-PROTEGE, a collection of Prot\'eg\'e plugins for displaying those proofs in Prot\'eg\'e. We also give a short glimpse of the latest version of EVONNE, a more advanced standalone application for displaying and interacting with proofs computed with EVEE-LIBS.
Masked Siamese ConvNets
Authors: Li Jing, Jiachen Zhu, Yann Lecun
Date: 2022-06-15 00:00:00
Tasks: image-classification, Image Classification, Inductive Bias, object-detection, Object Detection, Representation Learning, Self-Supervised Learning

Self-supervised learning has shown superior performances over supervised methods on various vision benchmarks. The siamese network, which encourages embeddings to be invariant to distortions, is one of the most successful self-supervised visual representation learning approaches. Among all the augmentation methods, masking is the most general and straightforward method that has the potential to be applied to all kinds of input and requires the least amount of domain knowledge. However, masked siamese networks require particular inductive bias and practically only work well with Vision Transformers. This work empirically studies the problems behind masked siamese networks with ConvNets. We propose several empirical designs to overcome these problems gradually. Our method performs competitively on low-shot image classification and outperforms previous methods on object detection benchmarks. We discuss several remaining issues and hope this work can provide useful data points for future general-purpose self-supervised learning.
Region-enhanced Deep Graph Convolutional Networks for Rumor Detection
Authors: Ge Wang, Li Tan, Tianbao Song, Wei Wang, Ziliang Shang
Date: 2022-06-15 00:00:00
Tasks: Graph Neural Network

Social media has been rapidly developing in the public sphere due to its ease of spreading new information, which leads to the circulation of rumors. However, detecting rumors from such a massive amount of information is becoming an increasingly arduous challenge. Previous work generally obtained valuable features from propagation information. It should be noted that most methods only target the propagation structure while ignoring the rumor transmission pattern. This limited focus severely restricts the collection of spread data. To solve this problem, the authors of the present study are motivated to explore the regionalized propagation patterns of rumors. Specifically, a novel region-enhanced deep graph convolutional network (RDGCN) that enhances the propagation features of rumors by learning regionalized propagation patterns and trains to learn the propagation patterns by unsupervised learning is proposed. In addition, a source-enhanced residual graph convolution layer (SRGCL) is designed to improve the graph neural network (GNN) oversmoothness and increase the depth limit of the rumor detection methods-based GNN. Experiments on Twitter15 and Twitter16 show that the proposed model performs better than the baseline approach on rumor detection and early rumor detection.
Model-based RL with Optimistic Posterior Sampling: Structural Conditions and Sample Complexity
Authors: Alekh Agarwal, Tong Zhang
Date: 2022-06-15 00:00:00

We propose a general framework to design posterior sampling methods for model-based RL. We show that the proposed algorithms can be analyzed by reducing regret to Hellinger distance in conditional probability estimation. We further show that optimistic posterior sampling can control this Hellinger distance, when we measure model error via data likelihood. This technique allows us to design and analyze unified posterior sampling algorithms with state-of-the-art sample complexity guarantees for many model-based RL settings. We illustrate our general result in many special cases, demonstrating the versatility of our framework.
Hyperparameter Sensitivity in Deep Outlier Detection: Analysis and a Scalable Hyper-Ensemble Solution
Authors: Xueying Ding, Lingxiao Zhao, Leman Akoglu
Date: 2022-06-15 00:00:00
Tasks: Model Selection, Outlier Detection, Representation Learning, Sensitivity

Outlier detection (OD) literature exhibits numerous algorithms as it applies to diverse domains. However, given a new detection task, it is unclear how to choose an algorithm to use, nor how to set its hyperparameter(s) (HPs) in unsupervised settings. HP tuning is an ever-growing problem with the arrival of many new detectors based on deep learning, which usually come with a long list of HPs. Surprisingly, the issue of model selection in the outlier mining literature has been "the elephant in the room"; a significant factor in unlocking the utmost potential of deep methods, yet little said or done to systematically tackle the issue. In the first part of this paper, we conduct the first large-scale analysis on the HP sensitivity of deep OD methods, and through more than 35,000 trained models, quantitatively demonstrate that model selection is inevitable. Next, we design a HP-robust and scalable deep hyper-ensemble model called ROBOD that assembles models with varying HP configurations, bypassing the choice paralysis. Importantly, we introduce novel strategies to speed up ensemble training, such as parameter sharing, batch/simultaneous training, and data subsampling, that allow us to train fewer models with fewer parameters. Extensive experiments on both image and tabular datasets show that ROBOD achieves and retains robust, state-of-the-art detection performance as compared to its modern counterparts, while taking only $2$-$10$\% of the time by the naive hyper-ensemble with independent training.
Convergence and Price of Anarchy Guarantees of the Softmax Policy Gradient in Markov Potential Games
Authors: Dingyang Chen, Qi Zhang, Thinh T. Doan
Date: 2022-06-15 00:00:00
Tasks: Policy Gradient Methods

We study the performance of policy gradient methods for the subclass of Markov games known as Markov potential games (MPGs), which extends the notion of normal-form potential games to the stateful setting and includes the important special case of the fully cooperative setting where the agents share an identical reward function. Our focus in this paper is to study the convergence of the policy gradient method for solving MPGs under softmax policy parameterization, both tabular and parameterized with general function approximators such as neural networks. We first show the asymptotic convergence of this method to a Nash equilibrium of MPGs for tabular softmax policies. Second, we derive the finite-time performance of the policy gradient in two settings: 1) using the log-barrier regularization, and 2) using the natural policy gradient under the best-response dynamics (NPG-BR). Finally, extending the notion of price of anarchy (POA) and smoothness in normal-form games, we introduce the POA for MPGs and provide a POA bound for NPG-BR. To our knowledge, this is the first POA bound for solving MPGs. To support our theoretical results, we empirically compare the convergence rates and POA of policy gradient variants for both tabular and neural softmax policies.
AI Ethics Issues in Real World: Evidence from AI Incident Database
Authors: Mengyi Wei, Zhixuan Zhou
Date: 2022-06-15 00:00:00
Tasks: Autonomous Driving, Ethics

With the powerful performance of Artificial Intelligence (AI) also comes prevalent ethical issues. Though governments and corporations have curated multiple AI ethics guidelines to curb unethical behavior of AI, the effect has been limited, probably due to the vagueness of the guidelines. In this paper, we take a closer look at how AI ethics issues take place in real world, in order to have a more in-depth and nuanced understanding of different ethical issues as well as their social impact. With a content analysis of AI Incident Database, which is an effort to prevent repeated real world AI failures by cataloging incidents, we identified 13 application areas which often see unethical use of AI, with intelligent service robots, language/vision models and autonomous driving taking the lead. Ethical issues appear in 8 different forms, from inappropriate use and racial discrimination, to physical safety and unfair algorithm. With this taxonomy of AI ethics issues, we aim to provide AI practitioners with a practical guideline when trying to deploy AI applications ethically.
Real3D-Aug: Point Cloud Augmentation by Placing Real Objects with Occlusion Handling for 3D Detection and Segmentation
Authors: Petr Šebek, Šimon Pokorný, Patrik Vacek, Tomáš Svoboda
Date: 2022-06-15 00:00:00
Tasks: 3D Object Detection, Data Augmentation, Object, object-detection, Object Detection, Occlusion Handling, Segmentation, Semantic Segmentation

Object detection and semantic segmentation with the 3D lidar point cloud data require expensive annotation. We propose a data augmentation method that takes advantage of already annotated data multiple times. We propose an augmentation framework that reuses real data, automatically finds suitable placements in the scene to be augmented, and handles occlusions explicitly. Due to the usage of the real data, the scan points of newly inserted objects in augmentation sustain the physical characteristics of the lidar, such as intensity and raydrop. The pipeline proves competitive in training top-performing models for 3D object detection and semantic segmentation. The new augmentation provides a significant performance gain in rare and essential classes, notably 6.65% average precision gain for "Hard" pedestrian class in KITTI object detection or 2.14 mean IoU gain in the SemanticKITTI segmentation challenge over the state of the art.
ARES: Locally Adaptive Reconstruction-based Anomaly Scoring
Authors: Adam Goodge, Bryan Hooi, See Kiong Ng, Wee Siong Ng
Date: 2022-06-15 00:00:00
Tasks: Anomaly Detection, Dimensionality Reduction

How can we detect anomalies: that is, samples that significantly differ from a given set of high-dimensional data, such as images or sensor data? This is a practical problem with numerous applications and is also relevant to the goal of making learning algorithms more robust to unexpected inputs. Autoencoders are a popular approach, partly due to their simplicity and their ability to perform dimension reduction. However, the anomaly scoring function is not adaptive to the natural variation in reconstruction error across the range of normal samples, which hinders their ability to detect real anomalies. In this paper, we empirically demonstrate the importance of local adaptivity for anomaly scoring in experiments with real data. We then propose our novel Adaptive Reconstruction Error-based Scoring approach, which adapts its scoring based on the local behaviour of reconstruction error over the latent space. We show that this improves anomaly detection performance over relevant baselines in a wide variety of benchmark datasets.
Theory of Machine Learning with Limited Data
Authors: Marina Sapir
Date: 2022-06-15 00:00:00
Tasks: BIG-bench Machine Learning, Clustering, Decision Making, Learning Theory, regression

Application of machine learning may be understood as deriving new knowledge for practical use through explaining accumulated observations, training set. Peirce used the term abduction for this kind of inference. Here I formalize the concept of abduction for real valued hypotheses, and show that 14 of the most popular textbook ML learners (every learner I tested), covering classification, regression and clustering, implement this concept of abduction inference. The approach is proposed as an alternative to statistical learning theory, which requires an impractical assumption of indefinitely increasing training set for its justification.
NatGen: Generative pre-training by "Naturalizing" source code
Authors: Saikat Chakraborty, Toufique Ahmed, Yangruibo Ding, Premkumar Devanbu, Baishakhi Ray
Date: 2022-06-15 00:00:00
Tasks: Code Generation, Code Translation, Few-Shot Learning, Translation

Pre-trained Generative Language models (e.g. PLBART, CodeT5, SPT-Code) for source code yielded strong results on several tasks in the past few years, including code generation and translation. These models have adopted varying pre-training objectives to learn statistics of code construction from very large-scale corpora in a self-supervised fashion; the success of pre-trained models largely hinges on these pre-training objectives. This paper proposes a new pre-training objective, "Naturalizing" of source code, exploiting code's bimodal, dual-channel (formal & natural channels) nature. Unlike natural language, code's bimodal, dual-channel nature allows us to generate semantically equivalent code at scale. We introduce six classes of semantic preserving transformations to introduce un-natural forms of code, and then force our model to produce more natural original programs written by developers. Learning to generate equivalent, but more natural code, at scale, over large corpora of open-source code, without explicit manual supervision, helps the model learn to both ingest & generate code. We fine-tune our model in three generative Software Engineering tasks: code generation, code translation, and code refinement with limited human-curated labeled data and achieve state-of-the-art performance rivaling CodeT5. We show that our pre-trained model is especially competitive at zero-shot and few-shot learning, and better at learning code properties (e.g., syntax, data flow).