0/1 Deep Neural Networks via Block Coordinate Descent
Authors: HUI ZHANG, Shenglong Zhou, Geoffrey Ye Li, Naihua Xiu
Date: 2022-06-19 00:00:00
Tasks: 10-shot image generation, 16k, 2D Object Detection, 3D dense captioning, 3D Face Alignment, 3D Facial Expression Recognition, 3D Facial Landmark Localization, 3D Hand Pose Estimation, 3D Instance Segmentation, 3D Lane Detection, 3D Multi-Object Tracking, 3D Place Recognition, Abstractive Text Summarization, Action Recognition, Anomaly Detection, Arithmetic Reasoning, Articles, Asthmatic Lung Sound Classification, Audio Classification, Change Detection, Classification, Click-Through Rate Prediction, Code Generation, Color Image Denoising, Common Sense Reasoning, Cross-Domain Few-Shot Object Detection, Deblurring, DeepFake Detection, Denoising, Depth Estimation, Domain Generalization, Drug Discovery, EEG 4 classes, Face Detection, Face Recognition, Fake Image Detection, Fine-Grained Image Classification, Fracture detection, Fraud Detection, Gloss-free Sign Language Translation, Graph Classification, Handwritten Mathmatical Expression Recognition, Hateful Meme Classification, Highlight Detection, Image Captioning, Image Classification, Image Dehazing, Image Generation, Keyword Spotting, Language Modelling, License Plate Detection, Long-range modeling, Low-Light Image Enhancement, Machine Translation, Medical Image Segmentation, Meme Classification, Monocular Depth Estimation, Multi-Label Classification, Multimodal Emotion Recognition, Multimodal Intent Recognition, Multi-Object Tracking, Music Source Separation, NavSim, Novel View Synthesis, Object Detection, Object Detection In Aerial Images, Object Rearrangement, Object Tracking, Person Re-Identification, Phone-level pronunciation scoring, Pose Estimation, Question Answering, Railway Track Image Classification, Real-Time Object Detection, Rgb-T Tracking, Robot Manipulation, Robot Manipulation Generalization, Robot Task Planning, Semantic Segmentation, Speech Enhancement, Speech Recognition, Style Transfer, Table-to-Text Generation, Temporal Relation Extraction, Text to 3D, Text-to-Image Generation, Universal Domain Adaptation, Unsupervised Domain Adaptation, Video deraining, Video Generation, Video Question Answering, Virtual Try-on, Visual Object Tracking, Weakly Supervised Action Localization, Zero-Shot Video Question Answer

The step function is one of the simplest and most natural activation functions for deep neural networks (DNNs). As it counts 1 for positive variables and 0 for others, its intrinsic characteristics (e.g., discontinuity and no viable information of subgradients) impede its development for several decades. Even if there is an impressive body of work on designing DNNs with continuous activation functions that can be deemed as surrogates of the step function, it is still in the possession of some advantageous properties, such as complete robustness to outliers and being capable of attaining the best learning-theoretic guarantee of predictive accuracy. Hence, in this paper, we aim to train DNNs with the step function used as an activation function (dubbed as 0/1 DNNs). We first reformulate 0/1 DNNs as an unconstrained optimization problem and then solve it by a block coordinate descend (BCD) method. Moreover, we acquire closed-form solutions for sub-problems of BCD as well as its convergence properties. Furthermore, we also integrate $\ell_{2,0}$-regularization into 0/1 DNN to accelerate the training process and compress the network scale. As a result, the proposed algorithm has a high performance on classifying MNIST and Fashion-MNIST datasets. As a result, the proposed algorithm has a desirable performance on classifying MNIST, FashionMNIST, Cifar10, and Cifar100 datasets.