pwc-archive/papers-with-abstracts huggingface dataset:
Role of Noise in the Fairen-Velarde model of bacterial respiration
Authors: Soumyadeep Kundu, Muktish Acharyya
Date: 2024-04-12 00:00:00

Following our recent study (S. Kundu and M. Acharyya, International Journal of Modern Physics C, 35 (2024) 2450094), we have introduced stochasticity (random noise) in the Fairen-Velarde deterministic differential equations. The role of such noise on time scales is studied. The probability of reaching the domain of fixed points in a given interval is also calculated. The probability is studied as a function of the width of the noise, and a hyperbolic tangent fitting is proposed. The noise reduces the time scale for bringing the bacteria into an inactive state. The area of the fixed-point domain was found to be asympotically linear with noise. The time to reach the fixed-point domain is fitted to a Gaussian function with noise intensity.
Factor risk measures
Authors: Hirbod Assa, Peng Liu
Date: 2024-04-12 00:00:00

This paper introduces and studies factor risk measures. While risk measures only rely on the distribution of a loss random variable, in many cases risk needs to be measured relative to some major factors. In this paper, we introduce a double-argument mapping as a risk measure to assess the risk relative to a vector of factors, called factor risk measure. The factor risk measure only depends on the joint distribution of the risk and the factors. A set of natural axioms are discussed, and particularly distortion, quantile, linear and coherent factor risk measures are introduced and characterized. Moreover, we introduce a large set of concrete factor risk measures and many of them are new to the literature, which are interpreted in the context of regulatory capital requirement. Finally, the distortion factor risk measures are applied in the risk-sharing problem and some numerical examples are presented to show the difference between the Value-at-Risk and the quantile factor risk measures.
Revisiting Feature Prediction for Learning Visual Representations from Video
Authors: Adrien Bardes, Quentin Garrido, Jean Ponce, Xinlei Chen, Michael Rabbat, Yann Lecun, Mahmoud Assran, Nicolas Ballas
Date: 2024-02-15 00:00:00
Proceeding: arXiv preprint 2024 2
Tasks: Prediction

This paper explores feature prediction as a stand-alone objective for unsupervised learning from video and introduces V-JEPA, a collection of vision models trained solely using a feature prediction objective, without the use of pretrained image encoders, text, negative examples, reconstruction, or other sources of supervision. The models are trained on 2 million videos collected from public datasets and are evaluated on downstream image and video tasks. Our results show that learning by predicting video features leads to versatile visual representations that perform well on both motion and appearance-based tasks, without adaption of the model's parameters; e.g., using a frozen backbone. Our largest model, a ViT-H/16 trained only on videos, obtains 81.9% on Kinetics-400, 72.2% on Something-Something-v2, and 77.9% on ImageNet1K.
Dynamic Neural Control Flow Execution: An Agent-Based Deep Equilibrium Approach for Binary Vulnerability Detection
Authors: Litao Li, Steven H. H. Ding, Andrew Walenstein, Philippe Charland, Benjamin C. M. Fung
Date: 2024-04-03 00:00:00
Tasks: Graph Neural Network, Vulnerability Detection

Software vulnerabilities are a challenge in cybersecurity. Manual security patches are often difficult and slow to be deployed, while new vulnerabilities are created. Binary code vulnerability detection is less studied and more complex compared to source code, and this has important practical implications. Deep learning has become an efficient and powerful tool in the security domain, where it provides end-to-end and accurate prediction. Modern deep learning approaches learn the program semantics through sequence and graph neural networks, using various intermediate representation of programs, such as abstract syntax trees (AST) or control flow graphs (CFG). Due to the complex nature of program execution, the output of an execution depends on the many program states and inputs. Also, a CFG generated from static analysis can be an overestimation of the true program flow. Moreover, the size of programs often does not allow a graph neural network with fixed layers to aggregate global information. To address these issues, we propose DeepEXE, an agent-based implicit neural network that mimics the execution path of a program. We use reinforcement learning to enhance the branching decision at every program state transition and create a dynamic environment to learn the dependency between a vulnerability and certain program states. An implicitly defined neural network enables nearly infinite state transitions until convergence, which captures the structural information at a higher level. The experiments are conducted on two semi-synthetic and two real-world datasets. We show that DeepEXE is an accurate and efficient method and outperforms the state-of-the-art vulnerability detection methods.
CATP: Cross-Attention Token Pruning for Accuracy Preserved Multimodal Model Inference
Authors: Ruqi Liao, Chuqing Zhao, Jin Li, Weiqi Feng
Date: 2024-04-02 00:00:00
Tasks: Computational Efficiency

In response to the rising interest in large multimodal models, we introduce Cross-Attention Token Pruning (CATP), a precision-focused token pruning method. Our approach leverages cross-attention layers in multimodal models, exemplified by BLIP-2, to extract valuable information for token importance determination. CATP employs a refined voting strategy across model heads and layers. In evaluations, CATP achieves up to 12.1X higher accuracy compared to existing token pruning methods, addressing the trade-off between computational efficiency and model precision.
A Multi-Level Framework for Accelerating Training Transformer Models
Authors: Longwei Zou, Han Zhang, Yangdong Deng
Date: 2024-04-07 00:00:00

The fast growing capabilities of large-scale deep learning models, such as Bert, GPT and ViT, are revolutionizing the landscape of NLP, CV and many other domains. Training such models, however, poses an unprecedented demand for computing power, which incurs exponentially increasing energy cost and carbon dioxide emissions. It is thus critical to develop efficient training solutions to reduce the training costs. Motivated by a set of key observations of inter- and intra-layer similarities among feature maps and attentions that can be identified from typical training processes, we propose a multi-level framework for training acceleration. Specifically, the framework is based on three basic operators, Coalescing, De-coalescing and Interpolation, which can be orchestrated to build a multi-level training framework. The framework consists of a V-cycle training process, which progressively down- and up-scales the model size and projects the parameters between adjacent levels of models via coalescing and de-coalescing. The key idea is that a smaller model that can be trained for fast convergence and the trained parameters provides high-qualities intermediate solutions for the next level larger network. The interpolation operator is designed to break the symmetry of neurons incurred by de-coalescing for better convergence performance. Our experiments on transformer-based language models (e.g. Bert, GPT) as well as a vision model (e.g. DeiT) prove that the proposed framework reduces the computational cost by about 20% on training BERT/GPT-Base models and up to 51.6% on training the BERT-Large model while preserving the performance.
ApproxDARTS: Differentiable Neural Architecture Search with Approximate Multipliers
Authors: Michal Pinos, Lukas Sekanina, Vojtech Mrazek
Date: 2024-04-08 00:00:00
Tasks: GPU, Neural Architecture Search

Integrating the principles of approximate computing into the design of hardware-aware deep neural networks (DNN) has led to DNNs implementations showing good output quality and highly optimized hardware parameters such as low latency or inference energy. In this work, we present ApproxDARTS, a neural architecture search (NAS) method enabling the popular differentiable neural architecture search method called DARTS to exploit approximate multipliers and thus reduce the power consumption of generated neural networks. We showed on the CIFAR-10 data set that the ApproxDARTS is able to perform a complete architecture search within less than $10$ GPU hours and produce competitive convolutional neural networks (CNN) containing approximate multipliers in convolutional layers. For example, ApproxDARTS created a CNN showing an energy consumption reduction of (a) $53.84\%$ in the arithmetic operations of the inference phase compared to the CNN utilizing the native $32$-bit floating-point multipliers and (b) $5.97\%$ compared to the CNN utilizing the exact $8$-bit fixed-point multipliers, in both cases with a negligible accuracy drop. Moreover, the ApproxDARTS is $2.3\times$ faster than a similar but evolutionary algorithm-based method called EvoApproxNAS.
Interpretable Neural Temporal Point Processes for Modelling Electronic Health Records
Authors: Bingqing Liu
Date: 2024-04-09 00:00:00
Tasks: Point Processes, Temporal Sequences

Electronic Health Records (EHR) can be represented as temporal sequences that record the events (medical visits) from patients. Neural temporal point process (NTPP) has achieved great success in modeling event sequences that occur in continuous time space. However, due to the black-box nature of neural networks, existing NTPP models fall short in explaining the dependencies between different event types. In this paper, inspired by word2vec and Hawkes process, we propose an interpretable framework inf2vec for event sequence modelling, where the event influences are directly parameterized and can be learned end-to-end. In the experiment, we demonstrate the superiority of our model on event prediction as well as type-type influences learning.
Federated Distillation: A Survey
Authors: Lin Li, Jianping Gou, Baosheng Yu, Lan Du, Zhang Yiand Dacheng Tao
Date: 2024-04-02 00:00:00
Tasks: Federated Learning, Knowledge Distillation, Survey, Transfer Learning

Federated Learning (FL) seeks to train a model collaboratively without sharing private training data from individual clients. Despite its promise, FL encounters challenges such as high communication costs for large-scale models and the necessity for uniform model architectures across all clients and the server. These challenges severely restrict the practical applications of FL. To address these limitations, the integration of knowledge distillation (KD) into FL has been proposed, forming what is known as Federated Distillation (FD). FD enables more flexible knowledge transfer between clients and the server, surpassing the mere sharing of model parameters. By eliminating the need for identical model architectures across clients and the server, FD mitigates the communication costs associated with training large-scale models. This paper aims to offer a comprehensive overview of FD, highlighting its latest advancements. It delves into the fundamental principles underlying the design of FD frameworks, delineates FD approaches for tackling various challenges, and provides insights into the diverse applications of FD across different scenarios.
Complex-Valued Convolutional Gated Recurrent Neural Network for Ultrasound Beamforming
Authors: Zhiming Zhang, Zhenyu Lei, Mengchu Zhou, Hideyuki Hasegawa, Shangce Gao
Date: 2024-04-10 00:00:00
Proceeding: IEEE Transactions on Neural Networks and Learning Systems 2024 4

Ultrasound detection is a potent tool for the clinical diagnosis of various diseases due to its real-time, convenient, and noninvasive qualities. Yet, existing ultrasound beamforming and related methods face a big challenge to improve both the quality and speed of imaging for the required clinical applications. The most notable characteristic of ultrasound signal data is its spatial and temporal features. Because most signals are complex-valued, directly processing them by using real-valued networks leads to phase distortion and inaccurate output. In this study, for the first time, we propose a complex-valued convolutional gated recurrent (CCGR) neural network to handle ultrasound analytic signals with the aforementioned properties. The complex-valued network operations proposed in this study improve the beamforming accuracy of complex-valued ultrasound signals over traditional real-valued methods. Further, the proposed deep integration of convolution and recurrent neural networks makes a great contribution to extracting rich and informative ultrasound signal features. Our experimental results reveal its outstanding imaging quality over existing state-of-the-art methods. More significantly, its ultrafast processing speed of only 0.07 s per image promises considerable clinical application potential.
DKDFN: Domain Knowledge-Guided deep collaborative fusion network for multimodal unitemporal remote sensing land cover classification
Authors: Liheng Zhong
Date: 2022-02-13 00:00:00
Proceeding: ScienseDirect 2022 2
Tasks: Decoder, image-classification, Image Classification, Land Cover Classification, Remote Sensing Image Classification

Land use and land cover maps provide fundamental information that has been used in different types of studies,
ranging from public health to carbon cycling. However, the existing remote sensing image classification methods
thus far suffer from the insufficient usage of multiple modalities, underconsideration of prior domain knowledge,
and poor performance on minority classes. To alleviate these problems, we propose a novel domain knowledge-
guided deep collaborative fusion network (DKDFN) with performance boosting for minority categories for land
cover classification. More specifically, the DKDFN adopts a multihead encoder and a multibranch decoder
structure. The architecture of the encoder probablizes sufficient mining of complementary information from
multiple modalities, which are Sentinel-2, Sentinel-1, and SRTM Digital Elevation Data (SRTM) in our case. The
multibranch decoder enables land cover classification in a multitask learning setup, performing semantic seg­
mentation and reconstructing multimodal remote sensing indices, which are selected as representatives of
domain knowledge. This design incorporates domain knowledge in an effective end-to-end manner. The training
stage of our DKDFN is supervised by our proposed asymmetry loss function (ALF), which boosts performance on
nearly all categories, especially the categories with a low frequency of occurrence. Ablation studies of the
network suggest that our design logic is worth testing in any network with an encoder-decoder structure. The
study is conducted in Hunan, China and is verified using a self-labeled multimodal unitemporal remote sensing
image dataset. The comparative experiments between DKDFN and 6 state-of-the-art models (U-Net, SegNet,
PSPNet, DeepLab, HRNet, MP-ResNet) testify to the superiority of our method and suggest its potential to be
applied more widely to map land cover in other geographical areas given the availability of Sentinel-2, Sentinel-
1, and SRTM data. The dataset can be downloaded by https://github.com/LauraChow/HunanMultimodalDataset
Mapping Network-Coordinated Stacked Gated Recurrent Units for Turbulence Prediction
Authors: Zhiming Zhang, Shangce Gao, Mengchu Zhou, Mengtao Yan, Shuyang Cao
Date: 2024-04-08 00:00:00
Proceeding: IEEE/CAA Journal of Automatica Sinica 2024 4

Accurately predicting fluid forces acting on the surface of a structure is crucial in engineering design. However, this task becomes particularly challenging in turbulent flow, due to the complex and irregular changes in the flow field. In this study, we propose a novel deep learning method, named mapping network-coordinated stacked gated recurrent units (MSU), for predicting pressure on a circular cylinder from velocity data. Specifically, our coordinated learning strategy is designed to extract the most critical velocity point for prediction, a process that has not been explored before. In our experiments, MSU extracts one point from a velocity field containing 121 points and utilizes this point to accurately predict 100 pressure points on the cylinder. This method significantly reduces the workload of data measurement in practical engineering applications. Our experimental results demonstrate that MSU predictions are highly similar to the real turbulent data in both spatio-temporal and individual aspects. Furthermore, the comparison results show that MSU predicts more precise results, even outperforming models that use all velocity field points. Compared with state-of-the-art methods, MSU has an average improvement of more than 45% in various indicators such as root mean square error (RMSE). Through comprehensive and authoritative physical verification, we established that MSU’s prediction results closely align with pressure field data obtained in real turbulence fields. This confirmation underscores the considerable potential of MSU for practical applications in real engineering scenarios. The code is available at https://github.com/zhangzm0128/MSU.
Unlocking Deterministic Robustness Certification on ImageNet
Authors: Kai Hu, Andy Zou, Zifan Wang, Klas Leino, Matt Fredrikson
Date: 2023-01-29 00:00:00
Proceeding: NeurIPS 2023 11

Despite the promise of Lipschitz-based methods for provably-robust deep learning with deterministic guarantees, current state-of-the-art results are limited to feed-forward Convolutional Networks (ConvNets) on low-dimensional data, such as CIFAR-10. This paper investigates strategies for expanding certifiably robust training to larger, deeper models. A key challenge in certifying deep networks is efficient calculation of the Lipschitz bound for residual blocks found in ResNet and ViT architectures. We show that fast ways of bounding the Lipschitz constant for conventional ResNets are loose, and show how to address this by designing a new residual block, leading to the \emph{Linear ResNet} (LiResNet) architecture. We then introduce \emph{Efficient Margin MAximization} (EMMA), a loss function that stabilizes robust training by simultaneously penalizing worst-case adversarial examples from \emph{all} classes. Together, these contributions yield new \emph{state-of-the-art} robust accuracy on CIFAR-10/100 and Tiny-ImageNet under $\ell_2$ perturbations. Moreover, for the first time, we are able to scale up fast deterministic robustness guarantees to ImageNet, demonstrating that this approach to robust learning can be applied to real-world applications. We release our code on Github: \url{https://github.com/klasleino/gloro}.
Epidemic Learning: Boosting Decentralized Learning with Randomized Communication
Authors: Martijn de Vos, Sadegh Farhadkhani, Rachid Guerraoui, Anne-Marie Kermarrec, Rafael Pires, Rishi Sharma
Date: 2023-10-03 00:00:00
Proceeding: NeurIPS 2023 11

We present Epidemic Learning (EL), a simple yet powerful decentralized learning (DL) algorithm that leverages changing communication topologies to achieve faster model convergence compared to conventional DL approaches. At each round of EL, each node sends its model updates to a random sample of $s$ other nodes (in a system of $n$ nodes). We provide an extensive theoretical analysis of EL, demonstrating that its changing topology culminates in superior convergence properties compared to the state-of-the-art (static and dynamic) topologies. Considering smooth non-convex loss functions, the number of transient iterations for EL, i.e., the rounds required to achieve asymptotic linear speedup, is in $O(n^3/s^2)$ which outperforms the best-known bound $O(n^3)$ by a factor of $s^2$, indicating the benefit of randomized communication for DL. We empirically evaluate EL in a 96-node network and compare its performance with state-of-the-art DL approaches. Our results illustrate that EL converges up to $ 1.7\times$ quicker than baseline DL algorithms and attains $2.2 $\% higher accuracy for the same communication volume.
BERT-LSH: Reducing Absolute Compute For Attention
Authors: ZeZheng Li, Kingston Yip
Date: 2024-04-12 00:00:00
Tasks: Computational Efficiency

This study introduces a novel BERT-LSH model that incorporates Locality Sensitive Hashing (LSH) to approximate the attention mechanism in the BERT architecture. We examine the computational efficiency and performance of this model compared to a standard baseline BERT model. Our findings reveal that BERT-LSH significantly reduces computational demand for the self-attention layer while unexpectedly outperforming the baseline model in pretraining and fine-tuning tasks. These results suggest that the LSH-based attention mechanism not only offers computational advantages but also may enhance the model's ability to generalize from its training data. For more information, visit our GitHub repository: https://github.com/leo4life2/algoml-final
Intelligent Chemical Purification Technique Based on Machine Learning
Authors: Wenchao Wu, Hao Xu, Dongxiao Zhang, Fanyang Mo
Date: 2024-04-14 00:00:00
Tasks: Transfer Learning

We present an innovative of artificial intelligence with column chromatography, aiming to resolve inefficiencies and standardize data collection in chemical separation and purification domain. By developing an automated platform for precise data acquisition and employing advanced machine learning algorithms, we constructed predictive models to forecast key separation parameters, thereby enhancing the efficiency and quality of chromatographic processes. The application of transfer learning allows the model to adapt across various column specifications, broadening its utility. A novel metric, separation probability ($S_p$), quantifies the likelihood of effective compound separation, validated through experimental verification. This study signifies a significant step forward int the application of AI in chemical research, offering a scalable solution to traditional chromatography challenges and providing a foundation for future technological advancements in chemical analysis and purification.
Cross-Modal Alternating Learning with Task-Aware Representations for Continual Learning
Authors: Bin-Bin Gao
Date: 2023-12-07 00:00:00
Proceeding: IEEE TMM 2023 2023 12
Tasks: Continual Learning, image-classification, Image Classification, Incremental Learning on CIFAR100-B0(5steps of 20 classes), Lifelong learning

Continual learning is a research field of artificial neural networks to simulate human lifelong learning ability. Although a surge of investigations has achieved considerable performance, most rely only on image modality for incremental image recognition tasks. In this paper, we propose a novel yet effective framework coined cross-modal Alternating Learning with Task-Aware representations (ALTA) to make good use of visual and linguistic modal information and achieve more effective continual learning. To do so, ALTA presents a cross-modal joint learning mechanism that leverages simultaneous learning of image and text representations to provide more effective supervision. And it mitigates forgetting by endowing task-aware representations with continual learning capability. Concurrently, considering the dilemma of stability and plasticity, ALTA proposes a cross-modal alternating learning strategy that alternately learns the task-aware cross-modal representations to match the image-text pairs between tasks better, further enhancing the ability of continual learning. We conduct extensive experiments under various popular image classification benchmarks to demonstrate that our approach achieves state-of-the-art performance. At the same time, systematic ablation studies and visualization analyses validate the effectiveness and rationality of our method. Our code for ALTA is available at \url{https://github.com/vijaylee/ALTA}.