pwc-archive/papers-with-abstracts huggingface dataset:
LLM-Pruner: On the Structural Pruning of Large Language Models
Authors: Xinyin Ma, Gongfan Fang, Xinchao Wang
Date: 2023-05-19 00:00:00
Proceeding: NeurIPS 2023 11
Tasks: Text Generation, zero-shot-classification, Zero-Shot Learning

Large language models (LLMs) have shown remarkable capabilities in language understanding and generation. However, such impressive capability typically comes with a substantial model size, which presents significant challenges in both the deployment, inference, and training stages. With LLM being a general-purpose task solver, we explore its compression in a task-agnostic manner, which aims to preserve the multi-task solving and language generation ability of the original LLM. One challenge to achieving this is the enormous size of the training corpus of LLM, which makes both data transfer and model post-training over-burdensome. Thus, we tackle the compression of LLMs within the bound of two constraints: being task-agnostic and minimizing the reliance on the original training dataset. Our method, named LLM-Pruner, adopts structural pruning that selectively removes non-critical coupled structures based on gradient information, maximally preserving the majority of the LLM's functionality. To this end, the performance of pruned models can be efficiently recovered through tuning techniques, LoRA, in merely 3 hours, requiring only 50K data. We validate the LLM-Pruner on three LLMs, including LLaMA, Vicuna, and ChatGLM, and demonstrate that the compressed models still exhibit satisfactory capabilities in zero-shot classification and generation. The code is available at: https://github.com/horseee/LLM-Pruner
Structural Pruning for Diffusion Models
Authors: Gongfan Fang, Xinyin Ma, Xinchao Wang
Date: 2023-05-18 00:00:00
Proceeding: NeurIPS 2023 11

Generative modeling has recently undergone remarkable advancements, primarily propelled by the transformative implications of Diffusion Probabilistic Models (DPMs). The impressive capability of these models, however, often entails significant computational overhead during both training and inference. To tackle this challenge, we present Diff-Pruning, an efficient compression method tailored for learning lightweight diffusion models from pre-existing ones, without the need for extensive re-training. The essence of Diff-Pruning is encapsulated in a Taylor expansion over pruned timesteps, a process that disregards non-contributory diffusion steps and ensembles informative gradients to identify important weights. Our empirical assessment, undertaken across several datasets highlights two primary benefits of our proposed method: 1) Efficiency: it enables approximately a 50\% reduction in FLOPs at a mere 10\% to 20\% of the original training expenditure; 2) Consistency: the pruned diffusion models inherently preserve generative behavior congruent with their pre-trained models. Code is available at \url{https://github.com/VainF/Diff-Pruning}.
Data-driven rational design of single-atom materials for hydrogen evolution and sensing
Authors: Lei Zhou, Pengfei Tian, Bowei Zhang, and Fu-Zhen Xuan
Date: 2023-08-20 00:00:00
Proceeding: Nano Research 2023 8

Herein  we  proposed  a  data-driven  high-throughput  principle  to  screen  high-performance  single-atom  materials  for  hydrogen evolution  reaction  (HER)  and  hydrogen  sensing  by  combing  the  theoretical  computations  and  a  topology-based  multi-scale convolution kernel machine learning algorithm. After the rational training by 25 groups of data and prediction of all 168 groups of
single-atom materials for HER and sensing, respectively, a high prediction accuracy (> 0.931 R2
 score) was achieved by our
model. Results show that the promising HER catalysts include Pt atoms in C4
 and Sc atoms in C1N3
 coordination environment.
Moreover, Y atoms in C4
 coordination environment and Cd atoms in C2N2
-ortho coordination environment were predicted with
great potential as hydrogen sensing materials. This method provides a way to accelerate the discovery of innovative materials by
avoiding the time-consuming empirical principles in experiments
FiGURe: Simple and Efficient Unsupervised Node Representations with Filter Augmentations
Authors: Chanakya Ekbote, Ajinkya Pankaj Deshpande, Arun Iyer, Ramakrishna Bairi, Sundararajan Sellamanickam
Date: 2023-10-03 00:00:00
Proceeding: NeurIPS 2023 11
Tasks: Contrastive Learning

Unsupervised node representations learnt using contrastive learning-based methods have shown good performance on downstream tasks. However, these methods rely on augmentations that mimic low-pass filters, limiting their performance on tasks requiring different eigen-spectrum parts. This paper presents a simple filter-based augmentation method to capture different parts of the eigen-spectrum. We show significant improvements using these augmentations. Further, we show that sharing the same weights across these different filter augmentations is possible, reducing the computational load. In addition, previous works have shown that good performance on downstream tasks requires high dimensional representations. Working with high dimensions increases the computations, especially when multiple augmentations are involved. We mitigate this problem and recover good performance through lower dimensional embeddings using simple random Fourier feature projections. Our method, FiGURe achieves an average gain of up to 4.4%, compared to the state-of-the-art unsupervised models, across all datasets in consideration, both homophilic and heterophilic. Our code can be found at: https://github.com/microsoft/figure.
Performance Analysis of Machine Learning Algorithms in Chronic Kidney Disease Prediction
Authors: Iftekhar Ahmed, Tanzil Ebad Chowdhury, Biggo Bushon Routh, Nafisa Tasmiya, Shadman Sakib, Adil Ahmed Chowdhury
Date: 2022-11-22 00:00:00
Proceeding: Conference 2022 11
Tasks: Disease Prediction, Kidney Function, Missing Values, regression

Kidneys are the filter of the human body. About 10 % of the global population is thought to be affected by Chronic Kidney Disease (CKD), which causes kidney function to decline. To protect in danger patients from additional kidney damage, effective risk evaluation of CKD and appropriate CKD monitoring are crucial. Due to quick and precise detection capabilities, Machine Learning models can help practitioners accomplish this goal efficiently therefore an enormous number of diagnosis systems and processes in the healthcare sector nowadays are relying on machine learning due to its disease prediction capability. In this study, we designed and suggested disease predictive computer-aided designs for the diagnosis of CKD. The dataset for CKD is attained from the repository of machine learning of UCL, with a few missing values; those are filled in using “mean/mode” and “Random sampling method” strategies. After successfully achieving the missing data, eight ML techniques (Random Forest, SVM, Naive Bayes, Logistic Regression, KNN, XG _Boost, Decision Tree and AdaBoost) were used to establish models and the performance evaluation comparisons among the result accuracies are measured by the techniques to find the machine learning models with highest accuracies. Among them, Random Forest as well as Logistic Regression showed an outstanding 99% accuracy, followed by the Ada Boost, XG-Boost, Naive Bayes, Decision Tree, and SVM whereas the KNN classifier model stands last with an accuracy of 73%.
Scalable Approximate NonSymmetric Autoencoder for Collaborative Filtering
Authors: Martin Spišák, Radek Bartyzal, Antonín Hoskovec, Ladislav Peška, Miroslav Tůma
Date: 2023-09-14 00:00:00
Proceeding: ACM Conference on Recommender Systems 2023 9
Tasks: Collaborative Filtering, Recommendation Systems

In the field of recommender systems, shallow autoencoders have recently gained significant attention. One of the most highly acclaimed shallow autoencoders is EASE, favored for its competitive recommendation accuracy and simultaneous simplicity. However, the poor scalability of EASE (both in time and especially in memory) severely restricts its use in production environments with vast item sets. In this paper, we propose a hyperefficient factorization technique for sparse approximate inversion of the data-Gram matrix used in EASE. The resulting autoencoder, SANSA, is an end-to-end sparse solution with prescribable density and almost arbitrarily low memory requirements — even for training. As such, SANSA allows us to effortlessly scale the concept of EASE to millions of items and beyond.
An Algorithm for Solving Robot Inverse Kinematics Based on FOA Optimized BP Neural Network
Authors: Yonghua Bai, Minzhou Luo, * and Fenglin Pang
Date: 2021-08-02 00:00:00
Proceeding: Applied Sciences 2021 8

The solution of robot inverse kinematics has a direct impact on the control accuracy of the robot. Conventional inverse kinematics solution methods, such as numerical solution, algebraic solution, and geometric solution, have insufficient solution speed and solution accuracy, and the solution process is complicated. Due to the mapping ability of the neural network, the use of neural networks to solve robot inverse kinematics problems has attracted widespread attention. However, it has slow convergence speed and low accuracy. This paper proposes the FOA optimized BP neural network algorithm to solve inverse kinematics. It overcomes the shortcomings of low convergence accuracy, slow convergence speed, and easy to fall into local minima when using BP neural network to solve inverse kinematics. The experimental results show that using the trained FOA optimized BP neural network to solve the inverse kinematics, the maximum error range of the output joint angle is [−0.04686, 0.1271]. The output error of the FOA optimized BP neural network algorithm is smaller than that of the ordinary BP neural network algorithm and the PSO optimized BP neural network algorithm. Using the FOA optimized BP neural network algorithm to solve the robot kinematics can improve the control accuracy of the robot.
FedFed: Feature Distillation against Data Heterogeneity in Federated Learning
Authors: Zhiqin Yang, Yonggang Zhang, Yu Zheng, Xinmei Tian, Hao Peng, Tongliang Liu, Bo Han
Date: 2023-10-08 00:00:00
Proceeding: NeurIPS 2023 11
Tasks: Federated Learning

Federated learning (FL) typically faces data heterogeneity, i.e., distribution shifting among clients. Sharing clients' information has shown great potentiality in mitigating data heterogeneity, yet incurs a dilemma in preserving privacy and promoting model performance. To alleviate the dilemma, we raise a fundamental question: \textit{Is it possible to share partial features in the data to tackle data heterogeneity?} In this work, we give an affirmative answer to this question by proposing a novel approach called {\textbf{Fed}erated \textbf{Fe}ature \textbf{d}istillation} (FedFed). Specifically, FedFed partitions data into performance-sensitive features (i.e., greatly contributing to model performance) and performance-robust features (i.e., limitedly contributing to model performance). The performance-sensitive features are globally shared to mitigate data heterogeneity, while the performance-robust features are kept locally. FedFed enables clients to train models over local and shared data. Comprehensive experiments demonstrate the efficacy of FedFed in promoting model performance.
Dream the Impossible: Outlier Imagination with Diffusion Models
Authors: Xuefeng Du, Yiyou Sun, Xiaojin Zhu, Yixuan Li
Date: 2023-09-23 00:00:00
Proceeding: NeurIPS 2023 11
Tasks: Out of Distribution (OOD) Detection

Utilizing auxiliary outlier datasets to regularize the machine learning model has demonstrated promise for out-of-distribution (OOD) detection and safe prediction. Due to the labor intensity in data collection and cleaning, automating outlier data generation has been a long-desired alternative. Despite the appeal, generating photo-realistic outliers in the high dimensional pixel space has been an open challenge for the field. To tackle the problem, this paper proposes a new framework DREAM-OOD, which enables imagining photo-realistic outliers by way of diffusion models, provided with only the in-distribution (ID) data and classes. Specifically, DREAM-OOD learns a text-conditioned latent space based on ID data, and then samples outliers in the low-likelihood region via the latent, which can be decoded into images by the diffusion model. Different from prior works, DREAM-OOD enables visualizing and understanding the imagined outliers, directly in the pixel space. We conduct comprehensive quantitative and qualitative studies to understand the efficacy of DREAM-OOD, and show that training with the samples generated by DREAM-OOD can benefit OOD detection performance. Code is publicly available at https://github.com/deeplearning-wisc/dream-ood.
GraphAdapter: Tuning Vision-Language Models With Dual Knowledge Graph
Authors: Xin Li, Dongze Lian, Zhihe Lu, Jiawang Bai, Zhibo Chen, Xinchao Wang
Date: 2023-09-24 00:00:00
Proceeding: NeurIPS 2023 11
Tasks: Transfer Learning

Adapter-style efficient transfer learning (ETL) has shown excellent performance in the tuning of vision-language models (VLMs) under the low-data regime, where only a few additional parameters are introduced to excavate the task-specific knowledge based on the general and powerful representation of VLMs. However, most adapter-style works face two limitations: (i) modeling task-specific knowledge with a single modality only; and (ii) overlooking the exploitation of the inter-class relationships in downstream tasks, thereby leading to sub-optimal solutions. To mitigate that, we propose an effective adapter-style tuning strategy, dubbed GraphAdapter, which performs the textual adapter by explicitly modeling the dual-modality structure knowledge (i.e., the correlation of different semantics/classes in textual and visual modalities) with a dual knowledge graph. In particular, the dual knowledge graph is established with two sub-graphs, i.e., a textual knowledge sub-graph, and a visual knowledge sub-graph, where the nodes and edges represent the semantics/classes and their correlations in two modalities, respectively. This enables the textual feature of each prompt to leverage the task-specific structure knowledge from both textual and visual modalities, yielding a more effective classifier for downstream tasks. Extensive experimental results on 11 benchmark datasets reveal that our GraphAdapter significantly outperforms previous adapter-based methods. The code will be released at https://github.com/lixinustc/GraphAdapter
NeuroGF: A Neural Representation for Fast Geodesic Distance and Path Queries
Authors: Qijian Zhang, Junhui Hou, Yohanes Yudhi Adikusuma, Wenping Wang, Ying He
Date: 2023-06-01 00:00:00
Proceeding: NeurIPS 2023 11
Tasks: 3D geometry

Geodesics are essential in many geometry processing applications. However, traditional algorithms for computing geodesic distances and paths on 3D mesh models are often inefficient and slow. This makes them impractical for scenarios that require extensive querying of arbitrary point-to-point geodesics. Although neural implicit representations have emerged as a popular way of representing 3D shape geometries, there is still no research on representing geodesics with deep implicit functions. To bridge this gap, this paper presents the first attempt to represent geodesics on 3D mesh models using neural implicit functions. Specifically, we introduce neural geodesic fields (NeuroGFs), which are learned to represent the all-pairs geodesics of a given mesh. By using NeuroGFs, we can efficiently and accurately answer queries of arbitrary point-to-point geodesic distances and paths, overcoming the limitations of traditional algorithms. Evaluations on common 3D models show that NeuroGFs exhibit exceptional performance in solving the single-source all-destination (SSAD) and point-to-point geodesics, and achieve high accuracy consistently. Besides, NeuroGFs also offer the unique advantage of encoding both 3D geometry and geodesics in a unified representation. Moreover, we further extend generalizable learning frameworks of NeuroGFs by adding shape feature encoders, which also show satisfactory performances for unseen shapes and categories. Code is made available at https://github.com/keeganhk/NeuroGF/tree/master.
S-CLIP: Semi-supervised Vision-Language Learning using Few Specialist Captions
Authors: Sangwoo Mo, Minkyu Kim, Kyungmin Lee, Jinwoo Shin
Date: 2023-05-23 00:00:00
Proceeding: NeurIPS 2023 11
Tasks: Contrastive Learning, Image-text Retrieval, Partial Label Learning, Pseudo Label, Text Retrieval, zero-shot-classification, Zero-Shot Learning

Vision-language models, such as contrastive language-image pre-training (CLIP), have demonstrated impressive results in natural image domains. However, these models often struggle when applied to specialized domains like remote sensing, and adapting to such domains is challenging due to the limited number of image-text pairs available for training. To address this, we propose S-CLIP, a semi-supervised learning method for training CLIP that utilizes additional unpaired images. S-CLIP employs two pseudo-labeling strategies specifically designed for contrastive learning and the language modality. The caption-level pseudo-label is given by a combination of captions of paired images, obtained by solving an optimal transport problem between unpaired and paired images. The keyword-level pseudo-label is given by a keyword in the caption of the nearest paired image, trained through partial label learning that assumes a candidate set of labels for supervision instead of the exact one. By combining these objectives, S-CLIP significantly enhances the training of CLIP using only a few image-text pairs, as demonstrated in various specialist domains, including remote sensing, fashion, scientific figures, and comics. For instance, S-CLIP improves CLIP by 10% for zero-shot classification and 4% for image-text retrieval on the remote sensing benchmark, matching the performance of supervised CLIP while using three times fewer image-text pairs.
Framework and Benchmarks for Combinatorial and Mixed-variable Bayesian Optimization
Authors: Kamil Dreczkowski, Antoine Grosnit, Haitham Bou Ammar
Date: 2023-06-16 00:00:00
Proceeding: NeurIPS 2023 11
Tasks: Bayesian Optimization, Benchmarking

This paper introduces a modular framework for Mixed-variable and Combinatorial Bayesian Optimization (MCBO) to address the lack of systematic benchmarking and standardized evaluation in the field. Current MCBO papers often introduce non-diverse or non-standard benchmarks to evaluate their methods, impeding the proper assessment of different MCBO primitives and their combinations. Additionally, papers introducing a solution for a single MCBO primitive often omit benchmarking against baselines that utilize the same methods for the remaining primitives. This omission is primarily due to the significant implementation overhead involved, resulting in a lack of controlled assessments and an inability to showcase the merits of a contribution effectively. To overcome these challenges, our proposed framework enables an effortless combination of Bayesian Optimization components, and provides a diverse set of synthetic and real-world benchmarking tasks. Leveraging this flexibility, we implement 47 novel MCBO algorithms and benchmark them against seven existing MCBO solvers and five standard black-box optimization algorithms on ten tasks, conducting over 4000 experiments. Our findings reveal a superior combination of MCBO primitives outperforming existing approaches and illustrate the significance of model fit and the use of a trust region. We make our MCBO library available under the MIT license at \url{https://github.com/huawei-noah/HEBO/tree/master/MCBO}.
RTQ: Rethinking Video-language Understanding Based on Image-text Model
Authors: Xiao Wang, Yaoyu Li, Tian Gan, Zheng Zhang, Jingjing Lv, Liqiang Nie
Date: 2023-12-01 00:00:00
Tasks: Video Captioning, Video Question Answering, Video Retrieval

Recent advancements in video-language understanding have been established on the foundation of image-text models, resulting in promising outcomes due to the shared knowledge between images and videos. However, video-language understanding presents unique challenges due to the inclusion of highly complex semantic details, which result in information redundancy, temporal dependency, and scene complexity. Current techniques have only partially tackled these issues, and our quantitative analysis indicates that some of these methods are complementary. In light of this, we propose a novel framework called RTQ (Refine, Temporal model, and Query), which addresses these challenges simultaneously. The approach involves refining redundant information within frames, modeling temporal relations among frames, and querying task-specific information from the videos. Remarkably, our model demonstrates outstanding performance even in the absence of video-language pre-training, and the results are comparable with or superior to those achieved by state-of-the-art pre-training methods. Code is available at https://github.com/SCZwangxiao/RTQ-MM2023.
BCN: Batch Channel Normalization for Image Classification
Authors: Afifa Khaled, Chao Li, Jia Ning, Kun He
Date: 2023-12-01 00:00:00
Tasks: Classification, image-classification, Image Classification

Normalization techniques have been widely used in the field of deep learning due to their capability of enabling higher learning rates and are less careful in initialization. However, the effectiveness of popular normalization technologies is typically limited to specific areas. Unlike the standard Batch Normalization (BN) and Layer Normalization (LN), where BN computes the mean and variance along the (N,H,W) dimensions and LN computes the mean and variance along the (C,H,W) dimensions (N, C, H and W are the batch, channel, spatial height and width dimension, respectively), this paper presents a novel normalization technique called Batch Channel Normalization (BCN). To exploit both the channel and batch dependence and adaptively and combine the advantages of BN and LN based on specific datasets or tasks, BCN separately normalizes inputs along the (N, H, W) and (C, H, W) axes, then combines the normalized outputs based on adaptive parameters. As a basic block, BCN can be easily integrated into existing models for various applications in the field of computer vision. Empirical results show that the proposed technique can be seamlessly applied to various versions of CNN or Vision Transformer architecture. The code is publicly available at https://github.com/AfifaKhaled/BatchChannel-Normalization
Integrate Assignment of Multiple Heterogeneous Unmanned Aerial Vehicles Performing Dynamic Disaster Inspection and Validation Task With Dubins Path
Authors: WEINAN WUJIE XU YIMING SUN
Date: 2023-08-09 00:00:00
Proceeding: journal 2023 8

The actual performance of a task-assignment method of unmanned aerial vehicles (UAVs) shows a considerable dependence on the path planning process coupled with it; however, this topic has been rarely studied in the existing literature. This article considers the problem of maximizing the task-assignment reward of a fleet of heterogeneous UAVs for a dynamic reconnaissance and confirmation task under constraints of critical time and multi-UAV tasks where the coupled path optimization objectives also need to be considered. The existing consensus-based bundle algorithm is extended with an effective method for managing the multitask and multiagent constraints. The proposed method can optimize the inherent coupling path using the Dubins path to reduce the differences between the estimated path and the actual path, shortening the operation time by adopting the distributed genetic algorithm. The proposed method is verified by the sample run tests of a disaster area reconnaissance and confirmation task and Monte Carlo comparison simulations with the two existing algorithms. The results verify both the practicality and advantages of the proposed method.
NYCU ML Lab LLAMA: Prefix tuning using LLAMA for Informationa Retrieval
Authors: Zhi-Xuan Tai, Yu-Che Chang
Date: 2023-12-01 00:00:00
Proceeding: arXiv 2023 12
Tasks: Retrieval

TBD