pwc-archive/papers-with-abstracts huggingface dataset:
Cohort Squeeze: Beyond a Single Communication Round per Cohort in Cross-Device Federated Learning
Authors: Kai Yi, Timur Kharisov, Igor Sokolov, Peter Richtárik
Date: 2024-06-03 00:00:00
Tasks: Federated Learning

Virtually all federated learning (FL) methods, including FedAvg, operate in the following manner: i) an orchestrating server sends the current model parameters to a cohort of clients selected via certain rule, ii) these clients then independently perform a local training procedure (e.g., via SGD or Adam) using their own training data, and iii) the resulting models are shipped to the server for aggregation. This process is repeated until a model of suitable quality is found. A notable feature of these methods is that each cohort is involved in a single communication round with the server only. In this work we challenge this algorithmic design primitive and investigate whether it is possible to ``squeeze more juice" out of each cohort than what is possible in a single communication round. Surprisingly, we find that this is indeed the case, and our approach leads to up to 74% reduction in the total communication cost needed to train a FL model in the cross-device setting. Our method is based on a novel variant of the stochastic proximal point method (SPPM-AS) which supports a large collection of client sampling procedures some of which lead to further gains when compared to classical client selection approaches.
Multi-Objective Neural Architecture Search by Learning Search Space Partitions
Authors: Yiyang Zhao, Linnan Wang, Tian Guo
Date: 2024-06-01 00:00:00
Tasks: Bayesian Optimization, GPU, Neural Architecture Search

Deploying deep learning models requires taking into consideration neural network metrics such as model size, inference latency, and #FLOPs, aside from inference accuracy. This results in deep learning model designers leveraging multi-objective optimization to design effective deep neural networks in multiple criteria. However, applying multi-objective optimizations to neural architecture search (NAS) is nontrivial because NAS tasks usually have a huge search space, along with a non-negligible searching cost. This requires effective multi-objective search algorithms to alleviate the GPU costs. In this work, we implement a novel multi-objectives optimizer based on a recently proposed meta-algorithm called LaMOO on NAS tasks. In a nutshell, LaMOO speedups the search process by learning a model from observed samples to partition the search space and then focusing on promising regions likely to contain a subset of the Pareto frontier. Using LaMOO, we observe an improvement of more than 200% sample efficiency compared to Bayesian optimization and evolutionary-based multi-objective optimizers on different NAS datasets. For example, when combined with LaMOO, qEHVI achieves a 225% improvement in sample efficiency compared to using qEHVI alone in NasBench201. For real-world tasks, LaMOO achieves 97.36% accuracy with only 1.62M #Params on CIFAR10 in only 600 search samples. On ImageNet, our large model reaches 80.4% top-1 accuracy with only 522M #FLOPs.
Cross-Table Pretraining towards a Universal Function Space for Heterogeneous Tabular Data
Authors: Jintai Chen, Zhen Lin, Qiyuan Chen, Jimeng Sun
Date: 2024-06-01 00:00:00

Tabular data from different tables exhibit significant diversity due to varied definitions and types of features, as well as complex inter-feature and feature-target relationships. Cross-dataset pretraining, which learns reusable patterns from upstream data to support downstream tasks, have shown notable success in various fields. Yet, when applied to tabular data prediction, this paradigm faces challenges due to the limited reusable patterns among diverse tabular datasets (tables) and the general scarcity of tabular data available for fine-tuning. In this study, we fill this gap by introducing a cross-table pretrained Transformer, XTFormer, for versatile downstream tabular prediction tasks. Our methodology insight is pretraining XTFormer to establish a "meta-function" space that encompasses all potential feature-target mappings. In pre-training, a variety of potential mappings are extracted from pre-training tabular datasets and are embedded into the "meta-function" space, and suited mappings are extracted from the "meta-function" space for downstream tasks by a specified coordinate positioning approach. Experiments show that, in 190 downstream tabular prediction tasks, our cross-table pretrained XTFormer wins both XGBoost and Catboost on 137 (72%) tasks, and surpasses representative deep learning models FT-Transformer and the tabular pre-training approach XTab on 144 (76%) and 162 (85%) tasks.
$μ$LO: Compute-Efficient Meta-Generalization of Learned Optimizers
Authors: Benjamin Thérien, Charles-Étienne Joseph, Boris Knyazev, Edouard Oyallon, Irina Rish, Eugene Belilovsky
Date: 2024-05-31 00:00:00
Tasks: GPU, Zero-shot Generalization

Learned optimizers (LOs) can significantly reduce the wall-clock training time of neural networks, substantially reducing training costs. However, they often suffer from poor meta-generalization, especially when training networks larger than those seen during meta-training. To address this, we use the recently proposed Maximal Update Parametrization ($\mu$P), which allows zero-shot generalization of optimizer hyperparameters from smaller to larger models. We extend $\mu$P theory to learned optimizers, treating the meta-training problem as finding the learned optimizer under $\mu$P. Our evaluation shows that LOs meta-trained with $\mu$P substantially improve meta-generalization as compared to LOs trained under standard parametrization (SP). Notably, when applied to large-width models, our best $\mu$LO, trained for 103 GPU-hours, matches or exceeds the performance of VeLO, the largest publicly available learned optimizer, meta-trained with 4000 TPU-months of compute. Moreover, $\mu$LOs demonstrate better generalization than their SP counterparts to deeper networks and to much longer training horizons (25 times longer) than those seen during meta-training.
Multi-Class Road User Detection With 3+1D Radar in the View-of-Delft Dataset
Authors: Andras Palffy, Ewoud Pool, Srimannarayana Baratam, Julian F. P. Kooij, Dariu M. Gavrila
Date: 2022-04-01 00:00:00
Proceeding: IEEE ROBOTICS AND AUTOMATION LETTERS 2022 4
Tasks: 3D Object Detection, Benchmarking, Object, object-detection, Object Detection

Next-generation automotive radars provide elevation data in addition to range-, azimuth- and Doppler velocity. In this experimental study, we apply a state-of-the-art object detector (PointPillars), previously used for LiDAR 3D data, to such 3+1D radar data (where 1D refers to Doppler). In ablation studies, we first explore the benefits of the additional elevation information, together with that of Doppler, radar cross section and temporal accumulation, in the context of multi-class road user detection. We subsequently compare object detection performance on the radar and LiDAR point clouds, object class-wise and as a function of distance. To facilitate our experimental study, we present the novel View-of-Delft (VoD) automotive dataset. It contains 8693 frames of synchronized and calibrated 64-layer LiDAR-, (stereo) camera-, and 3+1D radar-data acquired in complex, urban traffic. It consists of 123106 3D bounding box annotations of both moving and static objects, including 26587 pedestrian, 10800 cyclist and 26949 car labels. Our results show that object detection on 64-layer LiDAR data still outperforms that on 3+1D radar data, but the addition of elevation information and integration of successive radar scans helps close the gap. The VoD dataset is made freely available for scientific benchmarking at https://intelligent-vehicles.org/datasets/view-of-delft/ .
Why the Metric Backbone Preserves Community Structure
Authors: Maximilien Dreveton, Charbel Chucri, Matthias Grossglauser, Patrick Thiran
Date: 2024-06-06 00:00:00

The metric backbone of a weighted graph is the union of all-pairs shortest paths. It is obtained by removing all edges $(u,v)$ that are not the shortest path between $u$ and $v$. In networks with well-separated communities, the metric backbone tends to preserve many inter-community edges, because these edges serve as bridges connecting two communities, but tends to delete many intra-community edges because the communities are dense. This suggests that the metric backbone would dilute or destroy the community structure of the network. However, this is not borne out by prior empirical work, which instead showed that the metric backbone of real networks preserves the community structure of the original network well. In this work, we analyze the metric backbone of a broad class of weighted random graphs with communities, and we formally prove the robustness of the community structure with respect to the deletion of all the edges that are not in the metric backbone. An empirical comparison of several graph sparsification techniques confirms our theoretical finding and shows that the metric backbone is an efficient sparsifier in the presence of communities.
RCFusion: Fusing 4-D Radar and Camera With Bird’s-Eye View Features for 3-D Object Detection
Authors: Lianqing Zheng, Sen Li, Bin Tan, Long Yan, Sihan Chen, Libo Huang, Jie Bai, Xichan Zhu, Zhixiong Ma
Date: 2023-05-23 00:00:00
Proceeding: IEEE Transactions on Instrumentation and Measurement 2023 5
Tasks: 3D Object Detection, 3D Object Detection (RoI), Autonomous Driving, object-detection, Object Detection

Camera and millimeter-wave (MMW) radar fusion is essential for accurate and robust autonomous driving systems. With the advancement of radar technology, next-generation high-resolution automotive radar, i.e., 4-D radar, has emerged. In addition to the target range, azimuth, and Doppler velocity measurements of traditional radar, 4-D radar provides elevation measurement to create a denser “point cloud.” In this study, we propose a camera and 4-D radar fusion network called RCFusion, which achieves multimodal feature fusion under a unified bird’s-eye view (BEV) space to accomplish 3-D object detection tasks. In the camera stream, multiscale feature maps are obtained by the image backbone and feature pyramid network (FPN); they are then converted into orthographic feature maps by an orthographic feature transform (OFT). Next, enhanced and fine-grained image BEV features are obtained via a designed shared attention encoder. Meanwhile, in the 4-D radar stream, a newly designed component named radar PillarNet efficiently encodes the radar features to generate radar pseudo-images, which are fed into the point cloud backbone to create radar BEV features. An interactive attention module (IAM) is proposed for the fusion stage, which outputs a valid fusion of the two-modal BEV features. Finally, a generic detection head predicts the object classes and locations. The proposed RCFusion is validated on the TJ4DRadSet and view-of-delft (VoD) datasets. The experimental results and analysis show that the proposed method can effectively fuse camera and 4-D radar features to achieve robust detection performance.
HMM Model for Brain Tumor Detection and Classification
Authors: Parth Sharma, Rakesh Sharma
Date: 2021-06-22 00:00:00
Proceeding: Intelligent Computing and Communication Systems 2021 6
Tasks: Brain Tumor Classification, Brain Tumor Segmentation, Classification, model

Brain tumor is one of the significant problems that has taken the life of a lot of people in recent times. The brain tumor can be effectively treated if detected at an early stage. However, brain tumor detection requires sophisticated algorithms. The weight-based classification techniques are commonly used for this purpose but have less detection accuracy. In this paper, the hidden Markov model (HMM)-based brain tumor classification algorithm is presented that increases the detection accuracy. The HMM classifier is simulated on MATLAB, and its efficiency in terms of PSNR, MSE, and accuracy is compared to the traditional weight-based classifier.
DynGraspVS: Servoing Aided Grasping for Dynamic Environments
Authors: Gunjan Gupta, Vedansh Mittal, K. Madhav Krishna
Date: 2023-12-22 00:00:00
Proceeding: ROBIO 2023 12
Tasks: Model Predictive Control, Robotic Grasping

Visual servoing has been gaining popularity in various real-world vision-centric robotic applications. Autonomous robotic grasping often deals with unseen and unstructured environments, and in this task, Visual Servoing has been able to generate improved end-effector control by providing visual feedback. However, existing Servoing-aided grasping methods tend to fail at the task of grasping in dynamic environments i.e. - moving objects.In this paper, we introduce DynGraspVS, a novel Image-based Visual Servoing-aided Grasping approach that models the motion of moving objects in its interaction matrix. Leveraging a single-step rollout strategy, our approach achieves a remarkable increase in success rate, while converging faster and achieving a smoother trajectory, while maintaining precise alignments in six degrees of freedom. By integrating the velocity information into the interaction matrix, our method is able to successfully complete the challenging task of robotic grasping in the case of dynamic objects, while outperforming existing deep Model Predictive Control (MPC) based methods in the PyBullet simulation environment. We test it with a range of objects in the YCB dataset with varying range of shapes, sizes, and material properties. We report various evaluation metrics such as photometric error, success rate, time taken, and trajectory length.
Weight-based Decomposition: A Case for Bilinear MLPs
Authors: Michael T. Pearce, Thomas Dooms, Alice Rigg
Date: 2024-06-06 00:00:00

Gated Linear Units (GLUs) have become a common building block in modern foundation models. Bilinear layers drop the non-linearity in the "gate" but still have comparable performance to other GLUs. An attractive quality of bilinear layers is that they can be fully expressed in terms of a third-order tensor and linear operations. Leveraging this, we develop a method to decompose the bilinear tensor into a set of sparsely interacting eigenvectors that show promising interpretability properties in preliminary experiments for shallow image classifiers (MNIST) and small language models (Tiny Stories). Since the decomposition is fully equivalent to the model's original computations, bilinear layers may be an interpretability-friendly architecture that helps connect features to the model weights. Application of our method may not be limited to pretrained bilinear models since we find that language models such as TinyLlama-1.1B can be finetuned into bilinear variants.
Evolving Efficient CNN Based Model for Image Classification
Authors: Afsaneh Shams, Drew Becker, Kyle Becker, Soheyla Amirian, Khaled Rasheed
Date: 2024-04-09 00:00:00
Proceeding: 2023 Congress in Computer Science, Computer Engineering, & Applied Computing (CSCE) 2024 4
Tasks: Classification, Evolutionary Algorithms, image-classification, Image Classification, model

Evolutionary algorithms, rooted in Darwin's theorem, can be considered as a base for implementing deep/machine learning models. This approach can noticeably increase the accuracy in most cases as shown in this paper. This experiment aims to evaluate the performance of two evolutionary algorithms, an evolutionary neural network (ENN) and an evolutionary CNN-based algorithm with mutation and crossover (ECNNB), on the Fashion-MNIST, MNIST, and EMNIST_Digits datasets. The performance of the ENN algorithm is examined for 10, 50, and 100 generations, with 50 generations being used due to computational limitations. The results show that the accuracy of the model improves as the number of generations increases. However, the ECNNB model consistently outperforms the ENN model on all three datasets, with an average accuracy of 92.58% on Fashion-MNIST, 99.32% on MNIST, and 99.50% on EMNIST_Digits, compared to 88.54 %, 98.05 %, and 98.95 %, respectively, for the ENN model. The performance of both models is compared with other state-of-the-art models in the literature. These results highlight the significance of well-designed models in achieving high accuracy in machine learning tasks.
Opinion Mining Using Pre-Trained Large Language Models: Identifying the Type, Polarity, Intensity, Expression, and Source of Private States
Authors: Saeed Ahmadnia, Arash Yousefi Jordehi, Mahsa Hosseini Khasheh Heyran, SeyedAbolghasem Mirroshandel, Owen Rambow
Date: 2024-05-01 00:00:00
Proceeding: LREC-Coling 2024 5
Tasks: Fine-Grained Opinion Analysis, Opinion Mining, Sentiment Analysis, Sentiment Classification

Opinion mining is an important task in natural language processing. The MPQA Opinion Corpus is a fine-grained and comprehensive dataset of private states (i.e., the condition of a source who has an attitude which may be directed toward a target) based on context. Although this dataset was released years ago, because of its complex definition of annotations and hard-to-read data format, almost all existing research works have only focused on a small subset of the dataset. In this paper, we present a comprehensive study of the entire MPQA 2.0 dataset. In order to achieve this goal, we first provide a clean version of MPQA 2.0 in a more interpretable format. Then, we propose two novel approaches for opinion mining, establishing new high baselines for future work. We use two pre-trained large language models, BERT and T5, to automatically identify the type, polarity, and intensity of private states expressed in phrases, and we use T5 to detect opinion expressions and their agents (i.e., sources).
Meta-Learning for Efficient Fine-Tuning of Large Language Models
Authors: Shriyansh Singh, Pramit Saha
Date: 2024-05-08 00:00:00
Proceeding: International Journal of Scientific and Research Publication (IJSRP) 2024 6
Tasks: Meta-Learning, Text Generation

In this project, we developed a parameter-efficient text generator model that generates text in the same way as a Reddit TIFU post. We used the Reddit TIFU dataset, which contains long-form text posts with titles and summaries. We explored the data and preprocessed it for training. We then designed and trained a BERT-based text generation model using PyTorch and the Hugging Face Transformers library. We tuned the hyperparameters to improve the model's performance and evaluated it using multiple metrics. Our final model achieved a high level of accuracy and can be used for various natural language processing tasks.
QUEST: Quality-Aware Metropolis-Hastings Sampling for Machine Translation
Authors: Gonçalo R. A. Faria, Sweta Agrawal, António Farinhas, Ricardo Rei, José G. C. de Souza, André F. T. Martins
Date: 2024-05-28 00:00:00
Tasks: Decoder, Machine Translation, Translation

An important challenge in machine translation (MT) is to generate high-quality and diverse translations. Prior work has shown that the estimated likelihood from the MT model correlates poorly with translation quality. In contrast, quality evaluation metrics (such as COMET or BLEURT) exhibit high correlations with human judgments, which has motivated their use as rerankers (such as quality-aware and minimum Bayes risk decoding). However, relying on a single translation with high estimated quality increases the chances of "gaming the metric''. In this paper, we address the problem of sampling a set of high-quality and diverse translations. We provide a simple and effective way to avoid over-reliance on noisy quality estimates by using them as the energy function of a Gibbs distribution. Instead of looking for a mode in the distribution, we generate multiple samples from high-density areas through the Metropolis-Hastings algorithm, a simple Markov chain Monte Carlo approach. The results show that our proposed method leads to high-quality and diverse outputs across multiple language pairs (English$\leftrightarrow${German, Russian}) with two strong decoder-only LLMs (Alma-7b, Tower-7b).
Diversity in Evolutionary Dynamics
Authors: Yuval Rabani, Leonard J. Schulman, Alistair Sinclair
Date: 2024-06-06 00:00:00
Tasks: Diversity

We consider the dynamics imposed by natural selection on the populations of two competing, sexually reproducing, haploid species. In this setting, the fitness of any genome varies over time due to the changing population mix of the competing species; crucially, this fitness variation arises naturally from the model itself, without the need for imposing it exogenously as is typically the case. Previous work on this model [14] showed that, in the special case where each of the two species exhibits just two phenotypes, genetic diversity is maintained at all times. This finding supported the tenet that sexual reproduction is advantageous because it promotes diversity, which increases the survivability of a species. In the present paper we consider the more realistic case where there are more than two phenotypes available to each species. The conclusions about diversity in general turn out to be very different from the two-phenotype case. Our first result is negative: namely, we show that sexual reproduction does not guarantee the maintenance of diversity at all times, i.e., the result of [14] does not generalize. Our counterexample consists of two competing species with just three phenotypes each. We show that, for any time~$t_0$ and any $\varepsilon>0$, there is a time $t\ge t_0$ at which the combined diversity of both species is smaller than~$\varepsilon$. Our main result is a complementary positive statement, which says that in any non-degenerate example, diversity is maintained in a weaker, "infinitely often" sense. Thus, our results refute the supposition that sexual reproduction ensures diversity at all times, but affirm a weaker assertion that extended periods of high diversity are necessarily a recurrent event.
A Statistical Characterization of Wireless Channels Conditioned on Side Information
Authors: Benedikt Böck, Michael Baur, Nurettin Turan, Dominik Semmler, Wolfgang Utschick
Date: 2024-06-06 00:00:00

Statistical prior channel knowledge, such as the wide-sense-stationary-uncorrelated-scattering (WSSUS) property, and additional side information both can be used to enhance physical layer applications in wireless communication. Generally, the wireless channel's strongly fluctuating path phases and WSSUS property characterize the channel by a zero mean and Toeplitz-structured covariance matrices in different domains. In this work, we derive a framework to comprehensively categorize side information based on whether it preserves or abandons these statistical features conditioned on the given side information. To accomplish this, we combine insights from a generic channel model with the representation of wireless channels as probabilistic graphs. Additionally, we exemplify several applications, ranging from channel modeling to estimation and clustering, which demonstrate how the proposed framework can practically enhance physical layer methods utilizing machine learning (ML).