pwc-archive/papers-with-abstracts huggingface dataset:
Edge computing service deployment and task offloading based on multi-task high-dimensional multi-objective optimization
Authors: Yanheng Guo, Yan Zhang, Linjie Wu, Mengxia Li, Xingjuan Cai, Jinjun Chen
Date: 2023-12-07 00:00:00
Tasks: Diversity, Edge-computing

The Mobile Edge Computing (MEC) system located close to the client allows mobile smart devices to offload their computations onto edge servers, enabling them to benefit from low-latency computing services. Both cloud service providers and users seek more comprehensive solutions, necessitating judicious decisions in service deployment and task offloading while balancing multiple objectives. This study investigates service deployment and task offloading challenges in a multi-user environment, framing them as a multi-task high-dimensional multi-objective optimization (MT-HD-MOO) problem within an edge environment. To ensure stable service provisioning, beyond considering latency, energy consumption, and cost as deployment objectives, network reliability is also incorporated. Furthermore, to promote equitable usage of edge servers, load balancing is introduced as a fourth task offloading objective, in addition to latency, energy consumption, and cost. Additionally, this paper designs a MT-HD-MOO algorithm based on a multi-selection strategy to address this model and its solution. By employing diverse selection strategies, an environment selection strategy pool is established to enhance population diversity within the high-dimensional objective space. Ultimately, the algorithm's effectiveness is verified through simulation experiments.
Propagating Knowledge Updates to LMs Through Distillation
Authors: Shankar Padmanabhan, Yasumasa Onoe, Michael J. Q. Zhang, Greg Durrett, Eunsol Choi
Date: 2023-06-15 00:00:00
Proceeding: NeurIPS 2023 11
Tasks: knowledge editing, Language Modelling

Modern language models have the capacity to store and use immense amounts of knowledge about real-world entities, but it remains unclear how to update such knowledge stored in model parameters. While prior methods for updating knowledge in LMs successfully inject atomic facts, updated LMs fail to make inferences based on injected facts. In this work, we demonstrate that a context distillation-based approach can both impart knowledge about entities and propagate that knowledge to enable broader inferences. Our approach consists of two stages: transfer set generation and distillation on the transfer set. We first generate a transfer set by prompting a language model to generate continuations from the entity definition. Then, we update the model parameters so that the distribution of the LM (the student) matches the distribution of the LM conditioned on the definition (the teacher) on the transfer set. Our experiments demonstrate that this approach is more effective at propagating knowledge updates than fine-tuning and other gradient-based knowledge-editing methods. Moreover, it does not compromise performance in other contexts, even when injecting the definitions of up to 150 entities at once.
Multi-Head Adapter Routing for Cross-Task Generalization
Authors: Lucas Caccia, Edoardo Ponti, Zhan Su, Matheus Pereira, Nicolas Le Roux, Alessandro Sordoni
Date: 2022-11-07 00:00:00
Proceeding: NeurIPS 2023 11
Tasks: parameter-efficient fine-tuning

Parameter-efficient fine-tuning (PEFT) for cross-task generalization consists in pre-training adapters on a multi-task training set before few-shot adaptation to test tasks. Polytropon [Ponti et al., 2023] ($\texttt{Poly}$) jointly learns an inventory of adapters and a routing function that selects a (variable-size) subset of adapters for each task during both pre-training and few-shot adaptation. In this paper, we investigate the role that adapter routing plays in its success and design new variants based on our findings. First, we build on the intuition that finer-grained routing provides more expressivity. Hence, we propose $\texttt{MHR}$ (Multi-Head Routing) which combines subsets of adapter parameters and outperforms $\texttt{Poly}$ under a comparable parameter budget; by only fine-tuning the routing function and not the adapters ($\texttt{MHR}$-$z$) we achieve competitive performance with extreme parameter efficiency. Second, we find that $\texttt{Poly}$/$\texttt{MHR}$ performance is a result of better multi-task optimization, rather than modular inductive biases that facilitate adapter recombination and local adaptation, as previously hypothesized. In fact, we find that $\texttt{MHR}$ exhibits high gradient alignment between training tasks. We find that routing is most beneficial during multi-task pre-training rather than during few-shot adaptation and propose $\texttt{MHR}$-$\mu$, which discards routing and fine-tunes the average of the pre-trained adapters on each downstream tasks. This establishes $\texttt{MHR}$-$\mu$ as an effective method for single-adapter fine-tuning. We also show that $\texttt{MHR}$-$\mu$ can be used as an effective zero-shot transfer method by training the average of the pre-trained adapters for a few additional steps on the multi-task training set: this yields gains up to 3% on absolute accuracy w.r.t. the baselines.
Fine-Tuning Language Models with Just Forward Passes
Authors: Sadhika Malladi, Tianyu Gao, Eshaan Nichani, Alex Damian, Jason D. Lee, Danqi Chen, Sanjeev Arora
Date: 2023-05-27 00:00:00
Proceeding: NeurIPS 2023 11
Tasks: GPU, In-Context Learning, Multiple-choice

Fine-tuning language models (LMs) has yielded success on diverse downstream tasks, but as LMs grow in size, backpropagation requires a prohibitively large amount of memory. Zeroth-order (ZO) methods can in principle estimate gradients using only two forward passes but are theorized to be catastrophically slow for optimizing large models. In this work, we propose a memory-efficient zerothorder optimizer (MeZO), adapting the classical ZO-SGD method to operate in-place, thereby fine-tuning LMs with the same memory footprint as inference. For example, with a single A100 80GB GPU, MeZO can train a 30-billion parameter model, whereas fine-tuning with backpropagation can train only a 2.7B LM with the same budget. We conduct comprehensive experiments across model types (masked and autoregressive LMs), model scales (up to 66B), and downstream tasks (classification, multiple-choice, and generation). Our results demonstrate that (1) MeZO significantly outperforms in-context learning and linear probing; (2) MeZO achieves comparable performance to fine-tuning with backpropagation across multiple tasks, with up to 12x memory reduction and up to 2x GPU-hour reduction in our implementation; (3) MeZO is compatible with both full-parameter and parameter-efficient tuning techniques such as LoRA and prefix tuning; (4) MeZO can effectively optimize non-differentiable objectives (e.g., maximizing accuracy or F1). We support our empirical findings with theoretical insights, highlighting how adequate pre-training and task prompts enable MeZO to fine-tune huge models, despite classical ZO analyses suggesting otherwise.
Speculative Decoding with Big Little Decoder
Authors: Sehoon Kim, Karttikeya Mangalam, Suhong Moon, Jitendra Malik, Michael W. Mahoney, Amir Gholami, Kurt Keutzer
Date: 2023-02-15 00:00:00
Proceeding: NeurIPS 2023 11
Tasks: Decoder, de-en, GPU, Machine Translation, Text Generation

The recent emergence of Large Language Models based on the Transformer architecture has enabled dramatic advancements in the field of Natural Language Processing. However, these models have long inference latency, which limits their deployment and makes them prohibitively expensive for various real-time applications. The inference latency is further exacerbated by autoregressive generative tasks, as models need to run iteratively to generate tokens sequentially without leveraging token-level parallelization. To address this, we propose Big Little Decoder (BiLD), a framework that can improve inference efficiency and latency for a wide range of text generation applications. The BiLD framework contains two models with different sizes that collaboratively generate text. The small model runs autoregressively to generate text with a low inference cost, and the large model is only invoked occasionally to refine the small model's inaccurate predictions in a non-autoregressive manner. To coordinate the small and large models, BiLD introduces two simple yet effective policies: (1) the fallback policy that determines when to hand control over to the large model; and (2) the rollback policy that determines when the large model needs to correct the small model's inaccurate predictions. To evaluate our framework across different tasks and models, we apply BiLD to various text generation scenarios encompassing machine translation on IWSLT 2017 De-En and WMT 2014 De-En, and summarization on XSUM and CNN/DailyMail. On an NVIDIA T4 GPU, our framework achieves a speedup of up to 2.12x speedup with minimal generation quality degradation. Furthermore, our framework is fully plug-and-play and can be applied without any modifications in the training process or model architecture. Our code is open-sourced
ProtoDiff: Learning to Learn Prototypical Networks by Task-Guided Diffusion
Authors: Yingjun Du, Zehao Xiao, Shengcai Liao, Cees Snoek
Date: 2023-06-26 00:00:00
Proceeding: NeurIPS 2023 11
Tasks: Few-Shot Learning, Meta-Learning

Prototype-based meta-learning has emerged as a powerful technique for addressing few-shot learning challenges. However, estimating a deterministic prototype using a simple average function from a limited number of examples remains a fragile process. To overcome this limitation, we introduce ProtoDiff, a novel framework that leverages a task-guided diffusion model during the meta-training phase to gradually generate prototypes, thereby providing efficient class representations. Specifically, a set of prototypes is optimized to achieve per-task prototype overfitting, enabling accurately obtaining the overfitted prototypes for individual tasks. Furthermore, we introduce a task-guided diffusion process within the prototype space, enabling the meta-learning of a generative process that transitions from a vanilla prototype to an overfitted prototype. ProtoDiff gradually generates task-specific prototypes from random noise during the meta-test stage, conditioned on the limited samples available for the new task. Furthermore, to expedite training and enhance ProtoDiff's performance, we propose the utilization of residual prototype learning, which leverages the sparsity of the residual prototype. We conduct thorough ablation studies to demonstrate its ability to accurately capture the underlying prototype distribution and enhance generalization. The new state-of-the-art performance on within-domain, cross-domain, and few-task few-shot classification further substantiates the benefit of ProtoDiff.
Navigating the Pitfalls of Active Learning Evaluation: A Systematic Framework for Meaningful Performance Assessment
Authors: Carsten T. Lüth, Till J. Bungert, Lukas Klein, Paul F. Jaeger
Date: 2023-01-25 00:00:00
Proceeding: NeurIPS 2023 11
Tasks: Active Learning, image-classification, Image Classification, Self-Supervised Learning

Active Learning (AL) aims to reduce the labeling burden by interactively selecting the most informative samples from a pool of unlabeled data. While there has been extensive research on improving AL query methods in recent years, some studies have questioned the effectiveness of AL compared to emerging paradigms such as semi-supervised (Semi-SL) and self-supervised learning (Self-SL), or a simple optimization of classifier configurations. Thus, today's AL literature presents an inconsistent and contradictory landscape, leaving practitioners uncertain about whether and how to use AL in their tasks. In this work, we make the case that this inconsistency arises from a lack of systematic and realistic evaluation of AL methods. Specifically, we identify five key pitfalls in the current literature that reflect the delicate considerations required for AL evaluation. Further, we present an evaluation framework that overcomes these pitfalls and thus enables meaningful statements about the performance of AL methods. To demonstrate the relevance of our protocol, we present a large-scale empirical study and benchmark for image classification spanning various data sets, query methods, AL settings, and training paradigms. Our findings clarify the inconsistent picture in the literature and enable us to give hands-on recommendations for practitioners. The benchmark is hosted at https://github.com/IML-DKFZ/realistic-al .
FeCAM: Exploiting the Heterogeneity of Class Distributions in Exemplar-Free Continual Learning
Authors: Dipam Goswami, Yuyang Liu, Bartłomiej Twardowski, Joost Van de Weijer
Date: 2023-09-25 00:00:00
Proceeding: NeurIPS 2023 11
Tasks: class-incremental learning, Class Incremental Learning, Continual Learning, Exemplar-Free, Incremental Learning

Exemplar-free class-incremental learning (CIL) poses several challenges since it prohibits the rehearsal of data from previous tasks and thus suffers from catastrophic forgetting. Recent approaches to incrementally learning the classifier by freezing the feature extractor after the first task have gained much attention. In this paper, we explore prototypical networks for CIL, which generate new class prototypes using the frozen feature extractor and classify the features based on the Euclidean distance to the prototypes. In an analysis of the feature distributions of classes, we show that classification based on Euclidean metrics is successful for jointly trained features. However, when learning from non-stationary data, we observe that the Euclidean metric is suboptimal and that feature distributions are heterogeneous. To address this challenge, we revisit the anisotropic Mahalanobis distance for CIL. In addition, we empirically show that modeling the feature covariance relations is better than previous attempts at sampling features from normal distributions and training a linear classifier. Unlike existing methods, our approach generalizes to both many- and few-shot CIL settings, as well as to domain-incremental settings. Interestingly, without updating the backbone network, our method obtains state-of-the-art results on several standard continual learning benchmarks. Code is available at https://github.com/dipamgoswami/FeCAM.
Joint Learning of Label and Environment Causal Independence for Graph Out-of-Distribution Generalization
Authors: Shurui Gui, Meng Liu, Xiner Li, Youzhi Luo, Shuiwang Ji
Date: 2023-06-01 00:00:00
Proceeding: NeurIPS 2023 11
Tasks: Out-of-Distribution Generalization

We tackle the problem of graph out-of-distribution (OOD) generalization. Existing graph OOD algorithms either rely on restricted assumptions or fail to exploit environment information in training data. In this work, we propose to simultaneously incorporate label and environment causal independence (LECI) to fully make use of label and environment information, thereby addressing the challenges faced by prior methods on identifying causal and invariant subgraphs. We further develop an adversarial training strategy to jointly optimize these two properties for causal subgraph discovery with theoretical guarantees. Extensive experiments and analysis show that LECI significantly outperforms prior methods on both synthetic and real-world datasets, establishing LECI as a practical and effective solution for graph OOD generalization. Our code is available at https://github.com/divelab/LECI.
PlanE: Representation Learning over Planar Graphs
Authors: Radoslav Dimitrov, Zeyang Zhao, Ralph Abboud, İsmail İlkan Ceylan
Date: 2023-07-03 00:00:00
Proceeding: NeurIPS 2023 11
Tasks: Isomorphism Testing, Representation Learning

Graph neural networks are prominent models for representation learning over graphs, where the idea is to iteratively compute representations of nodes of an input graph through a series of transformations in such a way that the learned graph function is isomorphism invariant on graphs, which makes the learned representations graph invariants. On the other hand, it is well-known that graph invariants learned by these class of models are incomplete: there are pairs of non-isomorphic graphs which cannot be distinguished by standard graph neural networks. This is unsurprising given the computational difficulty of graph isomorphism testing on general graphs, but the situation begs to differ for special graph classes, for which efficient graph isomorphism testing algorithms are known, such as planar graphs. The goal of this work is to design architectures for efficiently learning complete invariants of planar graphs. Inspired by the classical planar graph isomorphism algorithm of Hopcroft and Tarjan, we propose PlanE as a framework for planar representation learning. PlanE includes architectures which can learn complete invariants over planar graphs while remaining practically scalable. We empirically validate the strong performance of the resulting model architectures on well-known planar graph benchmarks, achieving multiple state-of-the-art results.
DYffusion: A Dynamics-informed Diffusion Model for Spatiotemporal Forecasting
Authors: Salva Rühling Cachay, Bo Zhao, Hailey Joren, Rose Yu
Date: 2023-06-03 00:00:00
Proceeding: NeurIPS 2023 11
Tasks: Computational Efficiency, Inductive Bias

While diffusion models can successfully generate data and make predictions, they are predominantly designed for static images. We propose an approach for efficiently training diffusion models for probabilistic spatiotemporal forecasting, where generating stable and accurate rollout forecasts remains challenging, Our method, DYffusion, leverages the temporal dynamics in the data, directly coupling it with the diffusion steps in the model. We train a stochastic, time-conditioned interpolator and a forecaster network that mimic the forward and reverse processes of standard diffusion models, respectively. DYffusion naturally facilitates multi-step and long-range forecasting, allowing for highly flexible, continuous-time sampling trajectories and the ability to trade-off performance with accelerated sampling at inference time. In addition, the dynamics-informed diffusion process in DYffusion imposes a strong inductive bias and significantly improves computational efficiency compared to traditional Gaussian noise-based diffusion models. Our approach performs competitively on probabilistic forecasting of complex dynamics in sea surface temperatures, Navier-Stokes flows, and spring mesh systems.
PriorBand: Practical Hyperparameter Optimization in the Age of Deep Learning
Authors: Neeratyoy Mallik, Edward Bergman, Carl Hvarfner, Danny Stoll, Maciej Janowski, Marius Lindauer, Luigi Nardi, Frank Hutter
Date: 2023-06-21 00:00:00
Proceeding: NeurIPS 2023 11
Tasks: Deep Learning, Hyperparameter Optimization

Hyperparameters of Deep Learning (DL) pipelines are crucial for their downstream performance. While a large number of methods for Hyperparameter Optimization (HPO) have been developed, their incurred costs are often untenable for modern DL. Consequently, manual experimentation is still the most prevalent approach to optimize hyperparameters, relying on the researcher's intuition, domain knowledge, and cheap preliminary explorations. To resolve this misalignment between HPO algorithms and DL researchers, we propose PriorBand, an HPO algorithm tailored to DL, able to utilize both expert beliefs and cheap proxy tasks. Empirically, we demonstrate PriorBand's efficiency across a range of DL benchmarks and show its gains under informative expert input and robustness against poor expert beliefs
Construction of Hierarchical Neural Architecture Search Spaces based on Context-free Grammars
Authors: Simon Schrodi, Danny Stoll, Binxin Ru, Rhea Sukthanker, Thomas Brox, Frank Hutter
Date: 2022-11-03 00:00:00
Proceeding: NeurIPS 2023 11
Tasks: AutoML, Bayesian Optimization, Neural Architecture Search

The discovery of neural architectures from simple building blocks is a long-standing goal of Neural Architecture Search (NAS). Hierarchical search spaces are a promising step towards this goal but lack a unifying search space design framework and typically only search over some limited aspect of architectures. In this work, we introduce a unifying search space design framework based on context-free grammars that can naturally and compactly generate expressive hierarchical search spaces that are 100s of orders of magnitude larger than common spaces from the literature. By enhancing and using their properties, we effectively enable search over the complete architecture and can foster regularity. Further, we propose an efficient hierarchical kernel design for a Bayesian Optimization search strategy to efficiently search over such huge spaces. We demonstrate the versatility of our search space design framework and show that our search strategy can be superior to existing NAS approaches. Code is available at https://github.com/automl/hierarchical_nas_construction.
Full Model-Free Control Architecture for Hybrid UAVs
Authors: Jacson M. O. Barth1, Jean-Philippe Condomines1
Date: 2019-10-10 00:00:00
Proceeding: journal 2019 10
Tasks: model, Position

This paper discusses the development of a control
architecture for hybrid Unmanned Aerial Vehicles (UAVs) based
on model-free control (MFC) algorithms. Hybrid UAVs combine
the beneficial features of fixed-wing UAVs with Vertical TakeOff and Landing (VTOL) capabilities to perform five different
flight phases during typical missions, such as vertical takeoff, transitioning flight, forward flight, hovering and vertical
landing. Based on model-free control principles, a novel control
architecture that handles the hybrid UAV dynamics at any flight
phase is presented. This unified controller allows autonomous
flights without discontinuities of switching for the entire flight
envelope with position tracking, velocity control and attitude
stabilization. Simulation results show that the proposed control
architecture provides an effective control performance for
the entire flight envelope and excellent disturbance rejections
during the critical flight phases, such as transitioning and
hovering flights in windy conditions.
A multi-layer refined network model for the identification of essential proteins
Authors: Haoyue Wang, Li Pan, Bo Yang, Junqiang Jiang, Wenbin Li
Date: 2023-12-06 00:00:00
Tasks: Specificity

The identification of essential proteins in protein-protein interaction networks (PINs) can help to discover drug targets and prevent disease. In order to improve the accuracy of the identification of essential proteins, researchers attempted to obtain a refined PIN by combining multiple biological information to filter out some unreliable interactions in the PIN. Unfortunately, such approaches drastically reduce the number of nodes in the PIN after multiple refinements and result in a sparser PIN. It makes a considerable portion of essential proteins unidentifiable. In this paper, we propose a multi-layer refined network (MR-PIN) that addresses this problem. Firstly, four refined networks are constructed by respectively integrating different biological information into the static PIN to form a multi-layer heterogeneous network. Then scores of proteins in each network layer are calculated by the existing node ranking method, and the importance score of a protein in the MR-PIN is evaluated in terms of the geometric mean of its scores in all layers. Finally, all nodes are sorted by their importance scores to determine their essentiality. To evaluate the effectiveness of the multi-layer refined network model, we apply 16 node ranking methods on the MR-PIN, and compare the results with those on the SPIN, DPIN and RDPIN. Then the predictive performances of these ranking methods are validated in terms of the identification number of essential protein at top100 - top600, sensitivity, specificity, positive predictive value, negative predictive value, F-measure, accuracy, Jackknife, ROCAUC and PRAUC. The experimental results show that the MR-PIN is superior to the existing refined PINs in the identification accuracy of essential proteins.