pwc-archive/papers-with-abstracts huggingface dataset:
Towards MoE Deployment: Mitigating Inefficiencies in Mixture-of-Expert (MoE) Inference
Authors: Haiyang Huang, Newsha Ardalani, Anna Sun, Liu Ke, Hsien-Hsin S. Lee, Anjali Sridhar, Shruti Bhosale, Carole-Jean Wu, Benjamin Lee
Date: 2023-03-10 00:00:00
Tasks: CPU, Decoder, GPU, Language Modeling, Language Modelling, Machine Translation, Mixture-of-Experts

Mixture-of-Experts (MoE) models have gained popularity in achieving state-of-the-art performance in a wide range of tasks in computer vision and natural language processing. They effectively expand the model capacity while incurring a minimal increase in computation cost during training. However, deploying such models for inference is difficult due to their large size and complex communication pattern. In this work, we provide a characterization of two MoE workloads, namely Language Modeling (LM) and Machine Translation (MT) and identify their sources of inefficiencies at deployment. We propose three optimization techniques to mitigate sources of inefficiencies, namely (1) Dynamic gating, (2) Expert Buffering, and (3) Expert load balancing. We show that dynamic gating improves maximum throughput by 6.21-11.23$\times$ for LM, 5.75-10.98$\times$ for MT Encoder and 2.58-5.71$\times$ for MT Decoder. It also reduces memory usage by up to 1.36$\times$ for LM and up to 1.1$\times$ for MT. We further propose Expert Buffering, a new caching mechanism that only keeps hot, active experts in GPU memory while buffering the rest in CPU memory. This reduces static memory allocation by up to 1.47$\times$. We finally propose a load balancing methodology that provides additional scalability to the workload.
DP-Fast MH: Private, Fast, and Accurate Metropolis-Hastings for Large-Scale Bayesian Inference
Authors: Wanrong Zhang, Ruqi Zhang
Date: 2023-03-10 00:00:00
Tasks: Bayesian Inference, Drug Design, Medical Diagnosis, Privacy Preserving

Bayesian inference provides a principled framework for learning from complex data and reasoning under uncertainty. It has been widely applied in machine learning tasks such as medical diagnosis, drug design, and policymaking. In these common applications, data can be highly sensitive. Differential privacy (DP) offers data analysis tools with powerful worst-case privacy guarantees and has been developed as the leading approach in privacy-preserving data analysis. In this paper, we study Metropolis-Hastings (MH), one of the most fundamental MCMC methods, for large-scale Bayesian inference under differential privacy. While most existing private MCMC algorithms sacrifice accuracy and efficiency to obtain privacy, we provide the first exact and fast DP MH algorithm, using only a minibatch of data in most iterations. We further reveal, for the first time, a three-way trade-off among privacy, scalability (i.e. the batch size), and efficiency (i.e. the convergence rate), theoretically characterizing how privacy affects the utility and computational cost in Bayesian inference. We empirically demonstrate the effectiveness and efficiency of our algorithm in various experiments.
MOELA: A Multi-Objective Evolutionary/Learning Design Space Exploration Framework for 3D Heterogeneous Manycore Platforms
Authors: Sirui Qi, Yingheng Li, Sudeep Pasricha, Ryan Gary Kim
Date: 2023-03-10 00:00:00

To enable emerging applications such as deep machine learning and graph processing, 3D network-on-chip (NoC) enabled heterogeneous manycore platforms that can integrate many processing elements (PEs) are needed. However, designing such complex systems with multiple objectives can be challenging due to the huge associated design space and long evaluation times. To optimize such systems, we propose a new multi-objective design space exploration framework called MOELA that combines the benefits of evolutionary-based search with a learning-based local search to quickly determine PE and communication link placement to optimize multiple objectives (e.g., latency, throughput, and energy) in 3D NoC enabled heterogeneous manycore systems. Compared to state-of-the-art approaches, MOELA increases the speed of finding solutions by up to 128x, leads to a better Pareto Hypervolume (PHV) by up to 12.14x and improves energy-delay-product (EDP) by up to 7.7% in a 5-objective scenario.
Automated classification of pre-defined movement patterns: A comparison between GNSS and UWB technology
Authors: Rodi Laanen, Maedeh Nasri, Richard van Dijk, Mitra Baratchi, Alexander Koutamanis, Carolien Rieffe
Date: 2023-03-10 00:00:00
Tasks: SMAC, SMAC+

Advanced real-time location systems (RTLS) allow for collecting spatio-temporal data from human movement behaviours. Tracking individuals in small areas such as schoolyards or nursing homes might impose difficulties for RTLS in terms of positioning accuracy. However, to date, few studies have investigated the performance of different localisation systems regarding the classification of human movement patterns in small areas. The current study aims to design and evaluate an automated framework to classify human movement trajectories obtained from two different RTLS: Global Navigation Satellite System (GNSS) and Ultra-wideband (UWB), in areas of approximately 100 square meters. Specifically, we designed a versatile framework which takes GNSS or UWB data as input, extracts features from these data and classifies them according to the annotated spatial patterns. The automated framework contains three choices for applying noise removal: (i) no noise removal, (ii) Savitzky Golay filter on the raw location data or (iii) Savitzky Golay filter on the extracted features, as well as three choices regarding the classification algorithm: Decision Tree (DT), Random Forest (RF) or Support Vector Machine (SVM). We integrated different stages within the framework with the Sequential Model-Based Algorithm Configuration (SMAC) to perform automated hyperparameter optimisation. The best performance is achieved with a pipeline consisting of noise removal applied to the raw location data with an RF model for the GNSS and no noise removal with an SVM model for the UWB. We further demonstrate through statistical analysis that the UWB achieves significantly higher results than the GNSS in classifying movement patterns.
Operating data of a specific Aquatic Center as a Benchmark for dynamic model learning: search for a valid prediction model over an 8-hour horizon
Authors: François Gauthier-Clerc, Hoel Le Capitaine, Fabien Claveau, Philippe Chevrel
Date: 2023-03-10 00:00:00
Tasks: valid

This article presents an identification benchmark based on data from a public swimming pool in operation. Such a system is both a complex process and easily understandable by all with regard to the stakes. Ultimately, the objective is to reduce the energy bill while maintaining the level of quality of service. This objective is general in scope and is not limited to public swimming pools. This can be done effectively through what is known as economic predictive control. This type of advanced control is based on a process model. It is the aim of this article and the considered benchmark to show that such a dynamic model can be obtained from operating data. For this, operational data is formatted and shared, and model quality indicators are proposed. On this basis, the first identification results illustrate the results obtained by a linear multivariable model on the one hand, and by a neural dynamic model on the other hand. The benchmark calls for other proposals and results from control and data scientists for comparison.
Control of synaptic plasticity in neural networks
Authors: Mohammad Modiri
Date: 2023-03-10 00:00:00

The brain is a nonlinear and highly Recurrent Neural Network (RNN). This RNN is surprisingly plastic and supports our astonishing ability to learn and execute complex tasks. However, learning is incredibly complicated due to the brain's nonlinear nature and the obscurity of mechanisms for determining the contribution of each synapse to the output error. This issue is known as the Credit Assignment Problem (CAP) and is a fundamental challenge in neuroscience and Artificial Intelligence (AI). Nevertheless, in the current understanding of cognitive neuroscience, it is widely accepted that a feedback loop systems play an essential role in synaptic plasticity. With this as inspiration, we propose a computational model by combining Neural Networks (NN) and nonlinear optimal control theory. The proposed framework involves a new NN-based actor-critic method which is used to simulate the error feedback loop systems and projections on the NN's synaptic plasticity so as to ensure that the output error is minimized.
Uncertainty quantification in neural network classifiers -- a local linear approach
Authors: Magnus Malmström, Isaac Skog, Daniel Axehill, Fredrik Gustafsson
Date: 2023-03-10 00:00:00
Tasks: image-classification, Image Classification, Uncertainty Quantification

Classifiers based on neural networks (NN) often lack a measure of uncertainty in the predicted class. We propose a method to estimate the probability mass function (PMF) of the different classes, as well as the covariance of the estimated PMF. First, a local linear approach is used during the training phase to recursively compute the covariance of the parameters in the NN. Secondly, in the classification phase another local linear approach is used to propagate the covariance of the learned NN parameters to the uncertainty in the output of the last layer of the NN. This allows for an efficient Monte Carlo (MC) approach for: (i) estimating the PMF; (ii) calculating the covariance of the estimated PMF; and (iii) proper risk assessment and fusion of multiple classifiers. Two classical image classification tasks, i.e., MNIST, and CFAR10, are used to demonstrate the efficiency the proposed method.
Accounting for multiplicity in machine learning benchmark performance
Authors: Kajsa Møllersen, Einar Holsbø
Date: 2023-03-10 00:00:00

Machine learning methods are commonly evaluated and compared by their performance on data sets from public repositories. This allows for multiple methods, oftentimes several thousands, to be evaluated under identical conditions and across time. The highest ranked performance on a problem is referred to as state-of-the-art (SOTA) performance, and is used, among other things, as a reference point for publication of new methods. Using the highest-ranked performance as an estimate for SOTA is a biased estimator, giving overly optimistic results. The mechanisms at play are those of multiplicity, a topic that is well-studied in the context of multiple comparisons and multiple testing, but has, as far as the authors are aware of, been nearly absent from the discussion regarding SOTA estimates. The optimistic state-of-the-art estimate is used as a standard for evaluating new methods, and methods with substantial inferior results are easily overlooked. In this article, we provide a probability distribution for the case of multiple classifiers so that known analyses methods can be engaged and a better SOTA estimate can be provided. We demonstrate the impact of multiplicity through a simulated example with independent classifiers. We show how classifier dependency impacts the variance, but also that the impact is limited when the accuracy is high. Finally, we discuss three real-world examples; Kaggle competitions that demonstrate various aspects.
Adaptive Weight Assignment Scheme For Multi-task Learning
Authors: Aminul Huq, Mst Tasnim Pervin
Date: 2023-03-10 00:00:00
Tasks: Multi-Task Learning

Deep learning based models are used regularly in every applications nowadays. Generally we train a single model on a single task. However, we can train multiple tasks on a single model under multi-task learning settings. This provides us many benefits like lesser training time, training a single model for multiple tasks, reducing overfitting, improving performances etc. To train a model in multi-task learning settings we need to sum the loss values from different tasks. In vanilla multi-task learning settings we assign equal weights but since not all tasks are of similar difficulty we need to allocate more weight to tasks which are more difficult. Also improper weight assignment reduces the performance of the model. We propose a simple weight assignment scheme in this paper which improves the performance of the model and puts more emphasis on difficult tasks. We tested our methods performance on both image and textual data and also compared performance against two popular weight assignment methods. Empirical results suggest that our proposed method achieves better results compared to other popular methods.
Resource saving taxonomy classification with k-mer distributions and machine learning
Authors: Wolfgang Fuhl, Susanne Zabel, Kay Nieselt
Date: 2023-03-10 00:00:00

Modern high throughput sequencing technologies like metagenomic sequencing generate millions of sequences which have to be classified based on their taxonomic rank. Modern approaches either apply local alignment and comparison to existing data sets like MMseqs2 or use deep neural networks as it is done in DeepMicrobes and BERTax. Alignment-based approaches are costly in terms of runtime, especially since databases get larger and larger. For the deep learning-based approaches, specialized hardware is necessary for a computation, which consumes large amounts of energy. In this paper, we propose to use $k$-mer distributions obtained from DNA as features to classify its taxonomic origin using machine learning approaches like the subspace $k$-nearest neighbors algorithm, neural networks or bagged decision trees. In addition, we propose a feature space data set balancing approach, which allows reducing the data set for training and improves the performance of the classifiers. By comparing performance, time, and memory consumption of our approach to those of state-of-the-art algorithms (BERTax and MMseqs2) using several datasets, we show that our approach improves the classification on the genus level and achieves comparable results for the superkingdom and phylum level. Link: https://es-cloud.cs.uni-tuebingen.de/d/8e2ab8c3fdd444e1a135/?p=%2FTaxonomyClassification&mode=list
Neural Partial Differential Equations with Functional Convolution
Authors: Ziqian Wu, Xingzhe He, Yijun Li, Cheng Yang, Rui Liu, Shiying Xiong, Bo Zhu
Date: 2023-03-10 00:00:00

We present a lightweighted neural PDE representation to discover the hidden structure and predict the solution of different nonlinear PDEs. Our key idea is to leverage the prior of ``translational similarity'' of numerical PDE differential operators to drastically reduce the scale of learning model and training data. We implemented three central network components, including a neural functional convolution operator, a Picard forward iterative procedure, and an adjoint backward gradient calculator. Our novel paradigm fully leverages the multifaceted priors that stem from the sparse and smooth nature of the physical PDE solution manifold and the various mature numerical techniques such as adjoint solver, linearization, and iterative procedure to accelerate the computation. We demonstrate the efficacy of our method by robustly discovering the model and accurately predicting the solutions of various types of PDEs with small-scale networks and training sets. We highlight that all the PDE examples we showed were trained with up to 8 data samples and within 325 network parameters.
Transformer-based approaches to Sentiment Detection
Authors: Olumide Ebenezer Ojo, Hoang Thang Ta, Alexander Gelbukh, Hiram Calvo, Olaronke Oluwayemisi Adebanji, Grigori Sidorov
Date: 2023-03-13 00:00:00
Tasks: text-classification, Text Classification, Transfer Learning

The use of transfer learning methods is largely responsible for the present breakthrough in Natural Learning Processing (NLP) tasks across multiple domains. In order to solve the problem of sentiment detection, we examined the performance of four different types of well-known state-of-the-art transformer models for text classification. Models such as Bidirectional Encoder Representations from Transformers (BERT), Robustly Optimized BERT Pre-training Approach (RoBERTa), a distilled version of BERT (DistilBERT), and a large bidirectional neural network architecture (XLNet) were proposed. The performance of the four models that were used to detect disaster in the text was compared. All the models performed well enough, indicating that transformer-based models are suitable for the detection of disaster in text. The RoBERTa transformer model performs best on the test dataset with a score of 82.6% and is highly recommended for quality predictions. Furthermore, we discovered that the learning algorithms' performance was influenced by the pre-processing techniques, the nature of words in the vocabulary, unbalanced labeling, and the model parameters.
Are Models Trained on Indian Legal Data Fair?
Authors: Sahil Girhepuje, Anmol Goel, Gokul S Krishnan, Shreya Goyal, Satyendra Pandey, Ponnurangam Kumaraguru, Balaraman Ravindran
Date: 2023-03-13 00:00:00
Tasks: Fairness, Prediction

Recent advances and applications of language technology and artificial intelligence have enabled much success across multiple domains like law, medical and mental health. AI-based Language Models, like Judgement Prediction, have recently been proposed for the legal sector. However, these models are strife with encoded social biases picked up from the training data. While bias and fairness have been studied across NLP, most studies primarily locate themselves within a Western context. In this work, we present an initial investigation of fairness from the Indian perspective in the legal domain. We highlight the propagation of learnt algorithmic biases in the bail prediction task for models trained on Hindi legal documents. We evaluate the fairness gap using demographic parity and show that a decision tree model trained for the bail prediction task has an overall fairness disparity of 0.237 between input features associated with Hindus and Muslims. Additionally, we highlight the need for further research and studies in the avenues of fairness/bias in applying AI in the legal sector with a specific focus on the Indian context.
A Comprehensive Empirical Evaluation of Existing Word Embedding Approaches
Authors: Obaidullah Zaland, Muhammad Abulaish, Mohd. Fazil
Date: 2023-03-13 00:00:00

Vector-based word representations help countless Natural Language Processing (NLP) tasks capture the language's semantic and syntactic regularities. In this paper, we present the characteristics of existing word embedding approaches and analyze them with regard to many classification tasks. We categorize the methods into two main groups - Traditional approaches mostly use matrix factorization to produce word representations, and they are not able to capture the semantic and syntactic regularities of the language very well. On the other hand, Neural-network-based approaches can capture sophisticated regularities of the language and preserve the word relationships in the generated word representations. We report experimental results on multiple classification tasks and highlight the scenarios where one approach performs better than the rest.
Large Language Models in the Workplace: A Case Study on Prompt Engineering for Job Type Classification
Authors: Benjamin Clavié, Alexandru Ciceu, Frederick Naylor, Guillaume Soulié, Thomas Brightwell
Date: 2023-03-13 00:00:00
Tasks: Job classification, Prompt Engineering, text-classification, Text Classification, zero-shot-classification, Zero-Shot Learning

This case study investigates the task of job classification in a real-world setting, where the goal is to determine whether an English-language job posting is appropriate for a graduate or entry-level position. We explore multiple approaches to text classification, including supervised approaches such as traditional models like Support Vector Machines (SVMs) and state-of-the-art deep learning methods such as DeBERTa. We compare them with Large Language Models (LLMs) used in both few-shot and zero-shot classification settings. To accomplish this task, we employ prompt engineering, a technique that involves designing prompts to guide the LLMs towards the desired output. Specifically, we evaluate the performance of two commercially available state-of-the-art GPT-3.5-based language models, text-davinci-003 and gpt-3.5-turbo. We also conduct a detailed analysis of the impact of different aspects of prompt engineering on the model's performance. Our results show that, with a well-designed prompt, a zero-shot gpt-3.5-turbo classifier outperforms all other models, achieving a 6% increase in Precision@95% Recall compared to the best supervised approach. Furthermore, we observe that the wording of the prompt is a critical factor in eliciting the appropriate "reasoning" in the model, and that seemingly minor aspects of the prompt significantly affect the model's performance.
Generating multiple-choice questions for medical question answering with distractors and cue-masking
Authors: Damien Sileo, Kanimozhi Uma, Marie-Francine Moens
Date: 2023-03-13 00:00:00
Tasks: Language Modeling, Language Modelling, Masked Language Modeling, Medical Question Answering, MedQA, Multiple-choice, Multiple Choice Question Answering (MCQA), Question Answering

Medical multiple-choice question answering (MCQA) is particularly difficult. Questions may describe patient symptoms and ask for the correct diagnosis, which requires domain knowledge and complex reasoning. Standard language modeling pretraining alone is not sufficient to achieve the best results. \citet{jin2020disease} showed that focusing masked language modeling on disease name prediction when using medical encyclopedic paragraphs as input leads to considerable MCQA accuracy improvement. In this work, we show that (1) fine-tuning on generated MCQA dataset outperforms the masked language modeling based objective and (2) correctly masking the cues to the answers is critical for good performance. We release new pretraining datasets and achieve state-of-the-art results on 4 MCQA datasets, notably +5.7\% with base-size model on MedQA-USMLE.
A Human Subject Study of Named Entity Recognition (NER) in Conversational Music Recommendation Queries
Authors: Elena V. Epure, Romain Hennequin
Date: 2023-03-13 00:00:00
Tasks: Music Recommendation, named-entity-recognition, Named Entity Recognition, Named Entity Recognition (NER), NER

We conducted a human subject study of named entity recognition on a noisy corpus of conversational music recommendation queries, with many irregular and novel named entities. We evaluated the human NER linguistic behaviour in these challenging conditions and compared it with the most common NER systems nowadays, fine-tuned transformers. Our goal was to learn about the task to guide the design of better evaluation methods and NER algorithms. The results showed that NER in our context was quite hard for both human and algorithms under a strict evaluation schema; humans had higher precision, while the model higher recall because of entity exposure especially during pre-training; and entity types had different error patterns (e.g. frequent typing errors for artists). The released corpus goes beyond predefined frames of interaction and can support future work in conversational music recommendation.