pwc-archive/papers-with-abstracts huggingface dataset:
GraphGLOW: Universal and Generalizable Structure Learning for Graph Neural Networks
Authors: Wentao Zhao, Qitian Wu, Chenxiao Yang, Junchi Yan
Date: 2023-06-20 00:00:00
Tasks: Graph structure learning

Graph structure learning is a well-established problem that aims at optimizing graph structures adaptive to specific graph datasets to help message passing neural networks (i.e., GNNs) to yield effective and robust node embeddings. However, the common limitation of existing models lies in the underlying \textit{closed-world assumption}: the testing graph is the same as the training graph. This premise requires independently training the structure learning model from scratch for each graph dataset, which leads to prohibitive computation costs and potential risks for serious over-fitting. To mitigate these issues, this paper explores a new direction that moves forward to learn a universal structure learning model that can generalize across graph datasets in an open world. We first introduce the mathematical definition of this novel problem setting, and describe the model formulation from a probabilistic data-generative aspect. Then we devise a general framework that coordinates a single graph-shared structure learner and multiple graph-specific GNNs to capture the generalizable patterns of optimal message-passing topology across datasets. The well-trained structure learner can directly produce adaptive structures for unseen target graphs without any fine-tuning. Across diverse datasets and various challenging cross-graph generalization protocols, our experiments show that even without training on target graphs, the proposed model i) significantly outperforms expressive GNNs trained on input (non-optimized) topology, and ii) surprisingly performs on par with state-of-the-art models that independently optimize adaptive structures for specific target graphs, with notably orders-of-magnitude acceleration for training on the target graph.
HK-LegiCoST: Leveraging Non-Verbatim Transcripts for Speech Translation
Authors: Cihan Xiao, Henry Li Xinyuan, Jinyi Yang, Dongji Gao, Matthew Wiesner, Kevin Duh, Sanjeev Khudanpur
Date: 2023-06-20 00:00:00
Tasks: Cross-corpus, Sentence, speech-recognition, Speech Recognition, Translation

We introduce HK-LegiCoST, a new three-way parallel corpus of Cantonese-English translations, containing 600+ hours of Cantonese audio, its standard traditional Chinese transcript, and English translation, segmented and aligned at the sentence level. We describe the notable challenges in corpus preparation: segmentation, alignment of long audio recordings, and sentence-level alignment with non-verbatim transcripts. Such transcripts make the corpus suitable for speech translation research when there are significant differences between the spoken and written forms of the source language. Due to its large size, we are able to demonstrate competitive speech translation baselines on HK-LegiCoST and extend them to promising cross-corpus results on the FLEURS Cantonese subset. These results deliver insights into speech recognition and translation research in languages for which non-verbatim or ``noisy'' transcription is common due to various factors, including vernacular and dialectal speech.
InRank: Incremental Low-Rank Learning
Authors: Jiawei Zhao, Yifei Zhang, Beidi Chen, Florian Schäfer, Anima Anandkumar
Date: 2023-06-20 00:00:00
Tasks: Computational Efficiency

The theory of greedy low-rank learning (GLRL) aims to explain the impressive generalization capabilities of deep learning. It proves that stochastic gradient-based training implicitly regularizes neural networks towards low-rank solutions through a gradual increase of the rank during training. However, there is a gap between theory and practice since GLRL requires an infinitesimal initialization of the weights, which is not practical due to the fact that it is a saddle point. In this work, we remove the assumption of infinitesimal initialization by focusing on cumulative weight updates. We prove the cumulative weight updates follow an incremental low-rank trajectory for arbitrary orthogonal initialization of weights in a three-layer linear network. Empirically, we demonstrate that our theory holds on a broad range of neural networks (e.g., transformers) and standard training algorithms (e.g., SGD, Adam). However, existing training algorithms do not exploit the low-rank property to improve computational efficiency as the networks are not parameterized in low-rank. To remedy this, we design a new training algorithm Incremental Low-Rank Learning (InRank), which explicitly expresses cumulative weight updates as low-rank matrices while incrementally augmenting their ranks during training. We evaluate InRank on GPT-2, and our results indicate that InRank achieves comparable prediction performance as the full-rank counterpart while requiring at most 33% of the total ranks throughout training. We also propose an efficient version of InRank that achieves a reduction of 37% in total training time and 36% in model size when training GPT-medium on WikiText-103 from scratch.
LoSparse: Structured Compression of Large Language Models based on Low-Rank and Sparse Approximation
Authors: Yixiao Li, Yifan Yu, Qingru Zhang, Chen Liang, Pengcheng He, Weizhu Chen, Tuo Zhao
Date: 2023-06-20 00:00:00
Tasks: Diversity, Model Compression, Natural Language Understanding, Question Answering, Text Generation

Transformer models have achieved remarkable results in various natural language tasks, but they are often prohibitively large, requiring massive memories and computational resources. To reduce the size and complexity of these models, we propose LoSparse (Low-Rank and Sparse approximation), a novel model compression technique that approximates a weight matrix by the sum of a low-rank matrix and a sparse matrix. Our method combines the advantages of both low-rank approximations and pruning, while avoiding their limitations. Low-rank approximation compresses the coherent and expressive parts in neurons, while pruning removes the incoherent and non-expressive parts in neurons. Pruning enhances the diversity of low-rank approximations, and low-rank approximation prevents pruning from losing too many expressive neurons. We evaluate our method on natural language understanding, question answering, and natural language generation tasks. We show that it significantly outperforms existing compression methods.
CF-GODE: Continuous-Time Causal Inference for Multi-Agent Dynamical Systems
Authors: Song Jiang, Zijie Huang, Xiao Luo, Yizhou Sun
Date: 2023-06-20 00:00:00
Tasks: Causal Inference, counterfactual

Multi-agent dynamical systems refer to scenarios where multiple units interact with each other and evolve collectively over time. To make informed decisions in multi-agent dynamical systems, such as determining the optimal vaccine distribution plan, it is essential for decision-makers to estimate the continuous-time counterfactual outcomes. However, existing studies of causal inference over time rely on the assumption that units are mutually independent, which is not valid for multi-agent dynamical systems. In this paper, we aim to bridge this gap and study how to estimate counterfactual outcomes in multi-agent dynamical systems. Causal inference in a multi-agent dynamical system has unique challenges: 1) Confounders are time-varying and are present in both individual unit covariates and those of other units; 2) Units are affected by not only their own but also others' treatments; 3) The treatments are naturally dynamic, such as receiving vaccines and boosters in a seasonal manner. We model a multi-agent dynamical system as a graph and propose CounterFactual GraphODE (CF-GODE), a causal model that estimates continuous-time counterfactual outcomes in the presence of inter-dependencies between units. To facilitate continuous-time estimation, we propose Treatment-Induced GraphODE, a novel ordinary differential equation based on GNN, which incorporates dynamical treatments as additional inputs to predict potential outcomes over time. To remove confounding bias, we propose two domain adversarial learning based objectives that learn balanced continuous representation trajectories, which are not predictive of treatments and interference. We further provide theoretical justification to prove their effectiveness. Experiments on two semi-synthetic datasets confirm that CF-GODE outperforms baselines on counterfactual estimation. We also provide extensive analyses to understand how our model works.
Adaptive Federated Learning with Auto-Tuned Clients
Authors: Junhyung Lyle Kim, Mohammad Taha Toghani, César A. Uribe, Anastasios Kyrillidis
Date: 2023-06-19 00:00:00
Tasks: Federated Learning

Federated learning (FL) is a distributed machine learning framework where the global model of a central server is trained via multiple collaborative steps by participating clients without sharing their data. While being a flexible framework, where the distribution of local data, participation rate, and computing power of each client can greatly vary, such flexibility gives rise to many new challenges, especially in the hyperparameter tuning on the client side. We propose $\Delta$-SGD, a simple step size rule for SGD that enables each client to use its own step size by adapting to the local smoothness of the function each client is optimizing. We provide theoretical and empirical results where the benefit of the client adaptivity is shown in various FL scenarios.
Co-design Hardware and Algorithm for Vector Search
Authors: Wenqi Jiang, Shigang Li, Yu Zhu, Johannes De Fine Licht, Zhenhao He, Runbin Shi, Cedric Renggli, Shuai Zhang, Theodoros Rekatsinas, Torsten Hoefler, Gustavo Alonso
Date: 2023-06-19 00:00:00
Tasks: CPU, Information Retrieval, Retrieval

Vector search has emerged as the foundation for large-scale information retrieval and machine learning systems, with search engines like Google and Bing processing tens of thousands of queries per second on petabyte-scale document datasets by evaluating vector similarities between encoded query texts and web documents. As performance demands for vector search systems surge, accelerated hardware offers a promising solution in the post-Moore's Law era. We introduce \textit{FANNS}, an end-to-end and scalable vector search framework on FPGAs. Given a user-provided recall requirement on a dataset and a hardware resource budget, \textit{FANNS} automatically co-designs hardware and algorithm, subsequently generating the corresponding accelerator. The framework also supports scale-out by incorporating a hardware TCP/IP stack in the accelerator. \textit{FANNS} attains up to 23.0$\times$ and 37.2$\times$ speedup compared to FPGA and CPU baselines, respectively, and demonstrates superior scalability to GPUs, achieving 5.5$\times$ and 7.6$\times$ speedup in median and 95\textsuperscript{th} percentile (P95) latency within an eight-accelerator configuration. The remarkable performance of \textit{FANNS} lays a robust groundwork for future FPGA integration in data centers and AI supercomputers.
Insufficiently Justified Disparate Impact: A New Criterion for Subgroup Fairness
Authors: Neil Menghani, Edward McFowland III, Daniel B. Neill
Date: 2023-06-19 00:00:00
Tasks: Fairness

In this paper, we develop a new criterion, "insufficiently justified disparate impact" (IJDI), for assessing whether recommendations (binarized predictions) made by an algorithmic decision support tool are fair. Our novel, utility-based IJDI criterion evaluates false positive and false negative error rate imbalances, identifying statistically significant disparities between groups which are present even when adjusting for group-level differences in base rates. We describe a novel IJDI-Scan approach which can efficiently identify the intersectional subpopulations, defined across multiple observed attributes of the data, with the most significant IJDI. To evaluate IJDI-Scan's performance, we conduct experiments on both simulated and real-world data, including recidivism risk assessment and credit scoring. Further, we implement and evaluate approaches to mitigating IJDI for the detected subpopulations in these domains.
Human Limits in Machine Learning: Prediction of Plant Phenotypes Using Soil Microbiome Data
Authors: Rosa Aghdam, Xudong Tang, Shan Shan, Richard Lankau, Claudia Solís-Lemus
Date: 2023-06-19 00:00:00
Tasks: Data Augmentation, Model Selection

The preservation of soil health is a critical challenge in the 21st century due to its significant impact on agriculture, human health, and biodiversity. We provide the first deep investigation of the predictive potential of machine learning models to understand the connections between soil and biological phenotypes. We investigate an integrative framework performing accurate machine learning-based prediction of plant phenotypes from biological, chemical, and physical properties of the soil via two models: random forest and Bayesian neural network. We show that prediction is improved when incorporating environmental features like soil physicochemical properties and microbial population density into the models, in addition to the microbiome information. Exploring various data preprocessing strategies confirms the significant impact of human decisions on predictive performance. We show that the naive total sum scaling normalization that is commonly used in microbiome research is not the optimal strategy to maximize predictive power. Also, we find that accurately defined labels are more important than normalization, taxonomic level or model characteristics. In cases where humans are unable to classify samples accurately, machine learning model performance is limited. Lastly, we provide domain scientists via a full model selection decision tree to identify the human choices that optimize model prediction power. Our work is accompanied by open source reproducible scripts (https://github.com/solislemuslab/soil-microbiome-nn) for maximum outreach among the microbiome research community.
Evaluating Loss Functions and Learning Data Pre-Processing for Climate Downscaling Deep Learning Models
Authors: Xingying Huang
Date: 2023-06-19 00:00:00
Tasks: Deep Learning, Image Super-Resolution, Super-Resolution

Deep learning models have gained popularity in climate science, following their success in computer vision and other domains. For instance, researchers are increasingly employing deep learning techniques for downscaling climate data, drawing inspiration from image super-resolution models. However, there are notable differences between image data and climate data. While image data typically falls within a specific range (e.g., [0, 255]) and exhibits a relatively uniform or normal distribution, climate data can possess arbitrary value ranges and highly uneven distributions, such as precipitation data. This non-uniform distribution presents challenges when attempting to directly apply existing computer vision models to climate science tasks. Few studies have addressed this issue thus far. In this study, we explore the effects of loss functions and non-linear data pre-processing methods for deep learning models in the context of climate downscaling. We employ a climate downscaling experiment as an example to evaluate these factors. Our findings reveal that L1 loss and L2 loss perform similarly on some more balanced data like temperature data while for some imbalanced data like precipitation data, L2 loss performs significantly better than L1 loss. Additionally, we propose an approach to automatically learn the non-linear pre-processing function, which further enhances model accuracy and achieves the best results.
Nonlinear Feature Aggregation: Two Algorithms driven by Theory
Authors: Paolo Bonetti, Alberto Maria Metelli, Marcello Restelli
Date: 2023-06-19 00:00:00
Tasks: Dimensionality Reduction, feature selection, regression

Many real-world machine learning applications are characterized by a huge number of features, leading to computational and memory issues, as well as the risk of overfitting. Ideally, only relevant and non-redundant features should be considered to preserve the complete information of the original data and limit the dimensionality. Dimensionality reduction and feature selection are common preprocessing techniques addressing the challenge of efficiently dealing with high-dimensional data. Dimensionality reduction methods control the number of features in the dataset while preserving its structure and minimizing information loss. Feature selection aims to identify the most relevant features for a task, discarding the less informative ones. Previous works have proposed approaches that aggregate features depending on their correlation without discarding any of them and preserving their interpretability through aggregation with the mean. A limitation of methods based on correlation is the assumption of linearity in the relationship between features and target. In this paper, we relax such an assumption in two ways. First, we propose a bias-variance analysis for general models with additive Gaussian noise, leading to a dimensionality reduction algorithm (NonLinCFA) which aggregates non-linear transformations of features with a generic aggregation function. Then, we extend the approach assuming that a generalized linear model regulates the relationship between features and target. A deviance analysis leads to a second dimensionality reduction algorithm (GenLinCFA), applicable to a larger class of regression problems and classification settings. Finally, we test the algorithms on synthetic and real-world datasets, performing regression and classification tasks, showing competitive performances.
Modular Simulation Environment Towards OTN AI-based Solutions
Authors: Sam Aleyadeh, Abbas Javadtalab, Abdallah Shami
Date: 2023-06-19 00:00:00

The current trend for highly dynamic and virtualized networking infrastructure made automated networking a critical requirement. Multiple solutions have been proposed to address this, including the most sought-after machine learning ML-based solutions. However, the main hurdle when developing Next Generation Network is the availability of large datasets, especially in 5G and beyond and Optical Transport Networking (OTN) traffic. This need led researchers to look for viable simulation environments to generate the necessary volume with highly configurable real-life scenarios, which can be costly in setup and require subscription-based products and even the purchase of dedicated hardware, depending on the supplier. We aim to address this issue by generating high-volume and fidelity datasets by proposing a modular solution to adapt to the user's available resources. These datasets can be used to develop better-aforementioned ML solutions resulting in higher accuracy and adaptation to real-life networking traffic.
Fairness-aware Message Passing for Graph Neural Networks
Authors: Huaisheng Zhu, Guoji Fu, Zhimeng Guo, Zhiwei Zhang, Teng Xiao, Suhang Wang
Date: 2023-06-19 00:00:00
Tasks: Fairness

Graph Neural Networks (GNNs) have shown great power in various domains. However, their predictions may inherit societal biases on sensitive attributes, limiting their adoption in real-world applications. Although many efforts have been taken for fair GNNs, most existing works just adopt widely used fairness techniques in machine learning to graph domains and ignore or don't have a thorough understanding of the message passing mechanism with fairness constraints, which is a distinctive feature of GNNs. To fill the gap, we propose a novel fairness-aware message passing framework GMMD, which is derived from an optimization problem that considers both graph smoothness and representation fairness. GMMD can be intuitively interpreted as encouraging a node to aggregate representations of other nodes from different sensitive groups while subtracting representations of other nodes from the same sensitive group, resulting in fair representations. We also provide a theoretical analysis to justify that GMMD can guarantee fairness, which leads to a simpler and theory-guided variant GMMD-S. Extensive experiments on graph benchmarks show that our proposed framework can significantly improve the fairness of various backbone GNN models while maintaining high accuracy.
CAMMARL: Conformal Action Modeling in Multi Agent Reinforcement Learning
Authors: Nikunj Gupta, Somjit Nath, Samira Ebrahimi Kahou
Date: 2023-06-19 00:00:00
Tasks: Conformal Prediction, Decision Making, Multi-agent Reinforcement Learning, reinforcement-learning, Reinforcement Learning

Before taking actions in an environment with more than one intelligent agent, an autonomous agent may benefit from reasoning about the other agents and utilizing a notion of a guarantee or confidence about the behavior of the system. In this article, we propose a novel multi-agent reinforcement learning (MARL) algorithm CAMMARL, which involves modeling the actions of other agents in different situations in the form of confident sets, i.e., sets containing their true actions with a high probability. We then use these estimates to inform an agent's decision-making. For estimating such sets, we use the concept of conformal predictions, by means of which, we not only obtain an estimate of the most probable outcome but get to quantify the operable uncertainty as well. For instance, we can predict a set that provably covers the true predictions with high probabilities (e.g., 95%). Through several experiments in two fully cooperative multi-agent tasks, we show that CAMMARL elevates the capabilities of an autonomous agent in MARL by modeling conformal prediction sets over the behavior of other agents in the environment and utilizing such estimates to enhance its policy learning.
Accelerated, physics-inspired inference of skeletal muscle microstructure from diffusion-weighted MRI
Authors: Noel Naughton, Stacey Cahoon, Brad Sutton, John G. Georgiadis
Date: 2023-06-19 00:00:00

Muscle health is a critical component of overall health and quality of life. However, current measures of skeletal muscle health take limited account of microstructural variations within muscle, which play a crucial role in mediating muscle function. To address this, we present a physics-inspired, machine learning-based framework for the non-invasive and in vivo estimation of microstructural organization in skeletal muscle from diffusion-weighted MRI (dMRI). To reduce the computational expense associated with direct numerical simulations of dMRI physics, a polynomial meta-model is developed that accurately represents the input/output relationships of a high-fidelity numerical model. This meta-model is used to develop a Gaussian process (GP) model to provide voxel-wise estimates and confidence intervals of microstructure organization in skeletal muscle. Given noise-free data, the GP model accurately estimates microstructural parameters. In the presence of noise, the diameter, intracellular diffusion coefficient, and membrane permeability are accurately estimated with narrow confidence intervals, while volume fraction and extracellular diffusion coefficient are poorly estimated and exhibit wide confidence intervals. A reduced-acquisition GP model, consisting of one-third the diffusion-encoding measurements, is shown to predict parameters with similar accuracy to the original model. The fiber diameter and volume fraction estimated by the reduced GP model is validated via histology, with both parameters within their associated confidence intervals, demonstrating the capability of the proposed framework as a promising non-invasive tool for assessing skeletal muscle health and function.