pwc-archive/papers-with-abstracts huggingface dataset:
GD doesn't make the cut: Three ways that non-differentiability affects neural network training
Authors: Siddharth Krishna Kumar
Date: 2024-01-16 00:00:00
Tasks: Network Pruning

This paper critically examines the fundamental distinctions between gradient methods applied to non-differentiable functions (NGDMs) and classical gradient descents (GDs) for differentiable functions, revealing significant gaps in current deep learning optimization theory. We demonstrate that NGDMs exhibit markedly different convergence properties compared to GDs, strongly challenging the applicability of extensive neural network convergence literature based on $L-smoothness$ to non-smooth neural networks. Our analysis reveals paradoxical behavior of NDGM solutions for $L_{1}$-regularized problems, where increasing regularization counterintuitively leads to larger $L_{1}$ norms of optimal solutions. This finding calls into question widely adopted $L_{1}$ penalization techniques for network pruning. We further challenge the common assumption that optimization algorithms like RMSProp behave similarly in differentiable and non-differentiable contexts. Expanding on the Edge of Stability phenomenon, we demonstrate its occurrence in a broader class of functions, including Lipschitz continuous convex differentiable functions. This finding raises important questions about its relevance and interpretation in non-convex, non-differentiable neural networks, particularly those using ReLU activations. Our work identifies critical misunderstandings of NDGMs in influential literature, stemming from an overreliance on strong smoothness assumptions. These findings necessitate a reevaluation of optimization dynamics in deep learning, emphasizing the crucial need for more nuanced theoretical foundations in analyzing these complex systems.
Improving Limited Supervised Foot Ulcer Segmentation Using Cross-Domain Augmentation
Authors: Shang-Jui Kuo, Po-Han Huang, Chia-Ching Lin, Jeng-Lin Li, Ming-Ching Chang
Date: 2024-01-16 00:00:00
Tasks: Diversity, Segmentation

Diabetic foot ulcers pose health risks, including higher morbidity, mortality, and amputation rates. Monitoring wound areas is crucial for proper care, but manual segmentation is subjective due to complex wound features and background variation. Expert annotations are costly and time-intensive, thus hampering large dataset creation. Existing segmentation models relying on extensive annotations are impractical in real-world scenarios with limited annotated data. In this paper, we propose a cross-domain augmentation method named TransMix that combines Augmented Global Pre-training AGP and Localized CutMix Fine-tuning LCF to enrich wound segmentation data for model learning. TransMix can effectively improve the foot ulcer segmentation model training by leveraging other dermatology datasets not on ulcer skins or wounds. AGP effectively increases the overall image variability, while LCF increases the diversity of wound regions. Experimental results show that TransMix increases the variability of wound regions and substantially improves the Dice score for models trained with only 40 annotated images under various proportions.
From Coarse to Fine: Efficient Training for Audio Spectrogram Transformers
Authors: Jiu Feng, Mehmet Hamza Erol, Joon Son Chung, Arda Senocak
Date: 2024-01-16 00:00:00
Tasks: Audio Classification

Transformers have become central to recent advances in audio classification. However, training an audio spectrogram transformer, e.g. AST, from scratch can be resource and time-intensive. Furthermore, the complexity of transformers heavily depends on the input audio spectrogram size. In this work, we aim to optimize AST training by linking to the resolution in the time-axis. We introduce multi-phase training of audio spectrogram transformers by connecting the seminal idea of coarse-to-fine with transformer models. To achieve this, we propose a set of methods for temporal compression. By employing one of these methods, the transformer model learns from lower-resolution (coarse) data in the initial phases, and then is fine-tuned with high-resolution data later in a curriculum learning strategy. Experimental results demonstrate that the proposed training mechanism for AST leads to improved (or on-par) performance with faster convergence, i.e. requiring fewer computational resources and less time. This approach is also generalizable to other AST-based methods regardless of their learning paradigms.
Enhancing Dynamical System Modeling through Interpretable Machine Learning Augmentations: A Case Study in Cathodic Electrophoretic Deposition
Authors: Christian Jacobsen, Jiayuan Dong, Mehdi Khalloufi, Xun Huan, Karthik Duraisamy, Maryam Akram, Wanjiao Liu
Date: 2024-01-16 00:00:00
Tasks: Interpretable Machine Learning

We introduce a comprehensive data-driven framework aimed at enhancing the modeling of physical systems, employing inference techniques and machine learning enhancements. As a demonstrative application, we pursue the modeling of cathodic electrophoretic deposition (EPD), commonly known as e-coating. Our approach illustrates a systematic procedure for enhancing physical models by identifying their limitations through inference on experimental data and introducing adaptable model enhancements to address these shortcomings. We begin by tackling the issue of model parameter identifiability, which reveals aspects of the model that require improvement. To address generalizability , we introduce modifications which also enhance identifiability. However, these modifications do not fully capture essential experimental behaviors. To overcome this limitation, we incorporate interpretable yet flexible augmentations into the baseline model. These augmentations are parameterized by simple fully-connected neural networks (FNNs), and we leverage machine learning tools, particularly Neural Ordinary Differential Equations (Neural ODEs), to learn these augmentations. Our simulations demonstrate that the machine learning-augmented model more accurately captures observed behaviors and improves predictive accuracy. Nevertheless, we contend that while the model updates offer superior performance and capture the relevant physics, we can reduce off-line computational costs by eliminating certain dynamics without compromising accuracy or interpretability in downstream predictions of quantities of interest, particularly film thickness predictions. The entire process outlined here provides a structured approach to leverage data-driven methods. Firstly, it helps us comprehend the root causes of model inaccuracies, and secondly, it offers a principled method for enhancing model performance.
Ask the experts: sourcing high-quality datasets for nutritional counselling through Human-AI collaboration
Authors: Simone Balloccu, Ehud Reiter, Vivek Kumar, Diego Reforgiato Recupero, Daniele Riboni
Date: 2024-01-16 00:00:00
Tasks: Nutrition

Large Language Models (LLMs), with their flexible generation abilities, can be powerful data sources in domains with few or no available corpora. However, problems like hallucinations and biases limit such applications. In this case study, we pick nutrition counselling, a domain lacking any public resource, and show that high-quality datasets can be gathered by combining LLMs, crowd-workers and nutrition experts. We first crowd-source and cluster a novel dataset of diet-related issues, then work with experts to prompt ChatGPT into producing related supportive text. Finally, we let the experts evaluate the safety of the generated text. We release HAI-coaching, the first expert-annotated nutrition counselling dataset containing ~2.4K dietary struggles from crowd workers, and ~97K related supportive texts generated by ChatGPT. Extensive analysis shows that ChatGPT while producing highly fluent and human-like text, also manifests harmful behaviours, especially in sensitive topics like mental health, making it unsuitable for unsupervised use.
Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translation
Authors: Haoran Xu, Amr Sharaf, Yunmo Chen, Weiting Tan, Lingfeng Shen, Benjamin Van Durme, Kenton Murray, Young Jin Kim
Date: 2024-01-16 00:00:00
Tasks: Decoder, Machine Translation, Translation

Moderate-sized large language models (LLMs) -- those with 7B or 13B parameters -- exhibit promising machine translation (MT) performance. However, even the top-performing 13B LLM-based translation models, like ALMA, does not match the performance of state-of-the-art conventional encoder-decoder translation models or larger-scale LLMs such as GPT-4. In this study, we bridge this performance gap. We first assess the shortcomings of supervised fine-tuning for LLMs in the MT task, emphasizing the quality issues present in the reference data, despite being human-generated. Then, in contrast to SFT which mimics reference translations, we introduce Contrastive Preference Optimization (CPO), a novel approach that trains models to avoid generating adequate but not perfect translations. Applying CPO to ALMA models with only 22K parallel sentences and 12M parameters yields significant improvements. The resulting model, called ALMA-R, can match or exceed the performance of the WMT competition winners and GPT-4 on WMT'21, WMT'22 and WMT'23 test datasets.
Harnessing Artificial Intelligence for Sustainable Agricultural Development in Africa: Opportunities, Challenges, and Impact
Authors: Kinyua Gikunda
Date: 2024-01-03 00:00:00

This paper explores the transformative potential of artificial intelligence (AI) in the context of sustainable agricultural development across diverse regions in Africa. Delving into opportunities, challenges, and impact, the study navigates through the dynamic landscape of AI applications in agriculture. Opportunities such as precision farming, crop monitoring, and climate-resilient practices are examined, alongside challenges related to technological infrastructure, data accessibility, and skill gaps. The article analyzes the impact of AI on smallholder farmers, supply chains, and inclusive growth. Ethical considerations and policy implications are also discussed, offering insights into responsible AI integration. By providing a nuanced understanding, this paper contributes to the ongoing discourse on leveraging AI for fostering sustainability in African agriculture.
A debiasing technique for place-based algorithmic patrol management
Authors: Alexander Einarsson, Simen Oestmo, Lester Wollman, Duncan Purves, Ryan Jenkins
Date: 2023-12-22 00:00:00
Tasks: Decision Making, Fairness, Management

In recent years, there has been a revolution in data-driven policing. With that has come scrutiny on how bias in historical data affects algorithmic decision making. In this exploratory work, we introduce a debiasing technique for place-based algorithmic patrol management systems. We show that the technique efficiently eliminates racially biased features while retaining high accuracy in the models. Finally, we provide a lengthy list of potential future research in the realm of fairness and data-driven policing which this work uncovered.
Trustworthy human-centric based Automated Decision-Making Systems
Authors: Marcelino Cabrera, Carlos Cruz, Pavel Novoa-Hernández, David A. Pelta, José Luis Verdegay
Date: 2023-12-22 00:00:00
Tasks: Decision Making

Automated Decision-Making Systems (ADS) have become pervasive across various fields, activities, and occupations, to enhance performance. However, this widespread adoption introduces potential risks, including the misuse of ADS. Such misuse may manifest when ADS is employed in situations where it is unnecessary or when essential requirements, conditions, and terms are overlooked, leading to unintended consequences. This research paper presents a thorough examination of the implications, distinctions, and ethical considerations associated with digitalization, digital transformation, and the utilization of ADS in contemporary society and future contexts. Emphasis is placed on the imperative need for regulation, transparency, and ethical conduct in the deployment of ADS.
Future-proofing Education: A Prototype for Simulating Oral Examinations Using Large Language Models
Authors: André Nitze
Date: 2023-12-22 00:00:00

This study explores the impact of Large Language Models (LLMs) in higher education, focusing on an automated oral examination simulation using a prototype. The design considerations of the prototype are described, and the system is evaluated with a select group of educators and students. Technical and pedagogical observations are discussed. The prototype proved to be effective in simulating oral exams, providing personalized feedback, and streamlining educators' workloads. The promising results of the prototype show the potential for LLMs in democratizing education, inclusion of diverse student populations, and improvement of teaching quality and efficiency.
Do backrun auctions protect traders?
Authors: Andrew W. Macpherson
Date: 2024-01-16 00:00:00

We study a new "laminated" queueing model for orders on batched trading venues such as decentralised exchanges. The model aims to capture and generalise transaction queueing infrastructure that has arisen to organise MEV activity on public blockchains such as Ethereum, providing convenient channels for sophisticated agents to extract value by acting on end-user order flow by performing arbitrage and related HFT activities. In our model, market orders are interspersed with orders created by arbitrageurs that under idealised conditions reset the marginal price to a global equilibrium between each trade, improving predictability of execution for liquidity traders. If an arbitrageur has a chance to land multiple opportunities in a row, he may attempt to manipulate the execution price of the intervening market order by a probabilistic blind sandwiching strategy. To study how bad this manipulation can get, we introduce and bound a price manipulation coefficient that measures the deviation from global equilibrium of local pricing quoted by a rational arbitrageur. We exhibit cases in which this coefficient is well approximated by a "zeta value' with interpretable and empirically measurable parameters.
Over-the-Air Federated Learning with Phase Noise: Analysis and Countermeasures
Authors: Martin Dahl, Erik G. Larsson
Date: 2024-01-16 00:00:00
Tasks: Federated Learning

Wirelessly connected devices can collaborately train a machine learning model using federated learning, where the aggregation of model updates occurs using over-the-air computation. Carrier frequency offset caused by imprecise clocks in devices will cause the phase of the over-the-air channel to drift randomly, such that late symbols in a coherence block are transmitted with lower quality than early symbols. To mitigate the effect of degrading symbol quality, we propose a scheme where one of the permutations Roll, Flip and Sort are applied on gradients before transmission. Through simulations we show that the permutations can both improve and degrade learning performance. Furthermore, we derive the expectation and variance of the gradient estimate, which is shown to grow exponentially with the number of symbols in a coherence block.
Metasurface-Based Receivers with $1$-bit ADCs for Multi-User Uplink Communications
Authors: Panagiotis Gavriilidis, Italo Atzeni, George C. Alexandropoulos
Date: 2024-01-15 00:00:00

The massive Multiple-Input Multiple-Output (mMIMO) concept has been recently moving forward to extreme scales to address the envisioned requirements of next generation networks. However, the extension of conventional architectures will result in significant cost and power consumption. To this end, metasurface-based transceivers, consisting of microstrips of metamaterials, have recently emerged as an efficient enabler of extreme mMIMO systems. In this paper, we consider metasurface-based receivers with a $1$-bit Analog-to-Digital Converter (ADC) per microstrip and develop an analytical framework for the optimization of the analog and digital combining matrices. Our numerical results, including comparisons with fully digital, infinite-resolution MIMO, provide useful insights into the role of various system parameters.
Individual and Collective Welfare in Risk Sharing with Many States
Authors: Federico Echenique, Farzad Pourbabaee
Date: 2024-01-14 00:00:00

We investigate the effects of a large (but finite) state space on models of efficient risk sharing. A group of risk-averse agents agree on a risk-sharing agreement in an economy without aggregate risk. The economy is subject to a perturbation, or shock, that prompts a renegotiation of the agreement. If agents insist on an $\ep$-utility improvement to accept a new agreement, then the probability of a post-shock acceptable agreement vanishes exponentially to zero as the number of states grows. We use similar arguments to consider a model where agents have multiple prior preferences, and show that the existence of an $\ep$-Pareto improving trade requires that some sets of priors have vanishingly small measure. Our results hinge on the "shape does not matter" message of high dimensional isoperimetric inequalities.
A Survey on Game Theory Optimal Poker
Authors: Prathamesh Sonawane, Arav Chheda
Date: 2024-01-02 00:00:00
Tasks: Survey

Poker is in the family of imperfect information games unlike other games such as chess, connect four, etc which are perfect information game instead. While many perfect information games have been solved, no non-trivial imperfect information game has been solved to date. This makes poker a great test bed for Artificial Intelligence research. In this paper we firstly compare Game theory optimal poker to Exploitative poker. Secondly, we discuss the intricacies of abstraction techniques, betting models, and specific strategies employed by successful poker bots like Tartanian[1] and Pluribus[6]. Thirdly, we also explore 2-player vs multi-player games and the limitations that come when playing with more players. Finally, this paper discusses the role of machine learning and theoretical approaches in developing winning strategies and suggests future directions for this rapidly evolving field.
A new model of trust based on neural information processing
Authors: Scott E. Allen, René F. Kizilcec, A. David Redish
Date: 2024-01-16 00:00:00
Tasks: Decision Making

More than 30 years of research has firmly established the vital role of trust in human organizations and relationships, but the underlying mechanisms by which people build, lose, and rebuild trust remains incompletely understood. We propose a mechanistic model of trust that is grounded in the modern neuroscience of decision making. Since trust requires anticipating the future actions of others, any mechanistic model must be built upon up-to-date theories on how the brain learns, represents, and processes information about the future within its decision-making systems. Contemporary neuroscience has revealed that decision making arises from multiple parallel systems that perform distinct, complementary information processing. Each system represents information in different forms, and therefore learns via different mechanisms. When an act of trust is reciprocated or violated, this provides new information that can be used to anticipate future actions. The taxonomy of neural information representations that is the basis for the system boundaries between neural decision-making systems provides a taxonomy for categorizing different forms of trust and generating mechanistic predictions about how these forms of trust are learned and manifested in human behavior. Three key predictions arising from our model are (1) strategic risk-taking can reveal how to best proceed in a relationship, (2) human organizations and environments can be intentionally designed to encourage trust among their members, and (3) violations of trust need not always degrade trust, but can also provide opportunities to build trust.
Far- versus Near-Field RIS Modeling and Beam Design
Authors: Mohamadreza Delbari, George C. Alexandropoulos, Robert Schober, Vahid Jamali
Date: 2024-01-16 00:00:00

In this chapter, we investigate the mathematical foundation of the modeling and design of reconfigurable intelligent surfaces (RIS) in both the far- and near-field regimes. More specifically, we first present RIS-assisted wireless channel models for the far- and near-field regimes, discussing relevant phenomena, such as line-of-sight (LOS) and non-LOS links, rich and poor scattering, channel correlation, and array manifold. Subsequently, we introduce two general approaches for the RIS reflective beam design, namely optimization-based and analytical, which offer different degrees of design flexibility and computational complexity. Furthermore, we provide a comprehensive set of simulation results for the performance evaluation of the studied RIS beam designs and the investigation of the impact of the system parameters.
Index Modulation for Integrated Sensing and Communications: A Signal Processing Perspective
Authors: Ahmet M. Elbir, Abdulkadir Celik, Ahmed M. Eltawil, Moeness G. Amin
Date: 2024-01-16 00:00:00
Tasks: ISAC

A joint design of both sensing and communication can lead to substantial enhancement for both subsystems in terms of size, cost as well as spectrum and hardware efficiency. In the last decade, integrated sensing and communications (ISAC) has emerged as a means to efficiently utilize the spectrum on a single and shared hardware platform. Recent studies focused on developing multi-function approaches to share the spectrum between radar sensing and communications. Index modulation (IM) is one particular approach to incorporate information-bearing communication symbols into the emitted radar waveforms. While IM has been well investigated in communications-only systems, the implementation adoption of IM concept in ISAC has recently attracted researchers to achieve improved energy/spectral efficiency while maintaining satisfactory radar sensing performance. This article focuses on recent studies on IM-ISAC, and presents in detail the analytical background and relevance of the major IM-ISAC applications.
Cramer-Rao bound and absolute sensitivity in chemical reaction networks
Authors: Dimitri Loutchko, Yuki Sughiyama, Tetsuya J. Kobayashi
Date: 2024-01-13 00:00:00
Tasks: Math, Sensitivity

Chemical reaction networks (CRN) comprise an important class of models to understand biological functions such as cellular information processing, the robustness and control of metabolic pathways, circadian rhythms, and many more. However, any CRN describing a certain function does not act in isolation but is a part of a much larger network and as such is constantly subject to external changes. In [Shinar, Alon, and Feinberg. "Sensitivity and robustness in chemical reaction networks." SIAM J App Math (2009): 977-998.], the responses of CRN to changes in the linear conserved quantities, called sensitivities, were studied in and the question of how to construct absolute, i.e., basis-independent, sensitivities was raised. In this article, by applying information geometric methods, such a construction is provided. The idea is to track how concentration changes in a particular chemical propagate to changes of all concentrations within a steady state. This is encoded in the matrix of absolute sensitivities. As the main technical tool, a multivariate Cramer-Rao bound for CRN is proven, which is based on the the analogy between quasi-thermostatic steady states and the exponential family of probability distributions. This leads to a linear algebraic characterization of the matrix of absolute sensitivities for quasi-thermostatic CRN. As an example, the core module of the IDHKP-IDH glyoxylate bypass regulation system is analyzed analytically and numerically by extensive random sampling of the concentration space. The experimentally known findings for the robustness of the IDH enzyme are confirmed and a hidden symmetry at the level of distributions is revealed, providing a blueprint for the analysis of the robustness properties of CRNs.