pwc-archive/papers-with-abstracts huggingface dataset:
Syntax-Guided Transformers: Elevating Compositional Generalization and Grounding in Multimodal Environments
Authors: Danial Kamali, Parisa Kordjamshidi
Date: 2023-11-07 00:00:00
Tasks: Compositional Generalization (AVG), Dependency Parsing

Compositional generalization, the ability of intelligent models to extrapolate understanding of components to novel compositions, is a fundamental yet challenging facet in AI research, especially within multimodal environments. In this work, we address this challenge by exploiting the syntactic structure of language to boost compositional generalization. This paper elevates the importance of syntactic grounding, particularly through attention masking techniques derived from text input parsing. We introduce and evaluate the merits of using syntactic information in the multimodal grounding problem. Our results on grounded compositional generalization underscore the positive impact of dependency parsing across diverse tasks when utilized with Weight Sharing across the Transformer encoder. The results push the state-of-the-art in multimodal grounding and parameter-efficient modeling and provide insights for future research.
Uncovering Intermediate Variables in Transformers using Circuit Probing
Authors: Michael A. Lepori, Thomas Serre, Ellie Pavlick
Date: 2023-11-07 00:00:00
Tasks: Language Modeling, Language Modelling, Sentence

Neural network models have achieved high performance on a wide variety of complex tasks, but the algorithms that they implement are notoriously difficult to interpret. In order to understand these algorithms, it is often necessary to hypothesize intermediate variables involved in the network's computation. For example, does a language model depend on particular syntactic properties when generating a sentence? However, existing analysis tools make it difficult to test hypotheses of this type. We propose a new analysis technique -- circuit probing -- that automatically uncovers low-level circuits that compute hypothesized intermediate variables. This enables causal analysis through targeted ablation at the level of model parameters. We apply this method to models trained on simple arithmetic tasks, demonstrating its effectiveness at (1) deciphering the algorithms that models have learned, (2) revealing modular structure within a model, and (3) tracking the development of circuits over training. We compare circuit probing to other methods across these three experiments, and find it on par or more effective than existing analysis methods. Finally, we demonstrate circuit probing on a real-world use case, uncovering circuits that are responsible for subject-verb agreement and reflexive anaphora in GPT2-Small and Medium.
Formal Aspects of Language Modeling
Authors: Ryan Cotterell, Anej Svete, Clara Meister, Tianyu Liu, Li Du
Date: 2023-11-07 00:00:00
Tasks: Language Modeling, Language Modelling

Large language models have become one of the most commonly deployed NLP inventions. In the past half-decade, their integration into core natural language processing tools has dramatically increased the performance of such tools, and they have entered the public discourse surrounding artificial intelligence. Consequently, it is important for both developers and researchers alike to understand the mathematical foundations of large language models, as well as how to implement them. These notes are the accompaniment to the theoretical portion of the ETH Z\"urich course on large language models, covering what constitutes a language model from a formal, theoretical perspective.
Aspect-based Meeting Transcript Summarization: A Two-Stage Approach with Weak Supervision on Sentence Classification
Authors: Zhongfen Deng, Seunghyun Yoon, Trung Bui, Franck Dernoncourt, Quan Hung Tran, Shuaiqi Liu, Wenting Zhao, Tao Zhang, Yibo Wang, Philip S. Yu
Date: 2023-11-07 00:00:00
Tasks: Sentence, Sentence Classification

Aspect-based meeting transcript summarization aims to produce multiple summaries, each focusing on one aspect of content in a meeting transcript. It is challenging as sentences related to different aspects can mingle together, and those relevant to a specific aspect can be scattered throughout the long transcript of a meeting. The traditional summarization methods produce one summary mixing information of all aspects, which cannot deal with the above challenges of aspect-based meeting transcript summarization. In this paper, we propose a two-stage method for aspect-based meeting transcript summarization. To select the input content related to specific aspects, we train a sentence classifier on a dataset constructed from the AMI corpus with pseudo-labeling. Then we merge the sentences selected for a specific aspect as the input for the summarizer to produce the aspect-based summary. Experimental results on the AMI corpus outperform many strong baselines, which verifies the effectiveness of our proposed method.
Free-Space Optical Spiking Neural Network
Authors: Reyhane Ahmadi, Amirreza Ahmadnejad, Somayyeh Koohi
Date: 2023-11-08 00:00:00
Tasks: object-detection, Object Detection

Neuromorphic engineering has emerged as a promising avenue for developing brain-inspired computational systems. However, conventional electronic AI-based processors often encounter challenges related to processing speed and thermal dissipation. As an alternative, optical implementations of such processors have been proposed, capitalizing on the intrinsic information-processing capabilities of light. Within the realm of optical neuromorphic engineering, various optical neural networks (ONNs) have been explored. Among these, Spiking Neural Networks (SNNs) have exhibited notable success in emulating the computational principles of the human brain. Nevertheless, the integration of optical SNN processors has presented formidable obstacles, mainly when dealing with the computational demands of large datasets. In response to these challenges, we introduce a pioneering concept: the Free-space Optical deep Spiking Convolutional Neural Network (OSCNN). This novel approach draws inspiration from computational models of the human eye. We have meticulously designed various optical components within the OSCNN to tackle object detection tasks across prominent benchmark datasets, including MNIST, ETH 80, and Caltech. Our results demonstrate promising performance with minimal latency and power consumption compared to their electronic ONN counterparts. Additionally, we conducted several pertinent simulations, such as optical intensity to-latency conversion and synchronization. Of particular significance is the evaluation of the feature extraction layer, employing a Gabor filter bank, which stands to impact the practical deployment of diverse ONN architectures significantly.
On the estimation of the number of components in multivariate functional principal component analysis
Authors: Steven Golovkine, Edward Gunning, Andrew J. Simpkin, Norma Bargary
Date: 2023-11-08 00:00:00

Happ and Greven (2018) developed a methodology for principal components analysis of multivariate functional data observed on different dimensional domains. Their approach relies on an estimation of univariate functional principal components for each univariate functional feature. In this paper, we present extensive simulations to investigate choosing the number of principal components to retain. We show empirically that the conventional approach of using a percentage of variance explained threshold for each univariate functional feature may be unreliable when aiming to explain an overall percentage of variance in the multivariate functional data, and thus we advise practitioners to exercise caution.
Online Learning Quantum States with the Logarithmic Loss via VB-FTRL
Authors: Wei-Fu Tseng, Kai-Chun Chen, Zi-Hong Xiao, Yen-Huan Li
Date: 2023-11-06 00:00:00
Tasks: Quantum State Tomography, Stochastic Optimization

Online learning of quantum states with the logarithmic loss (LL-OLQS) is a quantum generalization of online portfolio selection (OPS), a classic open problem in online learning for over three decades. This problem also emerges in designing stochastic optimization algorithms for maximum-likelihood quantum state tomography. Recently, Jezequel et al. (arXiv:2209.13932) proposed the VB-FTRL algorithm, the first regret-optimal algorithm for OPS with moderate computational complexity. In this paper, we generalize VB-FTRL for LL-OLQS. Let $d$ denote the dimension and $T$ the number of rounds. The generalized algorithm achieves a regret rate of $O ( d^2 \log ( d + T ) )$ for LL-OLQS. Each iteration of the algorithm consists of solving a semidefinite program that can be implemented in polynomial time by, for example, cutting-plane methods. For comparison, the best-known regret rate for LL-OLQS is currently $O ( d^2 \log T )$, achieved by an exponential weight method. However, no explicit implementation is available for the exponential weight method for LL-OLQS. To facilitate the generalization, we introduce the notion of VB-convexity. VB-convexity is a sufficient condition for the volumetric barrier associated with any function to be convex and is of independent interest.
Towards Deeper, Lighter and Interpretable Cross Network for CTR Prediction
Authors: Fangye Wang, Hansu Gu, Dongsheng Li, Tun Lu, Peng Zhang, Ning Gu
Date: 2023-11-08 00:00:00
Proceeding: Proceedings of the 32nd ACM International Conference on Information and Knowledge Management 2023 10
Tasks: Click-Through Rate Prediction, Recommendation Systems

Click Through Rate (CTR) prediction plays an essential role in recommender systems and online advertising. It is crucial to effectively model feature interactions to improve the prediction performance of CTR models. However, existing methods face three significant challenges. First, while most methods can automatically capture high-order feature interactions, their performance tends to diminish as the order of feature interactions increases. Second, existing methods lack the ability to provide convincing interpretations of the prediction results, especially for high-order feature interactions, which limits the trustworthiness of their predictions. Third, many methods suffer from the presence of redundant parameters, particularly in the embedding layer. This paper proposes a novel method called Gated Deep Cross Network (GDCN) and a Field-level Dimension Optimization (FDO) approach to address these challenges. As the core structure of GDCN, Gated Cross Network (GCN) captures explicit high-order feature interactions and dynamically filters important interactions with an information gate in each order. Additionally, we use the FDO approach to learn condensed dimensions for each field based on their importance. Comprehensive experiments on five datasets demonstrate the effectiveness, superiority and interpretability of GDCN. Moreover, we verify the effectiveness of FDO in learning various dimensions and reducing model parameters. The code is available on \url{https://github.com/anonctr/GDCN}.
A Comprehensive Summarization and Evaluation of Feature Refinement Modules for CTR Prediction
Authors: Fangye Wang, Hansu Gu, Dongsheng Li, Tun Lu, Peng Zhang, Li Shang, Ning Gu
Date: 2023-11-08 00:00:00
Tasks: Benchmarking, Click-Through Rate Prediction, GPU

Click-through rate (CTR) prediction is widely used in academia and industry. Most CTR tasks fall into a feature embedding \& feature interaction paradigm, where the accuracy of CTR prediction is mainly improved by designing practical feature interaction structures. However, recent studies have argued that the fixed feature embedding learned only through the embedding layer limits the performance of existing CTR models. Some works apply extra modules on top of the embedding layer to dynamically refine feature representations in different instances, making it effective and easy to integrate with existing CTR methods. Despite the promising results, there is a lack of a systematic review and summarization of this new promising direction on the CTR task. To fill this gap, we comprehensively summarize and define a new module, namely \textbf{feature refinement} (FR) module, that can be applied between feature embedding and interaction layers. We extract 14 FR modules from previous works, including instances where the FR module was proposed but not clearly defined or explained. We fully assess the effectiveness and compatibility of existing FR modules through comprehensive and extensive experiments with over 200 augmented models and over 4,000 runs for more than 15,000 GPU hours. The results offer insightful guidelines for researchers, and all benchmarking code and experimental results are open-sourced. In addition, we present a new architecture of assigning independent FR modules to separate sub-networks for parallel CTR models, as opposed to the conventional method of inserting a shared FR module on top of the embedding layer. Our approach is also supported by comprehensive experiments demonstrating its effectiveness.
Rethinking Cross-Domain Sequential Recommendation under Open-World Assumptions
Authors: Wujiang Xu, Qitian Wu, Runzhong Wang, Mingming Ha, Qiongxu Ma, Linxun Chen, Bing Han, Junchi Yan
Date: 2023-11-08 00:00:00
Tasks: Sequential Recommendation

Cross-Domain Sequential Recommendation (CDSR) methods aim to tackle the data sparsity and cold-start problems present in Single-Domain Sequential Recommendation (SDSR). Existing CDSR works design their elaborate structures relying on overlapping users to propagate the cross-domain information. However, current CDSR methods make closed-world assumptions, assuming fully overlapping users across multiple domains and that the data distribution remains unchanged from the training environment to the test environment. As a result, these methods typically result in lower performance on online real-world platforms due to the data distribution shifts. To address these challenges under open-world assumptions, we design an \textbf{A}daptive \textbf{M}ulti-\textbf{I}nterest \textbf{D}ebiasing framework for cross-domain sequential recommendation (\textbf{AMID}), which consists of a multi-interest information module (\textbf{MIM}) and a doubly robust estimator (\textbf{DRE}). Our framework is adaptive for open-world environments and can improve the model of most off-the-shelf single-domain sequential backbone models for CDSR. Our MIM establishes interest groups that consider both overlapping and non-overlapping users, allowing us to effectively explore user intent and explicit interest. To alleviate biases across multiple domains, we developed the DRE for the CDSR methods. We also provide a theoretical analysis that demonstrates the superiority of our proposed estimator in terms of bias and tail bound, compared to the IPS estimator used in previous work.
LATTE: Lattice ATTentive Encoding for Character-based Word Segmentation
Authors: Thodsaporn Chay-intr, Hidetaka Kamigaito, Kotaro Funakoshi, Manabu Okumura
Date: 2023-06-01 00:00:00
Proceeding: Journal of Natural Language Processing 2023 6
Tasks: Chinese Word Segmentation, Japanese Word Segmentation, Segmentation, Thai Word Segmentation

A character sequence comprises at least one or more segmentation alternatives. This can be considered segmentation ambiguity and may weaken segmentation performance in word segmentation. Proper handling of such ambiguity lessens ambiguous decisions on word boundaries. Previous works have achieved remarkable segmentation performance and alleviated the ambiguity problem by incorporating the lattice, owing to its ability to capture segmentation alternatives, along with graph-based and pre-trained models. However, multiple granularity information, including character and word, in a lattice that encodes with such models may not be attentively exploited. To strengthen multi-granularity representations in a lattice, we propose the Lattice ATTentive Encoding (LATTE) method for character-based word segmentation. Our model employs the lattice structure to handle segmentation alternatives and utilizes graph neural networks along with an attention mechanism to attentively extract multi-granularity representation from the lattice for complementing character representations. Our experimental results demonstrated improvements in segmentation performance on the BCCWJ, CTB6, and BEST2010 datasets in three languages, particularly Japanese, Chinese, and Thai.
Common Knowledge, Regained
Authors: Yannai A. Gonczarowski, Yoram Moses
Date: 2023-11-07 00:00:00

For common knowledge to arise in dynamic settings, all players must simultaneously come to know it has arisen. Consequently, common knowledge cannot arise in many realistic settings with timing frictions. This counterintuitive observation of Halpern and Moses (1990) was discussed by Arrow et al. (1987) and Aumann (1989), was called a paradox by Morris (2014), and has evaded satisfactory resolution for four decades. We resolve this paradox by proposing a new definition for common knowledge, which coincides with the traditional one in static settings but is more permissive in dynamic settings. Under our definition, common knowledge can arise without simultaneity, particularly in canonical examples of the Haplern-Moses paradox. We demonstrate its usefulness by deriving for it an agreement theorem \`a la Aumann (1976), showing it arises in the setting of Geanakoplos and Polemarchakis (1982) with timing frictions added, and applying it to characterize equilibrium behavior in a dynamic coordination game.
Synergy Among Flexible Demands: Forming a Coalition to Earn More from Reserve Market
Authors: Peter A. V. Gade, Trygve Skjøtskift, Henrik Bindner, Jalal Kazempour
Date: 2023-11-08 00:00:00

We address potential synergy among flexible demands and how they may earn more collectively than individually by forming a coalition and bidding to the reserve market. We consider frequency-supporting ancillary service markets, particularly the manual Frequency Restoration Reserve (mFRR) market. The coalition of flexible demands provides more reliable mFRR services, where in comparison to individual demands, is penalized less for their potential failure and is paid more for their successful activation. This synergy effect is quantified as a function of the number of homogeneous assets in the coalition. A subsequent payment allocation mechanism using Shapley values is proposed to distribute the total earnings of the coalition among demands, while incentivizing them to remain in the coalition. For our numerical study, we use real price data from the Danish mFRR market in 2022.
Discerning and Enhancing the Weighted Sum-Rate Maximization Algorithms in Communications
Authors: Zepeng Zhang, Ziping Zhao, Kaiming Shen, Daniel P. Palomar, Wei Yu
Date: 2023-11-08 00:00:00

Weighted sum-rate (WSR) maximization plays a critical role in communication system design. This paper examines three optimization methods for WSR maximization, which ensure convergence to stationary points: two block coordinate ascent (BCA) algorithms, namely, weighted sum-minimum mean-square error (WMMSE) and WSR maximization via fractional programming (WSR-FP), along with a minorization-maximization (MM) algorithm, WSR maximization via MM (WSR-MM). Our contributions are threefold. Firstly, we delineate the exact relationships among WMMSE, WSR-FP, and WSR-MM, which, despite their extensive use in the literature, lack a comprehensive comparative study. By probing the theoretical underpinnings linking the BCA and MM algorithmic frameworks, we reveal the direct correlations between the equivalent transformation techniques, essential to the development of WMMSE and WSR-FP, and the surrogate functions pivotal to WSR-MM. Secondly, we propose a novel algorithm, WSR-MM+, harnessing the flexibility of selecting surrogate functions in MM framework. By circumventing the repeated matrix inversions in the search for optimal Lagrange multipliers in existing algorithms, WSR-MM+ significantly reduces the computational load per iteration and accelerates convergence. Thirdly, we reconceptualize WSR-MM+ within the BCA framework, introducing a new equivalent transform, which gives rise to an enhanced version of WSR-FP, named as WSR-FP+. We further demonstrate that WSR-MM+ can be construed as the basic gradient projection method. This perspective yields a deeper understanding into its computational intricacies. Numerical simulations corroborate the connections between WMMSE, WSR-FP, and WSR-MM and confirm the efficacy of the proposed WSR-MM+ and WSR-FP+ algorithms.
NEAT-MUSIC: Auto-calibration of DOA Estimation for Terahertz-Band Massive MIMO Systems
Authors: Ahmet M. Elbir, Abdulkadir Celik, Ahmed M. Eltawil
Date: 2023-11-07 00:00:00

Terahertz (THz) band is envisioned for the future sixth generation wireless systems thanks to its abundant bandwidth and very narrow beamwidth. These features are one of the key enabling factors for high resolution sensing with milli-degree level direction-of-arrival (DOA) estimation. Therefore, this paper investigates the DOA estimation problem in THz systems in the presence of two major error sources: 1) gain-phase mismatches, which occur due to the deviations in the radio-frequency circuitry; 2) beam-squint, which is caused because of the deviations in the generated beams at different subcarriers due to ultra-wide bandwidth. An auto-calibration approach, namely NoisE subspAce correcTion technique for MUltiple SIgnal Classification (NEAT-MUSIC), is proposed based on the correction of the noise subspace for accurate DOA estimation in the presence of gain-phase mismatches and beam-squint. To gauge the performance of the proposed approach, the Cramer-Rao bounds are also derived. Numerical results show the effectiveness of the proposed approach.
Adaptive Stochastic Nonlinear Model Predictive Control with Look-ahead Deep Reinforcement Learning for Autonomous Vehicle Motion Control
Authors: Baha Zarrouki, Chenyang Wang, Johannes Betz
Date: 2023-11-07 00:00:00
Tasks: Decision Making, Deep Reinforcement Learning, Model Predictive Control, Reinforcement Learning (RL)

In this paper, we present a Deep Reinforcement Learning (RL)-driven Adaptive Stochastic Nonlinear Model Predictive Control (SNMPC) to optimize uncertainty handling, constraints robustification, feasibility, and closed-loop performance. To this end, we conceive an RL agent to proactively anticipate upcoming control tasks and to dynamically determine the most suitable combination of key SNMPC parameters - foremost the robustification factor $\kappa$ and the Uncertainty Propagation Horizon (UPH) $T_u$. We analyze the trained RL agent's decision-making process and highlight its ability to learn context-dependent optimal parameters. One key finding is that adapting the constraints robustification factor with the learned policy reduces conservatism and improves closed-loop performance while adapting UPH renders previously infeasible SNMPC problems feasible when faced with severe disturbances. We showcase the enhanced robustness and feasibility of our Adaptive SNMPC (aSNMPC) through the real-time motion control task of an autonomous passenger vehicle to follow an optimal race line when confronted with significant time-variant disturbances. Experimental findings demonstrate that our look-ahead RL-driven aSNMPC outperforms its Static SNMPC (sSNMPC) counterpart in minimizing the lateral deviation both with accurate and inaccurate disturbance assumptions and even when driving in previously unexplored environments.