pwc-archive/papers-with-abstracts huggingface dataset:
Reconstructing Global Daily CO2 Emissions via Machine Learning
Authors: Tao Li, Lixing Wang, Zihan Qiu, Philippe Ciais, Taochun Sun, Matthew W. Jones, Robbie M. Andrew, Glen P. Peters, Piyu Ke, Xiaoting Huang, Robert B. Jackson, Zhu Liu
Date: 2024-07-29 00:00:00

High temporal resolution CO2 emission data are crucial for understanding the drivers of emission changes, however, current emission dataset is only available on a yearly basis. Here, we extended a global daily CO2 emissions dataset backwards in time to 1970 using machine learning algorithm, which was trained to predict historical daily emissions on national scales based on relationships between daily emission variations and predictors established for the period since 2019. Variation in daily CO2 emissions far exceeded the smoothed seasonal variations. For example, the range of daily CO2 emissions equivalent to 31% of the year average daily emissions in China and 46% of that in India in 2022, respectively. We identified the critical emission-climate temperature (Tc) is 16.5 degree celsius for global average (18.7 degree celsius for China, 14.9 degree celsius for U.S., and 18.4 degree celsius for Japan), in which negative correlation observed between daily CO2 emission and ambient temperature below Tc and a positive correlation above it, demonstrating increased emissions associated with higher ambient temperature. The long-term time series spanning over fifty years of global daily CO2 emissions reveals an increasing trend in emissions due to extreme temperature events, driven by the rising frequency of these occurrences. This work suggests that, due to climate change, greater efforts may be needed to reduce CO2 emissions.
Risk management in multi-objective portfolio optimization under uncertainty
Authors: Yannick Becker, Pascal Halffmann, Anita Schöbel
Date: 2024-07-29 00:00:00
Tasks: Management, Portfolio Optimization

In portfolio optimization, decision makers face difficulties from uncertainties inherent in real-world scenarios. These uncertainties significantly influence portfolio outcomes in both classical and multi-objective Markowitz models. To address these challenges, our research explores the power of robust multi-objective optimization. Since portfolio managers frequently measure their solutions against benchmarks, we enhance the multi-objective min-regret robustness concept by incorporating these benchmark comparisons. This approach bridges the gap between theoretical models and real-world investment scenarios, offering portfolio managers more reliable and adaptable strategies for navigating market uncertainties. Our framework provides a more nuanced and practical approach to portfolio optimization under real-world conditions.
Seismic Random Noise Attenuation Based on Non-IID Pixel-Wise Gaussian Noise Modeling
Authors: Chuangji Meng, Jinghuai Gao, Member, IEEE, Yajun Tian, Zhiqiang Wang
Date: 2023-05-16 00:00:00
Proceeding: IEEE Transactions on Geoscience and Remote Sensing 2023 5
Tasks: Denoising, Geophysics, Variational Inference

Random noise attenuation (NA) is a key step in seismic field data processing. With the rise of artificial intelligence, deep learning (DL) algorithms are gradually introduced into seismic random noise suppression. The most commonly used DL paradigm takes mean squared error (MSE) as loss function,
the default assumption is that its error term obeys independently identically distribution (IID) Gaussian, and its noise level involved preset-hyperparameters in the local area of data cannot be adjusted adaptively in the training phase. This leads to the poor generalization of deep denoiser on Non-IID noises. In this study, we propose a DL framework based on Non-IID pixel-wise Gaussian noise modeling, which integrates NA and noise level estimation into a unique Bayesian framework. The framework can adaptively characterize the noise and data distribution in the local area of noisy data through the variational inference (VI) technique, which allows the network to see more noises of varying degrees and learn effective information from them. Thus, our proposed framework called VI-Non-IID inclines to have better noise characterization and generalization capabilities, which
brings better performance on seismic field NA. Furthermore, we conduct a series of experiments on seismic synthetic and field data to test the performance of two implementations of VI-NonIID: VI-Non-IID (Unet) and VI-Non-IID denoising convolution network (DnCNN). A lot of results validate the superiority of our proposed VI-Non-IID framework. Specifically, VI-Non-IID can explicitly predict the denoised data and its corresponding noise level map simultaneously, and succeed in attenuating unknown field noises while preserving the useful seismic signals.
Generative modeling of seismic data using score-based generative models
Authors: Chuangji Meng, Jinghuai Gao, Yajun Tian, Hongling Chen, Renyu Luo.
Date: 2024-03-17 00:00:00
Proceeding: EAGE Annual Conference & Exhibition 2024 3
Tasks: Geophysics, Image Generation

The availability of seismic data has specific implications for inversion and interpretation methods, especially for learning-based methods such as deep learning. Forward modeling or acquisition of field data are common means of obtaining seismic data, which can be costly, time-consuming and labor-intensive. This paper provides a generative modeling method for generating seismic data. We use a score-based generative model to learn the distribution of target seismic data set. Specifically, we learn the gradient of the target data set distribution through score matching using the noise conditional score network (NCSN). Then, we sample high-quality samples similar to the target data set through Langevin dynamics with learned NCSN. We take the seismic records synthesized by Marmousi as an example to show the powerful generative modeling capabilities of the generative model. By sampling from a prior distribution (Gaussian distribution), the generative model can generate diverse samples and have good interpretability. For example, by interpolating two random data points from the prior distribution, the generated data has manifold continuity in certain features (such as amplitude, inclination, polarity, number of events, etc.).
Stochastic Solutions for Simultaneous Seismic Data Denoising and Reconstruction via Score-based Generative Models
Authors: Chuangji Meng, Jinghuai Gao∗, Yajun Tian, Hongling Chen∗, Wei zhang, Renyu Luo.
Date: 2024-07-01 00:00:00
Proceeding: IEEE Transactions on Geoscience and Remote Sensing 2024 7
Tasks: Denoising, Geophysics, Image Generation

Usually, inverse problems are ill-posed. The solution to the inverse problem is indeterminate, meaning that for given observational data, there may be multiple possible solutions. It is not sufficient to give a definite solution to common seismic inverse problems. In this study, we provide stochastic solutions for seismic inverse problems (denoising and reconstruction). We sample a range of possible and high -quality solutions for a given observation with various degradations from the posterior distribution through Langevin dynamics with conditional score function, all shown to be reasonable results; for example, the stochastic solutions we sampled may contain as many geological structures of interest to the expert as possible. Experimental results on synthetic and field data verify the superiority of posterior sampling. In particular, our method has obvious advantages over other methods, such as traditional and (supervised, self-supervised, and unsupervised) deep learning (DL) methods, especially in denoising under extremely low signal-to-noise ratio (SNR) and reconstruction for data with consecutively missing traces and noise. We also analyze the advantages of our approach and concluded that successful generative modeling of seismic data by the score-based generative models (SGMs) is the key to posterior sampling for the inverse problems, which all benefit from the seismic data prior implicit in the trained score network in the SGMs.
Evaluating Human-AI Collaboration: A Review and Methodological Framework
Authors: George Fragiadakis, Christos Diou, George Kousiouris, Mara Nikolaidou
Date: 2024-07-09 00:00:00
Tasks: Decision Making

The use of artificial intelligence (AI) in working environments with individuals, known as Human-AI Collaboration (HAIC), has become essential in a variety of domains, boosting decision-making, efficiency, and innovation. Despite HAIC's wide potential, evaluating its effectiveness remains challenging due to the complex interaction of components involved. This paper provides a detailed analysis of existing HAIC evaluation approaches and develops a fresh paradigm for more effectively evaluating these systems. Our framework includes a structured decision tree which assists to select relevant metrics based on distinct HAIC modes (AI-Centric, Human-Centric, and Symbiotic). By including both quantitative and qualitative metrics, the framework seeks to represent HAIC's dynamic and reciprocal nature, enabling the assessment of its impact and success. This framework's practicality can be examined by its application in an array of domains, including manufacturing, healthcare, finance, and education, each of which has unique challenges and requirements. Our hope is that this study will facilitate further research on the systematic evaluation of HAIC in real-world applications.
Comparison of Large Language Models for Generating Contextually Relevant Questions
Authors: Ivo Lodovico Molina, Valdemar Švábenský, Tsubasa Minematsu, Li Chen, Fumiya Okubo, Atsushi Shimada
Date: 2024-07-30 00:00:00
Tasks: Question Generation, Question-Generation

This study explores the effectiveness of Large Language Models (LLMs) for Automatic Question Generation in educational settings. Three LLMs are compared in their ability to create questions from university slide text without fine-tuning. Questions were obtained in a two-step pipeline: first, answer phrases were extracted from slides using Llama 2-Chat 13B; then, the three models generated questions for each answer. To analyze whether the questions would be suitable in educational applications for students, a survey was conducted with 46 students who evaluated a total of 246 questions across five metrics: clarity, relevance, difficulty, slide relation, and question-answer alignment. Results indicate that GPT-3.5 and Llama 2-Chat 13B outperform Flan T5 XXL by a small margin, particularly in terms of clarity and question-answer alignment. GPT-3.5 especially excels at tailoring questions to match the input answers. The contribution of this research is the analysis of the capacity of LLMs for Automatic Question Generation in education.
A2SF: Accumulative Attention Scoring with Forgetting Factor for Token Pruning in Transformer Decoder
Authors: Hyun-rae Jo, Dongkun Shin
Date: 2024-07-30 00:00:00
Tasks: Decoder, Fairness

Recently, large language models (LLM) based on transformers are facing memory bottleneck issues due to KV cache, especially in long sequence handling. Previous researches proposed KV cache compression techniques that identify insignificant tokens based on Accumulative Attention Scores and removes their items from KV cache, noting that only few tokens play an important role in attention operations. However, we have observed that the existing Accumulative Attention Score is not suitable for the transformer decoder structure. In the decoder model, the number of times the Attention Score accumulates varies depending on the order of token appearance due to the effect of masking, causing an uneven comparison between tokens. To solve this, we propose Accumulative Attention Score with Forgetting Factor (A2SF) technique, which introduces a Forgetting Factor in the Attention Score accumulation process. A2SF applies a penalty to the past Attention Score generated from old tokens by repeatedly multiplying the Forgetting Factor to the Attention Score over time. Therefore, older tokens receive a larger penalty, providing fairness among different ages of tokens. Through the fair comparison among tokens, we can more effectively select important tokens. We have verified the accuracy improvement through A2SF in the OPT and LLaMA models and A2SF improves the accuracy of LLaMA 2 by up to 7.8% and 5.1% on 1-shot and 0-shot.
A Bayesian Flow Network Framework for Chemistry Tasks
Authors: Nianze Tao, Minori Abe
Date: 2024-07-28 00:00:00
Tasks: Diversity, Language Modeling, Language Modelling, Molecular Property Prediction, Text-based de novo Molecule Generation

In this work, we introduce ChemBFN, a language model that handles chemistry tasks based on Bayesian flow networks working on discrete data. A new accuracy schedule is proposed to improve the sampling quality by significantly reducing the reconstruction loss. We show evidence that our method is appropriate for generating molecules with satisfied diversity even when a smaller number of sampling steps is used. A classifier-free guidance method is adapted for conditional generation. It is also worthwhile to point out that after generative training, our model can be fine-tuned on regression and classification tasks with the state-of-the-art performance, which opens the gate of building all-in-one models in a single module style. Our model has been open sourced at https://github.com/Augus1999/bayesian-flow-network-for-chemistry.
StackFLOW: Monocular Human-Object Reconstruction by Stacked Normalizing Flow with Offset
Authors: Chaofan Huo, Ye Shi, Yuexin Ma, Lan Xu, Jingyi Yu, Jingya Wang
Date: 2024-07-30 00:00:00
Tasks: Human-Object Interaction Detection, Object, Object Reconstruction

Modeling and capturing the 3D spatial arrangement of the human and the object is the key to perceiving 3D human-object interaction from monocular images. In this work, we propose to use the Human-Object Offset between anchors which are densely sampled from the surface of human mesh and object mesh to represent human-object spatial relation. Compared with previous works which use contact map or implicit distance filed to encode 3D human-object spatial relations, our method is a simple and efficient way to encode the highly detailed spatial correlation between the human and object. Based on this representation, we propose Stacked Normalizing Flow (StackFLOW) to infer the posterior distribution of human-object spatial relations from the image. During the optimization stage, we finetune the human body pose and object 6D pose by maximizing the likelihood of samples based on this posterior distribution and minimizing the 2D-3D corresponding reprojection loss. Extensive experimental results show that our method achieves impressive results on two challenging benchmarks, BEHAVE and InterCap datasets.
Efficient Multi-Objective Neural Architecture Search via Pareto Dominance-based Novelty Search
Authors: An Vo, Ngoc Hoang Luong
Date: 2024-07-30 00:00:00
Tasks: Diversity, Neural Architecture Search

Neural Architecture Search (NAS) aims to automate the discovery of high-performing deep neural network architectures. Traditional objective-based NAS approaches typically optimize a certain performance metric (e.g., prediction accuracy), overlooking large parts of the architecture search space that potentially contain interesting network configurations. Furthermore, objective-driven population-based metaheuristics in complex search spaces often quickly exhaust population diversity and succumb to premature convergence to local optima. This issue becomes more complicated in NAS when performance objectives do not fully align with the actual performance of the candidate architectures, as is often the case with training-free metrics. While training-free metrics have gained popularity for their rapid performance estimation of candidate architectures without incurring computation-heavy network training, their effective incorporation into NAS remains a challenge. This paper presents the Pareto Dominance-based Novelty Search for multi-objective NAS with Multiple Training-Free metrics (MTF-PDNS). Unlike conventional NAS methods that optimize explicit objectives, MTF-PDNS promotes population diversity by utilizing a novelty score calculated based on multiple training-free performance and complexity metrics, thereby yielding a broader exploration of the search space. Experimental results on standard NAS benchmark suites demonstrate that MTF-PDNS outperforms conventional methods driven by explicit objectives in terms of convergence speed, diversity maintenance, architecture transferability, and computational costs.
Monocular Human-Object Reconstruction in the Wild
Authors: Chaofan Huo, Ye Shi, Jingya Wang
Date: 2024-07-30 00:00:00
Tasks: Diversity, Human-Object Interaction Detection, Object, Object Reconstruction

Learning the prior knowledge of the 3D human-object spatial relation is crucial for reconstructing human-object interaction from images and understanding how humans interact with objects in 3D space. Previous works learn this prior from datasets collected in controlled environments, but due to the diversity of domains, they struggle to generalize to real-world scenarios. To overcome this limitation, we present a 2D-supervised method that learns the 3D human-object spatial relation prior purely from 2D images in the wild. Our method utilizes a flow-based neural network to learn the prior distribution of the 2D human-object keypoint layout and viewports for each image in the dataset. The effectiveness of the prior learned from 2D images is demonstrated on the human-object reconstruction task by applying the prior to tune the relative pose between the human and the object during the post-optimization stage. To validate and benchmark our method on in-the-wild images, we collect the WildHOI dataset from the YouTube website, which consists of various interactions with 8 objects in real-world scenarios. We conduct the experiments on the indoor BEHAVE dataset and the outdoor WildHOI dataset. The results show that our method achieves almost comparable performance with fully 3D supervised methods on the BEHAVE dataset, even if we have only utilized the 2D layout information, and outperforms previous methods in terms of generality and interaction diversity on in-the-wild images.
Assessing Graphical Perception of Image Embedding Models using Channel Effectiveness
Authors: Soohyun Lee, Minsuk Chang, SeokHyeon Park, Jinwook Seo
Date: 2024-07-30 00:00:00
Tasks: Chart Understanding, Question Answering

Recent advancements in vision models have greatly improved their ability to handle complex chart understanding tasks, like chart captioning and question answering. However, it remains challenging to assess how these models process charts. Existing benchmarks only roughly evaluate model performance without evaluating the underlying mechanisms, such as how models extract image embeddings. This limits our understanding of the model's ability to perceive fundamental graphical components. To address this, we introduce a novel evaluation framework to assess the graphical perception of image embedding models. For chart comprehension, we examine two main aspects of channel effectiveness: accuracy and discriminability of various visual channels. Channel accuracy is assessed through the linearity of embeddings, measuring how well the perceived magnitude aligns with the size of the stimulus. Discriminability is evaluated based on the distances between embeddings, indicating their distinctness. Our experiments with the CLIP model show that it perceives channel accuracy differently from humans and shows unique discriminability in channels like length, tilt, and curvature. We aim to develop this work into a broader benchmark for reliable visual encoders, enhancing models for precise chart comprehension and human-like perception in future applications.
ThinK: Thinner Key Cache by Query-Driven Pruning
Authors: Yuhui Xu, Zhanming Jie, Hanze Dong, Lei Wang, Xudong Lu, Aojun Zhou, Amrita Saha, Caiming Xiong, Doyen Sahoo
Date: 2024-07-30 00:00:00
Tasks: GPU, Quantization

Large Language Models (LLMs) have revolutionized the field of natural language processing, achieving unprecedented performance across a variety of applications. However, their increased computational and memory demands present significant challenges, especially when handling long sequences. This paper focuses on the long-context scenario, addressing the inefficiencies in KV cache memory consumption during inference. Unlike existing approaches that optimize the memory based on the sequence length, we identify substantial redundancy in the channel dimension of the KV cache, as indicated by an uneven magnitude distribution and a low-rank structure in the attention weights. In response, we propose ThinK, a novel query-dependent KV cache pruning method designed to minimize attention weight loss while selectively pruning the least significant channels. Our approach not only maintains or enhances model accuracy but also achieves a reduction in KV cache memory costs by over 20% compared with vanilla KV cache eviction and quantization methods. For instance, ThinK integrated with KIVI can achieve a 2.8x reduction in peak memory usage while maintaining nearly the same quality, enabling up to a 5x increase in batch size when using a single GPU. Extensive evaluations on the LLaMA and Mistral models across various long-sequence datasets verified the efficiency of ThinK, establishing a new baseline algorithm for efficient LLM deployment without compromising performance.
CLEFT: Language-Image Contrastive Learning with Efficient Large Language Model and Prompt Fine-Tuning
Authors: Yuexi Du, Brian Chang, Nicha C. Dvornek
Date: 2024-07-30 00:00:00
Tasks: Contrastive Learning, Diagnostic, GPU, Language Modeling, Language Modelling, Large Language Model, Representation Learning

Recent advancements in Contrastive Language-Image Pre-training (CLIP) have demonstrated notable success in self-supervised representation learning across various tasks. However, the existing CLIP-like approaches often demand extensive GPU resources and prolonged training times due to the considerable size of the model and dataset, making them poor for medical applications, in which large datasets are not always common. Meanwhile, the language model prompts are mainly manually derived from labels tied to images, potentially overlooking the richness of information within training samples. We introduce a novel language-image Contrastive Learning method with an Efficient large language model and prompt Fine-Tuning (CLEFT) that harnesses the strengths of the extensive pre-trained language and visual models. Furthermore, we present an efficient strategy for learning context-based prompts that mitigates the gap between informative clinical diagnostic data and simple class labels. Our method demonstrates state-of-the-art performance on multiple chest X-ray and mammography datasets compared with various baselines. The proposed parameter efficient framework can reduce the total trainable model size by 39% and reduce the trainable language model to only 4% compared with the current BERT encoder.