pwc-archive/papers-with-abstracts huggingface dataset:
Improving Vector-Quantized Image Modeling with Latent Consistency-Matching Diffusion
Authors: Bac Nguyen, Chieh-Hsin Lai, Yuhta Takida, Naoki Murata, Toshimitsu Uesaka, Stefano Ermon, Yuki Mitsufuji
Date: 2024-10-18 00:00:00
Tasks: Conditional Image Generation, Image Generation, Machine Translation, Text Generation

By embedding discrete representations into a continuous latent space, we can leverage continuous-space latent diffusion models to handle generative modeling of discrete data. However, despite their initial success, most latent diffusion methods rely on fixed pretrained embeddings, limiting the benefits of joint training with the diffusion model. While jointly learning the embedding (via reconstruction loss) and the latent diffusion model (via score matching loss) could enhance performance, end-to-end training risks embedding collapse, degrading generation quality. To mitigate this issue, we introduce VQ-LCMD, a continuous-space latent diffusion framework within the embedding space that stabilizes training. VQ-LCMD uses a novel training objective combining the joint embedding-diffusion variational lower bound with a consistency-matching (CM) loss, alongside a shifted cosine noise schedule and random dropping strategy. Experiments on several benchmarks show that the proposed VQ-LCMD yields superior results on FFHQ, LSUN Churches, and LSUN Bedrooms compared to discrete-state latent diffusion models. In particular, VQ-LCMD achieves an FID of 6.81 for class-conditional image generation on ImageNet with 50 steps.
TextSleuth: A New Dataset and Baseline for Scene Text Manipulation Detection
Authors: Abhineet Kumar Pandey, Ming-Ching Chang Xin, Li
Date: 2024-08-07 00:00:00
Proceeding: Conference on Multimedia Information Processing and Retrieval 2024 8
Tasks: Image Manipulation, Image Manipulation Detection

With the rise of digital content on social media and the advancement of image editing tools, tampering with scene text has become a serious concern. Scene text manipulation detection (STMD) is a kind of image manipulation detection (IMD) with focus on the tampering of scene text pixels, which is crucial for image content integrity and media forensics. In this paper, we present TextSleuth, a novel benchmark dataset specifically designed for STMD, by integrating three public datasets with newly introduced manipulation and annotations. We introduce professional edits on the Total-Text dataset (~1K images) with four levels of manipulated region perceptibility, and a large synthetic manipulation set (858K images) on the SynthText dataset, as well the integration of the Tampered-IC13 dataset (378 images). We established a new STMD baseline based on TextSleuth using MMFusion-IML, the state-of-the-art image manipulation detection model. We performed extensive experiments, reporting the AUC from ROC analysis and the balanced accuracy (bACC) metrics to maintain a balanced performance evaluation. The MMFusion-IML baseline achieves 0.641 AUC and 0.588 bACC on the Total-Text subset. In comparison, it achieves 0.89 AUC and 0.8272 bACC on the Tampered-IC13 subset. This showcases the real-world STMD challenges reflected in our new dataset. TextSleuth is a valuable resource for future research in scene text manipulation detection and forensics. The dataset is available at https://github.com/abhineet-pandey/Text-Sleuth.
Context-Aware Predictive Coding: A Representation Learning Framework for WiFi Sensing
Authors: B. Barahimi, H. Tabassum, M. Omer, O. Waqar
Date: 2024-09-16 00:00:00
Tasks: Activity Recognition, Human Activity Recognition, Representation Learning, Self-Supervised Learning, Transfer Learning

WiFi sensing is an emerging technology that utilizes wireless signals for various sensing applications. However, the reliance on supervised learning, the scarcity of labelled data, and the incomprehensible channel state information (CSI) pose significant challenges. These issues affect deep learning models' performance and generalization across different environments. Consequently, self-supervised learning (SSL) is emerging as a promising strategy to extract meaningful data representations with minimal reliance on labelled samples. In this paper, we introduce a novel SSL framework called Context-Aware Predictive Coding (CAPC), which effectively learns from unlabelled data and adapts to diverse environments. CAPC integrates elements of Contrastive Predictive Coding (CPC) and the augmentation-based SSL method, Barlow Twins, promoting temporal and contextual consistency in data representations. This hybrid approach captures essential temporal information in CSI, crucial for tasks like human activity recognition (HAR), and ensures robustness against data distortions. Additionally, we propose a unique augmentation, employing both uplink and downlink CSI to isolate free space propagation effects and minimize the impact of electronic distortions of the transceiver. Our evaluations demonstrate that CAPC not only outperforms other SSL methods and supervised approaches, but also achieves superior generalization capabilities. Specifically, CAPC requires fewer labelled samples while significantly outperforming supervised learning and surpassing SSL baselines. Furthermore, our transfer learning studies on an unseen dataset with a different HAR task and environment showcase an accuracy improvement of 1.8 percent over other SSL baselines and 24.7 percent over supervised learning, emphasizing its exceptional cross-domain adaptability.
BIPEFT: Budget-Guided Iterative Search for Parameter Efficient Fine-Tuning of Large Pretrained Language Models
Authors: Aofei Chang, Jiaqi Wang, Han Liu, Parminder Bhatia, Cao Xiao, Ting Wang, Fenglong Ma
Date: 2024-10-04 00:00:00
Tasks: parameter-efficient fine-tuning

Parameter Efficient Fine-Tuning (PEFT) offers an efficient solution for fine-tuning large pretrained language models for downstream tasks. However, most PEFT strategies are manually designed, often resulting in suboptimal performance. Recent automatic PEFT approaches aim to address this but face challenges such as search space entanglement, inefficiency, and lack of integration between parameter budgets and search processes. To overcome these issues, we introduce a novel Budget-guided Iterative search strategy for automatic PEFT (BIPEFT), significantly enhancing search efficiency. BIPEFT employs a new iterative search strategy to disentangle the binary module and rank dimension search spaces. Additionally, we design early selection strategies based on parameter budgets, accelerating the learning process by gradually removing unimportant modules and fixing rank dimensions. Extensive experiments on public benchmarks demonstrate the superior performance of BIPEFT in achieving efficient and effective PEFT for downstream tasks with a low parameter budget.
BACH: A Tool for Analyzing Blockchain Transactions Using Address Clustering Heuristics
Authors: Michele Caringella, Francesco Violante, Francesco De Lucci, Stefano Galantucci, Matteo Costantini
Date: 2024-09-26 00:00:00
Proceeding: Information, 15(10), 589 2024 9
Tasks: Clustering

Cryptocurrencies have now become an emerging blockchain-based payment technology; among them, bitcoin is the best known and most widely used. Users on these networks are pseudo-anonymous, meaning that while all transactions from an address are transparent and searchable by anyone, the users’ true identities are not directly revealed; to preserve their privacy, users often use many different addresses. In recent years, some studies have been conducted regarding analyzing clusters of bitcoin addresses that, according to certain heuristics, belong to the same entity. This capability provides law enforcement with valuable information for investigating illegal activities involving cryptocurrencies. Clustering methods that rely on a single heuristic often fail to accurately and comprehensively cluster multiple addresses. This paper proposes Bitcoin Address Clustering based on multiple Heuristics (BACH): a tool that uses three different clustering heuristics to identify clusters of bitcoin addresses, which are displayed through a three-dimensional graph. The results lead to several analyses, including a comparative evaluation of WalletExplorer, which is a similar address clustering tool. BACH introduces the innovative feature of visualizing the internal structure of clusters in a graphical format. The study also shows how the combined use of different heuristics provides better results and more complete clusters than those obtained from their individual use.
PUCA: Patch-Unshuffle and Channel Attention for Enhanced Self-Supervised Image Denoising
Authors: Hyemi Jang, Junsung Park, Dahuin Jung, Jaihyun Lew, Ho Bae, Sungroh Yoon
Date: 2023-10-16 00:00:00
Proceeding: NeurIPS 2023 11
Tasks: Denoising, Image Denoising, Self-Supervised Learning

Although supervised image denoising networks have shown remarkable performance on synthesized noisy images, they often fail in practice due to the difference between real and synthesized noise. Since clean-noisy image pairs from the real world are extremely costly to gather, self-supervised learning, which utilizes noisy input itself as a target, has been studied. To prevent a self-supervised denoising model from learning identical mapping, each output pixel should not be influenced by its corresponding input pixel; This requirement is known as J-invariance. Blind-spot networks (BSNs) have been a prevalent choice to ensure J-invariance in self-supervised image denoising. However, constructing variations of BSNs by injecting additional operations such as downsampling can expose blinded information, thereby violating J-invariance. Consequently, convolutions designed specifically for BSNs have been allowed only, limiting architectural flexibility. To overcome this limitation, we propose PUCA, a novel J-invariant U-Net architecture, for self-supervised denoising. PUCA leverages patch-unshuffle/shuffle to dramatically expand receptive fields while maintaining J-invariance and dilated attention blocks (DABs) for global context incorporation. Experimental results demonstrate that PUCA achieves state-of-the-art performance, outperforming existing methods in self-supervised image denoising.
Achilles, Neural Network to Predict the Gold Vs US Dollar Integration with Trading Bot for Automatic Trading
Authors: Angel Varela1
Date: 2024-10-29 00:00:00
Proceeding: NA 2024 10

Predicting the stock market is a big challenge for the machine
learning world. It is known how difficult it is to have accurate
and consistent predictions with ML models. Some
architectures are able to capture the movement of stocks but
almost never are able to be launched to the production world.
We present Achilles, with a classical architecture of
LSTM(Long Short Term Memory) neural network this model
is able to predict the Gold vs USD commodity. With the
predictions minute-per-minute of this model we implemented
a trading bot to run during a month of testing excluding
weekends. At the end of the testing period we generated
$1623.52 in profit with the methodology used. The results of
our method demonstrate Machine Learning can successfully
be implemented to predict the Gold vs USD commodity
Transformers Utilization in Chart Understanding: A Review of Recent Advances & Future Trends
Authors: Mirna Al-Shetairy, Hanan Hindy, Dina Khattab, Mostafa M. Aref
Date: 2024-10-05 00:00:00
Tasks: Benchmarking, Chart Understanding, Optical Character Recognition (OCR), Prompt Engineering, Visual Reasoning

In recent years, interest in vision-language tasks has grown, especially those involving chart interactions. These tasks are inherently multimodal, requiring models to process chart images, accompanying text, underlying data tables, and often user queries. Traditionally, Chart Understanding (CU) relied on heuristics and rule-based systems. However, recent advancements that have integrated transformer architectures significantly improved performance. This paper reviews prominent research in CU, focusing on State-of-The-Art (SoTA) frameworks that employ transformers within End-to-End (E2E) solutions. Relevant benchmarking datasets and evaluation techniques are analyzed. Additionally, this article identifies key challenges and outlines promising future directions for advancing CU solutions. Following the PRISMA guidelines, a comprehensive literature search is conducted across Google Scholar, focusing on publications from Jan'20 to Jun'24. After rigorous screening and quality assessment, 32 studies are selected for in-depth analysis. The CU tasks are categorized into a three-layered paradigm based on the cognitive task required. Recent advancements in the frameworks addressing various CU tasks are also reviewed. Frameworks are categorized into single-task or multi-task based on the number of tasks solvable by the E2E solution. Within multi-task frameworks, pre-trained and prompt-engineering-based techniques are explored. This review overviews leading architectures, datasets, and pre-training tasks. Despite significant progress, challenges remain in OCR dependency, handling low-resolution images, and enhancing visual reasoning. Future directions include addressing these challenges, developing robust benchmarks, and optimizing model efficiency. Additionally, integrating explainable AI techniques and exploring the balance between real and synthetic data are crucial for advancing CU research.
Large Body Language Models
Authors: Saif Punjwani, Larry Heck
Date: 2024-10-21 00:00:00
Tasks: Gesture Generation, Language Modeling, Language Modelling, Large Language Model, Neural Rendering

As virtual agents become increasingly prevalent in human-computer interaction, generating realistic and contextually appropriate gestures in real-time remains a significant challenge. While neural rendering techniques have made substantial progress with static scripts, their applicability to human-computer interactions remains limited. To address this, we introduce Large Body Language Models (LBLMs) and present LBLM-AVA, a novel LBLM architecture that combines a Transformer-XL large language model with a parallelized diffusion model to generate human-like gestures from multimodal inputs (text, audio, and video). LBLM-AVA incorporates several key components enhancing its gesture generation capabilities, such as multimodal-to-pose embeddings, enhanced sequence-to-sequence mapping with redefined attention mechanisms, a temporal smoothing module for gesture sequence coherence, and an attention-based refinement module for enhanced realism. The model is trained on our large-scale proprietary open-source dataset Allo-AVA. LBLM-AVA achieves state-of-the-art performance in generating lifelike and contextually appropriate gestures with a 30% reduction in Fr\'echet Gesture Distance (FGD), and a 25% improvement in Fr\'echet Inception Distance compared to existing approaches.
Accelerating Object Detection with YOLOv4 for Real-Time Applications
Authors: K. Senthil Kumar, K. M. B. Abdullah Safwan
Date: 2024-10-17 00:00:00
Tasks: Object, object-detection, Object Detection

Object Detection is related to Computer Vision. Object detection enables detecting instances of objects in images and videos. Due to its increased utilization in surveillance, tracking system used in security and many others applications have propelled researchers to continuously derive more efficient and competitive algorithms. However, problems emerges while implementing it in real-time because of their dynamic environment and complex algorithms used in object detection. In the last few years, Convolution Neural Network (CNN) have emerged as a powerful tool for recognizing image content and in computer vision approach for most problems. In this paper, We revived begins the brief introduction of deep learning and object detection framework like Convolutional Neural Network(CNN), You only look once - version 4 (YOLOv4). Then we focus on our proposed object detection architectures along with some modifications. The traditional model detects a small object in images. We have some modifications to the model. Our proposed method gives the correct result with accuracy.
EVC-MF: End-to-end Video Captioning Network with Multi-scale Features
Authors: Tian-Zi Niu, Zhen-Duo Chen, Xin Luo, Xin-Shun Xu
Date: 2024-10-22 00:00:00
Tasks: Decoder, Video Captioning

Conventional approaches for video captioning leverage a variety of offline-extracted features to generate captions. Despite the availability of various offline-feature-extractors that offer diverse information from different perspectives, they have several limitations due to fixed parameters. Concretely, these extractors are solely pre-trained on image/video comprehension tasks, making them less adaptable to video caption datasets. Additionally, most of these extractors only capture features prior to the classifier of the pre-training task, ignoring a significant amount of valuable shallow information. Furthermore, employing multiple offline-features may introduce redundant information. To address these issues, we propose an end-to-end encoder-decoder-based network (EVC-MF) for video captioning, which efficiently utilizes multi-scale visual and textual features to generate video descriptions. Specifically, EVC-MF consists of three modules. Firstly, instead of relying on multiple feature extractors, we directly feed video frames into a transformer-based network to obtain multi-scale visual features and update feature extractor parameters. Secondly, we fuse the multi-scale features and input them into a masked encoder to reduce redundancy and encourage learning useful features. Finally, we utilize an enhanced transformer-based decoder, which can efficiently leverage shallow textual information, to generate video descriptions. To evaluate our proposed model, we conduct extensive experiments on benchmark datasets. The results demonstrate that EVC-MF yields competitive performance compared with the state-of-theart methods.
Dhoroni: Exploring Bengali Climate Change and Environmental Views with a Multi-Perspective News Dataset and Natural Language Processing
Authors: Azmine Toushik Wasi, Wahid Faisal, Taj Ahmad, Abdur Rahman, Mst Rafia Islam
Date: 2024-10-22 00:00:00
Tasks: Articles, Authority Involvement Detection, Data Sources Detection, Detecting Climate/Environmental Topics, Impacted Location Detection, News Authenticity Identification, News Target Detection, Political Influence Detection, Scientific Data Usage Detection, Stance Detection, Statistical Data Usage Detection

Climate change poses critical challenges globally, disproportionately affecting low-income countries that often lack resources and linguistic representation on the international stage. Despite Bangladesh's status as one of the most vulnerable nations to climate impacts, research gaps persist in Bengali-language studies related to climate change and NLP. To address this disparity, we introduce Dhoroni, a novel Bengali (Bangla) climate change and environmental news dataset, comprising a 2300 annotated Bangla news articles, offering multiple perspectives such as political influence, scientific/statistical data, authenticity, stance detection, and stakeholder involvement. Furthermore, we present an in-depth exploratory analysis of Dhoroni and introduce BanglaBERT-Dhoroni family, a novel baseline model family for climate and environmental opinion detection in Bangla, fine-tuned on our dataset. This research contributes significantly to enhancing accessibility and analysis of climate discourse in Bengali (Bangla), addressing crucial communication and research gaps in climate-impacted regions like Bangladesh with 180 million people.
JMMMU: A Japanese Massive Multi-discipline Multimodal Understanding Benchmark for Culture-aware Evaluation
Authors: Shota Onohara, Atsuyuki Miyai, Yuki Imajuku, Kazuki Egashira, Jeonghun Baek, Xiang Yue, Graham Neubig, Kiyoharu Aizawa
Date: 2024-10-22 00:00:00
Tasks: Math

Accelerating research on Large Multimodal Models (LMMs) in non-English languages is crucial for enhancing user experiences across broader populations. In this paper, we introduce JMMMU (Japanese MMMU), the first large-scale Japanese benchmark designed to evaluate LMMs on expert-level tasks based on the Japanese cultural context. To facilitate comprehensive culture-aware evaluation, JMMMU features two complementary subsets: (i) culture-agnostic (CA) subset, where the culture-independent subjects (e.g., Math) are selected and translated into Japanese, enabling one-to-one comparison with its English counterpart MMMU; and (ii) culture-specific (CS) subset, comprising newly crafted subjects that reflect Japanese cultural context. Using the CA subset, we observe performance drop in many LMMs when evaluated in Japanese, which is purely attributable to language variation. Using the CS subset, we reveal their inadequate Japanese cultural understanding. Further, by combining both subsets, we identify that some LMMs perform well on the CA subset but not on the CS subset, exposing a shallow understanding of the Japanese language that lacks depth in cultural understanding. We hope this work will not only help advance LMM performance in Japanese but also serve as a guideline to create high-standard, culturally diverse benchmarks for multilingual LMM development. The project page is https://mmmu-japanese-benchmark.github.io/JMMMU/.
HyperspectralViTs: General Hyperspectral Models for On-board Remote Sensing
Authors: Vít Růžička, Andrew Markham
Date: 2024-10-22 00:00:00
Tasks: Scheduling

On-board processing of hyperspectral data with machine learning models would enable unprecedented amount of autonomy for a wide range of tasks, for example methane detection or mineral identification. This can enable early warning system and could allow new capabilities such as automated scheduling across constellations of satellites. Classical methods suffer from high false positive rates and previous deep learning models exhibit prohibitive computational requirements. We propose fast and accurate machine learning architectures which support end-to-end training with data of high spectral dimension without relying on hand-crafted products or spectral band compression preprocessing. We evaluate our models on two tasks related to hyperspectral data processing. With our proposed general architectures, we improve the F1 score of the previous methane detection state-of-the-art models by 27% on a newly created synthetic dataset and by 13% on the previously released large benchmark dataset. We also demonstrate that training models on the synthetic dataset improves performance of models finetuned on the dataset of real events by 6.9% in F1 score in contrast with training from scratch. On a newly created dataset for mineral identification, our models provide 3.5% improvement in the F1 score in contrast to the default versions of the models. With our proposed models we improve the inference speed by 85% in contrast to previous classical and deep learning approaches by removing the dependency on classically computed features. With our architecture, one capture from the EMIT sensor can be processed within 30 seconds on realistic proxy of the ION-SCV 004 satellite.