pwc-archive/papers-with-abstracts huggingface dataset:
Link, Synthesize, Retrieve: Universal Document Linking for Zero-Shot Information Retrieval
Authors: Dae Yon Hwang, Bilal Taha, Harshit Pande, Yaroslav Nechaev
Date: 2024-10-24 00:00:00
Tasks: Information Retrieval, named-entity-recognition, Named Entity Recognition, Named Entity Recognition (NER), NER, Retrieval

Despite the recent advancements in information retrieval (IR), zero-shot IR remains a significant challenge, especially when dealing with new domains, languages, and newly-released use cases that lack historical query traffic from existing users. For such cases, it is common to use query augmentations followed by fine-tuning pre-trained models on the document data paired with synthetic queries. In this work, we propose a novel Universal Document Linking (UDL) algorithm, which links similar documents to enhance synthetic query generation across multiple datasets with different characteristics. UDL leverages entropy for the choice of similarity models and named entity recognition (NER) for the link decision of documents using similarity scores. Our empirical studies demonstrate the effectiveness and universality of the UDL across diverse datasets and IR models, surpassing state-of-the-art methods in zero-shot cases. The developed code for reproducibility is included in https://github.com/eoduself/UDL
Stable Consistency Tuning: Understanding and Improving Consistency Models
Authors: Fu-Yun Wang, Zhengyang Geng, Hongsheng Li
Date: 2024-10-24 00:00:00
Tasks: Denoising, Image Generation

Diffusion models achieve superior generation quality but suffer from slow generation speed due to the iterative nature of denoising. In contrast, consistency models, a new generative family, achieve competitive performance with significantly faster sampling. These models are trained either through consistency distillation, which leverages pretrained diffusion models, or consistency training/tuning directly from raw data. In this work, we propose a novel framework for understanding consistency models by modeling the denoising process of the diffusion model as a Markov Decision Process (MDP) and framing consistency model training as the value estimation through Temporal Difference~(TD) Learning. More importantly, this framework allows us to analyze the limitations of current consistency training/tuning strategies. Built upon Easy Consistency Tuning (ECT), we propose Stable Consistency Tuning (SCT), which incorporates variance-reduced learning using the score identity. SCT leads to significant performance improvements on benchmarks such as CIFAR-10 and ImageNet-64. On ImageNet-64, SCT achieves 1-step FID 2.42 and 2-step FID 1.55, a new SoTA for consistency models.
Interweaving Insights: High-Order Feature Interaction for Fine-Grained Visual Recognition
Authors: Arindam Sikdar, Yonghuai Liu, Siddhardha Kedarisetty, Yitian Zhao, Amr Ahmed & Ardhendu Behera
Date: 2024-10-20 00:00:00
Proceeding: International Journal of Computer Vision 2024 10
Tasks: Computational Efficiency, Fine-Grained Image Classification, Fine-Grained Visual Recognition

This paper presents a novel approach for Fine-Grained Visual Classification (FGVC) by exploring Graph Neural Networks (GNNs) to facilitate high-order feature interactions, with a specific focus on constructing both inter- and intra-region graphs. Unlike previous FGVC techniques that often isolate global and local features, our method combines both features seamlessly during learning via graphs. Inter-region graphs capture long-range dependencies to recognize global patterns, while intra-region graphs delve into finer details within specific regions of an object by exploring high-dimensional convolutional features. A key innovation is the use of shared GNNs with an attention mechanism coupled with the Approximate Personalized Propagation of Neural Predictions (APPNP) message-passing algorithm, enhancing information propagation efficiency for better discriminability and simplifying the model architecture for computational efficiency. Additionally, the introduction of residual connections improves performance and training stability. Comprehensive experiments showcase state-of-the-art results on benchmark FGVC datasets, affirming the efficacy of our approach. This work underscores the potential of GNN in modeling high-level feature interactions, distinguishing it from previous FGVC methods that typically focus on singular aspects of feature representation. Our source code is available at https://github.com/Arindam-1991/I2-HOFI.
Real-time 3D-aware Portrait Video Relighting
Authors: Ziqi Cai, Kaiwen Jiang, Shu-Yu Chen, Yu-Kun Lai, Hongbo Fu, Boxin Shi, Lin Gao
Date: 2024-10-24 00:00:00
Proceeding: CVPR 2024 1
Tasks: NeRF

Synthesizing realistic videos of talking faces under custom lighting conditions and viewing angles benefits various downstream applications like video conferencing. However, most existing relighting methods are either time-consuming or unable to adjust the viewpoints. In this paper, we present the first real-time 3D-aware method for relighting in-the-wild videos of talking faces based on Neural Radiance Fields (NeRF). Given an input portrait video, our method can synthesize talking faces under both novel views and novel lighting conditions with a photo-realistic and disentangled 3D representation. Specifically, we infer an albedo tri-plane, as well as a shading tri-plane based on a desired lighting condition for each video frame with fast dual-encoders. We also leverage a temporal consistency network to ensure smooth transitions and reduce flickering artifacts. Our method runs at 32.98 fps on consumer-level hardware and achieves state-of-the-art results in terms of reconstruction quality, lighting error, lighting instability, temporal consistency and inference speed. We demonstrate the effectiveness and interactivity of our method on various portrait videos with diverse lighting and viewing conditions.
TabDPT: Scaling Tabular Foundation Models
Authors: Junwei Ma, Valentin Thomas, Rasa Hosseinzadeh, Hamidreza Kamkari, Alex Labach, Jesse C. Cresswell, Keyvan Golestan, Guangwei Yu, Maksims Volkovs, Anthony L. Caterini
Date: 2024-10-23 00:00:00
Tasks: In-Context Learning, Self-Supervised Learning

The challenges faced by neural networks on tabular data are well-documented and have hampered the progress of tabular foundation models. Techniques leveraging in-context learning (ICL) have shown promise here, allowing for dynamic adaptation to unseen data. ICL can provide predictions for entirely new datasets without further training or hyperparameter tuning, therefore providing very fast inference when encountering a novel task. However, scaling ICL for tabular data remains an issue: approaches based on large language models cannot efficiently process numeric tables, and tabular-specific techniques have not been able to effectively harness the power of real data to improve performance and generalization. We are able to overcome these challenges by training tabular-specific ICL-based architectures on real data with self-supervised learning and retrieval, combining the best of both worlds. Our resulting model -- the Tabular Discriminative Pre-trained Transformer (TabDPT) -- achieves state-of-the-art performance on the CC18 (classification) and CTR23 (regression) benchmarks with no task-specific fine-tuning, demonstrating the adapatability and speed of ICL once the model is pre-trained. TabDPT also demonstrates strong scaling as both model size and amount of available data increase, pointing towards future improvements simply through the curation of larger tabular pre-training datasets and training larger models.
A variational selection mechanism for article comment generation
Authors: Jiamiao Liu, Pengsen Cheng, Jinqiao Dai, Jiayong Liu
Date: 2023-09-12 00:00:00
Proceeding: Expert Systems with Applications 2023 9
Tasks: Articles, Comment Generation, Diversity, Informativeness

Article comment generation is a new and challenging task in natural language processing, which has recently received much attention from researchers. In the article comment generation, there are apparent distinctions and different perspectives on the comments of a single article. However, current researches ignore the one-to-many relationship between articles and comments, resulting in a lack of diversity and coherence in generated comments. To solve this problem, a variational selection mechanism model (VSMM) is proposed in our research. In this model, we construct a Gaussian mixture prior network to capture a richer latent space and generate comments with more diversity and informativeness. At the same time, VSMM maps latent variables into different semantic spaces through the selection mechanism to capture one-to-many relationships. Then we introduce a discriminator to distinguish whether the selected content is consistent with the reference comment content, thus improving the coherence of the generated comments. In addition, a hierarchical encoder with attention is introduced in the VSMM model, which can effectively solve the problem of long document encoding. Furthermore, we propose a multi-category article comment dataset to align closely with practical applications. Experiments on three datasets demonstrate that VSMM outperforms existing state-of-the-art comment generation methods in terms of diversity in single and multiple comment generations. Moreover, VSMM can generate fluent, diverse, and coherent comments on multi-category and topic-rich datasets.
DiffuCom: A novel diffusion model for comment generation
Authors: Jiamiao Liu, Pengsen Cheng, Jinqiao Dai, Jiayong Liu
Date: 2023-10-16 00:00:00
Proceeding: Knowledge-Based Systems 2023 10
Tasks: Comment Generation, model, Sentence, Text Generation

Given that the diffusion model has good advantages in diversity, an increasing number of researchers have recently begun exploring natural language generation. However, despite significant progress in modeling based on text diffusion, there are still several challenges to be overcome in comment tasks. On the one hand, the source sentences in the comment task are long. If each timestep in the diffusion carries and computes the source sentence, it will significantly increase the computational overhead. On the other hand, the diffusion model is essentially an iterative non-autoregressive model, which lacks target decomposition in the decoding. This makes vanilla crossattention prone to overlook local perceptual content, leading to the problem of poor correlation. To solve the aforementioned problem, we propose a novel diffusion model for comment generation (named DiffuCom). Specifically, DiffuCom employs the transformer- based encoder-decoder structure, encoding only once during the reverse diffusion process, effectively reducing computational overhead. Additionally, we design a context - aware cross -attention module (CACAM) that integrates global and local cross-attention information through a gating mechanism, which more accurately captures relevant fragment information in the source sentence and improves the correlation. Second, selfconditioning technology was introduced in the training process, which can effectively improve the quality of the samples. Furthermore, to enhance the controllability of DiffuCom, we also propose a blended control. It combines class-conditional and classifier guidance by utilizing the difference between training and inference. Extensive experiments on four comment datasets demonstrate that the proposed DiffuCom model can generate comments that are both diverse and relevant.
Where Am I and What Will I See: An Auto-Regressive Model for Spatial Localization and View Prediction
Authors: Junyi Chen, Di Huang, Weicai Ye, Wanli Ouyang, Tong He
Date: 2024-10-24 00:00:00
Tasks: Novel View Synthesis, Pose Estimation, Prediction, Spatial Reasoning

Spatial intelligence is the ability of a machine to perceive, reason, and act in three dimensions within space and time. Recent advancements in large-scale auto-regressive models have demonstrated remarkable capabilities across various reasoning tasks. However, these models often struggle with fundamental aspects of spatial reasoning, particularly in answering questions like "Where am I?" and "What will I see?". While some attempts have been done, existing approaches typically treat them as separate tasks, failing to capture their interconnected nature. In this paper, we present Generative Spatial Transformer (GST), a novel auto-regressive framework that jointly addresses spatial localization and view prediction. Our model simultaneously estimates the camera pose from a single image and predicts the view from a new camera pose, effectively bridging the gap between spatial awareness and visual prediction. The proposed innovative camera tokenization method enables the model to learn the joint distribution of 2D projections and their corresponding spatial perspectives in an auto-regressive manner. This unified training paradigm demonstrates that joint optimization of pose estimation and novel view synthesis leads to improved performance in both tasks, for the first time, highlighting the inherent relationship between spatial awareness and visual prediction.
AutoStep: Locally adaptive involutive MCMC
Authors: Tiange Liu, Nikola Surjanovic, Miguel Biron-Lattes, Alexandre Bouchard-Côté, Trevor Campbell
Date: 2024-10-24 00:00:00

Many common Markov chain Monte Carlo (MCMC) kernels can be formulated using a deterministic involutive proposal with a step size parameter. Selecting an appropriate step size is often a challenging task in practice; and for complex multiscale targets, there may not be one choice of step size that works well globally. In this work, we address this problem with a novel class of involutive MCMC methods -- AutoStep MCMC -- that selects an appropriate step size at each iteration adapted to the local geometry of the target distribution. We prove that under mild conditions AutoStep MCMC is $\pi$-invariant, irreducible, and aperiodic, and obtain bounds on expected energy jump distance and cost per iteration. Empirical results examine the robustness and efficacy of our proposed step size selection procedure, and show that AutoStep MCMC is competitive with state-of-the-art methods in terms of effective sample size per unit cost on a range of challenging target distributions.
PixelGaussian: Generalizable 3D Gaussian Reconstruction from Arbitrary Views
Authors: Xin Fei, Wenzhao Zheng, Yueqi Duan, Wei Zhan, Masayoshi Tomizuka, Kurt Keutzer, Jiwen Lu
Date: 2024-10-24 00:00:00

We propose PixelGaussian, an efficient feed-forward framework for learning generalizable 3D Gaussian reconstruction from arbitrary views. Most existing methods rely on uniform pixel-wise Gaussian representations, which learn a fixed number of 3D Gaussians for each view and cannot generalize well to more input views. Differently, our PixelGaussian dynamically adapts both the Gaussian distribution and quantity based on geometric complexity, leading to more efficient representations and significant improvements in reconstruction quality. Specifically, we introduce a Cascade Gaussian Adapter to adjust Gaussian distribution according to local geometry complexity identified by a keypoint scorer. CGA leverages deformable attention in context-aware hypernetworks to guide Gaussian pruning and splitting, ensuring accurate representation in complex regions while reducing redundancy. Furthermore, we design a transformer-based Iterative Gaussian Refiner module that refines Gaussian representations through direct image-Gaussian interactions. Our PixelGaussian can effectively reduce Gaussian redundancy as input views increase. We conduct extensive experiments on the large-scale ACID and RealEstate10K datasets, where our method achieves state-of-the-art performance with good generalization to various numbers of views. Code: https://github.com/Barrybarry-Smith/PixelGaussian.
CAMEL-Bench: A Comprehensive Arabic LMM Benchmark
Authors: Sara Ghaboura, Ahmed Heakl, Omkar Thawakar, Ali Alharthi, Ines Riahi, Abduljalil Saif, Jorma Laaksonen, Fahad S. Khan, Salman Khan, Rao M. Anwer
Date: 2024-10-24 00:00:00
Tasks: document understanding, Video Understanding, Visual Reasoning

Recent years have witnessed a significant interest in developing large multimodal models (LMMs) capable of performing various visual reasoning and understanding tasks. This has led to the introduction of multiple LMM benchmarks to evaluate LMMs on different tasks. However, most existing LMM evaluation benchmarks are predominantly English-centric. In this work, we develop a comprehensive LMM evaluation benchmark for the Arabic language to represent a large population of over 400 million speakers. The proposed benchmark, named CAMEL-Bench, comprises eight diverse domains and 38 sub-domains including, multi-image understanding, complex visual perception, handwritten document understanding, video understanding, medical imaging, plant diseases, and remote sensing-based land use understanding to evaluate broad scenario generalizability. Our CAMEL-Bench comprises around 29,036 questions that are filtered from a larger pool of samples, where the quality is manually verified by native speakers to ensure reliable model assessment. We conduct evaluations of both closed-source, including GPT-4 series, and open-source LMMs. Our analysis reveals the need for substantial improvement, especially among the best open-source models, with even the closed-source GPT-4o achieving an overall score of 62%. Our benchmark and evaluation scripts are open-sourced.
Unbounded: A Generative Infinite Game of Character Life Simulation
Authors: Jialu Li, Yuanzhen Li, Neal Wadhwa, Yael Pritch, David E. Jacobs, Michael Rubinstein, Mohit Bansal, Nataniel Ruiz
Date: 2024-10-24 00:00:00
Tasks: Instruction Following, Language Modelling, Large Language Model

We introduce the concept of a generative infinite game, a video game that transcends the traditional boundaries of finite, hard-coded systems by using generative models. Inspired by James P. Carse's distinction between finite and infinite games, we leverage recent advances in generative AI to create Unbounded: a game of character life simulation that is fully encapsulated in generative models. Specifically, Unbounded draws inspiration from sandbox life simulations and allows you to interact with your autonomous virtual character in a virtual world by feeding, playing with and guiding it - with open-ended mechanics generated by an LLM, some of which can be emergent. In order to develop Unbounded, we propose technical innovations in both the LLM and visual generation domains. Specifically, we present: (1) a specialized, distilled large language model (LLM) that dynamically generates game mechanics, narratives, and character interactions in real-time, and (2) a new dynamic regional image prompt Adapter (IP-Adapter) for vision models that ensures consistent yet flexible visual generation of a character across multiple environments. We evaluate our system through both qualitative and quantitative analysis, showing significant improvements in character life simulation, user instruction following, narrative coherence, and visual consistency for both characters and the environments compared to traditional related approaches.
3D-Adapter: Geometry-Consistent Multi-View Diffusion for High-Quality 3D Generation
Authors: Hansheng Chen, Bokui Shen, Yulin Liu, Ruoxi Shi, Linqi Zhou, Connor Z. Lin, Jiayuan Gu, Hao Su, Gordon Wetzstein, Leonidas Guibas
Date: 2024-10-24 00:00:00
Tasks: 3D Generation, 3D geometry, Denoising, Image to 3D, Text to 3D

Multi-view image diffusion models have significantly advanced open-domain 3D object generation. However, most existing models rely on 2D network architectures that lack inherent 3D biases, resulting in compromised geometric consistency. To address this challenge, we introduce 3D-Adapter, a plug-in module designed to infuse 3D geometry awareness into pretrained image diffusion models. Central to our approach is the idea of 3D feedback augmentation: for each denoising step in the sampling loop, 3D-Adapter decodes intermediate multi-view features into a coherent 3D representation, then re-encodes the rendered RGBD views to augment the pretrained base model through feature addition. We study two variants of 3D-Adapter: a fast feed-forward version based on Gaussian splatting and a versatile training-free version utilizing neural fields and meshes. Our extensive experiments demonstrate that 3D-Adapter not only greatly enhances the geometry quality of text-to-multi-view models such as Instant3D and Zero123++, but also enables high-quality 3D generation using the plain text-to-image Stable Diffusion. Furthermore, we showcase the broad application potential of 3D-Adapter by presenting high quality results in text-to-3D, image-to-3D, text-to-texture, and text-to-avatar tasks.
Deep Insights into Cognitive Decline: A Survey of Leveraging Non-Intrusive Modalities with Deep Learning Techniques
Authors: David Ortiz-Perez, Manuel Benavent-Lledo, Jose Garcia-Rodriguez, David Tomás, M. Flores Vizcaya-Moreno
Date: 2024-10-24 00:00:00

Cognitive decline is a natural part of aging, often resulting in reduced cognitive abilities. In some cases, however, this decline is more pronounced, typically due to disorders such as Alzheimer's disease. Early detection of anomalous cognitive decline is crucial, as it can facilitate timely professional intervention. While medical data can help in this detection, it often involves invasive procedures. An alternative approach is to employ non-intrusive techniques such as speech or handwriting analysis, which do not necessarily affect daily activities. This survey reviews the most relevant methodologies that use deep learning techniques to automate the cognitive decline estimation task, including audio, text, and visual processing. We discuss the key features and advantages of each modality and methodology, including state-of-the-art approaches like Transformer architecture and foundation models. In addition, we present works that integrate different modalities to develop multimodal models. We also highlight the most significant datasets and the quantitative results from studies using these resources. From this review, several conclusions emerge. In most cases, the textual modality achieves the best results and is the most relevant for detecting cognitive decline. Moreover, combining various approaches from individual modalities into a multimodal model consistently enhances performance across nearly all scenarios.
ConceptDrift: Uncovering Biases through the Lens of Foundation Models
Authors: Cristian Daniel Păduraru, Antonio Bărbălau, Radu Filipescu, Andrei Liviu Nicolicioiu, Elena Burceanu
Date: 2024-10-24 00:00:00

An important goal of ML research is to identify and mitigate unwanted biases intrinsic to datasets and already incorporated into pre-trained models. Previous approaches have identified biases using highly curated validation subsets, that require human knowledge to create in the first place. This limits the ability to automate the discovery of unknown biases in new datasets. We solve this by using interpretable vision-language models, combined with a filtration method using LLMs and known concept hierarchies. More exactly, for a dataset, we use pre-trained CLIP models that have an associated embedding for each class and see how it drifts through learning towards embeddings that disclose hidden biases. We call this approach ConceptDrift and show that it can be scaled to automatically identify biases in datasets like ImageNet without human prior knowledge. We propose two bias identification evaluation protocols to fill the gap in the previous work and show that our method significantly improves over SoTA methods, both using our protocol and classical evaluations. Alongside validating the identified biases, we also show that they can be leveraged to improve the performance of different methods. Our method is not bounded to a single modality, and we empirically validate it both on image (Waterbirds, CelebA, ImageNet), and text datasets (CivilComments).