pwc-archive/papers-with-abstracts huggingface dataset:
FrameFusion: Combining Similarity and Importance for Video Token Reduction on Large Visual Language Models
Authors: Tianyu Fu, Tengxuan Liu, Qinghao Han, Guohao Dai, Shengen Yan, Huazhong Yang, Xuefei Ning, Yu Wang
Date: 2024-12-30 00:00:00
Tasks: Question Answering, Token Reduction, Video Understanding

The increasing demand to process long and high-resolution videos significantly burdens Large Vision-Language Models (LVLMs) due to the enormous number of visual tokens. Existing token reduction methods primarily focus on importance-based token pruning, which overlooks the redundancy caused by frame resemblance and repetitive visual elements. In this paper, we analyze the high vision token similarities in LVLMs. We reveal that token similarity distribution condenses as layers deepen while maintaining ranking consistency. Leveraging the unique properties of similarity over importance, we introduce FrameFusion, a novel approach that combines similarity-based merging with importance-based pruning for better token reduction in LVLMs. FrameFusion identifies and merges similar tokens before pruning, opening up a new perspective for token reduction. We evaluate FrameFusion on diverse LVLMs, including Llava-Video-{7B,32B,72B}, and MiniCPM-V-8B, on video understanding, question-answering, and retrieval benchmarks. Experiments show that FrameFusion reduces vision tokens by 70$\%$, achieving 3.4-4.4x LLM speedups and 1.6-1.9x end-to-end speedups, with an average performance impact of less than 3$\%$. Our code is available at https://github.com/thu-nics/FrameFusion.
CRRG-CLIP: Automatic Generation of Chest Radiology Reports and Classification of Chest Radiographs
Authors: Jianfei Xu, Thanet Markchom, HuiZhi Liang
Date: 2024-12-31 00:00:00
Tasks: Classification

The complexity of stacked imaging and the massive number of radiographs make writing radiology reports complex and inefficient. Even highly experienced radiologists struggle to maintain accuracy and consistency in interpreting radiographs under prolonged high-intensity work. To address these issues, this work proposes the CRRG-CLIP Model (Chest Radiology Report Generation and Radiograph Classification Model), an end-to-end model for automated report generation and radiograph classification. The model consists of two modules: the radiology report generation module and the radiograph classification module. The generation module uses Faster R-CNN to identify anatomical regions in radiographs, a binary classifier to select key regions, and GPT-2 to generate semantically coherent reports. The classification module uses the unsupervised Contrastive Language Image Pretraining (CLIP) model, addressing the challenges of high-cost labelled datasets and insufficient features. The results show that the generation module performs comparably to high-performance baseline models on BLEU, METEOR, and ROUGE-L metrics, and outperformed the GPT-4o model on BLEU-2, BLEU-3, BLEU-4, and ROUGE-L metrics. The classification module significantly surpasses the state-of-the-art model in AUC and Accuracy. This demonstrates that the proposed model achieves high accuracy, readability, and fluency in report generation, while multimodal contrastive training with unlabelled radiograph-report pairs enhances classification performance.
A Hybrid Deep Learning and Model-Checking Framework for Accurate Brain Tumor Detection and Validation
Authors: Lahcen El fatimi, Elhoucine Elfatimi, Hanifa Bouchaneb
Date: 2024-12-31 00:00:00
Tasks: Medical Image Analysis, Segmentation

Model checking, a formal verification technique, ensures systems meet predefined requirements, playing a crucial role in minimizing errors and enhancing quality during development. This paper introduces a novel hybrid framework integrating model checking with deep learning for brain tumor detection and validation in medical imaging. By combining model-checking principles with CNN-based feature extraction and K-FCM clustering for segmentation, the proposed approach enhances the reliability of tumor detection and segmentation. Experimental results highlight the framework's effectiveness, achieving 98\% accuracy, 96.15\% precision, and 100\% recall, demonstrating its potential as a robust tool for advanced medical image analysis.
A Novel Convolution and Attention Mechanism-based Model for 6D Object Pose Estimation
Authors: Alexander Du, Yingwu Zhu
Date: 2024-12-31 00:00:00
Tasks: 6D Pose Estimation using RGB, feature selection, Pose Estimation

Estimating 6D object poses from RGB images is challenging because the lack of depth information requires inferring a three dimensional structure from 2D projections. Traditional methods often rely on deep learning with grid based data structures but struggle to capture complex dependencies among extracted features. To overcome this, we introduce a graph based representation derived directly from images, where spatial temporal features of each pixel serve as nodes, and relationships between them are defined through node connectivity and spatial interactions. We also employ feature selection mechanisms that use spatial attention and self attention distillation, along with a Legendre convolution layer leveraging the orthogonality of Legendre polynomials for numerical stability. Experiments on the LINEMOD, Occluded LINEMOD, and YCB Video datasets demonstrate that our method outperforms nine existing approaches and achieves state of the art benchmark in object pose estimation.
SmartSpatial: Enhancing the 3D Spatial Arrangement Capabilities of Stable Diffusion Models and Introducing a Novel 3D Spatial Evaluation Framework
Authors: Mao Xun Huang, Hen-Hsen Huang
Date: 2025-01-01 00:00:00
Tasks: Dependency Parsing, Image Generation

Stable Diffusion models have made remarkable strides in generating photorealistic images from text prompts but often falter when tasked with accurately representing complex spatial arrangements, particularly involving intricate 3D relationships. To address this limitation, we introduce SmartSpatial, an innovative approach that enhances the spatial arrangement capabilities of Stable Diffusion models through 3D-aware conditioning and attention-guided mechanisms. SmartSpatial incorporates depth information and employs cross-attention control to ensure precise object placement, delivering notable improvements in spatial accuracy metrics. In conjunction with SmartSpatial, we present SmartSpatialEval, a comprehensive evaluation framework designed to assess spatial relationships. This framework utilizes vision-language models and graph-based dependency parsing for performance analysis. Experimental results on the COCO and SpatialPrompts datasets show that SmartSpatial significantly outperforms existing methods, setting new benchmarks for spatial arrangement accuracy in image generation.
On the Utility of Equivariance and Symmetry Breaking in Deep Learning Architectures on Point Clouds
Authors: Sharvaree Vadgama, Mohammad Mohaiminul Islam, Domas Buracus, Christian Shewmake, Erik Bekkers
Date: 2025-01-01 00:00:00
Tasks: Benchmarking

This paper explores the key factors that influence the performance of models working with point clouds, across different tasks of varying geometric complexity. In this work, we explore the trade-offs between flexibility and weight-sharing introduced by equivariant layers, assessing when equivariance boosts or detracts from performance. It is often argued that providing more information as input improves a model's performance. However, if this additional information breaks certain properties, such as $\SE(3)$ equivariance, does it remain beneficial? We identify the key aspects of equivariant and non-equivariant architectures that drive success in different tasks by benchmarking them on segmentation, regression, and generation tasks across multiple datasets with increasing complexity. We observe a positive impact of equivariance, which becomes more pronounced with increasing task complexity, even when strict equivariance is not required.
Explainable Neural Networks with Guarantees: A Sparse Estimation Approach
Authors: Antoine Ledent, Peng Liu
Date: 2025-01-02 00:00:00
Tasks: Generalization Bounds

Balancing predictive power and interpretability has long been a challenging research area, particularly in powerful yet complex models like neural networks, where nonlinearity obstructs direct interpretation. This paper introduces a novel approach to constructing an explainable neural network that harmonizes predictiveness and explainability. Our model, termed SparXnet, is designed as a linear combination of a sparse set of jointly learned features, each derived from a different trainable function applied to a single 1-dimensional input feature. Leveraging the ability to learn arbitrarily complex relationships, our neural network architecture enables automatic selection of a sparse set of important features, with the final prediction being a linear combination of rescaled versions of these features. We demonstrate the ability to select significant features while maintaining comparable predictive performance and direct interpretability through extensive experiments on synthetic and real-world datasets. We also provide theoretical analysis on the generalization bounds of our framework, which is favorably linear in the number of selected features and only logarithmic in the number of input features. We further lift any dependence of sample complexity on the number of parameters or the architectural details under very mild conditions. Our research paves the way for further research on sparse and explainable neural networks with guarantee.
Information Subtraction: Learning Representations for Conditional Entropy
Authors: Keng Hou Leong, Yuxuan Xiu, Wai Kin, Chan
Date: 2025-01-02 00:00:00
Tasks: Domain Generalization

The representations of conditional entropy and conditional mutual information are significant in explaining the unique effects among variables. While previous studies based on conditional contrastive sampling have effectively removed information regarding discrete sensitive variables, they have not yet extended their scope to continuous cases. This paper introduces Information Subtraction, a framework designed to generate representations that preserve desired information while eliminating the undesired. We implement a generative-based architecture that outputs these representations by simultaneously maximizing an information term and minimizing another. With its flexibility in disentangling information, we can iteratively apply Information Subtraction to represent arbitrary information components between continuous variables, thereby explaining the various relationships that exist between them. Our results highlight the representations' ability to provide semantic features of conditional entropy. By subtracting sensitive and domain-specific information, our framework demonstrates effective performance in fair learning and domain generalization. The code for this paper is available at https://github.com/jh-liang/Information-Subtraction
OpenTable data with multi-criteria ratings
Authors: Yong Zheng
Date: 2024-11-22 00:00:00
Tasks: Recommendation Systems

With the development of recommender systems (RSs), several promising systems have emerged, such as context-aware RS, multi-criteria RS, and group RS. Multi-criteria recommender systems (MCRSs) are designed to provide personalized recommendations by considering user preferences in multiple attributes or criteria simultaneously. Unlike traditional RSs that typically focus on a single rating, these systems help users make more informed decisions by considering their diverse preferences and needs across various dimensions. In this article, we release the OpenTable data set which was crawled from OpenTable.com. The data set can be considered as a benchmark data set for multi-criteria recommendations.
A Privacy-Preserving Unsupervised Speaker Disentanglement Method for Depression Detection from Speech
Authors: Ravi, Vijay; Wang, Jinhan; Flint, Jonathan; Alwan, Abeer
Date: 2024-02-28 00:00:00
Proceeding: AAAI ML4CMH Workshop 2024 2
Tasks: De-identification, Depression Detection, Disentanglement, Privacy Preserving

The proposed method focuses on speaker disentanglement in the context of depression detection from speech signals.
Previous approaches require patient/speaker labels, encounter instability due to loss maximization, and introduce unnecessary
parameters for adversarial domain prediction. In contrast, the proposed unsupervised approach reduces cosine similarity
between latent spaces of depression and pre-trained speaker classification models. This method outperforms baseline models,
matches or exceeds adversarial methods in performance, and does so without relying on speaker labels or introducing
additional model parameters, leading to a reduction in model complexity. The higher the speaker de-identification score
(𝐷𝑒𝐼𝐷), the better the depression detection system is in masking a patient’s identity thereby enhancing the privacy attributes
of depression detection systems. On the DAIC-WOZ dataset with ComparE16 features and an LSTM-only model, our method
achieves an F1-Score of 0.776 and a 𝐷𝑒𝐼𝐷 score of 92.87%, outperforming its adversarial counterpart which has an F1-
Score of 0.762 and 68.37% 𝐷𝑒𝐼𝐷, respectively. Furthermore, we demonstrate that speaker-disentanglement methods are
complementary to text-based approaches, and a score-level fusion with a Word2vec-based depression detection model further
enhances the overall performance to an F1-Score of 0.830.
TLS_Finder: An algorithm for Identifying Tertiary Lymphoid Structures Using Immune Cell Spatial Coordinates
Authors: Ayse A Koksoy1, Maria Esther Salvatierra1, Patient Mosaic Team2, Luisa Maren Solis Soto1, Cara Haymaker1
Date: 2024-12-26 00:00:00
Proceeding: bioRxiv 2024 12

Tertiary lymphoid structures (TLS) are lymphoid formations that develop in non-lymphoid tissues during chronic inflammation, autoimmune diseases, and cancer. Accurate identification and quantification of TLS in tissue can provide crucial insights into the immune response of several disease processes including antitumor immune response. TLS are defined as aggregates of T cells, B cells and dendritic cells. In histological tissue sections stained with Hematoxylin and Eosin they are identified as aggregates of 50 or more lymphoid cells,  however immunohistochemical analysis are required to confirm presence of distinct immune cell patterns. Assessment of lymphoid aggregates can be done in H&E slides or in slides stained with single or multiplex immunohistochemistry or other tissue based high-plex approaches with key biomarkers such as CD3 (T cells) and CD20 (B cells), however manual assessment of them is time consuming thus limiting its full evaluation. To our knowledge, published algorithms that identify TLS within a tissue are based on histological assessment of H&E slides or through use of ML algorithms trained on images that show the TLS presence in tissues; and quantification and spatial analysis of TLS still remains a challenge. This study aims to develop a robust algorithm to recognize TLS using the spatial coordinates of immune cells in any given tissue.  The algorithm uses X and Y coordinates of T and B cells in tissues identified by a pathologist-supervised digital image analysis of multiplex chromogenic immunohistochemistry of tissues stained with CD3 and CD20. The algorithm is flexible to be used for detailed analysis of TLS stages; including other cell types within the definition of TLS; such as dendritic cells (DC) and high endothelial venules (HEV); to assess different stages of TLS formation.
LM-Net: A Light-weight and Multi-scale Network for Medical Image Segmentation
Authors: Zhenkun Lu, Chaoyin She, Wei Wang, Qinghua Huang
Date: 2025-01-07 00:00:00
Tasks: Image Segmentation, Medical Image Segmentation, Segmentation, Semantic Segmentation

Current medical image segmentation approaches have limitations in deeply exploring multi-scale information and effectively combining local detail textures with global contextual semantic information. This results in over-segmentation, under-segmentation, and blurred segmentation boundaries. To tackle these challenges, we explore multi-scale feature representations from different perspectives, proposing a novel, lightweight, and multi-scale architecture (LM-Net) that integrates advantages of both Convolutional Neural Networks (CNNs) and Vision Transformers (ViTs) to enhance segmentation accuracy. LM-Net employs a lightweight multi-branch module to capture multi-scale features at the same level. Furthermore, we introduce two modules to concurrently capture local detail textures and global semantics with multi-scale features at different levels: the Local Feature Transformer (LFT) and Global Feature Transformer (GFT). The LFT integrates local window self-attention to capture local detail textures, while the GFT leverages global self-attention to capture global contextual semantics. By combining these modules, our model achieves complementarity between local and global representations, alleviating the problem of blurred segmentation boundaries in medical image segmentation. To evaluate the feasibility of LM-Net, extensive experiments have been conducted on three publicly available datasets with different modalities. Our proposed model achieves state-of-the-art results, surpassing previous methods, while only requiring 4.66G FLOPs and 5.4M parameters. These state-of-the-art results on three datasets with different modalities demonstrate the effectiveness and adaptability of our proposed LM-Net for various medical image segmentation tasks.
VLM-driven Behavior Tree for Context-aware Task Planning
Authors: Naoki Wake, Atsushi Kanehira, Jun Takamatsu, Kazuhiro Sasabuchi, Katsushi Ikeuchi
Date: 2025-01-07 00:00:00
Tasks: Task Planning

The use of Large Language Models (LLMs) for generating Behavior Trees (BTs) has recently gained attention in the robotics community, yet remains in its early stages of development. In this paper, we propose a novel framework that leverages Vision-Language Models (VLMs) to interactively generate and edit BTs that address visual conditions, enabling context-aware robot operations in visually complex environments. A key feature of our approach lies in the conditional control through self-prompted visual conditions. Specifically, the VLM generates BTs with visual condition nodes, where conditions are expressed as free-form text. Another VLM process integrates the text into its prompt and evaluates the conditions against real-world images during robot execution. We validated our framework in a real-world cafe scenario, demonstrating both its feasibility and limitations.
Localizing AI: Evaluating Open-Weight Language Models for Languages of Baltic States
Authors: Jurgita Kapočiūtė-Dzikienė, Toms Bergmanis, Mārcis Pinnis
Date: 2025-01-07 00:00:00
Tasks: Machine Translation, Multiple-choice, Question Answering, Text Generation, Translation

Although large language models (LLMs) have transformed our expectations of modern language technologies, concerns over data privacy often restrict the use of commercially available LLMs hosted outside of EU jurisdictions. This limits their application in governmental, defence, and other data-sensitive sectors. In this work, we evaluate the extent to which locally deployable open-weight LLMs support lesser-spoken languages such as Lithuanian, Latvian, and Estonian. We examine various size and precision variants of the top-performing multilingual open-weight models, Llama~3, Gemma~2, Phi, and NeMo, on machine translation, multiple-choice question answering, and free-form text generation. The results indicate that while certain models like Gemma~2 perform close to the top commercially available models, many LLMs struggle with these languages. Most surprisingly, however, we find that these models, while showing close to state-of-the-art translation performance, are still prone to lexical hallucinations with errors in at least 1 in 20 words for all open-weight multilingual LLMs.
Synthetic Data Privacy Metrics
Authors: Amy Steier, Lipika Ramaswamy, Andre Manoel, Alexa Haushalter
Date: 2025-01-07 00:00:00

Recent advancements in generative AI have made it possible to create synthetic datasets that can be as accurate as real-world data for training AI models, powering statistical insights, and fostering collaboration with sensitive datasets while offering strong privacy guarantees. Effectively measuring the empirical privacy of synthetic data is an important step in the process. However, while there is a multitude of new privacy metrics being published every day, there currently is no standardization. In this paper, we review the pros and cons of popular metrics that include simulations of adversarial attacks. We also review current best practices for amending generative models to enhance the privacy of the data they create (e.g. differential privacy).
Not all tokens are created equal: Perplexity Attention Weighted Networks for AI generated text detection
Authors: Pablo Miralles-González, Javier Huertas-Tato, Alejandro Martín, David Camacho
Date: 2025-01-07 00:00:00
Tasks: All, Text Detection

The rapid advancement in large language models (LLMs) has significantly enhanced their ability to generate coherent and contextually relevant text, raising concerns about the misuse of AI-generated content and making it critical to detect it. However, the task remains challenging, particularly in unseen domains or with unfamiliar LLMs. Leveraging LLM next-token distribution outputs offers a theoretically appealing approach for detection, as they encapsulate insights from the models' extensive pre-training on diverse corpora. Despite its promise, zero-shot methods that attempt to operationalize these outputs have met with limited success. We hypothesize that one of the problems is that they use the mean to aggregate next-token distribution metrics across tokens, when some tokens are naturally easier or harder to predict and should be weighted differently. Based on this idea, we propose the Perplexity Attention Weighted Network (PAWN), which uses the last hidden states of the LLM and positions to weight the sum of a series of features based on metrics from the next-token distribution across the sequence length. Although not zero-shot, our method allows us to cache the last hidden states and next-token distribution metrics on disk, greatly reducing the training resource requirements. PAWN shows competitive and even better performance in-distribution than the strongest baselines (fine-tuned LMs) with a fraction of their trainable parameters. Our model also generalizes better to unseen domains and source models, with smaller variability in the decision boundary across distribution shifts. It is also more robust to adversarial attacks, and if the backbone has multilingual capabilities, it presents decent generalization to languages not seen during supervised training, with LLaMA3-1B reaching a mean macro-averaged F1 score of 81.46% in cross-validation with nine languages.