pwc-archive/papers-with-abstracts huggingface dataset:
COSMo: CLIP Talks on Open-Set Multi-Target Domain Adaptation
Authors: Munish Monga, Sachin Kumar Giroh, Ankit Jha, Mainak Singha, Biplab Banerjee, Jocelyn Chanussot
Date: 2024-08-31 00:00:00
Tasks: Domain Adaptation, Multi-target Domain Adaptation, Open-Set Multi-Target Domain Adaptation, Prompt Learning

Multi-Target Domain Adaptation (MTDA) entails learning domain-invariant information from a single source domain and applying it to multiple unlabeled target domains. Yet, existing MTDA methods predominantly focus on addressing domain shifts within visual features, often overlooking semantic features and struggling to handle unknown classes, resulting in what is known as Open-Set (OS) MTDA. While large-scale vision-language foundation models like CLIP show promise, their potential for MTDA remains largely unexplored. This paper introduces COSMo, a novel method that learns domain-agnostic prompts through source domain-guided prompt learning to tackle the MTDA problem in the prompt space. By leveraging a domain-specific bias network and separate prompts for known and unknown classes, COSMo effectively adapts across domain and class shifts. To the best of our knowledge, COSMo is the first method to address Open-Set Multi-Target DA (OSMTDA), offering a more realistic representation of real-world scenarios and addressing the challenges of both open-set and multi-target DA. COSMo demonstrates an average improvement of $5.1\%$ across three challenging datasets: Mini-DomainNet, Office-31, and Office-Home, compared to other related DA methods adapted to operate within the OSMTDA setting. Code is available at: https://github.com/munish30monga/COSMo
Smart Sleep Monitoring System via Passively Sensing Human Vibration Signals
Authors: Fangyu Li, Maria Valero, Jose Clemente, Zion Tse, WenZhan Song
Date: 2021-07-01 00:00:00
Proceeding: IEEE Sensors Journal 2021 7

In this paper, a bed-mounted vibration sensor-based system is proposed to monitor vital parameters during sleep, including heartbeat rate (HR) and respiratory rate (RR), body movements, and sleep postures. Our system enables smart healthcare that monitors daily sleep in a ubiquitous and non-invasive manner. Besides, the system is contact-free, as no external wearable devices and physical contacts are required. Furthermore, the vibration-based approach also avoids the privacy violation caused by the usage of surveillance cameras. To effectively monitor sleep status, a robust stable signal mode decomposition based HR and RR estimation method is developed for the complicated and noisy vibration signals. Besides, algorithms for body movement and sleep posture identification are also proposed based on vibration signal features. A prototype system is demonstrated with system details, showing great potentials in monitoring sleep status in a real-time user-friendly manner. Experimental results of short term and long term experiments with different participants and beds show that our system achieves satisfying accuracy compared with dedicated commercial devices.
Robust Portfolio Optimization using GOPALS: Geospatial Optimization and Portfolio Allocation using Landscape Segmentation
Authors: Jatin Patni, Ritabrata Bhattacharyya
Date: 2024-12-14 00:00:00
Proceeding: SSRN 2024 12
Tasks: Portfolio Optimization

Portfolio Optimization is an important area of research in Financial Investments. The traditional mean-variance framework for portfolio optimization is very sensitive to the estimation errors in the expectations of returns and covariances, and leads to suboptimal port folios. Even after using techniques such as shrinkage methods, robust optimization, and resampling methods, we are unable to address these limitations comprehensively. 

This study introduces GOPALS: Geospatial Optimization and Portfolio Allocation using Landscape Segmentation, a simulation-based portfolio optimization framework designed to overcome these limitations and improve the robustness of the portfolio optimization process. 
GOPALS methodology relies on simulating market scenarios and identifying ”stable regions” in the high-dimensional portfolio weight space (i.e. regions where portfolio allocations consistently perform favorably across varying market conditions) using clustering methods. By restricting the choice of portfolio allocations from such stable regions, the portfolio performance can be shown to be consistently favorable and robust to estimation errors.
GOPALS is applied to a handpicked universe of Indian Equity Mutual Funds, with practical constraints-such as long-only positions and concentration limits-in order to reflect real-world constraints of investing. The robustness of GOPALS is shown via backtests per formed over multiple historical regimes.
GOPALS offers a resilient approach to portfolio construction, hence it has considerable promise, not only in academic research, but also in the real world of financial investments. It offers investors and portfolio managers a new tool for navigating uncertain markets across different asset classes and global investment universes.
Tensor Products of Fell Bundles over Groups
Authors: Fernando Abadie
Date: 1997-12-16 00:00:00

We extend the theory of tensor products of C*-algebras to the larger category of Fell bundles over locally compact groups. We prove that, like in the case of C*-algebras, there exist maximal and minimal tensor products. Given two Fell bundles, we compare the tensor products of their cross-sectional algebras with the cross-sectional algebras of their tensor products. As applications we prove that, under certain conditions, the cross-sectional C*-algebra of a Fell bundle is nuclear or exact whenever so is its fiber over the unit element of the group.
A2RNet: Adversarial Attack Resilient Network for Robust Infrared and Visible Image Fusion
Authors: Jiawei Li, Hongwei Yu, Jiansheng Chen, Xinlong Ding, Jinlong Wang, JinYuan Liu, Bochao Zou, Huimin Ma
Date: 2024-12-13 00:00:00
Tasks: Adversarial Attack, Infrared And Visible Image Fusion

Infrared and visible image fusion (IVIF) is a crucial technique for enhancing visual performance by integrating unique information from different modalities into one fused image. Exiting methods pay more attention to conducting fusion with undisturbed data, while overlooking the impact of deliberate interference on the effectiveness of fusion results. To investigate the robustness of fusion models, in this paper, we propose a novel adversarial attack resilient network, called $\textrm{A}^{\textrm{2}}$RNet. Specifically, we develop an adversarial paradigm with an anti-attack loss function to implement adversarial attacks and training. It is constructed based on the intrinsic nature of IVIF and provide a robust foundation for future research advancements. We adopt a Unet as the pipeline with a transformer-based defensive refinement module (DRM) under this paradigm, which guarantees fused image quality in a robust coarse-to-fine manner. Compared to previous works, our method mitigates the adverse effects of adversarial perturbations, consistently maintaining high-fidelity fusion results. Furthermore, the performance of downstream tasks can also be well maintained under adversarial attacks. Code is available at https://github.com/lok-18/A2RNet.
IQViC: In-context, Question Adaptive Vision Compressor for Long-term Video Understanding LMMs
Authors: Sosuke Yamao, Natsuki Miyahara, Yuki Harazono, Shun Takeuchi
Date: 2024-12-13 00:00:00
Tasks: Question Answering, Video Question Answering, Video Understanding

With the increasing complexity of video data and the need for more efficient long-term temporal understanding, existing long-term video understanding methods often fail to accurately capture and analyze extended video sequences. These methods typically struggle to maintain performance over longer durations and to handle the intricate dependencies within the video content. To address these limitations, we propose a simple yet effective large multi-modal model framework for long-term video understanding that incorporates a novel visual compressor, the In-context, Question Adaptive Visual Compressor (IQViC). The key idea, inspired by humans' selective attention and in-context memory mechanisms, is to introduce a novel visual compressor and incorporate efficient memory management techniques to enhance long-term video question answering. Our framework utilizes IQViC, a transformer-based visual compressor, enabling question-conditioned in-context compression, unlike existing methods that rely on full video visual features. This selectively extracts relevant information, significantly reducing memory token requirements. Through extensive experiments on a new dataset based on InfiniBench for long-term video understanding, and standard benchmarks used for existing methods' evaluation, we demonstrate the effectiveness of our proposed IQViC framework and its superiority over state-of-the-art methods in terms of video understanding accuracy and memory efficiency.
Intervening on psychopathology networks: Evaluating intervention targets through simulations
Authors: Gabriela Lunansky
Date: 2022-08-01 00:00:00
Proceeding: Methods 2022 8

Identifying the different influences of symptoms in dynamic psychopathology models may hold promise for increasing treatment efficacy in clinical applications. Dynamic psychopathology models study the behavioral patterns of symptom networks, where symptoms mutually enforce each other. Interventions could be tailored to specific symptoms that are most effective at lowering symptom activity or that hinder the further development of psychopathology. Simulating interventions in psychopathology network models fits in a novel tradition where symptom-specific perturbations are used as in silico interventions. Here, we present the NodeIdentifyR algorithm (NIRA) to identify the projected most efficient, symptom-specific intervention target in a network model (i.e., the Ising model). We implemented NIRA in a freely available R package. The technique studies the projected effects of symptom-specific interventions by simulating data while symptom parameters (i.e., thresholds) are systematically altered. The projected effect of these interventions is defined in terms of the expected change in overall symptom activity across simulations. With this algorithm, it is possible to study (1) whether symptoms differ in their projected influence on the behavior of the symptom network and, if so, (2) which symptom has the largest projected effect in lowering or increasing overall symptom activation. As an illustration, we apply the algorithm to an empirical dataset containing Post-Traumatic Stress Disorder symptom assessments of participants who experienced the Wenchuan earthquake in 2008. The most important limitations of the method are discussed, as well as recommendations for future research, such as shifting towards modeling individual processes to validate these types of simulation-based intervention methods.
Evaluation Agent: Efficient and Promptable Evaluation Framework for Visual Generative Models
Authors: Fan Zhang, Shulin Tian, Ziqi Huang, Yu Qiao, Ziwei Liu
Date: 2024-12-10 00:00:00
Tasks: Video Generation

Recent advancements in visual generative models have enabled high-quality image and video generation, opening diverse applications. However, evaluating these models often demands sampling hundreds or thousands of images or videos, making the process computationally expensive, especially for diffusion-based models with inherently slow sampling. Moreover, existing evaluation methods rely on rigid pipelines that overlook specific user needs and provide numerical results without clear explanations. In contrast, humans can quickly form impressions of a model's capabilities by observing only a few samples. To mimic this, we propose the Evaluation Agent framework, which employs human-like strategies for efficient, dynamic, multi-round evaluations using only a few samples per round, while offering detailed, user-tailored analyses. It offers four key advantages: 1) efficiency, 2) promptable evaluation tailored to diverse user needs, 3) explainability beyond single numerical scores, and 4) scalability across various models and tools. Experiments show that Evaluation Agent reduces evaluation time to 10% of traditional methods while delivering comparable results. The Evaluation Agent framework is fully open-sourced to advance research in visual generative models and their efficient evaluation.
GaussianWorld: Gaussian World Model for Streaming 3D Occupancy Prediction
Authors: Sicheng Zuo, Wenzhao Zheng, Yuanhui Huang, Jie zhou, Jiwen Lu
Date: 2024-12-13 00:00:00
Proceeding: CVPR 2025 1
Tasks: Autonomous Driving, Prediction

3D occupancy prediction is important for autonomous driving due to its comprehensive perception of the surroundings. To incorporate sequential inputs, most existing methods fuse representations from previous frames to infer the current 3D occupancy. However, they fail to consider the continuity of driving scenarios and ignore the strong prior provided by the evolution of 3D scenes (e.g., only dynamic objects move). In this paper, we propose a world-model-based framework to exploit the scene evolution for perception. We reformulate 3D occupancy prediction as a 4D occupancy forecasting problem conditioned on the current sensor input. We decompose the scene evolution into three factors: 1) ego motion alignment of static scenes; 2) local movements of dynamic objects; and 3) completion of newly-observed scenes. We then employ a Gaussian world model (GaussianWorld) to explicitly exploit these priors and infer the scene evolution in the 3D Gaussian space considering the current RGB observation. We evaluate the effectiveness of our framework on the widely used nuScenes dataset. Our GaussianWorld improves the performance of the single-frame counterpart by over 2% in mIoU without introducing additional computations. Code: https://github.com/zuosc19/GaussianWorld.
GaussianAD: Gaussian-Centric End-to-End Autonomous Driving
Authors: Wenzhao Zheng, Junjie Wu, Yao Zheng, Sicheng Zuo, Zixun Xie, Longchao Yang, Yong Pan, Zhihui Hao, Peng Jia, Xianpeng Lang, Shanghang Zhang
Date: 2024-12-13 00:00:00
Tasks: Autonomous Driving, Decision Making, Motion Planning

Vision-based autonomous driving shows great potential due to its satisfactory performance and low costs. Most existing methods adopt dense representations (e.g., bird's eye view) or sparse representations (e.g., instance boxes) for decision-making, which suffer from the trade-off between comprehensiveness and efficiency. This paper explores a Gaussian-centric end-to-end autonomous driving (GaussianAD) framework and exploits 3D semantic Gaussians to extensively yet sparsely describe the scene. We initialize the scene with uniform 3D Gaussians and use surrounding-view images to progressively refine them to obtain the 3D Gaussian scene representation. We then use sparse convolutions to efficiently perform 3D perception (e.g., 3D detection, semantic map construction). We predict 3D flows for the Gaussians with dynamic semantics and plan the ego trajectory accordingly with an objective of future scene forecasting. Our GaussianAD can be trained in an end-to-end manner with optional perception labels when available. Extensive experiments on the widely used nuScenes dataset verify the effectiveness of our end-to-end GaussianAD on various tasks including motion planning, 3D occupancy prediction, and 4D occupancy forecasting. Code: https://github.com/wzzheng/GaussianAD.
Apollo: An Exploration of Video Understanding in Large Multimodal Models
Authors: Orr Zohar, Xiaohan Wang, Yann Dubois, Nikhil Mehta, Tong Xiao, Philippe Hansen-Estruch, Licheng Yu, Xiaofang Wang, Felix Juefei-Xu, Ning Zhang, Serena Yeung-Levy, Xide Xia
Date: 2024-12-13 00:00:00
Proceeding: CVPR 2025 1
Tasks: MME, Video MME, Video Understanding

Despite the rapid integration of video perception capabilities into Large Multimodal Models (LMMs), the underlying mechanisms driving their video understanding remain poorly understood. Consequently, many design decisions in this domain are made without proper justification or analysis. The high computational cost of training and evaluating such models, coupled with limited open research, hinders the development of video-LMMs. To address this, we present a comprehensive study that helps uncover what effectively drives video understanding in LMMs. We begin by critically examining the primary contributors to the high computational requirements associated with video-LMM research and discover Scaling Consistency, wherein design and training decisions made on smaller models and datasets (up to a critical size) effectively transfer to larger models. Leveraging these insights, we explored many video-specific aspects of video-LMMs, including video sampling, architectures, data composition, training schedules, and more. For example, we demonstrated that fps sampling during training is vastly preferable to uniform frame sampling and which vision encoders are the best for video representation. Guided by these findings, we introduce Apollo, a state-of-the-art family of LMMs that achieve superior performance across different model sizes. Our models can perceive hour-long videos efficiently, with Apollo-3B outperforming most existing $7$B models with an impressive 55.1 on LongVideoBench. Apollo-7B is state-of-the-art compared to 7B LMMs with a 70.9 on MLVU, and 63.3 on Video-MME.
A Library for Learning Neural Operators
Authors: Jean Kossaifi, Nikola Kovachki, Zongyi Li, David Pitt, Miguel Liu-Schiaffini, Robert Joseph George, Boris Bonev, Kamyar Azizzadenesheli, Julius Berner, Anima Anandkumar
Date: 2024-12-13 00:00:00
Tasks: Operator learning

We present NeuralOperator, an open-source Python library for operator learning. Neural operators generalize neural networks to maps between function spaces instead of finite-dimensional Euclidean spaces. They can be trained and inferenced on input and output functions given at various discretizations, satisfying a discretization convergence properties. Built on top of PyTorch, NeuralOperator provides all the tools for training and deploying neural operator models, as well as developing new ones, in a high-quality, tested, open-source package. It combines cutting-edge models and customizability with a gentle learning curve and simple user interface for newcomers.
A dual contrastive framework
Authors: Yuan Sun, Zhao Zhang, Jorge Ortiz
Date: 2024-12-13 00:00:00
Tasks: Contrastive Learning, Decoder, object-detection, Object Detection, Spatial Reasoning

In current multimodal tasks, models typically freeze the encoder and decoder while adapting intermediate layers to task-specific goals, such as region captioning. Region-level visual understanding presents significant challenges for large-scale vision-language models. While limited spatial awareness is a known issue, coarse-grained pretraining, in particular, exacerbates the difficulty of optimizing latent representations for effective encoder-decoder alignment. We propose AlignCap, a framework designed to enhance region-level understanding through fine-grained alignment of latent spaces. Our approach introduces a novel latent feature refinement module that enhances conditioned latent space representations to improve region-level captioning performance. We also propose an innovative alignment strategy, the semantic space alignment module, which boosts the quality of multimodal representations. Additionally, we incorporate contrastive learning in a novel manner within both modules to further enhance region-level captioning performance. To address spatial limitations, we employ a General Object Detection (GOD) method as a data preprocessing pipeline that enhances spatial reasoning at the regional level. Extensive experiments demonstrate that our approach significantly improves region-level captioning performance across various tasks
COMET: Benchmark for Comprehensive Biological Multi-omics Evaluation Tasks and Language Models
Authors: Yuchen Ren, Wenwei Han, Qianyuan Zhang, Yining Tang, Weiqiang Bai, Yuchen Cai, Lifeng Qiao, Hao Jiang, Dong Yuan, Tao Chen, Siqi Sun, Pan Tan, Wanli Ouyang, Nanqing Dong, Xinzhu Ma, Peng Ye
Date: 2024-12-13 00:00:00

As key elements within the central dogma, DNA, RNA, and proteins play crucial roles in maintaining life by guaranteeing accurate genetic expression and implementation. Although research on these molecules has profoundly impacted fields like medicine, agriculture, and industry, the diversity of machine learning approaches-from traditional statistical methods to deep learning models and large language models-poses challenges for researchers in choosing the most suitable models for specific tasks, especially for cross-omics and multi-omics tasks due to the lack of comprehensive benchmarks. To address this, we introduce the first comprehensive multi-omics benchmark COMET (Benchmark for Biological COmprehensive Multi-omics Evaluation Tasks and Language Models), designed to evaluate models across single-omics, cross-omics, and multi-omics tasks. First, we curate and develop a diverse collection of downstream tasks and datasets covering key structural and functional aspects in DNA, RNA, and proteins, including tasks that span multiple omics levels. Then, we evaluate existing foundational language models for DNA, RNA, and proteins, as well as the newly proposed multi-omics method, offering valuable insights into their performance in integrating and analyzing data from different biological modalities. This benchmark aims to define critical issues in multi-omics research and guide future directions, ultimately promoting advancements in understanding biological processes through integrated and different omics data analysis.
TraceVLA: Visual Trace Prompting Enhances Spatial-Temporal Awareness for Generalist Robotic Policies
Authors: Ruijie Zheng, Yongyuan Liang, Shuaiyi Huang, Jianfeng Gao, Hal Daumé III, Andrey Kolobov, Furong Huang, Jianwei Yang
Date: 2024-12-13 00:00:00
Tasks: Robot Manipulation, Vision-Language-Action

Although large vision-language-action (VLA) models pretrained on extensive robot datasets offer promising generalist policies for robotic learning, they still struggle with spatial-temporal dynamics in interactive robotics, making them less effective in handling complex tasks, such as manipulation. In this work, we introduce visual trace prompting, a simple yet effective approach to facilitate VLA models' spatial-temporal awareness for action prediction by encoding state-action trajectories visually. We develop a new TraceVLA model by finetuning OpenVLA on our own collected dataset of 150K robot manipulation trajectories using visual trace prompting. Evaluations of TraceVLA across 137 configurations in SimplerEnv and 4 tasks on a physical WidowX robot demonstrate state-of-the-art performance, outperforming OpenVLA by 10% on SimplerEnv and 3.5x on real-robot tasks and exhibiting robust generalization across diverse embodiments and scenarios. To further validate the effectiveness and generality of our method, we present a compact VLA model based on 4B Phi-3-Vision, pretrained on the Open-X-Embodiment and finetuned on our dataset, rivals the 7B OpenVLA baseline while significantly improving inference efficiency.