pwc-archive/papers-with-abstracts huggingface dataset:
Feature Aggregating Network with Inter-Frame Interaction for Efficient Video Super-Resolution
Authors: Yawei Li; Zhao Zhang; Suiyi Zhao; Jicong Fan; Haijun Zhang; Mingliang Xu
Date: 2023-12-01 00:00:00
Proceeding: Conference 2023 12
Tasks: Computational Efficiency, Image Super-Resolution, Super-Resolution, Video Super-Resolution

Video super-resolution (VSR) on mobile devices aims to restore high-resolution frames from their low-resolution counterparts, satisfying the requirements of performance, FLOPs and latency. On one hand, partial feature processing, as a classic and acknowledged strategy, is developed in current studies to reach an appropriate trade-off between FLOPs and accuracy. However, the splitting of partial feature processing strategy are usually performed in a blind manner, thereby reducing the computational efficiency and performance gains. On the other hand, current methods for mobile platforms primarily treat VSR as an extension of single-image super-resolution to reduce model calculation and inference latency. However, lacking inter-frame information interaction in current methods results in a suboptimal latency and accuracy trade-off. To this end, we propose a novel architecture, termed Feature Aggregating Network with Inter-frame Interaction (FANI), a lightweight yet considering frame-wise correlation VSR network, which could achieve real-time inference while maintaining superior performance. Our FANI accepts adjacent multi-frame low-resolution images as input and generally consists of several fully-connection-embedded modules, i.e., Multi-stage Partial Feature Distillation (MPFD) for capturing multi-level feature representations. Moreover, considering the importance of inter-frame alignment, we further employ a tiny Attention-based Frame Alignment (AFA) module to promote inter-frame information flow and aggregation efficiently. Extensive experiments on the well-known dataset and real-world mobile device demonstrate the superiority of our proposed FANI, which means that our FANI could be well adapted to mobile devices and produce visually pleasing results.
CropCap: Embedding Visual Cross-Partition Dependency for Image Captioning
Authors: Bo wang, Zhao Zhang, Suiyi Zhao, Haijun Zhang, Richang Hong, and Meng WangAuthors Info & Claims
Date: 2023-10-27 00:00:00
Proceeding: journal 2023 10
Tasks: Image Captioning

Transformer-based approaches to image captioning have shown great success by utilizing long-term dependency for visual embedding. However, their coarse long-term dependency, using the multi-head self-attention mechanism to capture the contextual interactions between the visual tokens on the time step and (or) embedded dimension, fail to distinguish fine-grained features of local partition. In this case, some similar features are captured, which leads to feature redundancy that decreases the performance. To respond to this issue, this paper proposes a novel image captioner embedding visual cross-partition dependency, dubbed CropCap. Specifically, the visual sequence generated from the Swin Transformer-based pre-embedding network is fed into the proposed cross-partition dependency module to refinedly model the interaction between partial representations on both the time step and embedded dimension. Furthermore, we formulaically reason the proposed cross-partition dependency, and theoretically prove its correctness. Extensive comparisons on the benchmark MS-COCO dataset demonstrated the effectiveness addressing the information redundancy issue, and verified the superior performance of our method.
Stabilizing multispectral pedestrian detection with evidential hybrid fusion
Authors: Q Li, C Zhang, Q Hu, P Zhu, H Fu, L Chen
Date: 2023-08-21 00:00:00
Proceeding: journal 2023 8
Tasks: Pedestrian Detection

Multispectral pedestrian detection is an important task due to its critical role in a wide spectrum of applications. Basically, the complementary information from color and thermal images could provide a more accurate and reliable pedestrian detection result. However, multimodal data usually suffer from the issue of dynamic change or corruption for some modalities. At the same time, as a safety-critical task, how to produce a stable and reliable detection result is also a key challenge. To address these challenges, we propose a stable multispectral pedestrian detection (SMPD) algorithm, providing a new paradigm for multispectral detection by dynamically integrating different modalities at an evidence level. Specifically, we introduce the Dirichlet distribution to characterize the distribution of the class probabilities, parameterized with evidence from different modalities. Then, multi-branch fusion, based on Dempster-Shafer theory, can integrate these pieces of evidence to obtain the detection result. In addition, a Plug-and-Play module, termed modal enhancement module, is introduced to enhance cross-modality interaction. This is an end-to-end framework, which can induce accurate detection and uncertainty estimation, and then endows the model with both reliability and robustness against noise or corruption. Extensive experimental results demonstrate the efficiency of our algorithm compared with state-of-the-art methods.
Jump Self-attention: Capturing High-order Statistics in Transformers
Authors: Haoyi Zhou, Siyang Xiao, Shanghang Zhang, Jieqi Peng, Shuai Zhang, JianXin Li
Date: 2024-04-03 00:00:00
Proceeding: journal 2024 4
Tasks: Image Captioning, Natural Language Understanding

The recent success of Transformer has benefited many real-world applications, with its capability of building long dependency through pairwise dot-products. However, the strong assumption that elements are directly attentive to each other limits the performance of tasks with high-order dependencies such as natural language understanding and Image captioning. To solve such problems, we are the first to define the Jump Self-attention (JAT) to build Transformers. Inspired by the pieces moving of English Draughts, we introduce the spectral convolutional technique to calculate JAT on the dot-product feature map. This technique allows JAT's propagation in each self-attention head and is interchangeable with the canonical self-attention. We further develop the higher-order variants under the multi-hop assumption to increase the generality. Moreover, the proposed architecture is compatible with the pre-trained models. With extensive experiments, we empirically show that our methods significantly increase the performance on ten different tasks.
A temporal enhanced semi-supervised segmentation network for needle detection in 3D ultrasound images
Authors: Mingwei Wen1, Pavel Shcherbakov2, Yang Xu1, 3, Jing Li3, Yi Hu3, Quan Zhou1, Huageng Liang4, Li Yuan5 and Xuming Zhang1
Date: 2024-05-21 00:00:00
Proceeding: journal 2024 5
Tasks: Image Segmentation, Medical Image Segmentation, Position, Segmentation, Semantic Segmentation

Objective. Automated biopsy needle segmentation in 3D ultrasound images can be used for biopsy navigation, but it is quite challenging due to the low ultrasound image resolution and interference similar to the needle appearance. For 3D medical image segmentation, such deep learning networks as convolutional neural network and transformer have been investigated. However, these segmentation methods require numerous labeled data for training, have difficulty in meeting the real-time segmentation requirement and involve high memory consumption. Approach. In this paper, we have proposed the temporal information-based semi-supervised training framework for fast and accurate needle segmentation. Firstly, a novel circle transformer module based on the static and dynamic features has been designed after the encoders for extracting and fusing the temporal information. Then, the consistency constraints of the outputs before and after combining temporal information are proposed to provide the semi-supervision for the unlabeled volume. Finally, the model is trained using the loss function which combines the cross-entropy and Dice similarity coefficient (DSC) based segmentation loss with mean square error based consistency loss. The trained model with the single ultrasound volume input is applied to realize the needle segmentation in ultrasound volume. Main results. Experimental results on three needle ultrasound datasets acquired during the beagle biopsy show that our approach is superior to the most competitive mainstream temporal segmentation model and semi-supervised method by providing higher DSC (77.1% versus 76.5%), smaller needle tip position (1.28 mm versus 1.87 mm) and length (1.78 mm versus 2.19 mm) errors on the kidney dataset as well as DSC (78.5% versus 76.9%), needle tip position (0.86 mm versus 1.12 mm) and length (1.01 mm versus 1.26 mm) errors on the prostate dataset. Significance. The proposed method can significantly enhance needle segmentation accuracy by training with sequential images at no additional cost. This enhancement may further improve the effectiveness of biopsy navigation systems.
Expanding the Prediction Capacity in Long Sequence Time-Series Forecasting
Authors: H Zhou, J Li, S Zhang, M Yan, H Xiong
Date: 2023-05-01 00:00:00
Proceeding: journal 2023 5
Tasks: Decoder, Time Series, Time Series Forecasting

Many real-world applications show growing demand for the prediction of long sequence time-series, such as electricity consumption planning. Long sequence time-series forecasting (LSTF) requires a higher prediction capacity of the model, which is the ability to capture precise long-range dependency coupling between output and input efficiently. Recent studies have shown the potential of Transformer to accommodate the capacity requirements. However, three real challenges that may have prevented expanding the prediction capacity in LSTF are that the Transformer is limited by quadratic time complexity, high memory usage, and slow inference speed under the encoder-decoder architecture. To address these issues, we design an efficient transformer-based model for LSTF, named Informer, with three distinctive characteristics.(i) a ProbSparse self-attention mechanism, which achieves O (L log⁡ L) in time complexity and memory usage, and has comparable performance on sequences' dependency alignment.(ii) the self-attention distilling promotes dominating attention by convolutional operators. Besides, the halving of layer width is intended to reduce the expense of building a deeper network on extremely long input sequences.(iii) the generative style decoder, while conceptually simple, predicts the long time-series sequences at one forward operation rather than a step-by-step way, which drastically improves the inference speed of long-sequence predictions. Extensive experiments on ten large-scale datasets demonstrate that Informer significantly outperforms existing methods and provides a new solution to the LSTF problem.
F-DARTS: Foveated Differentiable Architecture Search Based Multimodal Medical Image Fusion
Authors: Shaozhuang Ye; Tuo Wang; Mingyue Ding; Xuming Zhang
Date: 2023-06-07 00:00:00
Proceeding: journal 2023 6
Tasks: Foveation

Multimodal medical image fusion (MMIF) is highly significant in such fields as disease diagnosis and treatment. The traditional MMIF methods are difficult to provide satisfactory fusion accuracy and robustness due to the influence of such possible human-crafted components as image transform and fusion strategies. Existing deep learning based fusion methods are generally difficult to ensure image fusion effect due to the adoption of a human-designed network structure and a relatively simple loss function and the ignorance of human visual characteristics during weight learning. To address these issues, we have presented the foveated differentiable architecture search (F-DARTS) based unsupervised MMIF method. In this method, the foveation operator is introduced into the weight learning process to fully explore human visual characteristics for the effective image fusion. Meanwhile, a distinctive unsupervised loss function is designed for network training by integrating mutual information, sum of the correlations of differences, structural similarity and edge preservation value. Based on the presented foveation operator and loss function, an end-to-end encoder-decoder network architecture will be searched using the F-DARTS to produce the fused image. Experimental results on three multimodal medical image datasets demonstrate that the F-DARTS performs better than several traditional and deep learning based fusion methods by providing visually superior fused results and better objective evaluation metrics.
Financial Risk Spillover Effects Among China's Crude Oil Futures Market, Stock Market, and Bond Market During the COVID-19 Pandemic
Authors: Fengrou Sheng, Rulang Wang, Zhu Sun, Zonghu Liao
Date: 2024-03-02 00:00:00
Proceeding: Conference 2024 3

This paper unravels the dynamics of financial risk transmission across China’s crude oil futures, stock, and bond markets, spotlighting the period during and subsequent to the COVID-19 pandemic. Leveraging the generalized vector autoregressive (VAR) framework by Diebold & Yilmaz (2012), we analyze systemic risk spillovers, and investigate the static and evolving interconnections among these pivotal financial sectors from the initiation of crude oil futures trading in 2018 up to 2023. The core findings include: (1) Systemic Risk Contagion: There exists a pronounced systemic risk contagion among the markets studied, with the bond market being the predominant net recipient of risk spillovers, highlighting its vulnerability. In contrast, the stock market emerges as the main conduit of risk transmission, serving as the largest net sender of systemic risk; (2) Impact of the COVID-19 Pandemic: The pandemic significantly exacerbated the Total Spillover Index (TSI) across all markets, with a remarkable surge of approximately 8% to 26%. This amplification underscores the profound impact of global crises on enhancing risk contagion effects among financial markets; (3) Role of the Crude Oil Futures Market: The integration of the crude oil futures market into China’s financial ecosystem has not only augmented its influence but also introduced significant cross-market risks. This market’s growing impact highlights its essential role in the broader financial market dynamics and systemic risk management.
Towards Long-delayed Sparsity: Learning a Better Transformer through Reward Redistribution
Authors: Tianchen Zhu, Yue Qiu, Haoyi Zhou, JianXin Li
Date: 2023-02-01 00:00:00
Proceeding: journal 2023 2

Recently, Decision Transformer (DT) pioneered theoffine RL into a contextual conditional sequencemodeling paradigm, which leverages self-attendedautoregression to learn from global target rewards,states, and actions. However, many applicationshave a severe delay of the above signals, such as theagent can only obtain a reward signal at the end of each trajectory. This delay causes an unwanted biascumulating in autoregressive learning global signals. In this paper, we focused its virtual example on episodic reinforcement learning with trajectory feedback. We propose a new reward redistribution
algorithm for learning parameterized reward functions, and it decomposes the long-delayed reward onto each timestep. To improve the redistributing’s adaptation ability, we formulate the previous decomposition as a bi-level optimization problem for global optimal. We extensively evaluate the proposed method on various benchmarks and demonstrate an overwhelming performance improvement under long-delayed settings.
Load-aware Dynamic Partition of Deep Neutral Networks for Multi-User Offloading
Authors: Hongzhou Liu; Wenli Zheng; Li Li; Minyi Guo
Date: 2022-10-13 00:00:00
Proceeding: Conference 2022 10
Tasks: Edge-computing

The emerging edge computing technique provides support for the computation tasks that are delay-sensitive and compute-intensive, such as deep neural network inference, by offloading them from a user-end device to an edge server for fast execution. The increasing offloaded tasks on an edge server are gradually facing the contention of both the network and computation resources. The existing offloading approaches often partition the deep neural network at a place where the amount of data transmission is small to save network resource, but rarely consider the problem caused by computation resource shortage on the edge server. In this paper, we design LoADPart, a deep neural network offloading system. LoADPart can dynamically and jointly analyze both the available network bandwidth and the computation load of the edge server, and make proper decisions of deep neural network partition with a light-weighted algorithm, to minimize the end-to-end inference latency. We implement LoADPart for MindSpore, a deep learning framework supporting edge AI, and compare it with state-of-the-art solutions in the experiments on 6 deep neural networks. The results show that under the variation of server computation load, LoADPart can reduce the end-to-end latency by 14.2% on average and up to 32.3% in some specific cases.
Few-shot time-series anomaly detection with unsupervised domain adaptation
Authors: Hongbo Li, Wenli Zheng, Feilong Tang, Yanmin Zhu, and Jielong Huang
Date: 2023-11-01 00:00:00
Proceeding: journal 2023 11
Tasks: Anomaly Detection, Domain Adaptation, Few-Shot Learning, Time Series, Time Series Anomaly Detection, Time Series Classification, Triplet, Unsupervised Domain Adaptation

Anomaly detection for time-series data is crucial in the management of systems for streaming applications, computational services, and cloud platforms. The majority of current few-shot learning (FSL) approaches are supposed to discover the remarkably low fraction of anomaly samples in a large number of time-series samples. Furthermore, due to the tremendous effort required to label data, most time-series datasets lack data labels, necessitating unsupervised domain adaptation (UDA) methods. Therefore, time-series anomaly detection is a problem that combines the aforementioned two difficulties, termed FS-UDA. To solve the problem, we propose a Few-Shot time-series Anomaly Detection framework with unsupervised domAin adaPTation (FS-ADAPT), which consists of two modules: a dueling triplet network to address the constraints of unsupervised target information, and an incremental adaptation module for addressing the limitations of few anomaly samples in an online scenario. The dueling triplet network is adversarially trained with augmented data and unlabeled target samples to learn a classifier. The incremental adaptation module fully exploits both the critical anomaly samples and the freshest normal samples to keep the classifier up to date. Extensive experiments on five real-world time-series datasets are conducted to assess FS-ADAPT, which outperforms the state-of-the-art FSL and UDA based time-series classification models, as well as their naive combinations.
Towards Generic Image Manipulation Detection with Weakly-Supervised Self-Consistency Learning
Authors: Yuanhao Zhai, Tianyu Luan, David Doermann, Junsong Yuan
Date: 2023-09-03 00:00:00
Proceeding: ICCV 2023 1
Tasks: Image Manipulation, Image Manipulation Detection, Pseudo Label

As advanced image manipulation techniques emerge, detecting the manipulation becomes increasingly important. Despite the success of recent learning-based approaches for image manipulation detection, they typically require expensive pixel-level annotations to train, while exhibiting degraded performance when testing on images that are differently manipulated compared with training images. To address these limitations, we propose weakly-supervised image manipulation detection, such that only binary image-level labels (authentic or tampered with) are required for training purpose. Such a weakly-supervised setting can leverage more training images and has the potential to adapt quickly to new manipulation techniques. To improve the generalization ability, we propose weakly-supervised self-consistency learning (WSCL) to leverage the weakly annotated images. Specifically, two consistency properties are learned: multi-source consistency (MSC) and inter-patch consistency (IPC). MSC exploits different content-agnostic information and enables cross-source learning via an online pseudo label generation and refinement process. IPC performs global pair-wise patch-patch relationship reasoning to discover a complete region of manipulation. Extensive experiments validate that our WSCL, even though is weakly supervised, exhibits competitive performance compared with fully-supervised counterpart under both in-distribution and out-of-distribution evaluations, as well as reasonable manipulation localization ability.
United We Stand: Decentralized Multi-Agent Planning With Attrition
Authors: Nhat Nguyen, Duong Nguyen, Gianluca Rizzo, Hung Nguyen
Date: 2024-07-11 00:00:00

Decentralized planning is a key element of cooperative multi-agent systems for information gathering tasks. However, despite the high frequency of agent failures in realistic large deployment scenarios, current approaches perform poorly in the presence of failures, by not converging at all, and/or by making very inefficient use of resources (e.g. energy). In this work, we propose Attritable MCTS (A-MCTS), a decentralized MCTS algorithm capable of timely and efficient adaptation to changes in the set of active agents. It is based on the use of a global reward function for the estimation of each agent's local contribution, and regret matching for coordination. We evaluate its effectiveness in realistic data-harvesting problems under different scenarios. We show both theoretically and experimentally that A-MCTS enables efficient adaptation even under high failure rates. Results suggest that, in the presence of frequent failures, our solution improves substantially over the best existing approaches in terms of global utility and scalability.
Let Network Decide What to Learn: Symbolic Music Understanding Model Based on Large-scale Adversarial Pre-training
Authors: Zijian Zhao
Date: 2024-07-11 00:00:00
Tasks: Information Retrieval, Music Information Retrieval

As a crucial aspect of Music Information Retrieval (MIR), Symbolic Music Understanding (SMU) has garnered significant attention for its potential to assist both musicians and enthusiasts in learning and creating music. Recently, pre-trained language models have been widely adopted in SMU due to the substantial similarities between symbolic music and natural language, as well as the ability of these models to leverage limited music data effectively. However, some studies have shown the common pre-trained methods like Mask Language Model (MLM) may introduce bias issues like racism discrimination in Natural Language Process (NLP) and affects the performance of downstream tasks, which also happens in SMU. This bias often arises when masked tokens cannot be inferred from their context, forcing the model to overfit the training set instead of generalizing. To address this challenge, we propose Adversarial-MidiBERT for SMU, which adaptively determines what to mask during MLM via a masker network, rather than employing random masking. By avoiding the masking of tokens that are difficult to infer from context, our model is better equipped to capture contextual structures and relationships, rather than merely conforming to the training data distribution. We evaluate our method across four SMU tasks, and our approach demonstrates excellent performance in all cases. The code for our model is publicly available at https://github.com/RS2002/Adversarial-MidiBERT .