pwc-archive/papers-with-abstracts huggingface dataset:
Learning Better Registration to Learn Better Few-Shot Medical Image Segmentation: Authenticity, Diversity, and Robustness
Authors: Yuting He; Rongjun Ge; Xiaoming Qi; Yang Chen; Jiasong Wu; Jean-Louis Coatrieux; Guanyu Yang;Shuo Li
Date: 2022-07-22 00:00:00
Proceeding: IEEE Transactions on Neural Networks and Learning Systems 2022 7
Tasks: Diversity, Image Segmentation, Medical Image Segmentation, Segmentation, Semantic Segmentation

In this work, we address the task of few-shot medical image segmentation (MIS) with a novel proposed framework based on the learning registration to learn segmentation (LRLS) paradigm. To cope with the limitations of lack of authenticity, diversity, and robustness in the existing LRLS frameworks, we propose the better registration better segmentation (BRBS) framework with three main contributions that are experimentally shown to have substantial practical merit. First, we improve the authenticity in the registration-based generation program and propose the knowledge consistency constraint strategy that constrains the registration network to learn according to the domain knowledge. It brings the semantic-aligned and topology-preserved registration, thus allowing the generation program to output new data with great space and style authenticity. Second, we deeply studied the diversity of the generation process and propose the space-style sampling program, which introduces the modeling of the transformation path of style and space change between few atlases and numerous unlabeled images into the generation program. Therefore, the sampling on the transformation paths provides much more diverse space and style features to the generated data effectively improving the diversity. Third, we first highlight the robustness in the learning of segmentation in the LRLS paradigm and propose the mix misalignment regularization, which simulates the misalignment distortion and constrains the network to reduce the fitting degree of misaligned regions. Therefore, it builds regularization for these regions improving the robustness of segmentation learning. Without any bells and whistles, our approach achieves a new state-of-the-art performance in few-shot MIS on two challenging tasks that outperform the existing LRLS-based few-shot methods. We believe that this novel and effective framework will provide a powerful few-shot benchmark for the field of medical image and efficiently reduce the costs of medical image research. All of our code will be made publicly available online.
Contrastive Re-localization and History Distillation in Federated CMR Segmentation
Authors: Xiaoming Qi, Guanyu Yang, Yuting He, Wangyan Liu, Ali Islam, Shuo Li
Date: 2022-09-16 00:00:00
Proceeding: MICCAI 2022 2022 9
Tasks: Federated Learning, Image Segmentation, Medical Image Segmentation, Segmentation, Semantic Segmentation

Federated learning (FL) has shown value in multi-center multi-sequence cardiac magnetic resonance (CMR) segmentation, due to imbalanced CMR distributions and privacy preservation in clinical practice. However, the larger heterogeneity among multi-center multi-sequence CMR brings challenges to the FL framework: (1) Representation bias in the model fusion. The FL server model, which is generated by an average fusion of heterogeneous client models, is biased to representation close to the mean distribution and away from the long-distance distribution. Hence, the FL has poor representation ability. (2) Optimization stop in the model replacing. The heterogeneous server model replacing client model in FL directly causes the long-distance clients to utilize worse optimization to replace the original optimization. The client has to recover the optimization with the worse initialization, hence it lacks the continuous optimization ability. In this work, a cross-center cross-sequence medical image segmentation FL framework (FedCRLD) is proposed for the first time to facilitate multi-center multi-sequence CMR segmentation. (1) The contrastive re-localization module (CRL) of FedCRLD enables the correct representation from the heterogeneous model by embedding a novel contrastive difference metric of mutual information into a cross-attention localization transformer to transfer client-correlated knowledge from server model without bias. (2) The momentum distillation strategy (MD) of FedCRLD enables continuous optimization by conducting self-training on a dynamically updated client momentum bank to refine optimization by local correct optimization history. FedCRLD is validated on 420 CMR images 6 clients from 2 public datasets scanned by different hospitals, devices and contrast agents. Our FedCRLD achieves superior performance on multi-center multi-sequence CMR segmentation (average dice 85.96%).
A Unified Framework against Topology and Class Imbalance
Authors: Junyu Chen, Qianqian Xu, Zhiyong Yang, Xiaochun Cao, Qingming Huang
Date: 2022-10-01 00:00:00
Proceeding: Proceedings of the 30th ACM International Conference on Multimedia 2022 10
Tasks: Graph Learning

The Area Under ROC curve (AUC) is widely used as an evaluation metric in various applications. Due to its insensitivity towards class distribution, directly optimizing AUC performs well on the class imbalance problem. However, existing AUC optimization methods are limited to regular data such as text, images, and video. AUC optimization on graph data, which is ubiquitous and important, is seldom studied. Different from regular data, AUC optimization on graphs suffers from not only the class imbalance but also topology imbalance. To solve the complicated imbalance problem, we propose a unified topology-aware AUC optimization (TOPOAUC) framework, which could simultaneously deal with the topology and class imbalance problem in graph learning. We develop a multi-class AUC optimization work to deal with the class imbalance problem. With respect to topology imbalance, we propose a T opology-A ware I mportance L earning mechanism (TAIL), which considers the topology of pairwise nodes and different contributions of topology information to pairwise node neighbors. Extensive experiments on three real-world datasets demonstrate the effectiveness of our proposed method.
Multi-object Tracking: Decoupling Features to Solve the Contradictory Dilemma of Feature Requirements
Authors: Yan Jin; Fang Gao; Jun Yu; Jiabao Wang; Feng Shuang
Date: 2023-02-27 00:00:00
Proceeding: IEEE Transactions on Circuits and Systems for Video Technology 2023 2
Tasks: Multi-Object Tracking, Object Tracking

Multi-object tracking achieves the acquisition of target location information and identity information through two subtasks, detection and re-identification (ReID). The existing commonly used one-shot framework has speed advantages, but the two subtasks have different feature requirements, which leads to competitive learning in the training and thus weakens the feature quality. We propose a feature decoupling based multi-object tracking framework FDTrack for contradictory feature requirements. Through the mutual inhibition of the two subtasks, the features of the backbone network are decoupled. Then the decoupled features are self-constrained to enhance effective features. Considering the instability of the target state and the different confidence of the detections, a more reasonable association strategy is employed to maximize the matchings between detections, thus recovering low-confidence targets. FDTrack is extensively tested on the MOT17 and MOT20 benchmarks. The experimental results show that FDTrack surpasses the previous state-of-the-art (SOTA) methods and has good anti-interference and real-time performance. Moreover, our proposed modules have good portability and can be applied in other one-shot trackers to achieve performance improvement.
Adaptive Multi-scale Online Likelihood Network for AI-assisted Interactive Segmentation
Authors: Muhammad Asad, Helena Williams, Indrajeet Mandal, Sarim Ather, Jan Deprest, Jan D'hooge, Tom Vercauteren
Date: 2023-03-23 00:00:00
Tasks: Interactive Segmentation, Segmentation

Existing interactive segmentation methods leverage automatic segmentation and user interactions for label refinement, significantly reducing the annotation workload compared to manual annotation. However, these methods lack quick adaptability to ambiguous and noisy data, which is a challenge in CT volumes containing lung lesions from COVID-19 patients. In this work, we propose an adaptive multi-scale online likelihood network (MONet) that adaptively learns in a data-efficient online setting from both an initial automatic segmentation and user interactions providing corrections. We achieve adaptive learning by proposing an adaptive loss that extends the influence of user-provided interaction to neighboring regions with similar features. In addition, we propose a data-efficient probability-guided pruning method that discards uncertain and redundant labels in the initial segmentation to enable efficient online training and inference. Our proposed method was evaluated by an expert in a blinded comparative study on COVID-19 lung lesion annotation task in CT. Our approach achieved 5.86% higher Dice score with 24.67% less perceived NASA-TLX workload score than the state-of-the-art. Source code is available at: https://github.com/masadcv/MONet-MONAILabel
Feature Separation and Recalibration for Adversarial Robustness
Authors: Woo Jae Kim, Yoonki Cho, Junsik Jung, Sung-Eui Yoon
Date: 2023-03-24 00:00:00
Proceeding: CVPR 2023 1
Tasks: Adversarial Attack, Adversarial Robustness

Deep neural networks are susceptible to adversarial attacks due to the accumulation of perturbations in the feature level, and numerous works have boosted model robustness by deactivating the non-robust feature activations that cause model mispredictions. However, we claim that these malicious activations still contain discriminative cues and that with recalibration, they can capture additional useful information for correct model predictions. To this end, we propose a novel, easy-to-plugin approach named Feature Separation and Recalibration (FSR) that recalibrates the malicious, non-robust activations for more robust feature maps through Separation and Recalibration. The Separation part disentangles the input feature map into the robust feature with activations that help the model make correct predictions and the non-robust feature with activations that are responsible for model mispredictions upon adversarial attack. The Recalibration part then adjusts the non-robust activations to restore the potentially useful cues for model predictions. Extensive experiments verify the superiority of FSR compared to traditional deactivation techniques and demonstrate that it improves the robustness of existing adversarial training methods by up to 8.57% with small computational overhead. Codes are available at https://github.com/wkim97/FSR.
Bridging Stereo Geometry and BEV Representation with Reliable Mutual Interaction for Semantic Scene Completion
Authors: Bohan Li, Yasheng Sun, Zhujin Liang, Dalong Du, Zhuanghui Zhang, XiaoFeng Wang, Yunnan Wang, Xin Jin, Wenjun Zeng
Date: 2023-03-24 00:00:00
Tasks: 3D Semantic Scene Completion, Hallucination, Representation Learning, Stereo Matching

3D semantic scene completion (SSC) is an ill-posed perception task that requires inferring a dense 3D scene from limited observations. Previous camera-based methods struggle to predict accurate semantic scenes due to inherent geometric ambiguity and incomplete observations. In this paper, we resort to stereo matching technique and bird's-eye-view (BEV) representation learning to address such issues in SSC. Complementary to each other, stereo matching mitigates geometric ambiguity with epipolar constraint while BEV representation enhances the hallucination ability for invisible regions with global semantic context. However, due to the inherent representation gap between stereo geometry and BEV features, it is non-trivial to bridge them for dense prediction task of SSC. Therefore, we further develop a unified occupancy-based framework dubbed BRGScene, which effectively bridges these two representations with dense 3D volumes for reliable semantic scene completion. Specifically, we design a novel Mutual Interactive Ensemble (MIE) block for pixel-level reliable aggregation of stereo geometry and BEV features. Within the MIE block, a Bi-directional Reliable Interaction (BRI) module, enhanced with confidence re-weighting, is employed to encourage fine-grained interaction through mutual guidance. Besides, a Dual Volume Ensemble (DVE) module is introduced to facilitate complementary aggregation through channel-wise recalibration and multi-group voting. Our method outperforms all published camera-based methods on SemanticKITTI for semantic scene completion. Our code is available on https://github.com/Arlo0o/StereoScene.
HandNeRF: Neural Radiance Fields for Animatable Interacting Hands
Authors: Zhiyang Guo, Wengang Zhou, Min Wang, Li Li, Houqiang Li
Date: 2023-03-24 00:00:00
Proceeding: CVPR 2023 1
Tasks: NeRF

We propose a novel framework to reconstruct accurate appearance and geometry with neural radiance fields (NeRF) for interacting hands, enabling the rendering of photo-realistic images and videos for gesture animation from arbitrary views. Given multi-view images of a single hand or interacting hands, an off-the-shelf skeleton estimator is first employed to parameterize the hand poses. Then we design a pose-driven deformation field to establish correspondence from those different poses to a shared canonical space, where a pose-disentangled NeRF for one hand is optimized. Such unified modeling efficiently complements the geometry and texture cues in rarely-observed areas for both hands. Meanwhile, we further leverage the pose priors to generate pseudo depth maps as guidance for occlusion-aware density learning. Moreover, a neural feature distillation method is proposed to achieve cross-domain alignment for color optimization. We conduct extensive experiments to verify the merits of our proposed HandNeRF and report a series of state-of-the-art results both qualitatively and quantitatively on the large-scale InterHand2.6M dataset.
LON-GNN: Spectral GNNs with Learnable Orthonormal Basis
Authors: Qian Tao, Zhen Wang, Wenyuan Yu, Yaliang Li, Zhewei Wei
Date: 2023-03-24 00:00:00

In recent years, a plethora of spectral graph neural networks (GNN) methods have utilized polynomial basis with learnable coefficients to achieve top-tier performances on many node-level tasks. Although various kinds of polynomial bases have been explored, each such method adopts a fixed polynomial basis which might not be the optimal choice for the given graph. Besides, we identify the so-called over-passing issue of these methods and show that it is somewhat rooted in their less-principled regularization strategy and unnormalized basis. In this paper, we make the first attempts to address these two issues. Leveraging Jacobi polynomials, we design a novel spectral GNN, LON-GNN, with Learnable OrthoNormal bases and prove that regularizing coefficients becomes equivalent to regularizing the norm of learned filter function now. We conduct extensive experiments on diverse graph datasets to evaluate the fitting and generalization capability of LON-GNN, where the results imply its superiority.
Scaling Expert Language Models with Unsupervised Domain Discovery
Authors: Suchin Gururangan, Margaret Li, Mike Lewis, Weijia Shi, Tim Althoff, Noah A. Smith, Luke Zettlemoyer
Date: 2023-03-24 00:00:00
Tasks: All, Language Modeling, Language Modelling

Large language models are typically trained densely: all parameters are updated with respect to all inputs. This requires synchronization of billions of parameters across thousands of GPUs. We introduce a simple but effective method to asynchronously train large, sparse language models on arbitrary text corpora. Our method clusters a corpus into sets of related documents, trains a separate expert language model on each cluster, and combines them in a sparse ensemble for inference. This approach generalizes embarrassingly parallel training by automatically discovering the domains for each expert, and eliminates nearly all the communication overhead of existing sparse language models. Our technique outperforms dense baselines on multiple corpora and few-shot tasks, and our analysis shows that specializing experts to meaningful clusters is key to these gains. Performance also improves with the number of experts and size of training data, suggesting this is a highly efficient and accessible approach to training large language models.
A Hybrid ANN-SNN Architecture for Low-Power and Low-Latency Visual Perception
Authors: Asude Aydin, Mathias Gehrig, Daniel Gehrig, Davide Scaramuzza
Date: 2023-03-24 00:00:00
Tasks: 3D Human Pose Estimation, Pose Estimation

Spiking Neural Networks (SNN) are a class of bio-inspired neural networks that promise to bring low-power and low-latency inference to edge devices through asynchronous and sparse processing. However, being temporal models, SNNs depend heavily on expressive states to generate predictions on par with classical artificial neural networks (ANNs). These states converge only after long transient periods, and quickly decay without input data, leading to higher latency, power consumption, and lower accuracy. This work addresses this issue by initializing the state with an auxiliary ANN running at a low rate. The SNN then uses the state to generate predictions with high temporal resolution until the next initialization phase. Our hybrid ANN-SNN model thus combines the best of both worlds: It does not suffer from long state transients and state decay thanks to the ANN, and can generate predictions with high temporal resolution, low latency, and low power thanks to the SNN. We show for the task of event-based 2D and 3D human pose estimation that our method consumes 88% less power with only a 4% decrease in performance compared to its fully ANN counterparts when run at the same inference rate. Moreover, when compared to SNNs, our method achieves a 74% lower error. This research thus provides a new understanding of how ANNs and SNNs can be used to maximize their respective benefits.
BundleSDF: Neural 6-DoF Tracking and 3D Reconstruction of Unknown Objects
Authors: Bowen Wen, Jonathan Tremblay, Valts Blukis, Stephen Tyree, Thomas Muller, Alex Evans, Dieter Fox, Jan Kautz, Stan Birchfield
Date: 2023-03-24 00:00:00
Proceeding: CVPR 2023 1
Tasks: 3D Object Detection, 3D Object Tracking, 3D Reconstruction, 6D Pose Estimation, 6D Pose Estimation using RGBD, Object, Object Tracking, Pose Tracking

We present a near real-time method for 6-DoF tracking of an unknown object from a monocular RGBD video sequence, while simultaneously performing neural 3D reconstruction of the object. Our method works for arbitrary rigid objects, even when visual texture is largely absent. The object is assumed to be segmented in the first frame only. No additional information is required, and no assumption is made about the interaction agent. Key to our method is a Neural Object Field that is learned concurrently with a pose graph optimization process in order to robustly accumulate information into a consistent 3D representation capturing both geometry and appearance. A dynamic pool of posed memory frames is automatically maintained to facilitate communication between these threads. Our approach handles challenging sequences with large pose changes, partial and full occlusion, untextured surfaces, and specular highlights. We show results on HO3D, YCBInEOAT, and BEHAVE datasets, demonstrating that our method significantly outperforms existing approaches. Project page: https://bundlesdf.github.io
"Get ready for a party": Exploring smarter smart spaces with help from large language models
Authors: Evan King, Haoxiang Yu, Sangsu Lee, Christine Julien
Date: 2023-03-24 00:00:00

The right response to someone who says "get ready for a party" is deeply influenced by meaning and context. For a smart home assistant (e.g., Google Home), the ideal response might be to survey the available devices in the home and change their state to create a festive atmosphere. Current practical systems cannot service such requests since they require the ability to (1) infer meaning behind an abstract statement and (2) map that inference to a concrete course of action appropriate for the context (e.g., changing the settings of specific devices). In this paper, we leverage the observation that recent task-agnostic large language models (LLMs) like GPT-3 embody a vast amount of cross-domain, sometimes unpredictable contextual knowledge that existing rule-based home assistant systems lack, which can make them powerful tools for inferring user intent and generating appropriate context-dependent responses during smart home interactions. We first explore the feasibility of a system that places an LLM at the center of command inference and action planning, showing that LLMs have the capacity to infer intent behind vague, context-dependent commands like "get ready for a party" and respond with concrete, machine-parseable instructions that can be used to control smart devices. We furthermore demonstrate a proof-of-concept implementation that puts an LLM in control of real devices, showing its ability to infer intent and change device state appropriately with no fine-tuning or task-specific training. Our work hints at the promise of LLM-driven systems for context-awareness in smart environments, motivating future research in this area.
MindDiffuser: Controlled Image Reconstruction from Human Brain Activity with Semantic and Structural Diffusion
Authors: Yizhuo Lu, Changde Du, Dianpeng Wang, Huiguang He
Date: 2023-03-24 00:00:00
Tasks: Image Reconstruction

Reconstructing visual stimuli from measured functional magnetic resonance imaging (fMRI) has been a meaningful and challenging task. Previous studies have successfully achieved reconstructions with structures similar to the original images, such as the outlines and size of some natural images. However, these reconstructions lack explicit semantic information and are difficult to discern. In recent years, many studies have utilized multi-modal pre-trained models with stronger generative capabilities to reconstruct images that are semantically similar to the original ones. However, these images have uncontrollable structural information such as position and orientation. To address both of the aforementioned issues simultaneously, we propose a two-stage image reconstruction model called MindDiffuser, utilizing Stable Diffusion. In Stage 1, the VQ-VAE latent representations and the CLIP text embeddings decoded from fMRI are put into the image-to-image process of Stable Diffusion, which yields a preliminary image that contains semantic and structural information. In Stage 2, we utilize the low-level CLIP visual features decoded from fMRI as supervisory information, and continually adjust the two features in Stage 1 through backpropagation to align the structural information. The results of both qualitative and quantitative analyses demonstrate that our proposed model has surpassed the current state-of-the-art models in terms of reconstruction results on Natural Scenes Dataset (NSD). Furthermore, the results of ablation experiments indicate that each component of our model is effective for image reconstruction.