pwc-archive/papers-with-abstracts huggingface dataset:
UNetFormer: A Unified Vision Transformer Model and Pre-Training Framework for 3D Medical Image Segmentation
Authors: Ali Hatamizadeh, Ziyue Xu, Dong Yang, Wenqi Li, Holger Roth, Daguang Xu
Date: 2022-04-01 00:00:00
Tasks: Brain Tumor Segmentation, Image Segmentation, Medical Image Analysis, Medical Image Segmentation, Representation Learning, Segmentation, Semantic Segmentation, Tumor Segmentation

Vision Transformers (ViT)s have recently become popular due to their outstanding modeling capabilities, in particular for capturing long-range information, and scalability to dataset and model sizes which has led to state-of-the-art performance in various computer vision and medical image analysis tasks. In this work, we introduce a unified framework consisting of two architectures, dubbed UNetFormer, with a 3D Swin Transformer-based encoder and Convolutional Neural Network (CNN) and transformer-based decoders. In the proposed model, the encoder is linked to the decoder via skip connections at five different resolutions with deep supervision. The design of proposed architecture allows for meeting a wide range of trade-off requirements between accuracy and computational cost. In addition, we present a methodology for self-supervised pre-training of the encoder backbone via learning to predict randomly masked volumetric tokens using contextual information of visible tokens. We pre-train our framework on a cohort of $5050$ CT images, gathered from publicly available CT datasets, and present a systematic investigation of various components such as masking ratio and patch size that affect the representation learning capability and performance of downstream tasks. We validate the effectiveness of our pre-training approach by fine-tuning and testing our model on liver and liver tumor segmentation task using the Medical Segmentation Decathlon (MSD) dataset and achieve state-of-the-art performance in terms of various segmentation metrics. To demonstrate its generalizability, we train and test the model on BraTS 21 dataset for brain tumor segmentation using MRI images and outperform other methods in terms of Dice score. Code: https://github.com/Project-MONAI/research-contributions
RFVTM: A Recovery and Filtering Vertex Trichotomy Matching for Remote Sensing Image Registration
Authors: Ming Zhao, Bowen An, Yongpeng Wu, Huynh Van Luong, André Kaup
Date: 2022-04-02 00:00:00
Tasks: Image Registration

Reliable feature point matching is a vital yet challenging process in feature-based image registration. In this paper,a robust feature point matching algorithm called Recovery and Filtering Vertex Trichotomy Matching (RFVTM) is proposed to remove outliers and retain sufficient inliers for remote sensing images. A novel affine invariant descriptor called vertex trichotomy descriptor is proposed on the basis of that geometrical relations between any of vertices and lines are preserved after affine transformations, which is constructed by mapping each vertex into trichotomy sets. The outlier removals in Vertex Trichotomy Matching (VTM) are implemented by iteratively comparing the disparity of corresponding vertex trichotomy descriptors. Some inliers mistakenly validated by a large amount of outliers are removed in VTM iterations, and several residual outliers close to correct locations cannot be excluded with the same graph structures. Therefore, a recovery and filtering strategy is designed to recover some inliers based on identical vertex trichotomy descriptors and restricted transformation errors. Assisted with the additional recovered inliers, residual outliers can also be filtered out during the process of reaching identical graph for the expanded vertex sets. Experimental results demonstrate the superior performance on precision and stability of this algorithm under various conditions, such as remote sensing images with large transformations, duplicated patterns, or inconsistent spectral content.
Automatic Registration of Images with Inconsistent Content Through Line-Support Region Segmentation and Geometrical Outlier Removal
Authors: Ming Zhao, Yongpeng Wu, Shengda Pan, Fan Zhou, Bowen An, André Kaup
Date: 2022-04-02 00:00:00
Tasks: Image Registration

The implementation of automatic image registration is still difficult in various applications. In this paper, an automatic image registration approach through line-support region segmentation and geometrical outlier removal (ALRS-GOR) is proposed. This new approach is designed to address the problems associated with the registration of images with affine deformations and inconsistent content, such as remote sensing images with different spectral content or noise interference, or map images with inconsistent annotations. To begin with, line-support regions, namely a straight region whose points share roughly the same image gradient angle, are extracted to address the issues of inconsistent content existing in images. To alleviate the incompleteness of line segments, an iterative strategy with multi-resolution is employed to preserve global structures that are masked at full resolution by image details or noise. Then, Geometrical Outlier Removal (GOR) is developed to provide reliable feature point matching, which is based on affineinvariant geometrical classifications for corresponding matches initialized by SIFT. The candidate outliers are selected by comparing the disparity of accumulated classifications among all matches, instead of conventional methods which only rely on local geometrical relations. Various image sets have been considered in this paper for the evaluation of the proposed approach, including aerial images with simulated affine deformations, remote sensing optical and synthetic aperture radar images taken at different situations (multispectral, multisensor, and multitemporal), and map images with inconsistent annotations. Experimental results demonstrate the superior performance of the proposed method over the existing approaches for the whole data set.
Acoustic-to-articulatory Inversion based on Speech Decomposition and Auxiliary Feature
Authors: Jianrong Wang, Jinyu Liu, Longxuan Zhao, Shanyu Wang, Ruiguo Yu, Li Liu
Date: 2022-04-02 00:00:00

Acoustic-to-articulatory inversion (AAI) is to obtain the movement of articulators from speech signals. Until now, achieving a speaker-independent AAI remains a challenge given the limited data. Besides, most current works only use audio speech as input, causing an inevitable performance bottleneck. To solve these problems, firstly, we pre-train a speech decomposition network to decompose audio speech into speaker embedding and content embedding as the new personalized speech features to adapt to the speaker-independent case. Secondly, to further improve the AAI, we propose a novel auxiliary feature network to estimate the lip auxiliary features from the above personalized speech features. Experimental results on three public datasets show that, compared with the state-of-the-art only using the audio speech feature, the proposed method reduces the average RMSE by 0.25 and increases the average correlation coefficient by 2.0% in the speaker-dependent case. More importantly, the average RMSE decreases by 0.29 and the average correlation coefficient increases by 5.0% in the speaker-independent case.
Moment-based Adversarial Training for Embodied Language Comprehension
Authors: Shintaro Ishikawa, Komei Sugiura
Date: 2022-04-02 00:00:00

In this paper, we focus on a vision-and-language task in which a robot is instructed to execute household tasks. Given an instruction such as "Rinse off a mug and place it in the coffee maker," the robot is required to locate the mug, wash it, and put it in the coffee maker. This is challenging because the robot needs to break down the instruction sentences into subgoals and execute them in the correct order. On the ALFRED benchmark, the performance of state-of-the-art methods is still far lower than that of humans. This is partially because existing methods sometimes fail to infer subgoals that are not explicitly specified in the instruction sentences. We propose Moment-based Adversarial Training (MAT), which uses two types of moments for perturbation updates in adversarial training. We introduce MAT to the embedding spaces of the instruction, subgoals, and state representations to handle their varieties. We validated our method on the ALFRED benchmark, and the results demonstrated that our method outperformed the baseline method for all the metrics on the benchmark.
Gastrointestinal Polyps and Tumors Detection Based on Multi-scale Feature-fusion with WCE Sequences
Authors: Zhuo Falin, Liu Haihua, Pan Ning
Date: 2022-04-03 00:00:00
Tasks: Lesion Detection, Region Proposal

Wireless Capsule Endoscopy(WCE) has been widely used for the screening of gastrointestinal(GI) diseases, especially the small intestine, due to its advantages of non-invasive and painless imaging of the entire digestive tract.However, the huge amount of image data captured by WCE makes manual reading a process that requires a huge amount of tasks and can easily lead to missed detection and false detection of lesions.Therefore, In this paper, we propose a \textbf{T}wo-stage \textbf{M}ulti-scale \textbf{F}eature-fusion learning network(\textbf{TMFNet}) to automatically detect small intestinal polyps and tumors in WCE image sequences. Specifically, TMFNet consists of lesion detection network and lesion identification network. Among them, the former improves the feature extraction module and detection module based on the traditional Faster R-CNN network, and readjusts the parameters of the anchor in the region proposal network(RPN) module;the latter combines residual structure and feature pyramid structure are used to build a small intestinal lesion recognition network based on feature fusion, for reducing the false positive rate of the former and improve the overall accuracy.We used 22,335 WCE images in the experiment, with a total of 123,092 lesion regions used to train the detection framework of this paper. In the experiment, the detection framework is trained and tested on the real WCE image dataset provided by the hospital gastroenterology department. The sensitivity, false positive and accuracy of the final model on the RPM are 98.81$\%$, 7.43$\%$ and 92.57$\%$, respectively.Meanwhile,the corresponding results on the lesion images were 98.75$\%$, 5.62$\%$ and 94.39$\%$. The algorithm model proposed in this paper is obviously superior to other detection algorithms in detection effect and performance
A Novel Mask R-CNN Model to Segment Heterogeneous Brain Tumors through Image Subtraction
Authors: Sanskriti Singh
Date: 2022-04-04 00:00:00
Tasks: Image Segmentation, Pneumonia Detection, Segmentation, Semantic Segmentation

The segmentation of diseases is a popular topic explored by researchers in the field of machine learning. Brain tumors are extremely dangerous and require the utmost precision to segment for a successful surgery. Patients with tumors usually take 4 MRI scans, T1, T1gd, T2, and FLAIR, which are then sent to radiologists to segment and analyze for possible future surgery. To create a second segmentation, it would be beneficial to both radiologists and patients in being more confident in their conclusions. We propose using a method performed by radiologists called image segmentation and applying it to machine learning models to prove a better segmentation. Using Mask R-CNN, its ResNet backbone being pre-trained on the RSNA pneumonia detection challenge dataset, we can train a model on the Brats2020 Brain Tumor dataset. Center for Biomedical Image Computing & Analytics provides MRI data on patients with and without brain tumors and the corresponding segmentations. We can see how well the method of image subtraction works by comparing it to models without image subtraction through DICE coefficient (F1 score), recall, and precision on the untouched test set. Our model performed with a DICE coefficient of 0.75 in comparison to 0.69 without image subtraction. To further emphasize the usefulness of image subtraction, we compare our final model to current state-of-the-art models to segment tumors from MRI scans.
Differentiable Rendering for Synthetic Aperture Radar Imagery
Authors: Michael Wilmanski, Jonathan Tamir
Date: 2022-04-04 00:00:00
Tasks: 3D Object Reconstruction, Neural Rendering, Object Reconstruction

There is rising interest in differentiable rendering, which allows explicitly modeling geometric priors and constraints in optimization pipelines using first-order methods such as backpropagation. Incorporating such domain knowledge can lead to deep neural networks that are trained more robustly and with limited data, as well as the capability to solve ill-posed inverse problems. Existing efforts in differentiable rendering have focused on imagery from electro-optical sensors, particularly conventional RGB-imagery. In this work, we propose an approach for differentiable rendering of Synthetic Aperture Radar (SAR) imagery, which combines methods from 3D computer graphics with neural rendering. We demonstrate the approach on the inverse graphics problem of 3D Object Reconstruction from limited SAR imagery using high-fidelity simulated SAR data.
Extended Reality for Mental Health Evaluation -A Scoping Review
Authors: Omisore Olatunji, Ifeanyi Odenigbo, Joseph Orji, Amelia Beltran, Nilufar Baghaei, Meier Sandra, Rita Orji
Date: 2022-04-04 00:00:00
Tasks: Articles, Mixed Reality

Mental health disorders are the leading cause of health-related problems globally. It is projected that mental health disorders will be the leading cause of morbidity among adults as the incidence rates of anxiety and depression grows globally. Recently, extended reality (XR), a general term covering virtual reality (VR), augmented reality (AR) and mixed reality (MR), is paving a new way to deliver mental health care. In this paper, we conduct a scoping review on the development and application of XR in the area of mental disorders. We performed a scoping database search to identify the relevant studies indexed in Google Scholar, PubMed, and the ACM Digital Library. A search period between August 2016 and December 2023 was defined to select articles related to the usage of VR, AR, and MR in a mental health context. We identified a total of 85 studies from 27 countries across the globe. By performing data analysis, we found that most of the studies focused on developed countries such as the US (16.47%) and Germany (12.94%). None of the studies were for African countries. The majority of the articles reported that XR techniques led to a significant reduction in symptoms of anxiety or depression. More studies were published in the year 2021, i.e., 31.76% (n = 31). This could indicate that mental disorder intervention received a higher attention when COVID-19 emerged. Most studies (n = 65) focused on a population between 18 and 65 years old, only a few studies focused on teenagers (n = 2). Also, more studies were done experimentally (n = 67, 78.82%) rather than by analytical and modeling approaches (n = 8, 9.41%). This shows that there is a rapid development of XR technology for mental health care. Furthermore, these studies showed that XR technology can effectively be used for evaluating mental disorders in similar or better way as the conventional approaches.
Learning to solve Minimum Cost Multicuts efficiently using Edge-Weighted Graph Convolutional Neural Networks
Authors: Steffen Jung, Margret Keuper
Date: 2022-04-04 00:00:00
Tasks: Combinatorial Optimization

The minimum cost multicut problem is the NP-hard/APX-hard combinatorial optimization problem of partitioning a real-valued edge-weighted graph such as to minimize the total cost of the partition. While graph convolutional neural networks (GNN) have proven to be promising in the context of combinatorial optimization, most of them are only tailored to or tested on positive-valued edge weights, i.e. they do not comply to the nature of the multicut problem. We therefore adapt various GNN architectures including Graph Convolutional Networks, Signed Graph Convolutional Networks and Graph Isomorphic Networks to facilitate the efficient encoding of real-valued edge costs. Moreover, we employ a reformulation of the multicut ILP constraints to a polynomial program as loss function that allows to learn feasible multicut solutions in a scalable way. Thus, we provide the first approach towards end-to-end trainable multicuts. Our findings support that GNN approaches can produce good solutions in practice while providing lower computation times and largely improved scalability compared to LP solvers and optimized heuristics, especially when considering large instances.
Dressi: A Hardware-Agnostic Differentiable Renderer with Reactive Shader Packing and Soft Rasterization
Authors: Yusuke Takimoto, Hiroyuki Sato, Hikari Takehara, Keishiro Uragaki, Takehiro Tawara, Xiao Liang, Kentaro Oku, Wataru Kishimoto, Bo Zheng
Date: 2022-04-04 00:00:00
Tasks: Inverse Rendering

Differentiable rendering (DR) enables various computer graphics and computer vision applications through gradient-based optimization with derivatives of the rendering equation. Most rasterization-based approaches are built on general-purpose automatic differentiation (AD) libraries and DR-specific modules handcrafted using CUDA. Such a system design mixes DR algorithm implementation and algorithm building blocks, resulting in hardware dependency and limited performance. In this paper, we present a practical hardware-agnostic differentiable renderer called Dressi, which is based on a new full AD design. The DR algorithms of Dressi are fully written in our Vulkan-based AD for DR, Dressi-AD, which supports all primitive operations for DR. Dressi-AD and our inverse UV technique inside it bring hardware independence and acceleration by graphics hardware. Stage packing, our runtime optimization technique, can adapt hardware constraints and efficiently execute complex computational graphs of DR with reactive cache considering the render pass hierarchy of Vulkan. HardSoftRas, our novel rendering process, is designed for inverse rendering with a graphics pipeline. Under the limited functionalities of the graphics pipeline, HardSoftRas can propagate the gradients of pixels from the screen space to far-range triangle attributes. Our experiments and applications demonstrate that Dressi establishes hardware independence, high-quality and robust optimization with fast speed, and photorealistic rendering.
Computer-Aided Extraction of Select MRI Markers of Cerebral Small Vessel Disease: A Systematic Review
Authors: Jiyang Jiang, Dadong Wang, Yang song, Perminder S. Sachdev, Wei Wen
Date: 2022-04-04 00:00:00
Tasks: Diversity, Transfer Learning

Cerebral small vessel disease (CSVD) is a major vascular contributor to cognitive impairment in ageing, including dementias. Imaging remains the most promising method for in vivo studies of CSVD. To replace the subjective and laborious visual rating approaches, emerging studies have applied state-of-the-art artificial intelligence to extract imaging biomarkers of CSVD from MRI scans. We aimed to summarise published computer-aided methods to examine three imaging biomarkers of CSVD, namely cerebral microbleeds (CMB), dilated perivascular spaces (PVS), and lacunes of presumed vascular origin. Seventy-one classical image processing, classical machine learning, and deep learning studies were identified. CMB and PVS have been better studied, compared to lacunes. While good performance metrics have been achieved in local test datasets, there have not been generalisable pipelines validated in different research or clinical cohorts. Transfer learning and weak supervision techniques have been applied to accommodate the limitations in training data. Future studies could consider pooling data from multiple sources to increase diversity, and validating the performance of the methods using both image processing metrics and associations with clinical measures.
Optimizing the Consumption of Spiking Neural Networks with Activity Regularization
Authors: Simon Narduzzi, Siavash A. Bigdeli, Shih-Chii Liu, L. Andrea Dunbar
Date: 2022-04-04 00:00:00

Reducing energy consumption is a critical point for neural network models running on edge devices. In this regard, reducing the number of multiply-accumulate (MAC) operations of Deep Neural Networks (DNNs) running on edge hardware accelerators will reduce the energy consumption during inference. Spiking Neural Networks (SNNs) are an example of bio-inspired techniques that can further save energy by using binary activations, and avoid consuming energy when not spiking. The networks can be configured for equivalent accuracy on a task through DNN-to-SNN conversion frameworks but their conversion is based on rate coding therefore the synaptic operations can be high. In this work, we look into different techniques to enforce sparsity on the neural network activation maps and compare the effect of different training regularizers on the efficiency of the optimized DNNs and SNNs.
Satellite Monitoring of Terrestrial Plastic Waste
Authors: Caleb Kruse, Edward Boyda, Sully Chen, Krishna Karra, Tristan Bou-Nahra, Dan Hammer, Jennifer Mathis, Taylor Maddalene, Jenna Jambeck, Fabien Laurier
Date: 2022-03-24 00:00:00

Plastic waste is a significant environmental pollutant that is difficult to monitor. We created a system of neural networks to analyze spectral, spatial, and temporal components of Sentinel-2 satellite data to identify terrestrial aggregations of waste. The system works at continental scale. We evaluated performance in Indonesia and detected 374 waste aggregations, more than double the number of sites found in public databases. The same system deployed across twelve countries in Southeast Asia identifies 996 subsequently confirmed waste sites. For each detected site, we algorithmically monitor waste site footprints through time and cross-reference other datasets to generate physical and social metadata. 19% of detected waste sites are located within 200 m of a waterway. Numerous sites sit directly on riverbanks, with high risk of ocean leakage.
Internet-of-Things Architectures for Secure Cyber-Physical Spaces: the VISOR Experience Report
Authors: Daniel De Pascale, Giuseppe Cascavilla, Mirella Sangiovanni, Damian A. Tamburri, Willem-Jan van den Heuvel
Date: 2022-04-01 00:00:00

Internet of things (IoT) technologies are becoming a more and more widespread part of civilian life in common urban spaces, which are rapidly turning into cyber-physical spaces. Simultaneously, the fear of terrorism and crime in such public spaces is ever-increasing. Due to the resulting increased demand for security, video-based IoT surveillance systems have become an important area for research. Considering the large number of devices involved in the illicit recognition task, we conducted a field study in a Dutch Easter music festival in a national interest project called VISOR to select the most appropriate device configuration in terms of performance and results. We iteratively architected solutions for the security of cyber-physical spaces using IoT devices. We tested the performance of multiple federated devices encompassing drones, closed-circuit television, smart phone cameras, and smart glasses to detect real-case scenarios of potentially malicious activities such as mosh-pits and pick-pocketing. Our results pave the way to select optimal IoT architecture configurations -- i.e., a mix of CCTV, drones, smart glasses, and camera phones in our case -- to make safer cyber-physical spaces' a reality.