pwc-archive/papers-with-abstracts huggingface dataset:
IntentVizor: Towards Generic Query Guided Interactive Video Summarization
Authors: Guande Wu, Jianzhe Lin, Claudio T. Silva
Date: 2021-09-30 00:00:00
Proceeding: CVPR 2022 1
Tasks: Video Summarization, Video Understanding

The target of automatic video summarization is to create a short skim of the original long video while preserving the major content/events. There is a growing interest in the integration of user queries into video summarization or query-driven video summarization. This video summarization method predicts a concise synopsis of the original video based on the user query, which is commonly represented by the input text. However, two inherent problems exist in this query-driven way. First, the text query might not be enough to describe the exact and diverse needs of the user. Second, the user cannot edit once the summaries are produced, while we assume the needs of the user should be subtle and need to be adjusted interactively. To solve these two problems, we propose IntentVizor, an interactive video summarization framework guided by generic multi-modality queries. The input query that describes the user's needs are not limited to text but also the video snippets. We further represent these multi-modality finer-grained queries as user `intent', which is interpretable, interactable, editable, and can better quantify the user's needs. In this paper, we use a set of the proposed intents to represent the user query and design a new interactive visual analytic interface. Users can interactively control and adjust these mixed-initiative intents to obtain a more satisfying summary through the interface. Also, to improve the summarization quality via video understanding, a novel Granularity-Scalable Ego-Graph Convolutional Networks (GSE-GCN) is proposed. We conduct our experiments on two benchmark datasets. Comparisons with the state-of-the-art methods verify the effectiveness of the proposed framework. Code and dataset are available at https://github.com/jnzs1836/intent-vizor.
Semantic Dense Reconstruction with Consistent Scene Segments
Authors: Yingcai Wan, Yanyan Li, Yingxuan You, Cheng Guo, Lijin Fang, Federico Tombari
Date: 2021-09-30 00:00:00
Tasks: 3D Scene Reconstruction, Scene Understanding, Semantic Segmentation

In this paper, a method for dense semantic 3D scene reconstruction from an RGB-D sequence is proposed to solve high-level scene understanding tasks. First, each RGB-D pair is consistently segmented into 2D semantic maps based on a camera tracking backbone that propagates objects' labels with high probabilities from full scans to corresponding ones of partial views. Then a dense 3D mesh model of an unknown environment is incrementally generated from the input RGB-D sequence. Benefiting from 2D consistent semantic segments and the 3D model, a novel semantic projection block (SP-Block) is proposed to extract deep feature volumes from 2D segments of different views. Moreover, the semantic volumes are fused into deep volumes from a point cloud encoder to make the final semantic segmentation. Extensive experimental evaluations on public datasets show that our system achieves accurate 3D dense reconstruction and state-of-the-art semantic prediction performances simultaneously.
Spark in the Dark: Evaluating Encoder-Decoder Pairs for COVID-19 CT's Semantic Segmentation
Authors: Bruno A. Krinski, Daniel V. Ruiz, Eduardo Todt
Date: 2021-09-30 00:00:00
Tasks: Computed Tomography (CT), Decoder, Segmentation, Semantic Segmentation

With the COVID-19 global pandemic, computerassisted diagnoses of medical images have gained a lot of attention, and robust methods of Semantic Segmentation of Computed Tomography (CT) turned highly desirable. Semantic Segmentation of CT is one of many research fields of automatic detection of Covid-19 and was widely explored since the Covid19 outbreak. In the robotic field, Semantic Segmentation of organs and CTs are widely used in robots developed for surgery tasks. As new methods and new datasets are proposed quickly, it becomes apparent the necessity of providing an extensive evaluation of those methods. To provide a standardized comparison of different architectures across multiple recently proposed datasets, we propose in this paper an extensive benchmark of multiple encoders and decoders with a total of 120 architectures evaluated in five datasets, with each dataset being validated through a five-fold cross-validation strategy, totaling 3.000 experiments. To the best of our knowledge, this is the largest evaluation in number of encoders, decoders, and datasets proposed in the field of Covid-19 CT segmentation.
GT U-Net: A U-Net Like Group Transformer Network for Tooth Root Segmentation
Authors: Yunxiang Li, Shuai Wang, Jun Wang, Guodong Zeng, Wenjun Liu, Qianni Zhang, Qun Jin, Yaqi Wang
Date: 2021-09-30 00:00:00
Tasks: Anatomy, Segmentation

To achieve an accurate assessment of root canal therapy, a fundamental step is to perform tooth root segmentation on oral X-ray images, in that the position of tooth root boundary is significant anatomy information in root canal therapy evaluation. However, the fuzzy boundary makes the tooth root segmentation very challenging. In this paper, we propose a novel end-to-end U-Net like Group Transformer Network (GT U-Net) for the tooth root segmentation. The proposed network retains the essential structure of U-Net but each of the encoders and decoders is replaced by a group Transformer, which significantly reduces the computational cost of traditional Transformer architectures by using the grouping structure and the bottleneck structure. In addition, the proposed GT U-Net is composed of a hybrid structure of convolution and Transformer, which makes it independent of pre-training weights. For optimization, we also propose a shape-sensitive Fourier Descriptor (FD) loss function to make use of shape prior knowledge. Experimental results show that our proposed network achieves the state-of-the-art performance on our collected tooth root segmentation dataset and the public retina dataset DRIVE. Code has been released at https://github.com/Kent0n-Li/GT-U-Net.
Unsupervised Landmark Detection Based Spatiotemporal Motion Estimation for 4D Dynamic Medical Images
Authors: Yuyu Guo, Lei Bi, Dongming Wei, Liyun Chen, Zhengbin Zhu, Dagan Feng, Ruiyan Zhang, Qian Wang, Jinman Kim
Date: 2021-09-30 00:00:00
Tasks: Anatomy, Motion Estimation, Unsupervised Landmark Detection

Motion estimation is a fundamental step in dynamic medical image processing for the assessment of target organ anatomy and function. However, existing image-based motion estimation methods, which optimize the motion field by evaluating the local image similarity, are prone to produce implausible estimation, especially in the presence of large motion. In this study, we provide a novel motion estimation framework of Dense-Sparse-Dense (DSD), which comprises two stages. In the first stage, we process the raw dense image to extract sparse landmarks to represent the target organ anatomical topology and discard the redundant information that is unnecessary for motion estimation. For this purpose, we introduce an unsupervised 3D landmark detection network to extract spatially sparse but representative landmarks for the target organ motion estimation. In the second stage, we derive the sparse motion displacement from the extracted sparse landmarks of two images of different time points. Then, we present a motion reconstruction network to construct the motion field by projecting the sparse landmarks displacement back into the dense image domain. Furthermore, we employ the estimated motion field from our two-stage DSD framework as initialization and boost the motion estimation quality in light-weight yet effective iterative optimization. We evaluate our method on two dynamic medical imaging tasks to model cardiac motion and lung respiratory motion, respectively. Our method has produced superior motion estimation accuracy compared to existing comparative methods. Besides, the extensive experimental results demonstrate that our solution can extract well representative anatomical landmarks without any requirement of manual annotation. Our code is publicly available online.
Automated airway segmentation by learning graphical structure
Authors: Yihua Yang
Date: 2021-09-30 00:00:00
Tasks: Computed Tomography (CT), Graph Neural Network, Segmentation

In this research project, we put forward an advanced method for airway segmentation based on the existent convolutional neural network (CNN) and graph neural network (GNN). The method is originated from the vessel segmentation, but we ameliorate it and enable the novel model to perform better for datasets from computed tomography (CT) scans. Current methods for airway segmentation are considering the regular grid only. No matter what the detailed model is, including the 3-dimensional CNN or 2-dimensional CNN in three directions, the overall graph structures are not taken into consideration. In our model, with the neighbourhoods of airway taken into account, the graph structure is incorporated and the segmentation of airways are improved compared with the traditional CNN methods. We perform experiments on the chest CT scans, where the ground truth segmentation labels are produced manually. The proposed model shows that compared with the CNN-only method, the combination of CNN and GNN has a better performance in that the bronchi in the chest CT scans can be detected in most cases. In addition, the model we propose has a wide extension since the architecture is also utilitarian in fulfilling similar aims in other datasets. Hence, the state-of-the-art model is of great significance and highly applicable in our daily lives. Keywords: Airway segmentation, Convolutional neural network, Graph neural network
A Prior Knowledge Based Tumor and Tumoral Subregion Segmentation Tool for Pediatric Brain Tumors
Authors: Silu Zhang, Angela Edwards, Shubo Wang, Zoltan Patay, Asim Bag, Matthew A. Scoggins
Date: 2021-09-30 00:00:00
Tasks: Brain Tumor Segmentation, Segmentation, Transfer Learning, Tumor Segmentation

In the past few years, deep learning (DL) models have drawn great attention and shown superior performance on brain tumor and subregion segmentation tasks. However, the success is limited to segmentation of adult gliomas, where sufficient data have been collected, manually labeled, and published for training DL models. It is still challenging to segment pediatric tumors, because the appearances are different from adult gliomas. Hence, directly applying a pretained DL model on pediatric data usually generates unacceptable results. Because pediatric data is very limited, both labeled and unlabeled, we present a brain tumor segmentation model that is based on knowledge rather than learning from data. We also provide segmentation of more subregions for super heterogeneous tumor like atypical teratoid rhabdoid tumor (ATRT). Our proposed approach showed superior performance on both whole tumor and subregion segmentation tasks to DL based models on our pediatric data when training data is not available for transfer learning.
Chest X-Rays Image Classification from beta-Variational Autoencoders Latent Features
Authors: Leonardo Crespi, Daniele Loiacono, Arturo Chiti
Date: 2021-09-29 00:00:00
Tasks: Diagnostic, image-classification, Image Classification

Chest X-Ray (CXR) is one of the most common diagnostic techniques used in everyday clinical practice all around the world. We hereby present a work which intends to investigate and analyse the use of Deep Learning (DL) techniques to extract information from such images and allow to classify them, trying to keep our methodology as general as possible and possibly also usable in a real world scenario without much effort, in the future. To move in this direction, we trained several beta-Variational Autoencoder (beta-VAE) models on the CheXpert dataset, one of the largest publicly available collection of labeled CXR images; from these models, latent features have been extracted and used to train other Machine Learning models, able to classify the original images from the features extracted by the beta-VAE. Lastly, tree-based models have been combined together in ensemblings to improve the results without the necessity of further training or models engineering. Expecting some drop in pure performance with the respect to state of the art classification specific models, we obtained encouraging results, which show the viability of our approach and the usability of the high level features extracted by the autoencoders for classification tasks.
MetaHistoSeg: A Python Framework for Meta Learning in Histopathology Image Segmentation
Authors: Zheng Yuan, Andre Esteva, ran Xu
Date: 2021-09-29 00:00:00
Tasks: Domain Generalization, Few-Shot Learning, Image Segmentation, Meta-Learning, Semantic Segmentation, Transfer Learning

Few-shot learning is a standard practice in most deep learning based histopathology image segmentation, given the relatively low number of digitized slides that are generally available. While many models have been developed for domain specific histopathology image segmentation, cross-domain generalization remains a key challenge for properly validating models. Here, tooling and datasets to benchmark model performance across histopathological domains are lacking. To address this limitation, we introduce MetaHistoSeg - a Python framework that implements unique scenarios in both meta learning and instance based transfer learning. Designed for easy extension to customized datasets and task sampling schemes, the framework empowers researchers with the ability of rapid model design and experimentation. We also curate a histopathology meta dataset - a benchmark dataset for training and validating models on out-of-distribution performance across a range of cancer types. In experiments we showcase the usage of MetaHistoSeg with the meta dataset and find that both meta-learning and instance based transfer learning deliver comparable results on average, but in some cases tasks can greatly benefit from one over the other.
The Object at Hand: Automated Editing for Mixed Reality Video Guidance from Hand-Object Interactions
Authors: Yao Lu, Walterio W. Mayol-Cuevas
Date: 2021-09-29 00:00:00
Tasks: Mixed Reality, Object

In this paper, we concern with the problem of how to automatically extract the steps that compose real-life hand activities. This is a key competence towards processing, monitoring and providing video guidance in Mixed Reality systems. We use egocentric vision to observe hand-object interactions in real-world tasks and automatically decompose a video into its constituent steps. Our approach combines hand-object interaction (HOI) detection, object similarity measurement and a finite state machine (FSM) representation to automatically edit videos into steps. We use a combination of Convolutional Neural Networks (CNNs) and the FSM to discover, edit cuts and merge segments while observing real hand activities. We evaluate quantitatively and qualitatively our algorithm on two datasets: the GTEA\cite{li2015delving}, and a new dataset we introduce for Chinese Tea making. Results show our method is able to segment hand-object interaction videos into key step segments with high levels of precision.
Egocentric Hand-object Interaction Detection and Application
Authors: Yao Lu, Walterio W. Mayol-Cuevas
Date: 2021-09-29 00:00:00
Tasks: Hand-Object Interaction Detection, Object

In this paper, we present a method to detect the hand-object interaction from an egocentric perspective. In contrast to massive data-driven discriminator based method like \cite{Shan20}, we propose a novel workflow that utilises the cues of hand and object. Specifically, we train networks predicting hand pose, hand mask and in-hand object mask to jointly predict the hand-object interaction status. We compare our method with the most recent work from Shan et al. \cite{Shan20} on selected images from EPIC-KITCHENS \cite{damen2018scaling} dataset and achieve $89\%$ accuracy on HOI (hand-object interaction) detection which is comparative to Shan's ($92\%$). However, for real-time performance, with the same machine, our method can run over $\textbf{30}$ FPS which is much efficient than Shan's ($\textbf{1}\sim\textbf{2}$ FPS). Furthermore, with our approach, we are able to segment script-less activities from where we extract the frames with the HOI status detection. We achieve $\textbf{68.2\%}$ and $\textbf{82.8\%}$ F1 score on GTEA \cite{fathi2011learning} and the UTGrasp \cite{cai2015scalable} dataset respectively which are all comparative to the SOTA methods.
Targeted Gradient Descent: A Novel Method for Convolutional Neural Networks Fine-tuning and Online-learning
Authors: Junyu Chen, Evren Asma, Chung Chan
Date: 2021-09-29 00:00:00
Tasks: Denoising, Image Denoising

A convolutional neural network (ConvNet) is usually trained and then tested using images drawn from the same distribution. To generalize a ConvNet to various tasks often requires a complete training dataset that consists of images drawn from different tasks. In most scenarios, it is nearly impossible to collect every possible representative dataset as a priori. The new data may only become available after the ConvNet is deployed in clinical practice. ConvNet, however, may generate artifacts on out-of-distribution testing samples. In this study, we present Targeted Gradient Descent (TGD), a novel fine-tuning method that can extend a pre-trained network to a new task without revisiting data from the previous task while preserving the knowledge acquired from previous training. To a further extent, the proposed method also enables online learning of patient-specific data. The method is built on the idea of reusing a pre-trained ConvNet's redundant kernels to learn new knowledge. We compare the performance of TGD to several commonly used training approaches on the task of Positron emission tomography (PET) image denoising. Results from clinical images show that TGD generated results on par with training-from-scratch while significantly reducing data preparation and network training time. More importantly, it enables online learning on the testing study to enhance the network's generalization capability in real-world applications.
USIS: Unsupervised Semantic Image Synthesis
Authors: George Eskandar, Mohamed Abdelsamad, Karim Armanious, Bin Yang
Date: 2021-09-29 00:00:00
Tasks: Image Generation, Image-to-Image Translation, Translation

Semantic Image Synthesis (SIS) is a subclass of image-to-image translation where a photorealistic image is synthesized from a segmentation mask. SIS has mostly been addressed as a supervised problem. However, state-of-the-art methods depend on a huge amount of labeled data and cannot be applied in an unpaired setting. On the other hand, generic unpaired image-to-image translation frameworks underperform in comparison, because they color-code semantic layouts and feed them to traditional convolutional networks, which then learn correspondences in appearance instead of semantic content. In this initial work, we propose a new Unsupervised paradigm for Semantic Image Synthesis (USIS) as a first step towards closing the performance gap between paired and unpaired settings. Notably, the framework deploys a SPADE generator that learns to output images with visually separable semantic classes using a self-supervised segmentation loss. Furthermore, in order to match the color and texture distribution of real images without losing high-frequency information, we propose to use whole image wavelet-based discrimination. We test our methodology on 3 challenging datasets and demonstrate its ability to generate multimodal photorealistic images with an improved quality in the unpaired setting.
Convolutional Neural Network Compression through Generalized Kronecker Product Decomposition
Authors: Marawan Gamal Abdel Hameed, Marzieh S. Tahaei, Ali Mosleh, Vahid Partovi Nia
Date: 2021-09-29 00:00:00
Tasks: image-classification, Image Classification, Knowledge Distillation, Neural Network Compression

Modern Convolutional Neural Network (CNN) architectures, despite their superiority in solving various problems, are generally too large to be deployed on resource constrained edge devices. In this paper, we reduce memory usage and floating-point operations required by convolutional layers in CNNs. We compress these layers by generalizing the Kronecker Product Decomposition to apply to multidimensional tensors, leading to the Generalized Kronecker Product Decomposition (GKPD). Our approach yields a plug-and-play module that can be used as a drop-in replacement for any convolutional layer. Experimental results for image classification on CIFAR-10 and ImageNet datasets using ResNet, MobileNetv2 and SeNet architectures substantiate the effectiveness of our proposed approach. We find that GKPD outperforms state-of-the-art decomposition methods including Tensor-Train and Tensor-Ring as well as other relevant compression methods such as pruning and knowledge distillation.
LR-to-HR Face Hallucination with an Adversarial Progressive Attribute-Induced Network
Authors: Nitin Balachandran, Jun-Cheng Chen, Rama Chellappa
Date: 2021-09-29 00:00:00
Tasks: Attribute, Face Hallucination, Hallucination, Super-Resolution

Face super-resolution is a challenging and highly ill-posed problem since a low-resolution (LR) face image may correspond to multiple high-resolution (HR) ones during the hallucination process and cause a dramatic identity change for the final super-resolved results. Thus, to address this problem, we propose an end-to-end progressive learning framework incorporating facial attributes and enforcing additional supervision from multi-scale discriminators. By incorporating facial attributes into the learning process and progressively resolving the facial image, the mapping between LR and HR images is constrained more, and this significantly helps to reduce the ambiguity and uncertainty in one-to-many mapping. In addition, we conduct thorough evaluations on the CelebA dataset following the settings of previous works (i.e. super-resolving by a factor of 8x from tiny 16x16 face images.), and the results demonstrate that the proposed approach can yield satisfactory face hallucination images outperforming other state-of-the-art approaches.
Vision-Aided Beam Tracking: Explore the Proper Use of Camera Images with Deep Learning
Authors: Yu Tian, Chenwei Wang
Date: 2021-09-29 00:00:00
Tasks: Beam Prediction

We investigate the problem of wireless beam tracking on mmWave bands with the assistance of camera images. In particular, based on the user's beam indices used and camera images taken in the trajectory, we predict the optimal beam indices in the next few time spots. To resolve this problem, we first reformulate the "ViWi" dataset in [1] to get rid of the image repetition problem. Then we develop a deep learning approach and investigate various model components to achieve the best performance. Finally, we explore whether, when, and how to use the image for better beam prediction. To answer this question, we split the dataset into three clusters -- (LOS, light NLOS, serious NLOS)-like -- based on the standard deviation of the beam sequence. With experiments we demonstrate that using the image indeed helps beam tracking especially when the user is in serious NLOS, and the solution relies on carefully-designed dataset for training a model. Generally speaking, including NLOS-like data for training a model does not benefit beam tracking of the user in LOS, but including light NLOS-like data for training a model benefits beam tracking of the user in serious NLOS.