pwc-archive/papers-with-abstracts huggingface dataset:
Pyramid convolutional network for colorization in monochrome-color multi-lens camera system
Authors: XuanDong, WeixinLi, XiaojieWang
Date: 2021-04-13 00:00:00
Proceeding: Neurocomputing 2021 4
Tasks: Colorization

We study the problem of colorization in the monochrome-color multi-lens camera system. The recent convolutional network (CNN) based method learns a 3-D weight volume to solve this problem and gets very high accuracy. But the model size is very big due to the large-displacement problem, i.e. there are large displacements between some pixels in the input gray image and the pixels in the reference image that could provide correct colors. To overcome the limitations, we improve the recent CNN based method and propose to combine pyramid processing with CNNs for colorization. At each level of the pyramid, our method warps the reference image using the estimated warping information map from the previous level so that we can learn a much more compact 3-D weight volume for colorization. We also compute an update to the warping information map by a Markov Random Field method at each level. With the pyramid CNN structure, our model has much smaller model size, and experimental results show that our method outperforms all of the state-of-the-art methods in accuracy as well.
MIEHDR CNN: Main Image Enhancement based Ghost-Free High Dynamic Range Imaging using Dual-Lens Systems
Authors: Xuan Dong, Xiaoyan Hu, Weixin Li, Xiaojie Wang;Yunhong Wang
Date: 2021-05-18 00:00:00
Proceeding: AAAI Technical Track on Computer Vision I 2021 5
Tasks: Denoising, Image Enhancement

We study the High Dynamic Range (HDR) imaging problem using two Low Dynamic Range (LDR) images that are shot from dual-lens systems in a single shot time with different exposures. In most of the related HDR imaging methods, the problem is usually solved by Multiple Images Merging, i.e. the final HDR image is fused from pixels of all the input LDR images. However, ghost artifacts can be hardly avoided using this strategy. Instead of directly merging the multiple LDR inputs, we use an indirect way which enhances the main image, i.e. the short exposure image IS, using the long exposure image IL serving as guidance. In detail, we propose a new model, named MIEHDR CNN model, which consists of three subnets, i.e. Soft Warp CNN, 3D Guided Denoising CNN and Fusion CNN. The Soft Warp CNN aligns IL to get the aligned result ILA using the soft exposed result of IS as reference. The 3D Guided Denoising CNN denoises the soft exposed result of IS using ILA as guidance, whose result are fed into the Fusion CNN with IS to get the HDR result. The MIEHDR CNN model is implemented by MindSpore and experimental results show that we can outperform related methods largely and avoid ghost artifacts.
Cycle-CNN for Colorization towards Real Monochrome-Color Camera Systems
Authors: Xuan Dong, Weixin Li, Xiaojie Wang, Yunhong Wang
Date: 2020-04-03 00:00:00
Proceeding: AAAI Technical Track: Vision 2020 4
Tasks: Colorization

Colorization in monochrome-color camera systems aims to colorize the gray image IG from the monochrome camera using the color image RC from the color camera as reference. Since monochrome cameras have better imaging quality than color cameras, the colorization can help obtain higher quality color images. Related learning based methods usually simulate the monochrome-color camera systems to generate the synthesized data for training, due to the lack of ground-truth color information of the gray image in the real data. However, the methods that are trained relying on the synthesized data may get poor results when colorizing real data, because the synthesized data may deviate from the real data. We present a new CNN model, named cycle CNN, which can directly use the real data from monochrome-color camera systems for training. In detail, we use the colorization CNN model to do the colorization twice. First, we colorize IG using RC as reference to obtain the first-time colorization result IC. Second, we colorize the de-colored map of RC, i.e. RG, using the first-time colorization result IC as reference to obtain the second-time colorization result R′C. In this way, for the second-time colorization result R′C, we use the original color map RC as ground-truth and introduce the cycle consistency loss to push R′C ≈ RC. Also, for the first-time colorization result IC, we propose a structure similarity loss to encourage the luminance maps between IG and IC to have similar structures. In addition, we introduce a spatial smoothness loss within the colorization CNN model to encourage spatial smoothness of the colorization result. Combining all these losses, we could train the colorization CNN model using the real data in the absence of the ground-truth color information of IG. Experimental results show that we can outperform related methods largely for colorizing real data.
Memory-aware framework for fast and scalable second-order random walk over billion-edge natural graphs
Authors: Yingxia Shao, Shiyue Huang, Yawen Li, Xupeng Miao, Bin Cui & Lei Chen
Date: 2021-05-07 00:00:00
Proceeding: The VLDB Journal 2021 5
Tasks: Community Detection, Graph Embedding

Second-order random walk is an important technique for graph analysis. Many applications including graph embedding, proximity measure and community detection use it to capture higher-order patterns in the graph, thus improving the model accuracy. However, the memory explosion problem of this technique hinders it from analyzing large graphs. When processing a billion-edge graph like Twitter, existing solutions (e.g., alias method) of the second-order random walk may take up 1796TB memory. Such high memory consumption comes from the memory-unaware strategies for the node sampling during the random walk. In this paper, to clearly compare the efficiency of various node sampling methods, we first design a cost model and propose two new node sampling methods: one follows the acceptance-rejection paradigm to achieve a better balance between memory and time cost, and the other is optimized for fast sampling the skewed probability distributions existed in natural graphs. Second, to achieve the high efficiency of the second-order random walk within arbitrary memory budgets, we propose a novel memory-aware framework on the basis of the cost model. The framework applies a cost-based optimizer to assign desirable node sampling method for each node or edge in the graph within a memory budget meanwhile minimizing the time cost of the random walk. Finally, the framework provides general programming interfaces for users to define new second-order random walk models easily. The empirical studies demonstrate that our memory-aware framework is robust with respect to memory and is able to achieve considerable efficiency by reducing 90% of the memory cost.
Drawing Order Recovery from Trajectory Components
Authors: Minghao Yang, Xukang Zhou, Yangchang Sun, Jinglong Chen, Baohua Qiang
Date: 2021-05-13 00:00:00
Proceeding: IEEE International Conference on Acoustics, Speech and Signal Processing 2021 5
Tasks: Decoder

In spite of widely discussed, drawing order recovery (DOR) from static images is still a great challenge task. Based on the idea that drawing trajectories are able to be recovered by connecting their trajectory components in correct orders, this work proposes a novel DOR method from static images. The method contains two steps: firstly, we adopt a convolution neural network (CNN) to predict the next possible drawing components, which is able to covert the components in images to their reasonable sequences. We denote this architecture as Im2Seq-CNN; secondly, considering possible errors exist in the reasonable sequences generated by the first step, we construct a sequence to order structure (Seq2Order) to adjust the sequences to the correct orders. The main contributions include: (1) the Img2Seq-CNN step considers DOR from components instead of traditional pixels one by one along trajectories, which contributes to static images to component sequences; (2) the Seq2Order step adopts image position codes instead of traditional points’ coordinates in its encoder-decoder gated recurrent neural network (GRU-RNN). The proposed method is experienced on two well-known open handwriting databases, and yields robust and competitive results on handwriting DOR tasks compared to the state-of-arts.
Optical flow and scene flow estimation: A survey
Authors: MingliangZhai, XuezhiXiang, NingLv, XiangdongKong
Date: 2021-02-01 00:00:00
Proceeding: Pattern Recognition 2021 2
Tasks: Action Recognition, Autonomous Driving, Deep Learning, Optical Flow Estimation, Scene Flow Estimation, Scene Understanding, Survey

Motion analysis is one of the most fundamental and challenging problems in the field of computer vision, which can be widely applied in many areas, such as autonomous driving, action recognition, scene understanding, and robotics. In general, the displacement field between subsequent frames can be divided into two types: optical flow and scene flow. The optical flow represents the pixel motion of adjacent frames. In contrast, the scene flow is a 3D motion field of the dynamic scene between two frames. Traditional approaches for the estimation of optical flow and scene flow usually leverage the variational technique, which can be solved as an energy minimization process. In recent years, deep learning has emerged as a powerful technique for learning feature representations directly from data. It has led to remarkable progress in the field of optical flow and scene flow estimation. In this paper, we provide a comprehensive survey of optical flow and scene flow estimation. First, we briefly review the pioneering approaches that use variational technique and then we delve in detail into the deep learning-based approaches. Furthermore, we present insightful observations on evaluation issues, specifically benchmark datasets, evaluation metrics, and state-of-the-art performance. Finally, we give the promising directions for future research. To the best of our knowledge, we are the first to review both optical flow and scene flow estimation, and the first to cover both traditional and deep learning-based approaches.
Multi-object Tracking Method Based on Efficient Channel Attention and Switchable Atrous Convolution
Authors: Xuezhi Xiang, Wenkai Ren, Yujian Qiu, Kaixu Zhang & Ning Lv
Date: 2021-04-29 00:00:00
Proceeding: Neural Processing Letters 2021 4
Tasks: Multi-Object Tracking, Object, object-detection, Object Detection, Object Tracking

In recent years,object detection and data association have getting remarkable progress which are the core components for multi-object tracking. In multi-object tracking field,the main strategy is tracking-by-detection. Although the detection based tracking method can get great results, it is relies on the performance of the detector. In complex scene, detector can not provide reliable results. Moreover,due to the incorrect detection results, data association process can not be trusted. Based on this motivation, this paper focuses on improving the accuracy of detection and data association. We introduce the efficient channel attention module to the backbone network, which can adaptively extract important information in images. Furthermore, we apply switchable atrous convolution in the network to dynamically adjust the receptive field according to object changes. In data association process, the appearance features with minimum occlusion are saved for each existing trajectory, which are used for re-associate after the objects are lost. Extensive experiments on MOT16,MOT17 and MOT20 challenging datasets demonstrate that our method is comparable with the state-of-the-art multi-object tracking methods.
Stable and Effective One-Step Method for Person Search
Authors: Ning Lv, Xuezhi Xiang, Xinyao Wang, Jie Yang, Rokia Abdeen, Abdulmotaleb El Saddik
Date: 2021-05-13 00:00:00
Proceeding: IEEE International Conference on Acoustics, Speech and Signal Processing 2021 5
Tasks: Multi-Task Learning, Pedestrian Detection, Person Re-Identification, Person Search, Region Proposal

Person search, which requires both pedestrian detection and person re-identification, is a challenging computer vision task applied to real-world scenarios. The challenges faced by detection and re-identification, such as occlusion, poor illumination, confusing background, are still urgent for person search. In addition, one-step methods for person search need to deal with the divergence between two tasks. In this work, we propose an end-to-end model containing the feature extractor, the region proposal network, and the multi-task learning module. In order to process divergence between detection and re-identification, we introduce switchable normalization and gradient centralization to improve the stability of the model. To solve the imbalance problem of hard examples, we introduce focal loss as a classification loss in the multi-task learning module. The experimental results on two bench-marks, i.e., CUHK-SYSU and PRW, well demonstrate that our method outperforms the state-of-the-art one-step methods.
Harmonic Feature Activation for Few-Shot Semantic Segmentation
Authors: Binghao Liu, Jianbin Jiao, Qixiang Ye
Date: 2021-02-17 00:00:00
Proceeding: IEEE Transactions on Image Processing 2021 2
Tasks: Few-Shot Semantic Segmentation, Segmentation, Semantic Segmentation

Few-shot semantic segmentation remains an open problem because limited support (training) images are insufficient to represent the diverse semantics within target categories. Conventional methods typically model a target category solely using information from the support image(s), resulting in incomplete semantic activation. In this paper, we propose a novel few-shot segmentation approach, termed harmonic feature activation (HFA), with the aim to implement dense support-to-query semantic transform by incorporating the features of both query and support images. HFA is formulated as a bilinear model, which takes charge of the pixel-wise dense correlation (bilinear feature activation) between query and support images in a systematic way. HFA incorporates a low-rank decomposition procedure, which speeds up bilinear feature activation with negligible performance cost. In addition, a semantic diffusion procedure is fused with HFA, which further improves the global harmony and local consistency of the feature activation. Extensive experiments on commonly used datasets (PASCAL VOC and MS COCO) show that HFA improves the state-of-the-arts with significant margins. Code is available at https://github.com/Bibikiller/HFA.
Person Re-Identification by Context-aware Part Attention and Multi-Head Collaborative Learning
Authors: Dongming Wu, Mang Ye, Gaojie Lin, Xin Gao, Jianbing Shen
Date: 2021-04-26 00:00:00
Proceeding: IEEE TRANSACTIONS ON INFORMATION FORENSICS AND SECURITY 2021 4
Tasks: Person Re-Identification, Video-Based Person Re-Identification

Most existing works solve the video-based person re-identification (re-ID) problem by computing the representation of each frame independently and finally aggregate the frame-level features. However, these methods often suffer from the challenging factors in videos, such as serious occlusion, background clutter and pose variation. To address these issues, we propose a novel multi-level Context-aware Part Attention (CPA) model to learn discriminative and robust local part features. It is featured in two aspects: 1) the context-aware part attention module improves the robustness by capturing the global relationship among different body parts across different video frames, and 2) the attention module is further extended to multi-level attention mechanism which enhances the discriminability by simultaneously considering low- to high-level features in different convolutional layers. In addition, we propose a novel multi-head collaborative training scheme to improve the performance, which is collaboratively supervised by multiple heads with the same structure but different parameters. It contains two consistency regularization terms, which consider both multi-head and multi-frame consistency to achieve better results. The multi-level CPA model is designed for feature extraction, while the multi-head collaborative training scheme is designed for classifier supervision. They jointly improve our re-ID model from two complementary directions. Extensive experiments demonstrate that the proposed method achieves much better or at least comparable performance compared to the state-of-the-art on four video re-ID datasets.
Hyperspectral Band Selection via Spatial-Spectral Weighted Region-wise Multiple Graph Fusion-Based Spectral Clustering
Authors: Chang Tang, Xinwang Liu, En Zhu, Lizhe Wang, Albert Zomaya
Date: 2021-04-18 00:00:00
Proceeding: International Joint Conferences on Artificial Intelligence Organization 2021 4
Tasks: Clustering, Superpixels

In this paper, we propose a hyperspectral band selection method via spatial-spectral weighted region-wise multiple graph fusion-based spectral clustering, referred to as RMGF briefly. Considering that different objects have different reflection characteristics, we use a superpixel segmentation algorithm to segment the first principal component of original hyperspectral image cube into homogeneous regions. For each superpixel, we construct a corresponding similarity graph to reflect the similarity between band pairs. Then, a multiple graph diffusion strategy with theoretical convergence guarantee is designed to learn a unified graph for partitioning the whole hyperspectral cube into several subcubes via spectral clustering. During the graph diffusion process, the spatial and spectral information of each superpixel are embedded to make spatial/spectral similar superpixels contribute more to each other. Finally, the band containing minimum noise in each subcube is selected to represent the whole subcube. Extensive experiments are conducted on three public datasets to validate the superiority of the proposed method when compared with other state-of-the-art ones.
Tensor-Based Multi-View Block-Diagonal Structure Diffusion for Clustering Incomplete Multi-View Data
Authors: Zhenglai Li, Chang Tang, Xinwang Liu, Xiao Zheng, Wei zhang, En Zhu
Date: 2021-06-09 00:00:00
Proceeding: IEEE International Conference on Multimedia and Expo 2021 6
Tasks: Clustering, Incomplete multi-view clustering

In this paper, we propose a novel incomplete multi-view clustering method, in which a tensor nuclear norm regularizer elegantly diffuses the information of multi-view block-diagonal structure across different views. By exploring the membership between observed and missing samples and that between missing ones in each incomplete view with the guidance of the high-order view consistency, a global block-diagonal structure is well preserved in multiple spectral embedding matrices. Meanwhile, a consensus representation with strong separability is obtained for clustering. An iterative algorithm based on Augmented Lagrange Multiplier (ALM) is designed to solve the resultant model. Experimental results on six benchmark datasets indicate the superiority of the proposed method.
Consensus Graph Learning for Multi-view Clustering
Authors: Zhenglai Li, Chang Tang, Xinwang Liu, Xiao Zheng, Guanghui Yue, Wei zhang
Date: 2021-05-21 00:00:00
Proceeding: IEEE Transactions on Multimedia 2021 5
Tasks: Clustering, Graph Learning

Multi-view clustering, which exploits the multi-view information to partition data into their clusters, has attracted intense attention. However, most existing methods directly learn a similarity graph from original multi-view features, which inevitably contain noises and redundancy information. The learned similarity graph is inaccurate and is insufficient to depict the underlying cluster structure of multi-view data. To address this issue, we propose a novel multi-view clustering method that is able to construct an essential similarity graph in a spectral embedding space instead of the original feature space. Concretely, we first obtain multiple spectral embedding matrices from the view-specific similarity graphs, and reorganize the gram matrices constructed by the inner product of the normalized spectral embedding matrices into a tensor form. Then, we impose a weighted tensor nuclear norm constraint on the tensor to capture highorder consistent information among multiple views. Furthermore, we unify the spectral embedding and low rank tensor learning into a unified optimization framework to determine the spectral embedding matrices and tensor representation jointly. Finally, we obtain the consensus similarity graph from the gram matrices via an adaptive neighbor manner. An efficient optimization algorithm is designed to solve the resultant optimization problem. Extensive experiments on six benchmark datasets are conducted to verify the efficacy of the proposed method.
Toward Tweet Entity Linking with Heterogeneous Information Networks
Authors: Wei Shen, Yuwei Yin, Yang Yang, Jiawei Han, Jianyong Wang, Xiaojie Yuan
Date: 2021-03-23 00:00:00
Proceeding: IEEE Transactions on Knowledge and Data Engineering 2021 3
Tasks: Entity Linking, Metric Learning

Twitter, a microblogging platform, has developed into an increasingly invaluable information source, where millions of users post a great quantity of tweets with various topics per day. Heterogeneous information networks consisting of multi-type objects and relations are becoming more and more prevalent as an organization form of knowledge and information. The task of linking an entity mention in a tweet with its corresponding entity in a heterogeneous information network is of great importance, for the purpose of enriching heterogeneous information networks with the abundant and fresh knowledge embedded in tweets. However, the entity mention is ambiguous. Additionally, tweets are short and informal, making it difficult to mine enough information from a single tweet for entity linking. In this paper, we propose an unsupervised iterative clustering framework TELHIN to link multiple similar tweets with a heterogeneous information network jointly. Our framework takes three dimensions of tweet similarity into consideration: (1) content similarity, (2) temporal similarity, and (3) user similarity. The appropriate weights of different similarity dimensions for each entity mention are learned iteratively based on the metric learning algorithm by leveraging the pairwise constraints generated automatically. Experiments on real data demonstrate the effectiveness of our framework in comparison with the baselines.
Image Desaturation for SDO/AIA Using Deep Learning
Authors: Xuexin Yu, Long Xu & Yihua Yan
Date: 2021-03-26 00:00:00
Proceeding: Solar Physics 2021 3
Tasks: Deep Learning, Generative Adversarial Network, Image Inpainting, Image Restoration, valid

The Atmospheric Imaging Assembly (AIA) on board the Solar Dynamics Observatory (SDO) (launched in February 2010) provides uninterrupted full-disk solar images over 10 wavebands. In the case of violent solar flares, saturation would happen to SDO/AIA images in their core regions, which leads to signal loss, hindering us to understand physical mechanism behind solar flares. This paper introduces a deep learning based image restoration model which can recover signal of saturation region by referring to other normal/valid region within an image. The proposed model, namely PCGAN, combines partial convolution (PC) and conditional generative adversarial network (GAN). The PC module was originally designed for image inpainting, for repairing images with scratches and holes. In addition, a new comprehensive loss function consists of an adversarial loss, a pixel reconstruction loss, a gradient loss, a perceptual loss, a style loss and a total variation loss. Moreover, for validating the proposed model, a new dataset consisting of paired saturated and normal SDO/AIA images is established. Experimental results demonstrate that the proposed PCGAN can get appealing desaturated solar images with respect to both objective and subjective validations.