pwc-archive/papers-with-abstracts huggingface dataset:
Internet of Everything enabled solution for COVID-19, its new variants and future pandemics: Framework, Challenges, and Research Directions
Authors: Sunder Ali Khowaja, Parus Khuwaja, Kapal Dev
Date: 2021-01-02 00:00:00

After affecting the world in unexpected ways, COVID-19 has started mutating which is evident with the insurgence of its new variants. The governments, hospitals, schools, industries, and humans, in general, are looking for a potential solution in the vaccine which will eventually be available but its timeline for eradicating the virus is yet unknown. Several researchers have encouraged and recommended the use of good practices such as physical healthcare monitoring, immunity-boosting, personal hygiene, mental healthcare, and contact tracing for slowing down the spread of the virus. In this article, we propose the use of wearable/mobile sensors integrated with the Internet of Everything to cover the spectrum of good practices in an automated manner. We present hypothetical frameworks for each of the good practice modules and propose the COvid-19 Resistance Framework using the Internet of Everything (CORFIE) to tie all the individual modules in a unified architecture. We envision that CORFIE would be influential in assisting people with the new normal for current and future pandemics as well as instrumental in halting the economic losses, respectively. We also provide potential challenges and their probable solutions in compliance with the proposed CORFIE.
Socially Responsible AI Algorithms: Issues, Purposes, and Challenges
Authors: Lu Cheng, Kush R. Varshney, Huan Liu
Date: 2021-01-01 00:00:00
Tasks: Fairness

In the current era, people and society have grown increasingly reliant on artificial intelligence (AI) technologies. AI has the potential to drive us towards a future in which all of humanity flourishes. It also comes with substantial risks for oppression and calamity. Discussions about whether we should (re)trust AI have repeatedly emerged in recent years and in many quarters, including industry, academia, healthcare, services, and so on. Technologists and AI researchers have a responsibility to develop trustworthy AI systems. They have responded with great effort to design more responsible AI algorithms. However, existing technical solutions are narrow in scope and have been primarily directed towards algorithms for scoring or classification tasks, with an emphasis on fairness and unwanted bias. To build long-lasting trust between AI and human beings, we argue that the key is to think beyond algorithmic fairness and connect major aspects of AI that potentially cause AI's indifferent behavior. In this survey, we provide a systematic framework of Socially Responsible AI Algorithms that aims to examine the subjects of AI indifference and the need for socially responsible AI algorithms, define the objectives, and introduce the means by which we may achieve these objectives. We further discuss how to leverage this framework to improve societal well-being through protection, information, and prevention/mitigation.
Deep Reinforcement Learning with Quantum-inspired Experience Replay
Authors: Qing Wei, Hailan Ma, Chunlin Chen, Daoyi Dong
Date: 2021-01-06 00:00:00
Tasks: Atari Games, Deep Reinforcement Learning, Diversity, reinforcement-learning, Reinforcement Learning, Reinforcement Learning (RL)

In this paper, a novel training paradigm inspired by quantum computation is proposed for deep reinforcement learning (DRL) with experience replay. In contrast to traditional experience replay mechanism in DRL, the proposed deep reinforcement learning with quantum-inspired experience replay (DRL-QER) adaptively chooses experiences from the replay buffer according to the complexity and the replayed times of each experience (also called transition), to achieve a balance between exploration and exploitation. In DRL-QER, transitions are first formulated in quantum representations, and then the preparation operation and the depreciation operation are performed on the transitions. In this progress, the preparation operation reflects the relationship between the temporal difference errors (TD-errors) and the importance of the experiences, while the depreciation operation is taken into account to ensure the diversity of the transitions. The experimental results on Atari 2600 games show that DRL-QER outperforms state-of-the-art algorithms such as DRL-PER and DCRL on most of these games with improved training efficiency, and is also applicable to such memory-based DRL approaches as double network and dueling network.
CNN-based Ego-Motion Estimation for Fast MAV Maneuvers
Authors: Yingfu Xu, Guido C. H. E. de Croon
Date: 2021-01-06 00:00:00
Tasks: GPU, Motion Estimation, Self-Supervised Learning

In the field of visual ego-motion estimation for Micro Air Vehicles (MAVs), fast maneuvers stay challenging mainly because of the big visual disparity and motion blur. In the pursuit of higher robustness, we study convolutional neural networks (CNNs) that predict the relative pose between subsequent images from a fast-moving monocular camera facing a planar scene. Aided by the Inertial Measurement Unit (IMU), we mainly focus on translational motion. The networks we study have similar small model sizes (around 1.35MB) and high inference speeds (around 10 milliseconds on a mobile GPU). Images for training and testing have realistic motion blur. Departing from a network framework that iteratively warps the first image to match the second with cascaded network blocks, we study different network architectures and training strategies. Simulated datasets and a self-collected MAV flight dataset are used for evaluation. The proposed setup shows better accuracy over existing networks and traditional feature-point-based methods during fast maneuvers. Moreover, self-supervised learning outperforms supervised learning. Videos and open-sourced code are available at https://github.com/tudelft/PoseNet_Planar
ISETAuto: Detecting vehicles with depth and radiance information
Authors: Zhenyi Liu, Joyce Farrell, Brian Wandell
Date: 2021-01-06 00:00:00
Tasks: Autonomous Driving, vehicle detection

Autonomous driving applications use two types of sensor systems to identify vehicles - depth sensing LiDAR and radiance sensing cameras. We compare the performance (average precision) of a ResNet for vehicle detection in complex, daytime, driving scenes when the input is a depth map (D = d(x,y)), a radiance image (L = r(x,y)), or both [D,L]. (1) When the spatial sampling resolution of the depth map and radiance image are equal to typical camera resolutions, a ResNet detects vehicles at higher average precision from depth than radiance. (2) As the spatial sampling of the depth map declines to the range of current LiDAR devices, the ResNet average precision is higher for radiance than depth. (3) For a hybrid system that combines a depth map and radiance image, the average precision is higher than using depth or radiance alone. We established these observations in simulation and then confirmed them using realworld data. The advantage of combining depth and radiance can be explained by noting that the two type of information have complementary weaknesses. The radiance data are limited by dynamic range and motion blur. The LiDAR data have relatively low spatial resolution. The ResNet combines the two data sources effectively to improve overall vehicle detection.
Mesh Reconstruction from Aerial Images for Outdoor Terrain Mapping Using Joint 2D-3D Learning
Authors: Qiaojun Feng, Nikolay Atanasov
Date: 2021-01-06 00:00:00
Tasks: Depth Estimation

This paper addresses outdoor terrain mapping using overhead images obtained from an unmanned aerial vehicle. Dense depth estimation from aerial images during flight is challenging. While feature-based localization and mapping techniques can deliver real-time odometry and sparse points reconstruction, a dense environment model is generally recovered offline with significant computation and storage. This paper develops a joint 2D-3D learning approach to reconstruct local meshes at each camera keyframe, which can be assembled into a global environment model. Each local mesh is initialized from sparse depth measurements. We associate image features with the mesh vertices through camera projection and apply graph convolution to refine the mesh vertices based on joint 2-D reprojected depth and 3-D mesh supervision. Quantitative and qualitative evaluations using real aerial images show the potential of our method to support environmental monitoring and surveillance applications.
TGCN: Time Domain Graph Convolutional Network for Multiple Objects Tracking
Authors: Jie Zhang
Date: 2021-01-06 00:00:00
Tasks: Multiple Object Tracking, Object, Object Tracking

Multiple object tracking is to give each object an id in the video. The difficulty is how to match the predicted objects and detected objects in same frames. Matching features include appearance features, location features, etc. These features of the predicted object are basically based on some previous frames. However, few papers describe the relationship in the time domain between the previous frame features and the current frame features.In this paper, we proposed a time domain graph convolutional network for multiple objects tracking.The model is mainly divided into two parts, we first use convolutional neural network (CNN) to extract pedestrian appearance feature, which is a normal operation processing image in deep learning, then we use GCN to model some past frames' appearance feature to get the prediction appearance feature of the current frame. Due to this extension, we can get the pose features of the current frame according to the relationship between some frames in the past. Experimental evaluation shows that our extensions improve the MOTA by 1.3 on the MOT16, achieving overall competitive performance at high frame rates.
Multi-Stage Residual Hiding for Image-into-Audio Steganography
Authors: Wenxue Cui, Shaohui Liu, Feng Jiang, Yongliang Liu, Debin Zhao
Date: 2021-01-06 00:00:00

The widespread application of audio communication technologies has speeded up audio data flowing across the Internet, which made it a popular carrier for covert communication. In this paper, we present a cross-modal steganography method for hiding image content into audio carriers while preserving the perceptual fidelity of the cover audio. In our framework, two multi-stage networks are designed: the first network encodes the decreasing multilevel residual errors inside different audio subsequences with the corresponding stage sub-networks, while the second network decodes the residual errors from the modified carrier with the corresponding stage sub-networks to produce the final revealed results. The multi-stage design of proposed framework not only make the controlling of payload capacity more flexible, but also make hiding easier because of the gradual sparse characteristic of residual errors. Qualitative experiments suggest that modifications to the carrier are unnoticeable by human listeners and that the decoded images are highly intelligible.
Smile and Laugh Expressions Detection Based on Local Minimum Key Points
Authors: Mina Mohammadi Dashti, Majid Harouni
Date: 2021-01-06 00:00:00
Tasks: Dimensionality Reduction, Texture Classification

In this paper, a smile and laugh facial expression is presented based on dimension reduction and description process of the key points. The paper has two main objectives; the first is to extract the local critical points in terms of their apparent features, and the second is to reduce the system's dependence on training inputs. To achieve these objectives, three different scenarios on extracting the features are proposed. First of all, the discrete parts of a face are detected by local binary pattern method that is used to extract a set of global feature vectors for texture classification considering various regions of an input-image face. Then, in the first scenario and with respect to the correlation changes of adjacent pixels on the texture of a mouth area, a set of local key points are extracted using the Harris corner detector. In the second scenario, the dimension reduction of the extracted points of first scenario provided by principal component analysis algorithm leading to reduction in computational costs and overall complexity without loss of performance and flexibility, etc.
On-Device Document Classification using multimodal features
Authors: Sugam Garg, Harichandana, Sumit Kumar
Date: 2021-01-06 00:00:00
Tasks: Classification, Document Classification, General Classification, Model Compression, Optical Character Recognition, Optical Character Recognition (OCR)

From small screenshots to large videos, documents take up a bulk of space in a modern smartphone. Documents in a phone can accumulate from various sources, and with the high storage capacity of mobiles, hundreds of documents are accumulated in a short period. However, searching or managing documents remains an onerous task, since most search methods depend on meta-information or only text in a document. In this paper, we showcase that a single modality is insufficient for classification and present a novel pipeline to classify documents on-device, thus preventing any private user data transfer to server. For this task, we integrate an open-source library for Optical Character Recognition (OCR) and our novel model architecture in the pipeline. We optimise the model for size, a necessary metric for on-device inference. We benchmark our classification model with a standard multimodal dataset FOOD-101 and showcase competitive results with the previous State of the Art with 30% model compression.
MSD: Saliency-aware Knowledge Distillation for Multimodal Understanding
Authors: Woojeong Jin, Maziar Sanjabi, Shaoliang Nie, Liang Tan, Xiang Ren, Hamed Firooz
Date: 2021-01-06 00:00:00
Proceeding: Findings (EMNLP) 2021 11
Tasks: Knowledge Distillation, Meta-Learning

To reduce a model size but retain performance, we often rely on knowledge distillation (KD) which transfers knowledge from a large "teacher" model to a smaller "student" model. However, KD on multimodal datasets such as vision-language tasks is relatively unexplored, and digesting multimodal information is challenging since different modalities present different types of information. In this paper, we perform a large-scale empirical study to investigate the importance and effects of each modality in knowledge distillation. Furthermore, we introduce a multimodal knowledge distillation framework, modality-specific distillation (MSD), to transfer knowledge from a teacher on multimodal tasks by learning the teacher's behavior within each modality. The idea aims at mimicking a teacher's modality-specific predictions by introducing auxiliary loss terms for each modality. Furthermore, because each modality has different saliency for predictions, we define saliency scores for each modality and investigate saliency-based weighting schemes for the auxiliary losses. We further study a weight learning approach to learn the optimal weights on these loss terms. In our empirical analysis, we examine the saliency of each modality in KD, demonstrate the effectiveness of the weighting scheme in MSD, and show that it achieves better performance than KD on four multimodal datasets.
Line Segment Detection Using Transformers without Edges
Authors: Yifan Xu, Weijian Xu, David Cheung, Zhuowen Tu
Date: 2021-01-06 00:00:00
Proceeding: CVPR 2021 1
Tasks: Decoder, Line Segment Detection, Multi-Task Learning

In this paper, we present a joint end-to-end line segment detection algorithm using Transformers that is post-processing and heuristics-guided intermediate processing (edge/junction/region detection) free. Our method, named LinE segment TRansformers (LETR), takes advantages of having integrated tokenized queries, a self-attention mechanism, and an encoding-decoding strategy within Transformers by skipping standard heuristic designs for the edge element detection and perceptual grouping processes. We equip Transformers with a multi-scale encoder/decoder strategy to perform fine-grained line segment detection under a direct endpoint distance loss. This loss term is particularly suitable for detecting geometric structures such as line segments that are not conveniently represented by the standard bounding box representations. The Transformers learn to gradually refine line segments through layers of self-attention. In our experiments, we show state-of-the-art results on Wireframe and YorkUrban benchmarks.
Predicting Forest Fire Using Remote Sensing Data And Machine Learning
Authors: Suwei Yang, Massimo Lupascu, Kuldeep S. Meel
Date: 2021-01-06 00:00:00
Tasks: BIG-bench Machine Learning, Prediction

Over the last few decades, deforestation and climate change have caused increasing number of forest fires. In Southeast Asia, Indonesia has been the most affected country by tropical peatland forest fires. These fires have a significant impact on the climate resulting in extensive health, social and economic issues. Existing forest fire prediction systems, such as the Canadian Forest Fire Danger Rating System, are based on handcrafted features and require installation and maintenance of expensive instruments on the ground, which can be a challenge for developing countries such as Indonesia. We propose a novel, cost-effective, machine-learning based approach that uses remote sensing data to predict forest fires in Indonesia. Our prediction model achieves more than 0.81 area under the receiver operator characteristic (ROC) curve, performing significantly better than the baseline approach which never exceeds 0.70 area under ROC curve on the same tasks. Our model's performance remained above 0.81 area under ROC curve even when evaluated with reduced data. The results support our claim that machine-learning based approaches can lead to reliable and cost-effective forest fire prediction systems.
Multi-object Tracking with a Hierarchical Single-branch Network
Authors: Fan Wang, Lei Luo, En Zhu, Siwei Wang, Jun Long
Date: 2021-01-06 00:00:00
Tasks: Multi-Object Tracking, Multiple Object Tracking, Object, object-detection, Object Detection, Object Tracking, Online Multi-Object Tracking

Recent Multiple Object Tracking (MOT) methods have gradually attempted to integrate object detection and instance re-identification (Re-ID) into a united network to form a one-stage solution. Typically, these methods use two separated branches within a single network to accomplish detection and Re-ID respectively without studying the inter-relationship between them, which inevitably impedes the tracking performance. In this paper, we propose an online multi-object tracking framework based on a hierarchical single-branch network to solve this problem. Specifically, the proposed single-branch network utilizes an improved Hierarchical Online In-stance Matching (iHOIM) loss to explicitly model the inter-relationship between object detection and Re-ID. Our novel iHOIM loss function unifies the objectives of the two sub-tasks and encourages better detection performance and feature learning even in extremely crowded scenes. Moreover, we propose to introduce the object positions, predicted by a motion model, as region proposals for subsequent object detection, where the intuition is that detection results and motion predictions can complement each other in different scenarios. Experimental results on MOT16 and MOT20 datasets show that we can achieve state-of-the-art tracking performance, and the ablation study verifies the effectiveness of each proposed component.
Weakly-Supervised Multi-Face 3D Reconstruction
Authors: Jialiang Zhang, Lixiang Lin, Jianke Zhu, Steven C. H. Hoi
Date: 2021-01-06 00:00:00
Tasks: 3D Face Reconstruction, 3D Reconstruction, Face Alignment, Face Model, Face Reconstruction, Person Identification

3D face reconstruction plays a very important role in many real-world multimedia applications, including digital entertainment, social media, affection analysis, and person identification. The de-facto pipeline for estimating the parametric face model from an image requires to firstly detect the facial regions with landmarks, and then crop each face to feed the deep learning-based regressor. Comparing to the conventional methods performing forward inference for each detected instance independently, we suggest an effective end-to-end framework for multi-face 3D reconstruction, which is able to predict the model parameters of multiple instances simultaneously using single network inference. Our proposed approach not only greatly reduces the computational redundancy in feature extraction but also makes the deployment procedure much easier using the single network model. More importantly, we employ the same global camera model for the reconstructed faces in each image, which makes it possible to recover the relative head positions and orientations in the 3D scene. We have conducted extensive experiments to evaluate our proposed approach on the sparse and dense face alignment tasks. The experimental results indicate that our proposed approach is very promising on face alignment tasks without fully-supervision and pre-processing like detection and crop. Our implementation is publicly available at \url{https://github.com/kalyo-zjl/WM3DR}.
Unified Learning Approach for Egocentric Hand Gesture Recognition and Fingertip Detection
Authors: Mohammad Mahmudul Alam, Mohammad Tariqul Islam, S. M. Mahbubur Rahman
Date: 2021-01-06 00:00:00
Tasks: Fingertip Detection, Gesture Recognition, Hand Gesture Recognition, Position

Head-mounted device-based human-computer interaction often requires egocentric recognition of hand gestures and fingertips detection. In this paper, a unified approach of egocentric hand gesture recognition and fingertip detection is introduced. The proposed algorithm uses a single convolutional neural network to predict the probabilities of finger class and positions of fingertips in one forward propagation. Instead of directly regressing the positions of fingertips from the fully connected layer, the ensemble of the position of fingertips is regressed from the fully convolutional network. Subsequently, the ensemble average is taken to regress the final position of fingertips. Since the whole pipeline uses a single network, it is significantly fast in computation. Experimental results show that the proposed method outperforms the existing fingertip detection approaches including the Direct Regression and the Heatmap-based framework. The effectiveness of the proposed method is also shown in-the-wild scenario as well as in a use-case of virtual reality.
Shallow-UWnet : Compressed Model for Underwater Image Enhancement
Authors: Ankita Naik, Apurva Swarnakar, Kartik Mittal
Date: 2021-01-06 00:00:00
Tasks: Benchmarking, Image Enhancement, model

Over the past few decades, underwater image enhancement has attracted increasing amount of research effort due to its significance in underwater robotics and ocean engineering. Research has evolved from implementing physics-based solutions to using very deep CNNs and GANs. However, these state-of-art algorithms are computationally expensive and memory intensive. This hinders their deployment on portable devices for underwater exploration tasks. These models are trained on either synthetic or limited real world datasets making them less practical in real-world scenarios. In this paper we propose a shallow neural network architecture, \textbf{Shallow-UWnet} which maintains performance and has fewer parameters than the state-of-art models. We also demonstrated the generalization of our model by benchmarking its performance on combination of synthetic and real-world datasets.