pwc-archive/papers-with-abstracts huggingface dataset:
FExGAN-Meta: Facial Expression Generation with Meta Humans
Authors: J. Rafid Siddiqui
Date: 2022-02-17 00:00:00
Tasks: Facial expression generation

The subtleness of human facial expressions and a large degree of variation in the level of intensity to which a human expresses them is what makes it challenging to robustly classify and generate images of facial expressions. Lack of good quality data can hinder the performance of a deep learning model. In this article, we have proposed a Facial Expression Generation method for Meta-Humans (FExGAN-Meta) that works robustly with the images of Meta-Humans. We have prepared a large dataset of facial expressions exhibited by ten Meta-Humans when placed in a studio environment and then we have evaluated FExGAN-Meta on the collected images. The results show that FExGAN-Meta robustly generates and classifies the images of Meta-Humans for the simple as well as the complex facial expressions.
Contrastively-reinforced Attention Convolutional Neural Network for Fine-grained Image Recognition
Authors: Dichao Liu, Yu Wang, Jien Kato, Kenji Mase
Date: 2020-09-08 00:00:00
Proceeding: BMVC 2020 9
Tasks: Classification, Fine-Grained Image Classification, Fine-Grained Image Recognition

Fine-grained visual classification is inherently challenging because of its inter-class similarity and intra-class variance. However, by contrasting the images with same/different labels, a human can instinctively notice that the key clues lie in certain objects while other objects are ignorable. Inspired by this, we propose Contrastively-reinforced Attention Convolutional Neural Network (CRA-CNN), which reinforces the attention awareness of deep activations. CRA-CNN mainly contains two parts: the classification stream and attention regularization stream. The former classifies the input image and simultaneously divides the visual information of the input into attention and redundancy. The latter evaluates the attention/redundancy proposal by classifying the attention and contrasting the attention/redundancy of various inputs. The evaluation information is backpropagated and forces the classification stream to improve its awareness of visual attention, which helps classification. Experimental results on CUB-Birds and Stanford Cars show that CRA-CNN distinctly outperforms the baselines and is comparable with state-of-art studies despite its simplicity.
Real-Time Emotion Classification Using EEG Data Stream in E-Learning Contexts
Authors: Arijit Nandi, Fatos Xhafa, Laia Subirats, Santi Fort
Date: 2021-02-25 00:00:00
Proceeding: MDPI Sensors 2021 2
Tasks: Classification, EEG, Electroencephalogram (EEG), Emotional Intelligence, Emotion Classification

In face-to-face and online learning, emotions and emotional intelligence have an influence and play an essential role. Learners’ emotions are crucial for e-learning system because they promote or restrain the learning. Many researchers have investigated the impacts of emotions in enhancing and maximizing e-learning outcomes. Several machine learning and deep learning approaches have also been proposed to achieve this goal. All such approaches are suitable for an offline mode, where the data for emotion classification are stored and can be accessed infinitely. However, these offline mode approaches are inappropriate for real-time emotion classification when the data are coming in a continuous stream and data can be seen to the model at once only. We also need real-time responses according to the emotional state. For this, we propose a real-time emotion classification system (RECS)-based Logistic Regression (LR) trained in an online fashion using the Stochastic Gradient Descent (SGD) algorithm. The proposed RECS is capable of classifying emotions in real-time by training the model in an online fashion using an EEG signal stream. To validate the performance of RECS, we have used the DEAP data set, which is the most widely used benchmark data set for emotion classification. The results show that the proposed approach can effectively classify emotions in real-time from the EEG data stream, which achieved a better accuracy and F1-score than other offline and online approaches. The developed real-time emotion classification system is analyzed in an e-learning context scenario.
Multi-step dual control for exploration and exploitation in autonomous search with convergence guarantee
Authors: Yuan Tan, Jun Yang, Wen-Hua Chen, Shihua Li
Date: 2022-03-12 00:00:00
Tasks: Model Predictive Control

Motivated by the recently proposed dual control for exploration and exploitation (DCEE) concept, this paper presents a Multi-Step DCEE (MS-DCEE) framework with guaranteed convergence for autonomous search of a source of airborne dispersion. Different from the existing stochastic model predictive control (SMPC) algorithm and informative path planning (IPP) approaches, the proposed MS-DCEE approach uses the current and future input to not only drive the agent towards the estimated source location (exploitation) but also reduce its estimation uncertainty (exploration) by actively learning the operational environment. Unknown source target position, together with unknown environment, impose significant challenges in establishing the recursive feasibility and the convergence of the proposed algorithm. To address them, with the help of the property of Bayesian estimation, we develop a two-step approach where the unbiasedness of the mean estimation is assumed first and then the randomness of the mean estimate under each collected information sequence is accounted. Based on that, we develop a MS-DCEE scheme with suitable terminal ingredients where recursive feasibility and convergence are guaranteed. Two simulation scenarios are conducted, which show that the proposed MS-DCEE algorithm outperforms the SMPC, the IPP and the single-step DCEE approaches in terms of searching successful rates and efficiency.
Wasserstein Adversarial Transformer for Cloud Workload Prediction
Authors: Shivani Arbat, Vinodh Kumaran Jayakumar, Jaewoo Lee, Wei Wang, In Kee Kim
Date: 2022-03-12 00:00:00
Tasks: Prediction, Time Series, Time Series Analysis, Time Series Forecasting

Predictive Virtual Machine (VM) auto-scaling is a promising technique to optimize cloud applications operating costs and performance. Understanding the job arrival rate is crucial for accurately predicting future changes in cloud workloads and proactively provisioning and de-provisioning VMs for hosting the applications. However, developing a model that accurately predicts cloud workload changes is extremely challenging due to the dynamic nature of cloud workloads. Long-Short-Term-Memory (LSTM) models have been developed for cloud workload prediction. Unfortunately, the state-of-the-art LSTM model leverages recurrences to predict, which naturally adds complexity and increases the inference overhead as input sequences grow longer. To develop a cloud workload prediction model with high accuracy and low inference overhead, this work presents a novel time-series forecasting model called WGAN-gp Transformer, inspired by the Transformer network and improved Wasserstein-GANs. The proposed method adopts a Transformer network as a generator and a multi-layer perceptron as a critic. The extensive evaluations with real-world workload traces show WGAN-gp Transformer achieves 5 times faster inference time with up to 5.1 percent higher prediction accuracy against the state-of-the-art approach. We also apply WGAN-gp Transformer to auto-scaling mechanisms on Google cloud platforms, and the WGAN-gp Transformer-based auto-scaling mechanism outperforms the LSTM-based mechanism by significantly reducing VM over-provisioning and under-provisioning rates.
Diversifying Content Generation for Commonsense Reasoning with Mixture of Knowledge Graph Experts
Authors: Wenhao Yu, Chenguang Zhu, Lianhui Qin, Zhihan Zhang, Tong Zhao, Meng Jiang
Date: 2022-03-14 00:00:00
Proceeding: NAACL (DLG4NLP) 2022 7
Tasks: Diversity, Knowledge Graphs

Generative commonsense reasoning (GCR) in natural language is to reason about the commonsense while generating coherent text. Recent years have seen a surge of interest in improving the generation quality of commonsense reasoning tasks. Nevertheless, these approaches have seldom investigated diversity in the GCR tasks, which aims to generate alternative explanations for a real-world situation or predict all possible outcomes. Diversifying GCR is challenging as it expects to generate multiple outputs that are not only semantically different but also grounded in commonsense knowledge. In this paper, we propose MoKGE, a novel method that diversifies the generative reasoning by a mixture of expert (MoE) strategy on commonsense knowledge graphs (KG). A set of knowledge experts seek diverse reasoning on KG to encourage various generation outputs. Empirical experiments demonstrated that MoKGE can significantly improve the diversity while achieving on par performance on accuracy on two GCR benchmarks, based on both automatic and human evaluations.
MDMMT-2: Multidomain Multimodal Transformer for Video Retrieval, One More Step Towards Generalization
Authors: Alexander Kunitsyn, Maksim Kalashnikov, Maksim Dzabraev, Andrei Ivaniuta
Date: 2022-03-14 00:00:00
Tasks: Retrieval, Text to Video Retrieval, Video Retrieval

In this work we present a new State-of-The-Art on the text-to-video retrieval task on MSR-VTT, LSMDC, MSVD, YouCook2 and TGIF obtained by a single model. Three different data sources are combined: weakly-supervised videos, crowd-labeled text-image pairs and text-video pairs. A careful analysis of available pre-trained networks helps to choose the best prior-knowledge ones. We introduce three-stage training procedure that provides high transfer knowledge efficiency and allows to use noisy datasets during training without prior knowledge degradation. Additionally, double positional encoding is used for better fusion of different modalities and a simple method for non-square inputs processing is suggested.
One-stage Video Instance Segmentation: From Frame-in Frame-out to Clip-in Clip-out
Authors: Minghan Li, Lei Zhang
Date: 2022-03-12 00:00:00
Tasks: Instance Segmentation, Semantic Segmentation, Video Instance Segmentation

Many video instance segmentation (VIS) methods partition a video sequence into individual frames to detect and segment objects frame by frame. However, such a frame-in frame-out (FiFo) pipeline is ineffective to exploit the temporal information. Based on the fact that adjacent frames in a short clip are highly coherent in content, we propose to extend the one-stage FiFo framework to a clip-in clip-out (CiCo) one, which performs VIS clip by clip. Specifically, we stack FPN features of all frames in a short video clip to build a spatio-temporal feature cube, and replace the 2D conv layers in the prediction heads and the mask branch with 3D conv layers, forming clip-level prediction heads (CPH) and clip-level mask heads (CMH). Then the clip-level masks of an instance can be generated by feeding its box-level predictions from CPH and clip-level features from CMH into a small fully convolutional network. A clip-level segmentation loss is proposed to ensure that the generated instance masks are temporally coherent in the clip. The proposed CiCo strategy is free of inter-frame alignment, and can be easily embedded into existing FiFo based VIS approaches. To validate the generality and effectiveness of our CiCo strategy, we apply it to two representative FiFo methods, Yolact \cite{bolya2019yolact} and CondInst \cite{tian2020conditional}, resulting in two new one-stage VIS models, namely CiCo-Yolact and CiCo-CondInst, which achieve 37.1/37.3\%, 35.2/35.4\% and 17.2/18.0\% mask AP using the ResNet50 backbone, and 41.8/41.4\%, 38.0/38.9\% and 18.0/18.2\% mask AP using the Swin Transformer tiny backbone on YouTube-VIS 2019, 2021 and OVIS valid sets, respectively, recording new state-of-the-arts. Code and video demos of CiCo can be found at \url{https://github.com/MinghanLi/CiCo}.
Stubborn: A Strong Baseline for Indoor Object Navigation
Authors: Haokuan Luo, Albert Yue, Zhang-Wei Hong, Pulkit Agrawal
Date: 2022-03-14 00:00:00
Tasks: Object

We present a strong baseline that surpasses the performance of previously published methods on the Habitat Challenge task of navigating to a target object in indoor environments. Our method is motivated from primary failure modes of prior state-of-the-art: poor exploration, inaccurate object identification, and agent getting trapped due to imprecise map construction. We make three contributions to mitigate these issues: (i) First, we show that existing map-based methods fail to effectively use semantic clues for exploration. We present a semantic-agnostic exploration strategy (called Stubborn) without any learning that surprisingly outperforms prior work. (ii) We propose a strategy for integrating temporal information to improve object identification. (iii) Lastly, due to inaccurate depth observation the agent often gets trapped in small regions. We develop a multi-scale collision map for obstacle identification that mitigates this issue.
Federated Cycling (FedCy): Semi-supervised Federated Learning of Surgical Phases
Authors: Hasan Kassem, Deepak Alapatt, Pietro Mascagni, AI4SafeChole Consortium, Alexandros Karargyris, Nicolas Padoy
Date: 2022-03-14 00:00:00
Tasks: Federated Learning, Self-Supervised Learning, Surgical phase recognition

Recent advancements in deep learning methods bring computer-assistance a step closer to fulfilling promises of safer surgical procedures. However, the generalizability of such methods is often dependent on training on diverse datasets from multiple medical institutions, which is a restrictive requirement considering the sensitive nature of medical data. Recently proposed collaborative learning methods such as Federated Learning (FL) allow for training on remote datasets without the need to explicitly share data. Even so, data annotation still represents a bottleneck, particularly in medicine and surgery where clinical expertise is often required. With these constraints in mind, we propose FedCy, a federated semi-supervised learning (FSSL) method that combines FL and self-supervised learning to exploit a decentralized dataset of both labeled and unlabeled videos, thereby improving performance on the task of surgical phase recognition. By leveraging temporal patterns in the labeled data, FedCy helps guide unsupervised training on unlabeled data towards learning task-specific features for phase recognition. We demonstrate significant performance gains over state-of-the-art FSSL methods on the task of automatic recognition of surgical phases using a newly collected multi-institutional dataset of laparoscopic cholecystectomy videos. Furthermore, we demonstrate that our approach also learns more generalizable features when tested on data from an unseen domain.
Defending From Physically-Realizable Adversarial Attacks Through Internal Over-Activation Analysis
Authors: Giulio Rossolini, Federico Nesti, Fabio Brau, Alessandro Biondi, Giorgio Buttazzo
Date: 2022-03-14 00:00:00
Tasks: Adversarial Robustness, object-detection, Object Detection, Semantic Segmentation

This work presents Z-Mask, a robust and effective strategy to improve the adversarial robustness of convolutional networks against physically-realizable adversarial attacks. The presented defense relies on specific Z-score analysis performed on the internal network features to detect and mask the pixels corresponding to adversarial objects in the input image. To this end, spatially contiguous activations are examined in shallow and deep layers to suggest potential adversarial regions. Such proposals are then aggregated through a multi-thresholding mechanism. The effectiveness of Z-Mask is evaluated with an extensive set of experiments carried out on models for both semantic segmentation and object detection. The evaluation is performed with both digital patches added to the input images and printed patches positioned in the real world. The obtained results confirm that Z-Mask outperforms the state-of-the-art methods in terms of both detection accuracy and overall performance of the networks under attack. Additional experiments showed that Z-Mask is also robust against possible defense-aware attacks.
Mixed Evidence for Gestalt Grouping in Deep Neural Networks
Authors: Valerio Biscione, Jeffrey S. Bowers
Date: 2022-03-14 00:00:00
Tasks: Object Recognition

Gestalt psychologists have identified a range of conditions in which humans organize elements of a scene into a group or whole, and perceptual grouping principles play an essential role in scene perception and object identification. Recently, Deep Neural Networks (DNNs) trained on natural images (ImageNet) have been proposed as compelling models of human vision based on reports that they perform well on various brain and behavioral benchmarks. Here we test a total of 16 networks covering a variety of architectures and learning paradigms (convolutional, attention-based, supervised and self-supervised, feed-forward and recurrent) on dots (Experiment 1) and more complex shapes (Experiment 2) stimuli that produce strong Gestalts effects in humans. In Experiment 1 we found that convolutional networks were indeed sensitive in a human-like fashion to the principles of proximity, linearity, and orientation, but only at the output layer. In Experiment 2, we found that most networks exhibited Gestalt effects only for a few sets, and again only at the latest stage of processing. Overall, self-supervised and Vision-Transformer appeared to perform worse than convolutional networks in terms of human similarity. Remarkably, no model presented a grouping effect at the early or intermediate stages of processing. This is at odds with the widespread assumption that Gestalts occur prior to object recognition, and indeed, serve to organize the visual scene for the sake of object recognition. Our overall conclusion is that, albeit noteworthy that networks trained on simple 2D images support a form of Gestalt grouping for some stimuli at the output layer, this ability does not seem to transfer to more complex features. Additionally, the fact that this grouping only occurs at the last layer suggests that networks learn fundamentally different perceptual properties than humans.
GrIPS: Gradient-free, Edit-based Instruction Search for Prompting Large Language Models
Authors: Archiki Prasad, Peter Hase, Xiang Zhou, Mohit Bansal
Date: 2022-03-14 00:00:00

Providing natural language instructions in prompts is a useful new paradigm for improving task performance of large language models in a zero-shot setting. Recent work has aimed to improve such prompts via manual rewriting or gradient-based tuning. However, manual rewriting is time-consuming and requires subjective interpretation, while gradient-based tuning can be extremely computationally demanding for large models and may not be feasible for API-based models. In this work, we introduce Gradient-free Instructional Prompt Search (GrIPS), a gradient-free, edit-based search approach for improving task instructions for large language models. GrIPS takes in instructions designed for humans and automatically returns an improved, edited prompt, while allowing for API-based tuning. With InstructGPT models, GrIPS improves the average task performance by up to 4.30 percentage points on eight classification tasks from the Natural Instructions dataset (with similar improvements for OPT, BLOOM, and FLAN-T5). We see improvements for both instruction-only prompts and instruction + k-shot examples prompts. Notably, GrIPS outperforms manual rewriting and purely example-based prompts while controlling for the available compute and data budget. Further, performance of GrIPS is comparable to select gradient-based tuning approaches. Qualitatively, we show our edits can simplify instructions and at times make them incoherent but nonetheless improve accuracy. Our code is available at: https://github.com/archiki/GrIPS
On Cyclic Solutions to the Min-Max Latency Multi-Robot Patrolling Problem
Authors: Peyman Afshani, Mark De Berg, Kevin Buchin, Jie Gao, Maarten Loffler, Amir Nayyeri, Benjamin Raichel, Rik Sarkar, Haotian Wang, Hao-Tsung Yang
Date: 2022-03-14 00:00:00

We consider the following surveillance problem: Given a set $P$ of $n$ sites in a metric space and a set of $k$ robots with the same maximum speed, compute a patrol schedule of minimum latency for the robots. Here a patrol schedule specifies for each robot an infinite sequence of sites to visit (in the given order) and the latency $L$ of a schedule is the maximum latency of any site, where the latency of a site $s$ is the supremum of the lengths of the time intervals between consecutive visits to $s$. When $k=1$ the problem is equivalent to the travelling salesman problem (TSP) and thus it is NP-hard. We have two main results. We consider cyclic solutions in which the set of sites must be partitioned into $\ell$ groups, for some~$\ell \leq k$, and each group is assigned a subset of the robots that move along the travelling salesman tour of the group at equal distance from each other. Our first main result is that approximating the optimal latency of the class of cyclic solutions can be reduced to approximating the optimal travelling salesman tour on some input, with only a $1+\varepsilon$ factor loss in the approximation factor and an $O\left(\left( k/\varepsilon \right)^k\right)$ factor loss in the runtime, for any $\varepsilon >0$. Our second main result shows that an optimal cyclic solution is a $2(1-1/k)$-approximation of the overall optimal solution. Note that for $k=2$ this implies that an optimal cyclic solution is optimal overall. The results have a number of consequences. For the Euclidean version of the problem, for instance, combining our results with known results on Euclidean TSP, yields a PTAS for approximating an optimal cyclic solution, and it yields a $(2(1-1/k)+\varepsilon)$-approximation of the optimal unrestricted solution. If the conjecture mentioned above is true, then our algorithm is actually a PTAS for the general problem in the Euclidean setting.
Show Me More Details: Discovering Hierarchies of Procedures from Semi-structured Web Data
Authors: Shuyan Zhou, Li Zhang, Yue Yang, Qing Lyu, Pengcheng Yin, Chris Callison-Burch, Graham Neubig
Date: 2022-03-14 00:00:00
Proceeding: ACL 2022 5
Tasks: Articles, Retrieval, Video Retrieval

Procedures are inherently hierarchical. To "make videos", one may need to "purchase a camera", which in turn may require one to "set a budget". While such hierarchical knowledge is critical for reasoning about complex procedures, most existing work has treated procedures as shallow structures without modeling the parent-child relation. In this work, we attempt to construct an open-domain hierarchical knowledge-base (KB) of procedures based on wikiHow, a website containing more than 110k instructional articles, each documenting the steps to carry out a complex procedure. To this end, we develop a simple and efficient method that links steps (e.g., "purchase a camera") in an article to other articles with similar goals (e.g., "how to choose a camera"), recursively constructing the KB. Our method significantly outperforms several strong baselines according to automatic evaluation, human judgment, and application to downstream tasks such as instructional video retrieval. A demo with partial data can be found at https://wikihow-hierarchy.github.io. The code and the data are at https://github.com/shuyanzhou/wikihow_hierarchy.