pwc-archive/papers-with-abstracts huggingface dataset:
TEMOS: Generating diverse human motions from textual descriptions
Authors: Mathis Petrovich, Michael J. Black, Gül Varol
Date: 2022-04-25 00:00:00
Tasks: Motion Synthesis

We address the problem of generating diverse 3D human motions from textual descriptions. This challenging task requires joint modeling of both modalities: understanding and extracting useful human-centric information from the text, and then generating plausible and realistic sequences of human poses. In contrast to most previous work which focuses on generating a single, deterministic, motion from a textual description, we design a variational approach that can produce multiple diverse human motions. We propose TEMOS, a text-conditioned generative model leveraging variational autoencoder (VAE) training with human motion data, in combination with a text encoder that produces distribution parameters compatible with the VAE latent space. We show the TEMOS framework can produce both skeleton-based animations as in prior work, as well more expressive SMPL body motions. We evaluate our approach on the KIT Motion-Language benchmark and, despite being relatively straightforward, demonstrate significant improvements over the state of the art. Code and models are available on our webpage.
A Comparative Study of Meter Detection Methods for Automated Infrastructure Inspection
Authors: Yusuke Ohtsubo, Takuto Sato, Hirohiko Sagawa
Date: 2022-04-24 00:00:00

In order to read meter values from a camera on an autonomous inspection robot with positional errors, it is necessary to detect meter regions from the image. In this study, we developed shape-based, texture-based, and background information-based methods as meter area detection techniques and compared their effectiveness for meters of different shapes and sizes. As a result, we confirmed that the background information-based method can detect the farthest meters regardless of the shape and number of meters, and can stably detect meters with a diameter of 40px.
Avicenna: a challenge dataset for natural language generation toward commonsense syllogistic reasoning
Authors: Zeinab Aghahadi, Alireza Talebpour
Date: 2022-12-24 00:00:00
Proceeding: Journal of Applied Non-Classical Logics 2022 12
Tasks: Sentence, Text Generation, Transfer Learning

Syllogism is a type of everyday reasoning. For instance, given that "Avicenna wrote the famous book the Canon of Medicine " and " The Canon of Medicine has influenced modern medicine," it can be concluded that "Avicenna has influenced modern medicine." This study revolves around syllogistic natural language generation (NLG). The Avicenna corpus was developed as a benchmark for syllogistic NLG. In this respect, once the syllogistic relation between two premises is recognized, the Avicenna-trained models learn to generate the conclusion sentence (which is semantically unique). The experiments were performed using state-of-the-art pre-trained text generative models (TGMs). The state-of-the-art baseline was provided, and the accuracy was improved up to 32% when transfer learning was adopted. The models were evaluated using human and automatic procedures. The model’s confusion in detecting the middle-term (the duplicate part with the same meaning in the premises) was one of the main categories of errors that showed up in the error analysis.  This issue indicates that the model learns how to extract new facts based on the premises, but it faces a challenge in commonsense reasoning. The outcomes of the study demonstrated that although TGMs are significantly powerful, they do not yet have sufficient reasoning capabilities to generate text, based on commonsense knowledge. The Avicenna dataset poses a new challenge of commonsense inference that is easy for humans (98.1%) while difficult for state-of-the-art TGMs (32%).
Conditional β-VAE for De Novo Molecular Generation
Authors: Ryan J Richards, Austen M Groener
Date: 2022-05-02 00:00:00
Proceeding: ChemRxiv 2022 5
Tasks: Drug Discovery, Molecular Graph Generation, valid

Deep learning has significantly advanced and accelerated de novo molecular generation. Generative networks, namely Variational Autoencoders (VAEs) can not only randomly generate new molecules, but also alter molecular structures to optimize specific chemical properties which are pivotal for drug- discovery. While VAEs have been proposed and researched in the past for pharmaceutical applications, they possess deficiencies that limit their ability to both optimize properties and decode syntactically valid molecules. We present a recurrent, conditional β-VAE that disentangles the latent space to enhance post hoc molecule optimization. We create a mutual information driven training protocol and data augmentations to both increase molecular validity and promote longer sequence generation. We demonstrate the efficacy of our framework on the ZINC-250k dataset, achieving SOTA unconstrained optimization results on the penalized LogP (pLogP) and QED scores, while also matching current SOTA results for validity, novelty, and uniqueness scores for random generation. We match the current SOTA on QED for top-3 molecules at 0.948, while setting a new SOTA for pLogP optimization at 104.29, 90.12, 69.68 and demonstrating improved results on the constrained optimization task.
MemSeg: A semi-supervised method for image surface defect detection using differences and commonalities
Authors: Minghui Yang, Peng Wu, Jing Liu, Hui Feng
Date: 2022-05-02 00:00:00
Tasks: Anomaly Detection, Defect Detection, Robust classification

Under the semi-supervised framework, we propose an end-to-end memory-based segmentation network (MemSeg) to detect surface defects on industrial products. Considering the small intra-class variance of products in the same production line, from the perspective of differences and commonalities, MemSeg introduces artificially simulated abnormal samples and memory samples to assist the learning of the network. In the training phase, MemSeg explicitly learns the potential differences between normal and simulated abnormal images to obtain a robust classification hyperplane. At the same time, inspired by the mechanism of human memory, MemSeg uses a memory pool to store the general patterns of normal samples. By comparing the similarities and differences between input samples and memory samples in the memory pool to give effective guesses about abnormal regions; In the inference phase, MemSeg directly determines the abnormal regions of the input image in an end-to-end manner. Through experimental validation, MemSeg achieves the state-of-the-art (SOTA) performance on MVTec AD datasets with AUC scores of 99.56% and 98.84% at the image-level and pixel-level, respectively. In addition, MemSeg also has a significant advantage in inference speed benefiting from the end-to-end and straightforward network structure, which better meets the real-time requirement in industrial scenarios.
Contrastive Learning for Improving ASR Robustness in Spoken Language Understanding
Authors: Ya-Hsin Chang, Yun-Nung Chen
Date: 2022-05-02 00:00:00
Tasks: Contrastive Learning, Spoken Language Understanding

Spoken language understanding (SLU) is an essential task for machines to understand human speech for better interactions. However, errors from the automatic speech recognizer (ASR) usually hurt the understanding performance. In reality, ASR systems may not be easy to adjust for the target scenarios. Therefore, this paper focuses on learning utterance representations that are robust to ASR errors using a contrastive objective, and further strengthens the generalization ability by combining supervised contrastive learning and self-distillation in model fine-tuning. Experiments on three benchmark datasets demonstrate the effectiveness of our proposed approach.
CenterCLIP: Token Clustering for Efficient Text-Video Retrieval
Authors: Shuai Zhao, Linchao Zhu, Xiaohan Wang, Yi Yang
Date: 2022-05-02 00:00:00
Tasks: Clustering, Retrieval, Video Retrieval

Recently, large-scale pre-training methods like CLIP have made great progress in multi-modal research such as text-video retrieval. In CLIP, transformers are vital for modeling complex multi-modal relations. However, in the vision transformer of CLIP, the essential visual tokenization process, which produces discrete visual token sequences, generates many homogeneous tokens due to the redundancy nature of consecutive and similar frames in videos. This significantly increases computation costs and hinders the deployment of video retrieval models in web applications. In this paper, to reduce the number of redundant video tokens, we design a multi-segment token clustering algorithm to find the most representative tokens and drop the non-essential ones. As the frame redundancy occurs mostly in consecutive frames, we divide videos into multiple segments and conduct segment-level clustering. Center tokens from each segment are later concatenated into a new sequence, while their original spatial-temporal relations are well maintained. We instantiate two clustering algorithms to efficiently find deterministic medoids and iteratively partition groups in high dimensional space. Through this token clustering and center selection procedure, we successfully reduce computation costs by removing redundant visual tokens. This method further enhances segment-level semantic alignment between video and text representations, enforcing the spatio-temporal interactions of tokens from within-segment frames. Our method, coined as CenterCLIP, surpasses existing state-of-the-art by a large margin on typical text-video benchmarks, while reducing the training memory cost by 35\% and accelerating the inference speed by 14\% at the best case. The code is available at \href{{https://github.com/mzhaoshuai/CenterCLIP}}{{https://github.com/mzhaoshuai/CenterCLIP}}.
Rethinking Gradient Operator for Exposing AI-enabled Face Forgeries
Authors: Zhiqing Guo, Gaobo Yang, Dengyong Zhang, Ming Xia
Date: 2022-05-02 00:00:00
Tasks: Image Forensics

For image forensics, convolutional neural networks (CNNs) tend to learn content features rather than subtle manipulation traces, which limits forensic performance. Existing methods predominantly solve the above challenges by following a general pipeline, that is, subtracting the original pixel value from the predicted pixel value to make CNNs pay attention to the manipulation traces. However, due to the complicated learning mechanism, these methods may bring some unnecessary performance losses. In this work, we rethink the advantages of gradient operator in exposing face forgery, and design two plug-and-play modules by combining gradient operator with CNNs, namely tensor pre-processing (TP) and manipulation trace attention (MTA) module. Specifically, TP module refines the feature tensor of each channel in the network by gradient operator to highlight the manipulation traces and improve the feature representation. Moreover, MTA module considers two dimensions, namely channel and manipulation traces, to force the network to learn the distribution of manipulation traces. These two modules can be seamlessly integrated into CNNs for end-to-end training. Experiments show that the proposed network achieves better results than prior works on five public datasets. Especially, TP module greatly improves the accuracy by at least 4.60% compared with the existing pre-processing module only via simple tensor refinement. The code is available at: https://github.com/EricGzq/GocNet-pytorch.
AdapterBias: Parameter-efficient Token-dependent Representation Shift for Adapters in NLP Tasks
Authors: Chin-Lun Fu, Zih-Ching Chen, Yun-Ru Lee, Hung-Yi Lee
Date: 2022-04-30 00:00:00
Proceeding: Findings (NAACL) 2022 7

Transformer-based pre-trained models with millions of parameters require large storage. Recent approaches tackle this shortcoming by training adapters, but these approaches still require a relatively large number of parameters. In this study, AdapterBias, a surprisingly simple yet effective adapter architecture, is proposed. AdapterBias adds a token-dependent shift to the hidden output of transformer layers to adapt to downstream tasks with only a vector and a linear layer. Extensive experiments are conducted to demonstrate the effectiveness of AdapterBias. The experiments show that our proposed method can dramatically reduce the trainable parameters compared to the previous works with a minimal decrease in task performances compared with fine-tuned pre-trained models. We further find that AdapterBias automatically learns to assign more significant representation shifts to the tokens related to the task in consideration.
ComPhy: Compositional Physical Reasoning of Objects and Events from Videos
Authors: Zhenfang Chen, Kexin Yi, Yunzhu Li, Mingyu Ding, Antonio Torralba, Joshua B. Tenenbaum, Chuang Gan
Date: 2022-05-02 00:00:00
Proceeding: ICLR 2022 4

Objects' motions in nature are governed by complex interactions and their properties. While some properties, such as shape and material, can be identified via the object's visual appearances, others like mass and electric charge are not directly visible. The compositionality between the visible and hidden properties poses unique challenges for AI models to reason from the physical world, whereas humans can effortlessly infer them with limited observations. Existing studies on video reasoning mainly focus on visually observable elements such as object appearance, movement, and contact interaction. In this paper, we take an initial step to highlight the importance of inferring the hidden physical properties not directly observable from visual appearances, by introducing the Compositional Physical Reasoning (ComPhy) dataset. For a given set of objects, ComPhy includes few videos of them moving and interacting under different initial conditions. The model is evaluated based on its capability to unravel the compositional hidden properties, such as mass and charge, and use this knowledge to answer a set of questions posted on one of the videos. Evaluation results of several state-of-the-art video reasoning models on ComPhy show unsatisfactory performance as they fail to capture these hidden properties. We further propose an oracle neural-symbolic framework named Compositional Physics Learner (CPL), combining visual perception, physical property learning, dynamic prediction, and symbolic execution into a unified framework. CPL can effectively identify objects' physical properties from their interactions and predict their dynamics to answer questions.
HarmoF0: Logarithmic Scale Dilated Convolution For Pitch Estimation
Authors: Weixing Wei, Peilin Li, Yi Yu, Wei Li
Date: 2022-05-02 00:00:00

Sounds, especially music, contain various harmonic components scattered in the frequency dimension. It is difficult for normal convolutional neural networks to observe these overtones. This paper introduces a multiple rates dilated causal convolution (MRDC-Conv) method to capture the harmonic structure in logarithmic scale spectrograms efficiently. The harmonic is helpful for pitch estimation, which is important for many sound processing applications. We propose HarmoF0, a fully convolutional network, to evaluate the MRDC-Conv and other dilated convolutions in pitch estimation. The results show that this model outperforms the DeepF0, yields state-of-the-art performance in three datasets, and simultaneously reduces more than 90% parameters. We also find that it has stronger noise resistance and fewer octave errors. The code and pre-trained model are available at https://github.com/WX-Wei/HarmoF0.
Open-Set Semi-Supervised Learning for 3D Point Cloud Understanding
Authors: Xian Shi, Xun Xu, Wanyue Zhang, Xiatian Zhu, Chuan Sheng Foo, Kui Jia
Date: 2022-05-02 00:00:00
Tasks: 3D Point Cloud Classification, Point Cloud Classification

Semantic understanding of 3D point cloud relies on learning models with massively annotated data, which, in many cases, are expensive or difficult to collect. This has led to an emerging research interest in semi-supervised learning (SSL) for 3D point cloud. It is commonly assumed in SSL that the unlabeled data are drawn from the same distribution as that of the labeled ones; This assumption, however, rarely holds true in realistic environments. Blindly using out-of-distribution (OOD) unlabeled data could harm SSL performance. In this work, we propose to selectively utilize unlabeled data through sample weighting, so that only conducive unlabeled data would be prioritized. To estimate the weights, we adopt a bi-level optimization framework which iteratively optimizes a metaobjective on a held-out validation set and a task-objective on a training set. Faced with the instability of efficient bi-level optimizers, we further propose three regularization techniques to enhance the training stability. Extensive experiments on 3D point cloud classification and segmentation tasks verify the effectiveness of our proposed method. We also demonstrate the feasibility of a more efficient training strategy.
Knowledge Graph Contrastive Learning for Recommendation
Authors: Yuhao Yang, Chao Huang, Lianghao Xia, Chenliang Li
Date: 2022-05-02 00:00:00
Tasks: Contrastive Learning, General Knowledge, Knowledge Graphs, Recommendation Systems, World Knowledge

Knowledge Graphs (KGs) have been utilized as useful side information to improve recommendation quality. In those recommender systems, knowledge graph information often contains fruitful facts and inherent semantic relatedness among items. However, the success of such methods relies on the high quality knowledge graphs, and may not learn quality representations with two challenges: i) The long-tail distribution of entities results in sparse supervision signals for KG-enhanced item representation; ii) Real-world knowledge graphs are often noisy and contain topic-irrelevant connections between items and entities. Such KG sparsity and noise make the item-entity dependent relations deviate from reflecting their true characteristics, which significantly amplifies the noise effect and hinders the accurate representation of user's preference. To fill this research gap, we design a general Knowledge Graph Contrastive Learning framework (KGCL) that alleviates the information noise for knowledge graph-enhanced recommender systems. Specifically, we propose a knowledge graph augmentation schema to suppress KG noise in information aggregation, and derive more robust knowledge-aware representations for items. In addition, we exploit additional supervision signals from the KG augmentation process to guide a cross-view contrastive learning paradigm, giving a greater role to unbiased user-item interactions in gradient descent and further suppressing the noise. Extensive experiments on three public datasets demonstrate the consistent superiority of our KGCL over state-of-the-art techniques. KGCL also achieves strong performance in recommendation scenarios with sparse user-item interactions, long-tail and noisy KG entities. Our implementation codes are available at https://github.com/yuh-yang/KGCL-SIGIR22
CCLF: A Contrastive-Curiosity-Driven Learning Framework for Sample-Efficient Reinforcement Learning
Authors: Chenyu Sun, Hangwei Qian, Chunyan Miao
Date: 2022-05-02 00:00:00
Tasks: Data Augmentation, Q-Learning, reinforcement-learning, Reinforcement Learning (RL)

In reinforcement learning (RL), it is challenging to learn directly from high-dimensional observations, where data augmentation has recently been shown to remedy this via encoding invariances from raw pixels. Nevertheless, we empirically find that not all samples are equally important and hence simply injecting more augmented inputs may instead cause instability in Q-learning. In this paper, we approach this problem systematically by developing a model-agnostic Contrastive-Curiosity-Driven Learning Framework (CCLF), which can fully exploit sample importance and improve learning efficiency in a self-supervised manner. Facilitated by the proposed contrastive curiosity, CCLF is capable of prioritizing the experience replay, selecting the most informative augmented inputs, and more importantly regularizing the Q-function as well as the encoder to concentrate more on under-learned data. Moreover, it encourages the agent to explore with a curiosity-based reward. As a result, the agent can focus on more informative samples and learn representation invariances more efficiently, with significantly reduced augmented inputs. We apply CCLF to several base RL algorithms and evaluate on the DeepMind Control Suite, Atari, and MiniGrid benchmarks, where our approach demonstrates superior sample efficiency and learning performances compared with other state-of-the-art methods.
Assessing unconstrained surgical cuttings in VR using CNNs
Authors: Ilias Chrysovergis, Manos Kamarianakis, Mike Kentros, Dimitris Angelis, Antonis Protopsaltis, George Papagiannakis
Date: 2022-05-02 00:00:00
Tasks: Data Augmentation

We present a Convolutional Neural Network (CNN) suitable to assess unconstrained surgical cuttings, trained on a dataset created with a data augmentation technique.
A Novel Speech-Driven Lip-Sync Model with CNN and LSTM
Authors: Xiaohong Li, Xiang Wang, Kai Wang, Shiguo Lian
Date: 2022-05-02 00:00:00
Tasks: Face Model, speech-recognition, Speech Recognition

Generating synchronized and natural lip movement with speech is one of the most important tasks in creating realistic virtual characters. In this paper, we present a combined deep neural network of one-dimensional convolutions and LSTM to generate vertex displacement of a 3D template face model from variable-length speech input. The motion of the lower part of the face, which is represented by the vertex movement of 3D lip shapes, is consistent with the input speech. In order to enhance the robustness of the network to different sound signals, we adapt a trained speech recognition model to extract speech feature, and a velocity loss term is adopted to reduce the jitter of generated facial animation. We recorded a series of videos of a Chinese adult speaking Mandarin and created a new speech-animation dataset to compensate the lack of such public data. Qualitative and quantitative evaluations indicate that our model is able to generate smooth and natural lip movements synchronized with speech.
AI-Driven Contextual Advertising: A Technology Report and Implication Analysis
Authors: Emil Häglund, Johanna Björklund
Date: 2022-05-02 00:00:00

Programmatic advertising consists in automated auctioning of digital ad space. Every time a user requests a web page, placeholders on the page are populated with ads from the highest-bidding advertisers. The bids are typically based on information about the user, and to an increasing extent, on information about the surrounding media context. The growing interest in contextual advertising is in part a counterreaction to the current dependency on personal data, which is problematic from legal and ethical standpoints. The transition is further accelerated by developments in Artificial Intelligence (AI), which allow for a deeper semantic understanding of context and, by extension, more effective ad placement. In this article, we begin by identifying context factors that have been shown in previous research to positively influence how ads are received. We then continue to discuss applications of AI in contextual advertising, where it adds value by, e.g., extracting high-level information about media context and optimising bidding strategies. However, left unchecked, these new practices can lead to unfair ad delivery and manipulative use of context. We summarize these and other concerns for consumers, publishers and advertisers in an implication analysis.