pwc-archive/papers-with-abstracts huggingface dataset:
3D Meta-Segmentation Neural Network
Authors: Yu Hao, Yi Fang
Date: 2021-10-08 00:00:00
Tasks: 3D Part Segmentation, 3D Point Cloud Part Segmentation, Meta-Learning, Segmentation

Though deep learning methods have shown great success in 3D point cloud part segmentation, they generally rely on a large volume of labeled training data, which makes the model suffer from unsatisfied generalization abilities to unseen classes with limited data. To address this problem, we present a novel meta-learning strategy that regards the 3D shape segmentation function as a task. By training over a number of 3D part segmentation tasks, our method is capable to learn the prior over the respective 3D segmentation function space which leads to an optimal model that is rapidly adapting to new part segmentation tasks. To implement our meta-learning strategy, we propose two novel modules: meta part segmentation learner and part segmentation learner. During the training process, the part segmentation learner is trained to complete a specific part segmentation task in the few-shot scenario. In the meantime, the meta part segmentation learner is trained to capture the prior from multiple similar part segmentation tasks. Based on the learned information of task distribution, our meta part segmentation learner is able to dynamically update the part segmentation learner with optimal parameters which enable our part segmentation learner to rapidly adapt and have great generalization ability on new part segmentation tasks. We demonstrate that our model achieves superior part segmentation performance with the few-shot setting on the widely used dataset: ShapeNet.
Unsupervised Pose-Aware Part Decomposition for 3D Articulated Objects
Authors: Yuki Kawana, Yusuke Mukuta, Tatsuya Harada
Date: 2021-10-08 00:00:00
Tasks: Pose Estimation

Articulated objects exist widely in the real world. However, previous 3D generative methods for unsupervised part decomposition are unsuitable for such objects, because they assume a spatially fixed part location, resulting in inconsistent part parsing. In this paper, we propose PPD (unsupervised Pose-aware Part Decomposition) to address a novel setting that explicitly targets man-made articulated objects with mechanical joints, considering the part poses. We show that category-common prior learning for both part shapes and poses facilitates the unsupervised learning of (1) part decomposition with non-primitive-based implicit representation, and (2) part pose as joint parameters under single-frame shape supervision. We evaluate our method on synthetic and real datasets, and we show that it outperforms previous works in consistent part parsing of the articulated objects based on comparable part pose estimation performance to the supervised baseline.
Optical Interferometry in Astronomy
Authors: John D. Monnier
Date: 2003-07-02 00:00:00
Tasks: Astronomy

Here I review the current state of the field of optical stellar interferometry, concentrating on ground-based work although a brief report of space interferometry missions is included. We pause both to reflect on decades of immense progress in the field as well as to prepare for a new generation of large interferometers just now being commissioned (most notably, the CHARA, Keck and VLT Interferometers). First, this review summarizes the basic principles behind stellar interferometry needed by the lay-physicist and general astronomer to understand the scientific potential as well as technical challenges of interferometry. Next, the basic design principles of practical interferometers are discussed, using the experience of past and existing facilities to illustrate important points. Here there is significant discussion of current trends in the field, including the new facilities under construction and advanced technologies being debuted. This decade has seen the influence of stellar interferometry extend beyond classical regimes of stellar diameters and binary orbits to new areas such as mapping the accretion disks around young stars, novel calibration of the Cepheid Period-Luminosity relation, and imaging of stellar surfaces. The third section is devoted to the major scientific results from interferometry, grouped into natural categories reflecting these current developments. Lastly, I consider the future of interferometry, highlighting the kinds of new science promised by the interferometers coming on-line in the next few years. I also discuss the longer-term future of optical interferometry, including the prospects for space interferometry and the possibilities of large-scale ground-based projects. Critical technological developments are still needed to make these projects attractive and affordable.
Imitating Deep Learning Dynamics via Locally Elastic Stochastic Differential Equations
Authors: Jiayao Zhang, Hua Wang, Weijie J. Su
Date: 2021-10-11 00:00:00
Proceeding: NeurIPS 2021 12
Tasks: Deep Learning

Understanding the training dynamics of deep learning models is perhaps a necessary step toward demystifying the effectiveness of these models. In particular, how do data from different classes gradually become separable in their feature spaces when training neural networks using stochastic gradient descent? In this study, we model the evolution of features during deep learning training using a set of stochastic differential equations (SDEs) that each corresponds to a training sample. As a crucial ingredient in our modeling strategy, each SDE contains a drift term that reflects the impact of backpropagation at an input on the features of all samples. Our main finding uncovers a sharp phase transition phenomenon regarding the {intra-class impact: if the SDEs are locally elastic in the sense that the impact is more significant on samples from the same class as the input, the features of the training data become linearly separable, meaning vanishing training loss; otherwise, the features are not separable, regardless of how long the training time is. Moreover, in the presence of local elasticity, an analysis of our SDEs shows that the emergence of a simple geometric structure called the neural collapse of the features. Taken together, our results shed light on the decisive role of local elasticity in the training dynamics of neural networks. We corroborate our theoretical analysis with experiments on a synthesized dataset of geometric shapes and CIFAR-10.
Classification of anomalous gait using Machine Learning techniques and embedded sensors
Authors: T. R. D. Sa, C. M. S. Figueiredo
Date: 2021-10-08 00:00:00
Tasks: BIG-bench Machine Learning

Human gait can be a predictive factor for detecting pathologies that affect human locomotion according to studies. In addition, it is known that a high investment is demanded in order to raise a traditional clinical infrastructure able to provide human gait examinations, making them unaffordable for economically vulnerable patients. In face of this scenario, this work proposes an accessible and modern solution composed of a wearable device, to acquire 3D-accelerometer and 3D-gyroscope measurements, and machine learning techniques to classify between distinct categories of induced gait disorders. In order to develop the proposed research, it was created a dataset with the target label being 4 distinct and balanced categories of anomalous gait. The machine learning techniques that achieved the best performances (in terms of accuracy) in this dataset were through the application of Principal Component Analysis algorithm following of a Support Vector Machines classifier (94 \%). Further, an architecture based on a Feedforward Neural Network yielded even better results (96 \%). Finally, it is also presented computational performance comparison between the models implemented.
StARformer: Transformer with State-Action-Reward Representations for Visual Reinforcement Learning
Authors: Jinghuan Shang, Kumara Kahatapitiya, Xiang Li, Michael S. Ryoo
Date: 2021-10-12 00:00:00
Tasks: Imitation Learning, Inductive Bias, Offline RL, reinforcement-learning, Reinforcement Learning (RL)

Reinforcement Learning (RL) can be considered as a sequence modeling task: given a sequence of past state-action-reward experiences, an agent predicts a sequence of next actions. In this work, we propose State-Action-Reward Transformer (StARformer) for visual RL, which explicitly models short-term state-action-reward representations (StAR-representations), essentially introducing a Markovian-like inductive bias to improve long-term modeling. Our approach first extracts StAR-representations by self-attending image state patches, action, and reward tokens within a short temporal window. These are then combined with pure image state representations -- extracted as convolutional features, to perform self-attention over the whole sequence. Our experiments show that StARformer outperforms the state-of-the-art Transformer-based method on image-based Atari and DeepMind Control Suite benchmarks, in both offline-RL and imitation learning settings. StARformer is also more compliant with longer sequences of inputs. Our code is available at https://github.com/elicassion/StARformer.
MGH: Metadata Guided Hypergraph Modeling for Unsupervised Person Re-identification
Authors: Yiming Wu, Xintian Wu, Xi Li, Jian Tian
Date: 2021-10-12 00:00:00
Tasks: Person Re-Identification, Unsupervised Person Re-Identification

As a challenging task, unsupervised person ReID aims to match the same identity with query images which does not require any labeled information. In general, most existing approaches focus on the visual cues only, leaving potentially valuable auxiliary metadata information (e.g., spatio-temporal context) unexplored. In the real world, such metadata is normally available alongside captured images, and thus plays an important role in separating several hard ReID matches. With this motivation in mind, we propose~\textbf{MGH}, a novel unsupervised person ReID approach that uses meta information to construct a hypergraph for feature learning and label refinement. In principle, the hypergraph is composed of camera-topology-aware hyperedges, which can model the heterogeneous data correlations across cameras. Taking advantage of label propagation on the hypergraph, the proposed approach is able to effectively refine the ReID results, such as correcting the wrong labels or smoothing the noisy labels. Given the refined results, We further present a memory-based listwise loss to directly optimize the average precision in an approximate manner. Extensive experiments on three benchmarks demonstrate the effectiveness of the proposed approach against the state-of-the-art.
Multi-Modal Interaction Graph Convolutional Network for Temporal Language Localization in Videos
Authors: Zongmeng Zhang, Xianjing Han, Xuemeng Song, Yan Yan, Liqiang Nie
Date: 2021-10-12 00:00:00
Tasks: Semantic correspondence, Semantic Similarity, Semantic Textual Similarity, Sentence

This paper focuses on tackling the problem of temporal language localization in videos, which aims to identify the start and end points of a moment described by a natural language sentence in an untrimmed video. However, it is non-trivial since it requires not only the comprehensive understanding of the video and sentence query, but also the accurate semantic correspondence capture between them. Existing efforts are mainly centered on exploring the sequential relation among video clips and query words to reason the video and sentence query, neglecting the other intra-modal relations (e.g., semantic similarity among video clips and syntactic dependency among the query words). Towards this end, in this work, we propose a Multi-modal Interaction Graph Convolutional Network (MIGCN), which jointly explores the complex intra-modal relations and inter-modal interactions residing in the video and sentence query to facilitate the understanding and semantic correspondence capture of the video and sentence query. In addition, we devise an adaptive context-aware localization method, where the context information is taken into the candidate moments and the multi-scale fully connected layers are designed to rank and adjust the boundary of the generated coarse candidate moments with different lengths. Extensive experiments on Charades-STA and ActivityNet datasets demonstrate the promising performance and superior efficiency of our model.
LightSeq2: Accelerated Training for Transformer-based Models on GPUs
Authors: Xiaohui Wang, Yang Wei, Ying Xiong, Guyue Huang, Xian Qian, Yufei Ding, Mingxuan Wang, Lei LI
Date: 2021-10-12 00:00:00
Tasks: Decoder, GPU, Machine Translation, Speech Recognition, Translation

Transformer-based neural models are used in many AI applications. Training these models is expensive, as it takes huge GPU resources and long duration. It is challenging because typical data like sentences have variable lengths, and Transformer's computation patterns are more complex than convolutional neural networks. Existing systems either only focus on model inference or optimization for only BERT-like encoder models. In this paper, we present LightSeq2, a system to accelerate training for a general family of Transformer models on GPUs. We propose a series of GPU optimization techniques tailored to the specific computation flow and memory access patterns of Transformer models. LightSeq2 supports many model architectures, including BERT (encoder-only), GPT (decoder-only), Transformer (encoder-decoder), and vision Transformer. Our experiments for a variety of models and benchmarks show that LightSeq2 is consistently faster (1.4-3.5x) than previous systems on different GPUs. In particular, it gains 308% training speedup compared with existing systems on a large public machine translation benchmark (WMT14 English-German).
ABO: Dataset and Benchmarks for Real-World 3D Object Understanding
Authors: Jasmine Collins, Shubham Goel, Kenan Deng, Achleshwar Luthra, Leon Xu, Erhan Gundogdu, Xi Zhang, Tomas F. Yago Vicente, Thomas Dideriksen, Himanshu Arora, Matthieu Guillaumin, Jitendra Malik
Date: 2021-10-12 00:00:00
Proceeding: CVPR 2022 1
Tasks: 3D Reconstruction, Metric Learning, Object, Retrieval, Single-View 3D Reconstruction

We introduce Amazon Berkeley Objects (ABO), a new large-scale dataset designed to help bridge the gap between real and virtual 3D worlds. ABO contains product catalog images, metadata, and artist-created 3D models with complex geometries and physically-based materials that correspond to real, household objects. We derive challenging benchmarks that exploit the unique properties of ABO and measure the current limits of the state-of-the-art on three open problems for real-world 3D object understanding: single-view 3D reconstruction, material estimation, and cross-domain multi-view object retrieval.
Mention Memory: incorporating textual knowledge into Transformers through entity mention attention
Authors: Michiel de Jong, Yury Zemlyanskiy, Nicholas FitzGerald, Fei Sha, William Cohen
Date: 2021-10-12 00:00:00
Proceeding: ICLR 2022 4
Tasks: Claim Verification, Natural Language Understanding, Open-Domain Question Answering, Passage Retrieval, Question Answering

Natural language understanding tasks such as open-domain question answering often require retrieving and assimilating factual information from multiple sources. We propose to address this problem by integrating a semi-parametric representation of a large text corpus into a Transformer model as a source of factual knowledge. Specifically, our method represents knowledge with `mention memory', a table of dense vector representations of every entity mention in a corpus. The proposed model - TOME - is a Transformer that accesses the information through internal memory layers in which each entity mention in the input passage attends to the mention memory. This approach enables synthesis of and reasoning over many disparate sources of information within a single Transformer model. In experiments using a memory of 150 million Wikipedia mentions, TOME achieves strong performance on several open-domain knowledge-intensive tasks, including the claim verification benchmarks HoVer and FEVER and several entity-based QA benchmarks. We also show that the model learns to attend to informative mentions without any direct supervision. Finally we demonstrate that the model can generalize to new unseen entities by updating the memory without retraining.
Sign Language Recognition via Skeleton-Aware Multi-Model Ensemble
Authors: Songyao Jiang, Bin Sun, Lichen Wang, Yue Bai, Kunpeng Li, Yun Fu
Date: 2021-10-12 00:00:00
Tasks: Action Recognition, Sign Language Recognition, Skeleton Based Action Recognition

Sign language is commonly used by deaf or mute people to communicate but requires extensive effort to master. It is usually performed with the fast yet delicate movement of hand gestures, body posture, and even facial expressions. Current Sign Language Recognition (SLR) methods usually extract features via deep neural networks and suffer overfitting due to limited and noisy data. Recently, skeleton-based action recognition has attracted increasing attention due to its subject-invariant and background-invariant nature, whereas skeleton-based SLR is still under exploration due to the lack of hand annotations. Some researchers have tried to use off-line hand pose trackers to obtain hand keypoints and aid in recognizing sign language via recurrent neural networks. Nevertheless, none of them outperforms RGB-based approaches yet. To this end, we propose a novel Skeleton Aware Multi-modal Framework with a Global Ensemble Model (GEM) for isolated SLR (SAM-SLR-v2) to learn and fuse multi-modal feature representations towards a higher recognition rate. Specifically, we propose a Sign Language Graph Convolution Network (SL-GCN) to model the embedded dynamics of skeleton keypoints and a Separable Spatial-Temporal Convolution Network (SSTCN) to exploit skeleton features. The skeleton-based predictions are fused with other RGB and depth based modalities by the proposed late-fusion GEM to provide global information and make a faithful SLR prediction. Experiments on three isolated SLR datasets demonstrate that our proposed SAM-SLR-v2 framework is exceedingly effective and achieves state-of-the-art performance with significant margins. Our code will be available at https://github.com/jackyjsy/SAM-SLR-v2
Memory-Efficient CNN Accelerator Based on Interlayer Feature Map Compression
Authors: Zhuang Shao, Xiaoliang Chen, Li Du, Lei Chen, Yuan Du, Wei Zhuang, Huadong Wei, Chenjia Xie, Zhongfeng Wang
Date: 2021-10-12 00:00:00
Tasks: Feature Compression, Quantization

Existing deep convolutional neural networks (CNNs) generate massive interlayer feature data during network inference. To maintain real-time processing in embedded systems, large on-chip memory is required to buffer the interlayer feature maps. In this paper, we propose an efficient hardware accelerator with an interlayer feature compression technique to significantly reduce the required on-chip memory size and off-chip memory access bandwidth. The accelerator compresses interlayer feature maps through transforming the stored data into frequency domain using hardware-implemented 8x8 discrete cosine transform (DCT). The high-frequency components are removed after the DCT through quantization. Sparse matrix compression is utilized to further compress the interlayer feature maps. The on-chip memory allocation scheme is designed to support dynamic configuration of the feature map buffer size and scratch pad size according to different network-layer requirements. The hardware accelerator combines compression, decompression, and CNN acceleration into one computing stream, achieving minimal compressing and processing delay. A prototype accelerator is implemented on an FPGA platform and also synthesized in TSMC 28-nm COMS technology. It achieves 403GOPS peak throughput and 1.4x~3.3x interlayer feature map reduction by adding light hardware area overhead, making it a promising hardware accelerator for intelligent IoT devices.
Planning from Pixels in Environments with Combinatorially Hard Search Spaces
Authors: Marco Bagatella, Mirek Olšák, Michal Rolínek, Georg Martius
Date: 2021-10-12 00:00:00
Proceeding: NeurIPS 2021 12
Tasks: continuous-control, Continuous Control, Offline RL

The ability to form complex plans based on raw visual input is a litmus test for current capabilities of artificial intelligence, as it requires a seamless combination of visual processing and abstract algorithmic execution, two traditionally separate areas of computer science. A recent surge of interest in this field brought advances that yield good performance in tasks ranging from arcade games to continuous control; these methods however do not come without significant issues, such as limited generalization capabilities and difficulties when dealing with combinatorially hard planning instances. Our contribution is two-fold: (i) we present a method that learns to represent its environment as a latent graph and leverages state reidentification to reduce the complexity of finding a good policy from exponential to linear (ii) we introduce a set of lightweight environments with an underlying discrete combinatorial structure in which planning is challenging even for humans. Moreover, we show that our methods achieves strong empirical generalization to variations in the environment, even across highly disadvantaged regimes, such as "one-shot" planning, or in an offline RL paradigm which only provides low-quality trajectories.
Implicit Bias of Linear Equivariant Networks
Authors: Hannah Lawrence, Kristian Georgiev, Andrew Dienes, Bobak T. Kiani
Date: 2021-10-12 00:00:00
Tasks: Binary Classification

Group equivariant convolutional neural networks (G-CNNs) are generalizations of convolutional neural networks (CNNs) which excel in a wide range of technical applications by explicitly encoding symmetries, such as rotations and permutations, in their architectures. Although the success of G-CNNs is driven by their \emph{explicit} symmetry bias, a recent line of work has proposed that the \emph{implicit} bias of training algorithms on particular architectures is key to understanding generalization for overparameterized neural nets. In this context, we show that $L$-layer full-width linear G-CNNs trained via gradient descent for binary classification converge to solutions with low-rank Fourier matrix coefficients, regularized by the $2/L$-Schatten matrix norm. Our work strictly generalizes previous analysis on the implicit bias of linear CNNs to linear G-CNNs over all finite groups, including the challenging setting of non-commutative groups (such as permutations), as well as band-limited G-CNNs over infinite groups. We validate our theorems via experiments on a variety of groups, and empirically explore more realistic nonlinear networks, which locally capture similar regularization patterns. Finally, we provide intuitive interpretations of our Fourier space implicit regularization results in real space via uncertainty principles.