pwc-archive/papers-with-abstracts huggingface dataset:
Deconstructing Data Reconstruction: Multiclass, Weight Decay and General Losses
Date: 2023-09-21 00:00:00
Proceeding: NeurIPS 2023 11

Memorization of training data is an active research area, yet our understanding of the inner workings of neural networks is still in its infancy.
Recently, Haim et al. 2022 proposed a scheme to reconstruct training samples from multilayer perceptron binary classifiers, effectively demonstrating that a large portion of training samples are encoded in the parameters of such networks.
In this work, we extend their findings in several directions, including reconstruction from multiclass and convolutional neural networks. 
We derive a more general reconstruction scheme which is applicable to a wider range of loss functions such as regression losses. 
Moreover, we study the various factors that contribute to networks' susceptibility to such reconstruction schemes. 
Intriguingly, we observe that using weight decay during training increases reconstructability both in terms of quantity and quality. 
Additionally, we examine the influence of the number of neurons relative to the number of training samples on the reconstructability.
Code: https://github.com/gonbuzaglo/decoreco
Effect of prolonged use of Smartphones on neck and wrist muscle fatigue using surface EMG
Authors: D.Rodríguez, K.Zorrilla, L.Chirre, D.Palacios
Date: 2023-12-01 00:00:00
Proceeding: 2024 IEEE International Symposium on Medical Measurements and Applications (MeMeA) 2023 12
Tasks: Electromyography (EMG)

Prolonged smartphone usage is prevalent among young adults in Lima, Peru, with potential implications for musculoskeletal health. This study investigates muscle fatigue in university students aged 18 to 24, focusing on the upper trapezius and wrist flexor muscles. Utilizing electromyography (EMG), participants engaged in smartphone activities in various positions. Results highlight significant muscle fatigue associated with specific postures and tasks. Our findings underscore the need for preventive measures and ergonomic interventions to mitigate the impact of extensive smartphone use on the musculoskeletal health of young adults.
Flow-Attention-based Spatio-Temporal Aggregation Network for 3D Mask Detection
Date: 2023-09-21 00:00:00
Proceeding: NeurIPS 2023 11

Anti-spoofing detection has become a necessity for face recognition systems due to the security threat posed by spoofing attacks. Despite great success in traditional attacks, most deep-learning-based methods perform poorly in 3D masks, which can highly simulate real faces in appearance and structure, suffering generalizability insufficiency while focusing only on the spatial domain with single frame input. This has been mitigated by the recent introduction of a biomedical technology called rPPG (remote photoplethysmography). However, rPPG-based methods are sensitive to noisy interference and require at least one second (> 25 frames) of observation time, which induces high computational overhead. To address these challenges, we propose a novel 3D mask detection framework, called FASTEN (Flow-Attention-based Spatio-Temporal aggrEgation Network). We tailor the network for focusing more on fine-grained details in large movements, which can eliminate redundant spatio-temporal feature interference and quickly capture splicing traces of 3D masks in fewer frames. Our proposed network contains three key modules: 1) a facial optical flow network to obtain non-RGB inter-frame flow information; 2) flow attention to assign different significance to each frame; 3) spatio-temporal aggregation to aggregate high-level spatial features and temporal transition features. Through extensive experiments, FASTEN only requires five frames of input and outperforms eight competitors for both intra-dataset and cross-dataset evaluations in terms of multiple detection metrics. Moreover, FASTEN has been deployed in real-world mobile devices for practical 3D mask detection.Submission Number: 13585
Fast Partitioned Learned Bloom Filter
Date: 2023-09-21 00:00:00
Proceeding: NeurIPS 2023 11

A Bloom filter is a memory-efficient data structure for approximate membership queries used in numerous fields of computer science.
Recently, learned Bloom filters that achieve better memory efficiency using machine learning models have attracted attention.
One such filter, the partitioned learned Bloom filter (PLBF), achieves excellent memory efficiency.
However, PLBF requires a $\mathcal{O}(N^3k)$ time complexity to construct the data structure, where $N$ and $k$ are the hyperparameters of PLBF.
One can improve memory efficiency by increasing $N$, but the construction time becomes extremely long.
Thus, we propose two methods that can reduce the construction time while maintaining the memory efficiency of PLBF.
First, we propose fast PLBF, which can construct the same data structure as PLBF with a smaller time complexity $\mathcal{O}(N^2k)$.
Second, we propose fast PLBF++, which can construct the data structure with even smaller time complexity $\mathcal{O}(Nk\log N + Nk^2)$.
Fast PLBF++ does not necessarily construct the same data structure as PLBF.
Still, it is almost as memory efficient as PLBF, and it is proved that fast PLBF++ has the same data structure as PLBF when the distribution satisfies a certain constraint.
Our experimental results from real-world datasets show that (i) fast PLBF and fast PLBF++ can construct the data structure up to 233 and 761 times faster than PLBF, (ii) fast PLBF can achieve the same memory efficiency as PLBF, and (iii) fast PLBF++ can achieve almost the same memory efficiency as PLBF.
The codes are available at [this https URL](https://github.com/atsukisato/FastPLBF).
Unconstrained Dynamic Regret via Sparse Coding
Date: 2023-09-21 00:00:00
Proceeding: NeurIPS 2023 11

Motivated by the challenge of nonstationarity in sequential decision making, we study Online Convex Optimization (OCO) under the coupling of two problem structures: the domain is unbounded, and the comparator sequence $u_1,\ldots,u_T$ is arbitrarily time-varying. As no algorithm can guarantee low regret simultaneously against all comparator sequences, handling this setting requires moving from minimax optimality to comparator adaptivity. That is, sensible regret bounds should depend on certain complexity measures of the comparator relative to one's prior knowledge. This paper achieves a new type of such adaptive regret bounds leveraging a sparse coding framework. The complexity of the comparator is measured by its energy and its sparsity on a user-specified dictionary, which offers considerable versatility. For example, equipped with a wavelet dictionary, our framework improves the state-of-the-art bound (Jacobsen & Cutkosky, 2022) by adapting to both ($i$) the magnitude of the comparator average $||\bar u||=||\sum_{t=1}^Tu_t/T||$, rather than the maximum $\max_t||u_t||$; and ($ii$) the comparator variability $\sum_{t=1}^T||u_t-\bar u||$, rather than the uncentered sum $\sum_{t=1}^T||u_t||$. Furthermore, our proof is simpler due to decoupling function approximation from regret minimization.Submission Number: 8830
Lookaround Optimizer: $k$ steps around, 1 step average
Date: 2023-09-21 00:00:00
Proceeding: NeurIPS 2023 11

Weight Average (WA) is an active research topic due to its simplicity in ensembling deep networks and the effectiveness in promoting generalization. Existing weight average approaches, however, are often carried out along only one training trajectory in a post-hoc manner (i.e., the weights are averaged after the entire training process is finished), which significantly degrades the diversity between networks and thus impairs the effectiveness. In this paper, inspired by weight average, we propose Lookaround, a straightforward yet effective SGD-based optimizer leading to flatter minima with better generalization. Specifically, Lookaround iterates two steps during the whole training period: the around step and the average step. In each iteration, 1) the around step starts from a common point and trains multiple networks simultaneously, each on transformed data by a different data augmentation, and 2) the average step averages these trained networks to get the averaged network, which serves as the starting point for the next iteration. The around step improves the functionality diversity while the average step guarantees the weight locality of these networks during the whole training, which is essential for WA to work. We theoretically explain the superiority of Lookaround by convergence analysis, and make extensive experiments to evaluate Lookaround on popular benchmarks including CIFAR and ImageNet with both CNNs and ViTs, demonstrating clear superiority over state-of-the-arts. Our code is available at https://github.com/Ardcy/Lookaround.
Masked Space-Time Hash Encoding for Efficient Dynamic Scene Reconstruction
Date: 2023-09-21 00:00:00
Proceeding: NeurIPS 2023 11

In this paper, we propose the Masked Space-Time Hash encoding (MSTH), a novel method for efficiently reconstructing dynamic 3D scenes from multi-view or monocular videos. Based on the observation that dynamic scenes often contain substantial static areas that result in redundancy in storage and computations, MSTH represents a dynamic scene as a weighted combination of a 3D hash encoding and a 4D hash encoding. The weights for the two components are represented by a learnable mask which is guided by an uncertainty-based objective to reflect the spatial and temporal importance of each 3D position. With this design, our method can reduce the hash collision rate by avoiding redundant queries and modifications on static areas, making it feasible to represent a large number of space-time voxels by hash tables with small size.Besides, without the requirements to fit the large numbers of temporally redundant features independently, our method is easier to optimize and converge rapidly with only twenty minutes of training for a 300-frame dynamic scene. We evaluate our method on extensive dynamic scenes. As a result, MSTH obtains consistently better results than previous state-of-the-art methods with only 20 minutes of training time and 130 MB of memory storage.
Parameter-efficient Tuning of Large-scale Multimodal Foundation Model
Date: 2023-09-21 00:00:00
Proceeding: NeurIPS 2023 11

Driven by the progress of large-scale pre-training, parameter-efficient transfer learning has gained immense popularity across different subfields of Artificial Intelligence. The core is to adapt the model to downstream tasks with only a small set of parameters. Recently, researchers have leveraged such proven techniques in multimodal tasks and achieve promising results. However, two critical issues remain unresolved: how to further reduce the complexity with lightweight design and how to boost alignment between modalities under extremely low parameters. In this paper, we propose A gracefUl pRompt framewOrk for cRoss-modal trAnsfer (AURORA) to overcome these challenges. Considering the redundancy in existing architectures, we first utilize the mode approximation to generate 0.1M trainable parameters to implement the multimodal parameter-efficient tuning, which explores the low intrinsic dimension with only 0.04% parameters of the pre-trained model. Then, for better modality alignment, we propose the Informative Context Enhancement and Gated Query Transformation module under extremely few parameters scenes. A thorough evaluation on six cross-modal benchmarks shows that it not only outperforms the state-of-the-art but even outperforms the full fine-tuning approach. Our code is available at: https://github.com/WillDreamer/Aurora.
Learning Modulated Transformation in GANs
Date: 2023-09-21 00:00:00
Proceeding: NeurIPS 2023 11

The success of style-based generators largely benefits from style modulation,
which helps take care of the cross-instance variation within data. However, the
instance-wise stochasticity is typically introduced via regular convolution, where
kernels interact with features at some fixed locations, limiting its capacity for
modeling geometric variation. To alleviate this problem, we equip the generator
in generative adversarial networks (GANs) with a plug-and-play module, termed
as modulated transformation module (MTM). This module predicts spatial offsets
under the control of latent codes, based on which the convolution operation can
be applied at variable locations for different instances, and hence offers the model
an additional degree of freedom to handle geometry deformation. Extensive
experiments suggest that our approach can be faithfully generalized to various
generative tasks, including image generation, 3D-aware image synthesis, and
video generation, and get compatible with state-of-the-art frameworks without
any hyper-parameter tuning. It is noteworthy that, towards human generation on
the challenging TaiChi dataset, we improve the FID of StyleGAN3 from 21.36 to
13.60, demonstrating the efficacy of learning modulated geometry transformation.
Code and models are available at https://github.com/limbo0000/mtm.Submission Number: 3123
SODA: Robust Training of Test-Time Data Adaptors
Date: 2023-09-21 00:00:00
Proceeding: NeurIPS 2023 11

Adapting models deployed to test distributions can mitigate the performance degradation caused by distribution shifts. However, privacy concerns may render model parameters inaccessible. One promising approach involves utilizing zeroth-order optimization (ZOO) to train a data adaptor to adapt the test data to fit the deployed models. Nevertheless, the data adaptor trained with ZOO typically brings restricted improvements due to the potential corruption of data features caused by the data adaptor. To address this issue, we revisit ZOO in the context of test-time data adaptation. We find that the issue directly stems from the unreliable estimation of the gradients used to optimize the data adaptor, which is inherently due to the unreliable nature of the pseudo-labels assigned to the test data. Based on this observation, we propose pseudo-label-robust data adaptation (SODA) to improve the performance of data adaptation. Specifically, SODA leverages high-confidence predicted labels as reliable labels to optimize the data adaptor with ZOO for label prediction. For data with low-confidence predictions, SODA encourages the adaptor to preserve data information to mitigate data corruption. Empirical results indicate that SODA can significantly enhance the performance of deployed models in the presence of distribution shifts without requiring access to model parameters.
RIO: A Benchmark for Reasoning Intention-Oriented Objects in Open Environments
Date: 2023-09-26 00:00:00
Proceeding: NeurIPS 2023 11

Intention-oriented object detection aims to detect desired objects based on specific intentions or requirements. For instance, when we desire to "lie down and rest", we instinctively seek out a suitable option such as a "bed" or a "sofa" that can fulfill our needs. Previous work in this area is limited either by the number of intention descriptions or by the affordance vocabulary available for intention objects. These limitations make it challenging to handle intentions in open environments effectively. To facilitate this research, we construct a comprehensive dataset called Reasoning Intention-Oriented Objects (RIO). In particular, RIO is specifically designed to incorporate diverse real-world scenarios and a wide range of object categories. It offers the following key features: 1) intention descriptions in RIO are represented as natural sentences rather than a mere word or verb phrase, making them more practical and meaningful; 2) the intention descriptions are contextually relevant to the scene, enabling a broader range of potential functionalities associated with the objects; 3) the dataset comprises a total of 40,214 images and 130,585 intention-object pairs. With the proposed RIO, we evaluate the ability of some existing models to reason intention-oriented objects in open environments.Submission Number: 57
Training Your Image Restoration Network Better with Random Weight Network as Optimization Function
Date: 2023-09-21 00:00:00
Proceeding: NeurIPS 2023 11
Tasks: Denoising, Image Denoising, Image Enhancement, Image Restoration, Image Super-Resolution, Low-Light Image Enhancement, Super-Resolution

The blooming progress made in deep learning-based image restoration has been largely attributed to the availability of high-quality, large-scale datasets and advanced network structures. However, optimization functions such as L_1 and L_2 are still de facto. In this study, we propose to  investigate new optimization functions to improve image restoration performance. Our key insight is  that ``random weight network can be acted as a constraint for training better image restoration networks''. However, not all random weight networks are suitable as constraints. We draw inspiration from Functional theory and show that alternative random weight networks should be represented in the form of a strict mathematical manifold. We explore the potential of our random weight network prototypes that satisfy this requirement:  Taylor's unfolding network, invertible neural network, central difference convolution, and zero-order filtering. We investigate these prototypes from four aspects: 1)   random weight strategies, 2)  network architectures, 3)   network depths, and 4) combinations of random weight networks. Furthermore, we devise the random weight in two variants:  the weights are randomly initialized only once during the entire training procedure, and  the weights are randomly initialized in each training epoch. Our approach can be directly integrated into existing networks without incurring additional training and testing computational costs. We perform extensive experiments across multiple image restoration tasks, including image denoising, low-light image enhancement, and guided image super-resolution to demonstrate the consistent performance gains achieved by our method.  Upon acceptance of this paper, we will release the code.
MiliPoint: A Point Cloud Dataset for mmWave Radar
Date: 2023-09-26 00:00:00
Proceeding: NeurIPS 2023 11

Millimetre-wave (mmWave) radar has emerged as an attractive and cost-effective alternative for human activity sensing compared to traditional camera-based systems. mmWave radars are also non-intrusive, providing better protection for user privacy. However, as a Radio Frequency based technology, mmWave radars rely on capturing reflected signals from objects, making them more prone to noise compared to cameras. This raises an intriguing question for the deep learning community: Can we develop more effective point set-based deep learning methods for such attractive sensors? 
  
To answer this question, our work, termed MiliPoint, delves into this idea by providing a large-scale, open dataset for the community to explore how mmWave radars can be utilised for human activity recognition. Moreover, MiliPoint stands out as it is larger in size than existing datasets, has more diverse human actions represented, and encompasses all three key tasks in human activity recognition. We have also established a range of point-based deep neural networks such as DGCNN, PointNet++ and PointTransformer, on MiliPoint, which can serve to set the ground baseline for further development.Submission Number: 821
Seeing is not always believing: Benchmarking Human and Model Perception of AI-Generated Images
Date: 2023-09-26 00:00:00
Proceeding: NeurIPS 2023 11

Photos serve as a way for humans to record what they experience in their daily lives, and they are often regarded as trustworthy sources of information. However, there is a growing concern that the advancement of artificial intelligence (AI) technology may produce fake photos, which can create confusion and diminish trust in photographs. This study aims to comprehensively evaluate agents for distinguishing state-of-the-art AI-generated visual content. Our study benchmarks both human capability and cutting-edge fake image detection AI algorithms, using a newly collected large-scale fake image dataset Fake2M. In our human perception evaluation, titled HPBench, we discovered that humans struggle significantly to distinguish real photos from AI-generated ones, with a misclassification rate of 38.7\%. Along with this, we conduct the model capability of AI-Generated images detection evaluation MPBench and the top-performing model from MPBench achieves a 13\% failure rate under the same setting used in the human evaluation.
We hope that our study can raise awareness of the potential risks of AI-generated images and facilitate further research to prevent the spread of false information. More information can refer to https://github.com/Inf-imagine/Sentry.
AVOIDDS: Aircraft Vision-based Intruder Detection Dataset and Simulator
Date: 2023-09-26 00:00:00
Proceeding: NeurIPS 2023 11

Designing robust machine learning systems remains an open problem, and there is a need for benchmark problems that cover both environmental changes and evaluation on a downstream task. In this work, we introduce AVOIDDS, a realistic object detection benchmark for the vision-based aircraft detect-and-avoid problem. We provide a labeled dataset consisting of 72,000 photorealistic images of intruder aircraft with various lighting conditions, weather conditions, relative geometries, and geographic locations.  We also provide an interface that evaluates trained models on slices of this dataset to identify changes in performance with respect to changing environmental conditions. Finally, we implement a fully-integrated, closed-loop simulator of the vision-based detect-and-avoid problem to evaluate trained models with respect to the downstream collision avoidance task. This benchmark will enable further research in the design of robust machine learning systems for use in safety-critical applications. The AVOIDDS dataset and code are publicly available at https://purl.stanford.edu/hj293cv5980 and https://github.com/sisl/VisionBasedAircraftDAA, respectively.
ImageNet-Hard: The Hardest Images Remaining from a Study of the Power of Zoom and Spatial Biases in Image Classification
Date: 2023-09-26 00:00:00
Proceeding: NeurIPS 2023 11

Image classifiers are information-discarding machines, by design. Yet, how these models discard information remains mysterious. We hypothesize that one way for image classifiers to reach high accuracy is to first zoom to the most discriminative region in the image and then extract features from there to predict image labels, discarding the rest of the image. Studying six popular networks ranging from AlexNet to CLIP, we find that proper framing of the input image can lead to the correct classification of 98.91% of ImageNet images. Furthermore, we  uncover positional biases in various datasets, especially a strong center bias in two popular datasets: ImageNet-A and ObjectNet. Finally, leveraging our insights into the potential of zooming, we propose a test-time augmentation (TTA) technique that improves classification accuracy by forcing models to explicitly perform zoom-in operations before making predictions.
Our method is more interpretable, accurate, and faster than MEMO, a state-of-the-art (SOTA) TTA method. We introduce ImageNet-Hard, a new benchmark that challenges SOTA classifiers including large vision-language models even when optimal zooming is allowed.