pwc-archive/papers-with-abstracts huggingface dataset:
Prompt-ICM: A Unified Framework towards Image Coding for Machines with Task-driven Prompts
Authors: Ruoyu Feng, Jinming Liu, Xin Jin, Xiaohan Pan, Heming Sun, Zhibo Chen
Date: 2023-05-04 00:00:00

Image coding for machines (ICM) aims to compress images to support downstream AI analysis instead of human perception. For ICM, developing a unified codec to reduce information redundancy while empowering the compressed features to support various vision tasks is very important, which inevitably faces two core challenges: 1) How should the compression strategy be adjusted based on the downstream tasks? 2) How to well adapt the compressed features to different downstream tasks? Inspired by recent advances in transferring large-scale pre-trained models to downstream tasks via prompting, in this work, we explore a new ICM framework, termed Prompt-ICM. To address both challenges by carefully learning task-driven prompts to coordinate well the compression process and downstream analysis. Specifically, our method is composed of two core designs: a) compression prompts, which are implemented as importance maps predicted by an information selector, and used to achieve different content-weighted bit allocations during compression according to different downstream tasks; b) task-adaptive prompts, which are instantiated as a few learnable parameters specifically for tuning compressed features for the specific intelligent task. Extensive experiments demonstrate that with a single feature codec and a few extra parameters, our proposed framework could efficiently support different kinds of intelligent tasks with much higher coding efficiency.
Text Reading Order in Uncontrolled Conditions by Sparse Graph Segmentation
Authors: Renshen Wang, Yasuhisa Fujii, Alessandro Bissacco
Date: 2023-05-04 00:00:00
Tasks: Optical Character Recognition (OCR)

Text reading order is a crucial aspect in the output of an OCR engine, with a large impact on downstream tasks. Its difficulty lies in the large variation of domain specific layout structures, and is further exacerbated by real-world image degradations such as perspective distortions. We propose a lightweight, scalable and generalizable approach to identify text reading order with a multi-modal, multi-task graph convolutional network (GCN) running on a sparse layout based graph. Predictions from the model provide hints of bidimensional relations among text lines and layout region structures, upon which a post-processing cluster-and-sort algorithm generates an ordered sequence of all the text lines. The model is language-agnostic and runs effectively across multi-language datasets that contain various types of images taken in uncontrolled conditions, and it is small enough to be deployed on virtually any platform including mobile devices.
High-fidelity Generalized Emotional Talking Face Generation with Multi-modal Emotion Space Learning
Authors: Chao Xu, Junwei Zhu, Jiangning Zhang, Yue Han, Wenqing Chu, Ying Tai, Chengjie Wang, Zhifeng Xie, Yong liu
Date: 2023-05-04 00:00:00
Proceeding: CVPR 2023 1
Tasks: Face Generation, Talking Face Generation

Recently, emotional talking face generation has received considerable attention. However, existing methods only adopt one-hot coding, image, or audio as emotion conditions, thus lacking flexible control in practical applications and failing to handle unseen emotion styles due to limited semantics. They either ignore the one-shot setting or the quality of generated faces. In this paper, we propose a more flexible and generalized framework. Specifically, we supplement the emotion style in text prompts and use an Aligned Multi-modal Emotion encoder to embed the text, image, and audio emotion modality into a unified space, which inherits rich semantic prior from CLIP. Consequently, effective multi-modal emotion space learning helps our method support arbitrary emotion modality during testing and could generalize to unseen emotion styles. Besides, an Emotion-aware Audio-to-3DMM Convertor is proposed to connect the emotion condition and the audio sequence to structural representation. A followed style-based High-fidelity Emotional Face generator is designed to generate arbitrary high-resolution realistic identities. Our texture generator hierarchically learns flow fields and animated faces in a residual manner. Extensive experiments demonstrate the flexibility and generalization of our method in emotion control and the effectiveness of high-quality face synthesis.
Unsupervised Domain Adaptation for Neuron Membrane Segmentation based on Structural Features
Authors: Yuxiang An, Dongnan Liu, Weidong Cai
Date: 2023-05-04 00:00:00
Tasks: Domain Adaptation, Medical Image Analysis, Segmentation, Super-Resolution, Unsupervised Domain Adaptation

AI-enhanced segmentation of neuronal boundaries in electron microscopy (EM) images is crucial for automatic and accurate neuroinformatics studies. To enhance the limited generalization ability of typical deep learning frameworks for medical image analysis, unsupervised domain adaptation (UDA) methods have been applied. In this work, we propose to improve the performance of UDA methods on cross-domain neuron membrane segmentation in EM images. First, we designed a feature weight module considering the structural features during adaptation. Second, we introduced a structural feature-based super-resolution approach to alleviating the domain gap by adjusting the cross-domain image resolutions. Third, we proposed an orthogonal decomposition module to facilitate the extraction of domain-invariant features. Extensive experiments on two domain adaptive membrane segmentation applications have indicated the effectiveness of our method.
LayoutDM: Transformer-based Diffusion Model for Layout Generation
Authors: Shang Chai, Liansheng Zhuang, Fengying Yan
Date: 2023-05-04 00:00:00
Proceeding: CVPR 2023 1
Tasks: Denoising, Diversity, Layout Generation, model

Automatic layout generation that can synthesize high-quality layouts is an important tool for graphic design in many applications. Though existing methods based on generative models such as Generative Adversarial Networks (GANs) and Variational Auto-Encoders (VAEs) have progressed, they still leave much room for improving the quality and diversity of the results. Inspired by the recent success of diffusion models in generating high-quality images, this paper explores their potential for conditional layout generation and proposes Transformer-based Layout Diffusion Model (LayoutDM) by instantiating the conditional denoising diffusion probabilistic model (DDPM) with a purely transformer-based architecture. Instead of using convolutional neural networks, a transformer-based conditional Layout Denoiser is proposed to learn the reverse diffusion process to generate samples from noised layout data. Benefitting from both transformer and DDPM, our LayoutDM is of desired properties such as high-quality generation, strong sample diversity, faithful distribution coverage, and stationary training in comparison to GANs and VAEs. Quantitative and qualitative experimental results show that our method outperforms state-of-the-art generative models in terms of quality and diversity.
Sex Detection in the Early Stage of Fertilized Chicken Eggs via Image Recognition
Authors: Ufuk Asil, Efendi Nasibov
Date: 2023-05-03 00:00:00
Tasks: Morphological Analysis

Culling newly hatched male chicks in industrial hatcheries poses a serious ethical problem. Both laying and broiler breeders need males, but it is a problem because they are produced more than needed. Being able to determine the sex of chicks in the egg at the beginning or early stage of incubation can eliminate ethical problems as well as many additional costs. When we look at the literature, the methods used are very costly, low in applicability, invasive, inadequate in accuracy, or too late to eliminate ethical problems. Considering the embryo's development, the earliest observed candidate feature for sex determination is blood vessels. Detection from blood vessels can eliminate ethical issues, and these vessels can be seen when light is shined into the egg until the first seven days. In this study, sex determination was made by morphological analysis from embryonic vascular images obtained in the first week when the light was shined into the egg using a standard camera without any invasive procedure to the egg.
FormNetV2: Multimodal Graph Contrastive Learning for Form Document Information Extraction
Authors: Chen-Yu Lee, Chun-Liang Li, Hao Zhang, Timothy Dozat, Vincent Perot, Guolong Su, Xiang Zhang, Kihyuk Sohn, Nikolai Glushnev, Renshen Wang, Joshua Ainslie, Shangbang Long, Siyang Qin, Yasuhisa Fujii, Nan Hua, Tomas Pfister
Date: 2023-05-04 00:00:00
Tasks: Contrastive Learning, document understanding, Form, Language Modeling, Language Modelling

The recent advent of self-supervised pre-training techniques has led to a surge in the use of multimodal learning in form document understanding. However, existing approaches that extend the mask language modeling to other modalities require careful multi-task tuning, complex reconstruction target designs, or additional pre-training data. In FormNetV2, we introduce a centralized multimodal graph contrastive learning strategy to unify self-supervised pre-training for all modalities in one loss. The graph contrastive objective maximizes the agreement of multimodal representations, providing a natural interplay for all modalities without special customization. In addition, we extract image features within the bounding box that joins a pair of tokens connected by a graph edge, capturing more targeted visual cues without loading a sophisticated and separately pre-trained image embedder. FormNetV2 establishes new state-of-the-art performance on FUNSD, CORD, SROIE and Payment benchmarks with a more compact model size.
6G Integrated Sensing and Communication: From Vision to Realization
Authors: Thorsten Wild, Artjom Grudnitsky, Silvio Mandelli, Marcus Henninger, Junqing Guan, Frank Schaich
Date: 2023-05-03 00:00:00
Tasks: Integrated sensing and communication, ISAC, Pedestrian Detection

Integrated sensing and communications (ISAC) will be deployed into cellular communication systems possibly already with 5G-A and surely in 6G. This paper discusses ISAC use cases, key technology building blocks for system design with solutions and open research questions. Furthermore, we introduce our proof-of-concept (PoC) based on commercially available 5G communications hardware at mm-Wave frequencies, with sensing-specific algorithmic extensions. This new ISAC PoC can perform jointly high data-rate communications and OFDM radar sensing in the same frequency band. Initial pedestrian detection results are shown, indicating the practicability of ISAC in future cellular networks. The results also indicate our achievable sensing range and provide hints to the achievable range estimation accuracy, based on the stability of the PoC system communications hardware.
Scanpath Prediction in Panoramic Videos via Expected Code Length Minimization
Authors: Mu Li, Kanglong Fan, Kede Ma
Date: 2023-05-04 00:00:00
Tasks: Data Compression, Diversity, Imitation Learning, Scanpath prediction

Predicting human scanpaths when exploring panoramic videos is a challenging task due to the spherical geometry and the multimodality of the input, and the inherent uncertainty and diversity of the output. Most previous methods fail to give a complete treatment of these characteristics, and thus are prone to errors. In this paper, we present a simple new criterion for scanpath prediction based on principles from lossy data compression. This criterion suggests minimizing the expected code length of quantized scanpaths in a training set, which corresponds to fitting a discrete conditional probability model via maximum likelihood. Specifically, the probability model is conditioned on two modalities: a viewport sequence as the deformation-reduced visual input and a set of relative historical scanpaths projected onto respective viewports as the aligned path input. The probability model is parameterized by a product of discretized Gaussian mixture models to capture the uncertainty and the diversity of scanpaths from different users. Most importantly, the training of the probability model does not rely on the specification of "ground-truth" scanpaths for imitation learning. We also introduce a proportional-integral-derivative (PID) controller-based sampler to generate realistic human-like scanpaths from the learned probability model. Experimental results demonstrate that our method consistently produces better quantitative scanpath results in terms of prediction accuracy (by comparing to the assumed "ground-truths") and perceptual realism (through machine discrimination) over a wide range of prediction horizons. We additionally verify the perceptual realism improvement via a formal psychophysical experiment and the generalization improvement on several unseen panoramic video datasets.
Self-Supervised 3D Scene Flow Estimation Guided by Superpoints
Authors: Yaqi Shen, Le Hui, Jin Xie, Jian Yang
Date: 2023-05-04 00:00:00
Proceeding: CVPR 2023 1
Tasks: Scene Flow Estimation

3D scene flow estimation aims to estimate point-wise motions between two consecutive frames of point clouds. Superpoints, i.e., points with similar geometric features, are usually employed to capture similar motions of local regions in 3D scenes for scene flow estimation. However, in existing methods, superpoints are generated with the offline clustering methods, which cannot characterize local regions with similar motions for complex 3D scenes well, leading to inaccurate scene flow estimation. To this end, we propose an iterative end-to-end superpoint based scene flow estimation framework, where the superpoints can be dynamically updated to guide the point-level flow prediction. Specifically, our framework consists of a flow guided superpoint generation module and a superpoint guided flow refinement module. In our superpoint generation module, we utilize the bidirectional flow information at the previous iteration to obtain the matching points of points and superpoint centers for soft point-to-superpoint association construction, in which the superpoints are generated for pairwise point clouds. With the generated superpoints, we first reconstruct the flow for each point by adaptively aggregating the superpoint-level flow, and then encode the consistency between the reconstructed flow of pairwise point clouds. Finally, we feed the consistency encoding along with the reconstructed flow into GRU to refine point-level flow. Extensive experiments on several different datasets show that our method can achieve promising performance.
ANetQA: A Large-scale Benchmark for Fine-grained Compositional Reasoning over Untrimmed Videos
Authors: Zhou Yu, Lixiang Zheng, Zhou Zhao, Fei Wu, Jianping Fan, Kui Ren, Jun Yu
Date: 2023-05-04 00:00:00
Proceeding: CVPR 2023 1
Tasks: Question Answering, Spatio-temporal Scene Graphs, Video Question Answering

Building benchmarks to systemically analyze different capabilities of video question answering (VideoQA) models is challenging yet crucial. Existing benchmarks often use non-compositional simple questions and suffer from language biases, making it difficult to diagnose model weaknesses incisively. A recent benchmark AGQA poses a promising paradigm to generate QA pairs automatically from pre-annotated scene graphs, enabling it to measure diverse reasoning abilities with granular control. However, its questions have limitations in reasoning about the fine-grained semantics in videos as such information is absent in its scene graphs. To this end, we present ANetQA, a large-scale benchmark that supports fine-grained compositional reasoning over the challenging untrimmed videos from ActivityNet. Similar to AGQA, the QA pairs in ANetQA are automatically generated from annotated video scene graphs. The fine-grained properties of ANetQA are reflected in the following: (i) untrimmed videos with fine-grained semantics; (ii) spatio-temporal scene graphs with fine-grained taxonomies; and (iii) diverse questions generated from fine-grained templates. ANetQA attains 1.4 billion unbalanced and 13.4 million balanced QA pairs, which is an order of magnitude larger than AGQA with a similar number of videos. Comprehensive experiments are performed for state-of-the-art methods. The best model achieves 44.5% accuracy while human performance tops out at 84.5%, leaving sufficient room for improvement.
A Cross-direction Task Decoupling Network for Small Logo Detection
Authors: Hou, Sujuan, Xingzhuo, Min, Weiqing, Li, Jiacheng, Wang, Jing, Zheng, Yuanjie, Jiang, Shuqiang
Date: 2023-05-04 00:00:00

Logo detection plays an integral role in many applications. However, handling small logos is still difficult since they occupy too few pixels in the image, which burdens the extraction of discriminative features. The aggregation of small logos also brings a great challenge to the classification and localization of logos. To solve these problems, we creatively propose Cross-direction Task Decoupling Network (CTDNet) for small logo detection. We first introduce Cross-direction Feature Pyramid (CFP) to realize cross-direction feature fusion by adopting horizontal transmission and vertical transmission. In addition, Multi-frequency Task Decoupling Head (MTDH) decouples the classification and localization tasks into two branches. A multi frequency attention convolution branch is designed to achieve more accurate regression by combining discrete cosine transform and convolution creatively. Comprehensive experiments on four logo datasets demonstrate the effectiveness and efficiency of the proposed method.
Joint Activity Detection and Channel Estimation for Clustered Massive Machine Type Communications
Authors: Leatile Marata, Onel Luis Alcaraz López, Andreas Hauptmann, Hamza Djelouat, Hirley Alves
Date: 2023-05-04 00:00:00
Tasks: Action Detection, Activity Detection, compressed sensing

Compressed sensing multi-user detection (CS-MUD) algorithms play a key role in optimizing grant-free (GF) non-orthogonal multiple access (NOMA) for massive machine-type communications (mMTC). However, current CS-MUD algorithms cannot be efficiently parallelized, leading to computationally expensive implementations of joint activity detection and channel estimation (JADCE) as the number of deployed machine-type devices (MTDs) increases. To address this, the present work proposes novel JADCE algorithms that can be applied in parallel for different clusters of MTDs by exploiting the structure of the pilot sequences. These are the approximation error method (AEM)-alternating direction method of multipliers (ADMM), and AEM-sparse Bayesian learning (SBL). Results presented in terms of the normalized mean square error and the probability of miss detection show comparable performance to the conventional algorithms. However, both AEM-ADMM and AEM-SBL algorithms have significantly reduced computational complexity and run times, thus, facilitating network scalability.
Using Received Power in Microphone Arrays to Estimate Direction of Arrival
Authors: Gustav Zetterqvist, Fredrik Gustafsson, Gustaf Hendeby
Date: 2023-05-04 00:00:00

Conventional direction of arrival (DOA) estimators are based on array processing using either time differences or beamforming. The proposed approach is based on the received power at each microphone, which enables simple hardware, low sampling frequency and small arrays. The problem is recast into a linear regression framework where the least squares method applies, and the main drawback is that different sound sources are not readily separable. Our proposed approach is based on a training phase where the directional sensitivity of each microphone element is estimated. This model is then used as a fingerprint of the observed power vector in a real-time estimator. The learned power vector is here modeled by a Fourier series expansion, which enables Cram\'er-Rao lower bound computations. We demonstrate the performance using a circular array with eight microphones with promising results.
Estimation of symbol time offset using cyclic prefix in OFDM system
Authors: Ivan Cavlek, Sonja Grgic
Date: 2023-05-04 00:00:00

In this paper synchronization techniques using cyclic prefix (CP) are analyzed to remove the influence of symbol time offset (STO) for correct synchronization in Orthogonal Frequency Division Multiplex (OFDM) system. For correct detection of STO using CP two techniques are used. The first one is based on finding the maximum correlation between two blocks and the second one on finding the similarity between two blocks which is maximized when the difference between them is minimized. Two cases are observed. The first one uses only additive white Gaussian noise (AWGN), while the second also uses the channel impulse response (CIR) of Rayleigh channel. When channel effect is added both performances of estimation by correlation and difference deteriorate if sufficient level of signal to noise (SNR) and proper length of CP are not provided.
The Manifestation of Spatial Wideband Effect in Circular Array: From Beam Split to Beam Defocus
Authors: Zidong Wu, Linglong Dai
Date: 2023-05-04 00:00:00

Millimeter-wave (mmWave) and terahertz (THz) communications with hybrid precoding architectures have been regarded as energy-efficient solutions to fulfill the vision of high-speed transmissions for 6G communications. Benefiting from the advantages of providing a wide scan range and flat array gain, the uniform circular array (UCA) has attracted much attention. However, the growing bandwidth of mmWave and THz communications require frequency-independent phase shifts, which can not be perfectly realized through frequency-independent phase shifters (PSs) in classical hybrid precoding architectures. This mismatch causes the beam defocus effect in UCA wideband communications, where the high-gain beams could not form at non-central frequencies. In this paper, we first investigate the characteristics of the beam defocus effect distinguishing itself from the beam split effect in uniform linear array (ULA) systems. The beam pattern of UCA in both frequency domain and angular domain is analyzed, characterizing the beamforming loss caused by the beam defocus effect. Then, the delay-phase-precoding (DPP) architecture which leverages the true-time-delay (TTD) devices to generate frequency-dependent phase shifts is employed to mitigate the beam defocus effect. Finally, performance analysis and extensive simulation results are provided to evaluate the effectiveness of the DPP architecture in UCA systems.