pwc-archive/papers-with-abstracts huggingface dataset:
Detection of electric network frequency in audio recordings-from theory to practical detectors
Authors: Guang Hua, Han Liao, Qingyi Wang, Haijian Zhang, Dengpan Ye
Date: 2020-07-17 00:00:00
Proceeding: IEEE TRANSACTIONS ON INFORMATION FORENSICS AND SECURITY 2020 7
Tasks: ENF (Electric Network Frequency) Detection, ENF (Electric Network Frequency) Extraction

Recently, it has been discovered that the electric network frequency (ENF) could be captured by digital audio, video, or even image files, and could further be exploited in forensic investigations. However, the existence of the ENF in multimedia content is not a sure thing, and if the ENF is not present, ENF-based forensic analysis would become useless or even misleading. In this paper, we address the problem of ENF detection in digital audio recordings, which is modeled as the detection of a weak (ENF) signal contaminated by unknown colored wide-sense stationary (WSS) Gaussian noise, while the signal also contains multiple unknown random parameters. We first derive three Neyman-Pearson (NP) detectors, i.e., general matched filter (GMF), matched filter (MF)-like detector, and the asymptotic approximation of the GMF, and choose the MF-like detector as the clairvoyant detector. For practical detectors, we show that the generalized likelihood ratio test (GLRT) could not be efficiently obtained due to the unknown noise and large matrix inversion. Alternatively, we propose two least-squares (LS)-based time domain detectors termed as LS-likelihood ratio test (LRT) and naive-LRT. Further, we propose a time-frequency (TF) domain detector, termed as TF detector, which exploits the a priori knowledge of the ENF. The performances of the derived detectors are extensively analyzed in terms of test statistic distributions, threshold selection, and computational complexity. The naive-LRT detector is found to be only effective for very short recordings. As the data recording length increases, both LS-LRT and TF detectors yield effective detection results, while the latter is approximately a constant false alarm rate (CFAR) detector. Practical experiments using real audio recordings justify the effectiveness of the proposed detectors and our analysis.
Temporal Localization of Non-Static Digital Videos Using the Electrical Network Frequency
Authors: Samuel Fernández-Menduiña, Fernando Pérez-González
Date: 2020-04-20 00:00:00
Proceeding: IEEE Signal Processing Letters 2020 4
Tasks: ENF (Electric Network Frequency) Extraction from Video, Temporal Localization

Non-static scenes represent one of the main barriers for retrieving the electrical network frequency (ENF) from digital videos recorded in realistic scenarios, since movement influences the luminance of pixels, hindering the recovery of the desired information. Aiming to mitigate the effects of changes in the scene, in this letter a video processing stage, that detects and combines those pixels that are not affected by movement, is proposed. The sequence obtained thereby is delivered to a PLL-based FM demodulator, that estimates the ENF taking into account its autoregressive nature. Additionally, a frame rate estimation algorithm is employed to avoid time-consuming trial and error loops in the extraction process. The performance of the system is tested by computing the probability of estimating correctly the time of recording of a given video, using ground-truth sequences obtained directly from the mains or a reliable database.
Multimodal Emotion Recognition based on Facial Expressions, Speech, and EEG
Authors: JiaHui Pan, Member, Weijie Fang, Zhihang Zhang, Bingzhi Chen*, Zheng Zhang, Shuihua Wang, Senior Member, IEEE
Date: 2024-06-11 00:00:00
Proceeding: IEEE Open Journal of Engineering in Medicine and Biology 2024 6
Tasks: EEG, Emotion Recognition, Multimodal Emotion Recognition

As an essential human-machine interactive
task, emotion recognition has become an emerging area over the
decades. Although previous attempts to classify emotions have
achieved high performance, several challenges remain open: 1)

How to effectively recognize emotions using different modali-
ties remains challenging. 2) Due to the increasing amount of

computing power required for deep learning, how to provide
real-time detection and improve the robustness of deep neural
networks is important. Method: In this paper, we propose a deep
learning-based multimodal emotion recognition (MER) called

Deep-Emotion, which can adaptively integrate the most discrim-
inating features from facial expressions, speech, and electroen-
cephalogram (EEG) to improve the performance of the MER.

Specifically, the proposed Deep-Emotion framework consists of
three branches, i.e., the facial branch, speech branch, and EEG
branch. Correspondingly, the facial branch uses the improved
GhostNet neural network proposed in this paper for feature
extraction, which effectively alleviates the overfitting phenomenon
in the training process and improves the classification accuracy
compared with the original GhostNet network. For work on the

speech branch, this paper proposes a lightweight fully convolu-
tional neural network (LFCNN) for the efficient extraction of

speech emotion features. Regarding the study of EEG branches,
we proposed a tree-like LSTM (tLSTM) model capable of fusing
multi-stage features for EEG emotion feature extraction. Finally,
we adopted the strategy of decision-level fusion to integrate
the recognition results of the above three modes, resulting
in more comprehensive and accurate performance. Result and
Conclusions: Extensive experiments on the CK+, EMO-DB, and
MAHNOB-HCI datasets have demonstrated the advanced nature
of the Deep-Emotion method proposed in this paper, as well as
the feasibility and superiority of the MER approach.
SpecGaussian with Latent Features: A High-quality Modeling of the View-dependent Appearance for 3D Gaussian Splatting
Authors: Zhiru Wang, Shiyun Xie, Chengwei Pan, Guoping Wang
Date: 2024-08-23 00:00:00
Tasks: Decoder, Novel View Synthesis

Recently, the 3D Gaussian Splatting (3D-GS) method has achieved great success in novel view synthesis, providing real-time rendering while ensuring high-quality rendering results. However, this method faces challenges in modeling specular reflections and handling anisotropic appearance components, especially in dealing with view-dependent color under complex lighting conditions. Additionally, 3D-GS uses spherical harmonic to learn the color representation, which has limited ability to represent complex scenes. To overcome these challenges, we introduce Lantent-SpecGS, an approach that utilizes a universal latent neural descriptor within each 3D Gaussian. This enables a more effective representation of 3D feature fields, including appearance and geometry. Moreover, two parallel CNNs are designed to decoder the splatting feature maps into diffuse color and specular color separately. A mask that depends on the viewpoint is learned to merge these two colors, resulting in the final rendered image. Experimental results demonstrate that our method obtains competitive performance in novel view synthesis and extends the ability of 3D-GS to handle intricate scenarios with specular reflections.
A phase-based approach for ENF signal extraction from rolling shutter videos
Authors: Hyekyung Han, Youngbae Jeon, Baek-kyung Song, Ji Won Yoon
Date: 2022-07-08 00:00:00
Proceeding: IEEE Signal Processing Letters 2022 7
Tasks: ENF (Electric Network Frequency) Extraction, ENF (Electric Network Frequency) Extraction from Video

Electric Network Frequency (ENF) analysis has been an intriguing tool for multimedia forensics as former studies have paved the way for estimating ENF signals from digital audio, video, or even image files. However, for ENF signals to be widely used in extensive applications, supplementary research is needed so that ENF signals can be stably extracted without restrictions. In this letter, we propose a new phase-based approach for extracting ENF signals from CMOS sensor recordings. It uses phase differences between row signals from two consecutive frames, such that problems due to missing sample points during the idle periods are circumvented. The proposed method has substantial advantages in that it is applicable without a predefined read-out time and when the length of given videos is too short. Extensive experiments conducted with numerous devices demonstrate that the proposed method can take precedence over state-of-the-art methods because it robustly produces accurate ENF estimates in terms of alias frequency on the frame-level.
Zero-note samba: Self-supervised beat tracking
Authors: Dorian Desblancs, Vincent Lostanlen, Romain Hennequin
Date: 2023-07-21 00:00:00
Proceeding: IEEE/ACM Transactions on Audio, Speech, and Language Processing 2023 7
Tasks: Beat Tracking, Information Retrieval, Music Information Retrieval, Self-Supervised Learning

Supervised machine learning for music information retrieval requires a large annotated training set, and is thus an expensive and time-consuming process. To circumvent this problem, we propose to train deep neural networks to perceive beats in musical recordings despite having little or no access to human annotations. The key idea is to train two fully convolutional networks in parallel, which we name “Zero-Note Samba” (ZeroNS): the first analyzes the percussive part of a musical piece whilst the second analyzes its non-percussive part. These networks learn a self-supervised pretext task of synchrony prediction (sync-pred), which simulates the ability of musicians to groove together when playing in the same band. Sync-pred encourages the two networks to return similar outputs if the underlying musical parts are synchronized, yet dissimilar outputs if the parts are out of sync. In practice, we obtain the instrumental parts from commercial recordings via an off-the-shelf source separation system: Spleeter. After self-supervised learning with sync-pred, ZeroNS produces a sparse output that resembles a beat detection function. When used in conjunction with a dynamic Bayesian network, ZeroNS surpasses the state of the art in unsupervised beat tracking. Furthermore, fine-tuning ZeroNS to a small set of labeled data (of the order of one to ten songs) matches the performance of a fully supervised network on 96 songs. Lastly, we show that pre-training a supervised model with sync-pred mitigates dataset bias and thus improves cross-dataset generalization, at no extra annotation cost.
Filtering Approaches for Dealing with Noise in Anomaly Detection
Authors: Navid Hashemi, Eduardo Verdugo German, Jonatan Pena Ramirez, Justin Ruths
Date: 2019-09-03 00:00:00
Tasks: Anomaly Detection

The leading workhorse of anomaly (and attack) detection in the literature has been residual-based detectors, where the residual is the discrepancy between the observed output provided by the sensors (inclusive of any tampering along the way) and the estimated output provided by an observer. These techniques calculate some statistic of the residual and apply a threshold to determine whether or not to raise an alarm. To date, these methods have not leveraged the frequency content of the residual signal in making the detection problem easier, specifically dealing with the case of (e.g., measurement) noise. Here we demonstrate some opportunities to combine filtering to enhance the performance of residual-based detectors. We also demonstrate how filtering can provide a compelling alternative to residual-based methods when paired with a robust observer. In this process, we consider the class of attacks that are stealthy, or undetectable, by such filtered detection methods and the impact they can have on the system.
DetailCLIP: Detail-Oriented CLIP for Fine-Grained Tasks
Authors: Amin Karimi Monsefi, Kishore Prakash Sailaja, Ali Alilooee, Ser-Nam Lim, Rajiv Ramnath
Date: 2024-09-10 00:00:00
Tasks: Contrastive Learning, Image Reconstruction, Language Modeling, Language Modelling, Segmentation, Self-Supervised Learning

In this paper, we introduce DetailCLIP: A Detail-Oriented CLIP to address the limitations of contrastive learning-based vision-language models, particularly CLIP, in handling detail-oriented and fine-grained tasks like segmentation. While CLIP and its variants excel in the global alignment of image and text representations, they often struggle to capture the fine-grained details necessary for precise segmentation. To overcome these challenges, we propose a novel framework that employs patch-level comparison of self-distillation and pixel-level reconstruction losses, enhanced with an attention-based token removal mechanism. This approach selectively retains semantically relevant tokens, enabling the model to focus on the image's critical regions aligned with the specific functions of our model, including textual information processing, patch comparison, and image reconstruction, ensuring that the model learns high-level semantics and detailed visual features. Our experiments demonstrate that DetailCLIP surpasses existing CLIP-based and traditional self-supervised learning (SSL) models in segmentation accuracy and exhibits superior generalization across diverse datasets. DetailCLIP represents a significant advancement in vision-language modeling, offering a robust solution for tasks that demand high-level semantic understanding and detailed feature extraction. https://github.com/KishoreP1/DetailCLIP.
Think Together and Work Better: Combining Humans' and LLMs' Think-Aloud Outcomes for Effective Text Evaluation
Authors: SeongYeub Chu, JongWoo Kim, MunYong Yi
Date: 2024-09-11 00:00:00

This study introduces \textbf{InteractEval}, a framework that integrates human expertise and Large Language Models (LLMs) using the Think-Aloud (TA) method to generate attributes for checklist-based text evaluation. By combining human flexibility and reasoning with LLM consistency, InteractEval outperforms traditional non-LLM-based and LLM-based baselines across four distinct dimensions, consisting of Coherence, Fluency, Consistency, and Relevance. The experiment also investigates the effectiveness of the TA method, showing that it promotes divergent thinking in both humans and LLMs, leading to the generation of a wider range of relevant attributes and enhance text evaluation performance. Comparative analysis reveals that humans excel at identifying attributes related to internal quality (Coherence and Fluency), but LLMs perform better at those attributes related to external alignment (Consistency and Relevance). Consequently, leveraging both humans and LLMs together produces the best evaluation outcomes. In other words, this study emphasizes the necessity of effectively combining humans and LLMs in an automated checklist-based text evaluation framework. The code is available at \textbf{\url{https://github.com/BBeeChu/InteractEval.git}}.
Ontology-Free General-Domain Knowledge Graph-to-Text Generation Dataset Synthesis using Large Language Model
Authors: Daehee Kim, Deokhyung Kang, Sangwon Ryu, Gary Geunbae Lee
Date: 2024-09-11 00:00:00
Tasks: Data-to-Text Generation, Graph-to-Sequence, Knowledge Graphs, Language Modeling, Language Modelling, Large Language Model, Text Generation

Knowledge Graph-to-Text (G2T) generation involves verbalizing structured knowledge graphs into natural language text. Recent advancements in Pretrained Language Models (PLMs) have improved G2T performance, but their effectiveness depends on datasets with precise graph-text alignment. However, the scarcity of high-quality, general-domain G2T generation datasets restricts progress in the general-domain G2T generation research. To address this issue, we introduce Wikipedia Ontology-Free Graph-text dataset (WikiOFGraph), a new large-scale G2T dataset generated using a novel method that leverages Large Language Model (LLM) and Data-QuestEval. Our new dataset, which contains 5.85M general-domain graph-text pairs, offers high graph-text consistency without relying on external ontologies. Experimental results demonstrate that PLM fine-tuned on WikiOFGraph outperforms those trained on other datasets across various evaluation metrics. Our method proves to be a scalable and effective solution for generating high-quality G2T data, significantly advancing the field of G2T generation.
"My Grade is Wrong!": A Contestable AI Framework for Interactive Feedback in Evaluating Student Essays
Authors: Shengxin Hong, Chang Cai, Sixuan Du, Haiyue Feng, Siyuan Liu, Xiuyi Fan
Date: 2024-09-11 00:00:00

Interactive feedback, where feedback flows in both directions between teacher and student, is more effective than traditional one-way feedback. However, it is often too time-consuming for widespread use in educational practice. While Large Language Models (LLMs) have potential for automating feedback, they struggle with reasoning and interaction in an interactive setting. This paper introduces CAELF, a Contestable AI Empowered LLM Framework for automating interactive feedback. CAELF allows students to query, challenge, and clarify their feedback by integrating a multi-agent system with computational argumentation. Essays are first assessed by multiple Teaching-Assistant Agents (TA Agents), and then a Teacher Agent aggregates the evaluations through formal reasoning to generate feedback and grades. Students can further engage with the feedback to refine their understanding. A case study on 500 critical thinking essays with user studies demonstrates that CAELF significantly improves interactive feedback, enhancing the reasoning and interaction capabilities of LLMs. This approach offers a promising solution to overcoming the time and resource barriers that have limited the adoption of interactive feedback in educational settings.
A Novel Perturb-ability Score to Mitigate Evasion Adversarial Attacks on Flow-Based ML-NIDS
Authors: Mohamed elShehaby, Ashraf Matrawy
Date: 2024-09-11 00:00:00
Tasks: feature selection, Intrusion Detection, Network Intrusion Detection

As network security threats evolve, safeguarding flow-based Machine Learning (ML)-based Network Intrusion Detection Systems (NIDS) from evasion adversarial attacks is crucial. This paper introduces the notion of feature perturb-ability and presents a novel Perturb-ability Score (PS), which quantifies how susceptible NIDS features are to manipulation in the problem-space by an attacker. PS thereby identifies features structurally resistant to evasion attacks in flow-based ML-NIDS due to the semantics of network traffic fields, as these features are constrained by domain-specific limitations and correlations. Consequently, attempts to manipulate such features would likely either compromise the attack's malicious functionality, render the traffic invalid for processing, or potentially both outcomes simultaneously. We introduce and demonstrate the effectiveness of our PS-enabled defenses, PS-guided feature selection and PS-guided feature masking, in enhancing flow-based NIDS resilience. Experimental results across various ML-based NIDS models and public datasets show that discarding or masking highly manipulatable features (high-PS features) can maintain solid detection performance while significantly reducing vulnerability to evasion adversarial attacks. Our findings confirm that PS effectively identifies flow-based NIDS features susceptible to problem-space perturbations. This novel approach leverages problem-space NIDS domain constraints as lightweight universal defense mechanisms against evasion adversarial attacks targeting flow-based ML-NIDS.
SUPER: Evaluating Agents on Setting Up and Executing Tasks from Research Repositories
Authors: Ben Bogin, Kejuan Yang, Shashank Gupta, Kyle Richardson, Erin Bransom, Peter Clark, Ashish Sabharwal, Tushar Khot
Date: 2024-09-11 00:00:00

Given that Large Language Models (LLMs) have made significant progress in writing code, can they now be used to autonomously reproduce results from research repositories? Such a capability would be a boon to the research community, helping researchers validate, understand, and extend prior work. To advance towards this goal, we introduce SUPER, the first benchmark designed to evaluate the capability of LLMs in setting up and executing tasks from research repositories. SUPERaims to capture the realistic challenges faced by researchers working with Machine Learning (ML) and Natural Language Processing (NLP) research repositories. Our benchmark comprises three distinct problem sets: 45 end-to-end problems with annotated expert solutions, 152 sub problems derived from the expert set that focus on specific challenges (e.g., configuring a trainer), and 602 automatically generated problems for larger-scale development. We introduce various evaluation measures to assess both task success and progress, utilizing gold solutions when available or approximations otherwise. We show that state-of-the-art approaches struggle to solve these problems with the best model (GPT-4o) solving only 16.3% of the end-to-end set, and 46.1% of the scenarios. This illustrates the challenge of this task, and suggests that SUPER can serve as a valuable resource for the community to make and measure progress.
Synthetic continued pretraining
Authors: Zitong Yang, Neil Band, Shuangping Li, Emmanuel Candès, Tatsunori Hashimoto
Date: 2024-09-11 00:00:00
Tasks: Data Augmentation, Language Modelling, Retrieval-augmented Generation, World Knowledge

Pretraining on large-scale, unstructured internet text enables language models to acquire a significant amount of world knowledge. However, this knowledge acquisition is data-inefficient--to learn a given fact, models must be trained on hundreds to thousands of diverse representations of it. This poses a challenge when adapting a pretrained model to a small corpus of domain-specific documents, where each fact may appear rarely or only once. We propose to bridge this gap with synthetic continued pretraining: using the small domain-specific corpus to synthesize a large corpus more amenable to learning, and then performing continued pretraining on the synthesized corpus. We instantiate this proposal with EntiGraph, a synthetic data augmentation algorithm that extracts salient entities from the source documents and then generates diverse text by drawing connections between the sampled entities. Synthetic continued pretraining with EntiGraph enables a language model to answer questions and follow generic instructions related to the source documents without access to them. If, instead, the source documents are available at inference time, we show that the knowledge acquired through our approach compounds with retrieval-augmented generation. To better understand these results, we build a simple mathematical model of EntiGraph, and show how synthetic data augmentation can "rearrange" knowledge to enable more data-efficient learning.