pwc-archive/papers-with-abstracts huggingface dataset:
Open-World Social Event Classification
Authors: Shengsheng Qian, Hong Chen, Dizhan Xue, Quan Fang, Changsheng Xu
Date: 2023-04-30 00:00:00
Proceeding: WWW 2023 4
Tasks: Classification, Open-World Social Event Classification

With the rapid development of Internet and the expanding scale of social media, social event classification has attracted increasing attention. The key to social event classification is effectively leveraging the visual and textual semantics for classification. However, most of the existing approaches may suffer from the following limitations: (1) Most of them just simply concatenate the image features and text features to get the multimodal features and ignore the fine-grained semantic relationship between modalities. (2) The majority of them hold the closed-world assumption that all classes in test are already seen in training, while this assumption can be easily broken in real-world applications. In practice, new events on Internet may not belong to any existing/seen class, and therefore cannot be correctly identified by closed-world learning algorithms. To tackle these challenges, we propose an Open-World Social Event Classifier (OWSEC) model in this paper. Firstly, we design a multimodal mask transformer network to capture cross-modal semantic relations and fuse fine-grained multimodal features of social events while masking redundant information. Secondly, we design an open-world classifier and propose a cross-modal event mixture mechanism with a novel open-world classification loss to capture the potential distribution space of the unseen class. Extensive experiments on two public datasets demonstrate the superiority of our proposed OWSEC model for open-world social event classification.
Adaptive label-aware graph convolutional networks for cross-modal retrieval
Authors: Shengsheng Qian, Dizhan Xue, Quan Fang, Changsheng Xu
Date: 2021-08-06 00:00:00
Proceeding: IEEE Transactions on Multimedia 2021 8
Tasks: Cross-Modal Retrieval, Representation Learning, Retrieval

The cross-modal retrieval task has raised continuous attention in recent years with the increasing scale of multi-modal data, which has broad application prospects including multimedia data management and intelligent search engine. Most existing methods mainly project data of different modalities into a common representation space where label information is often exploited to distinguish samples from different semantic categories. However, they typically treat each label as an independent individual and ignore the underlying semantic structure of labels. In this paper, we propose an end-to-end adaptive label-aware graph convolutional network (ALGCN) by designing both the instance representation learning branch and the label representation learning branch, which can obtain modality-invariant and discriminative representations for cross-modal retrieval. Firstly, we construct an instance representation learning branch to transform instances of different modalities into a common representation space. Secondly, we adopt Graph Convolutional Network (GCN) to learn inter-dependent classifiers in the label representation learning branch. In addition, a novel adaptive correlation matrix is proposed to efficiently explore and preserve the semantic structure of labels in a data-driven manner. Together with a robust self-supervision loss for GCN, the GCN model can be supervised to learn an effective and robust correlation matrix for feature propagation. Comprehensive experimental results on three benchmark datasets, NUS-WIDE, MIRFlickr and MS-COCO, demonstrate the superiority of ALGCN, compared with the state-of-the-art methods in cross-modal retrieval.
Hierarchical memory decoder for visual narrating
Authors: Aming Wu, Yahong Han, Zhou Zhao, Yi Yang
Date: 2020-09-01 00:00:00
Proceeding: IEEE Transactions on Circuits and Systems for Video Technology 2020 9
Tasks: Decoder, Image Captioning, Video Captioning, Visual Storytelling

Visual narrating focuses on generating semantic descriptions to summarize visual content of images or videos, e.g., visual captioning and visual storytelling. The challenge mainly lies in how to design a decoder to generate accurate descriptions matching visual content. Recent advances often employ a recurrent neural network (RNN), e.g., Long-Short Term Memory (LSTM), as the decoder. However, RNN is prone to diluting long-term information, which weakens its performance of capturing long-term dependencies. Recent work has demonstrated memory network (MemNet) owns the advantage of storing long-term information. However, as the decoder, it has not been well exploited for visual narrating. The reason partially comes from the difficulty of multi-modal sequential decoding with MemNet. In this article, we devise a novel memory decoder for visual narrating. Concretely, to obtain a better multi-modal representation, we first design a new multi-modal fusion method to fully merge visual and lexical information. Then, based on the fusion result, during decoding, we construct a MemNet-based decoder consisting of multiple memory layers. Particularly, in each layer, we employ a memory set to store previous decoding information and utilize an attention mechanism to adaptively select the information related to the current output. Meanwhile, we also employ a memory set to store the decoding output of each memory layer at the current time step and still utilize an attention mechanism to select the related information. Thus, this decoder alleviates dilution of long-term information. Meanwhile, the hierarchical architecture leverages the latent information of each layer, which is helpful for generating accurate descriptions. Experimental results on two tasks of visual narrating, i.e., video captioning and visual storytelling, show that our decoder could obtain superior results and outperform the performance of conventional RNN-based decoder.
Re-ID-AR: Improved Person Re-identification in Video via Joint Weakly Supervised Action Recognition
Authors: Alsehaim, A., Breckon, T.P.
Date: 2021-11-01 00:00:00
Proceeding: BMVC 2021 11
Tasks: Action Recognition, Person Re-Identification, Video Understanding, Weakly-Supervised Action Recognition

We uniquely consider the task of joint person re-identification (Re-ID) and action recognition in video as a multi-task problem. In addition to the broader potential of joint Re-ID and action recognition within the context of automated multi-camera surveillance, we show that the consideration of action recognition in addition to Re-ID results in a model that learns discriminative feature representations that both improve Re-ID performance and are capable of providing viable per-view (clip-wise) action recognition. Our approach uses a single 2D Convolutional Neural Network (CNN) architecture comprising a common ResNet50-IBN backbone CNN architecture, to extract frame-level features with subsequent temporal attention for clip level feature extraction, followed by two sub-branches:- the IDentification (sub-)Network (IDN) for person Re-ID and the Action Recognition (sub-)Network for per-view action recognition. The IDN comprises a single fully connected layer while the ARN comprises multiple attention blocks on a one-to-one ratio with the number of actions to be recognised. This is subsequently trained as a joint Re-ID and action recognition task using a combination of two task-specific, multi-loss terms via weakly labelled actions obtained over two leading benchmark Re-ID datasets (MARS, LPW). Our consideration of Re-ID and action recognition as a multi-task problem results in a multi-branch 2D CNN architecture that outperforms prior work in the field (rank-1 (mAP) – MARS: 93.21%(87.23%), LPW: 79.60%) without any reliance 3D convolutions or multi-stream networks architectures as found in other contemporary work. Our work represents the first benchmark performance for such a joint Re-ID and action recognition video understanding task, hitherto unapproached in the literature, and is accompanied by a new public dataset of supplementary action labels for the seminal MARS and LPW Re-ID datasets.
VID-Trans-ReID: Enhanced Video Transformers for Person Re-identification
Authors: Alsehaim, A.; Breckon, T.
Date: 2022-11-01 00:00:00
Proceeding: BMVC 2022 11
Tasks: Person Re-Identification, Video-Based Person Re-Identification

Video-based person Re-identification (Re-ID) has received increasing attention recently due to its important role within surveillance video analysis. Video-based Re- ID expands upon earlier image-based methods by extracting person features temporally across multiple video image frames. The key challenge within person Re-ID is extracting a robust feature representation that is invariant to the challenges of pose and illumination variation across multiple camera viewpoints. Whilst most contemporary methods use a CNN based methodology, recent advances in vision transformer (ViT) architectures boost fine-grained feature discrimination via the use of both multi-head attention without any loss of feature robustness. To specifically enable ViT architectures to effectively address the challenges of video person Re-ID, we propose two novel modules constructs, Temporal Clip Shift and Shuffled (TCSS) and Video Patch Part Feature (VPPF), that boost the robustness of the resultant Re-ID feature representation. Furthermore, we combine our proposed approach with current best practices spanning both image and video based Re-ID including camera view embedding. Our proposed approach outperforms existing state-of-the-art work on the MARS, PRID2011, and iLIDS-VID Re-ID benchmark datasets achieving 96.36%, 96.63%, 94.67% rank-1 accuracy respectively and achieving 90.25% mAP on MARS.
Integrated Robotics Networks with Co-optimization of Drone Placement and Air-Ground Communications
Authors: Menghao Hu, Tong Zhang, Shuai Wang, Guoliang Li, Yingyang Chen, Qiang Li, Gaojie Chen
Date: 2023-09-09 00:00:00

Terrestrial robots, i.e., unmanned ground vehicles (UGVs), and aerial robots, i.e., unmanned aerial vehicles (UAVs), operate in separate spaces. To exploit their complementary features (e.g., fields of views, communication links, computing capabilities), a promising paradigm termed integrated robotics network emerges, which provides communications for cooperative UAVs-UGVs applications. However, how to efficiently deploy UAVs and schedule the UAVs-UGVs connections according to different UGV tasks become challenging. In this paper, we propose a sum-rate maximization problem, where UGVs plan their trajectories autonomously and are dynamically associated with UAVs according to their planned trajectories. Although the problem is a NP-hard mixed integer program, a fast polynomial time algorithm using alternating gradient descent and penalty-based binary relaxation, is devised. Simulation results demonstrate the effectiveness of the proposed algorithm.
Real-time Learning of Driving Gap Preference for Personalized Adaptive Cruise Control
Authors: Zhouqiao Zhao, Xishun Liao, Amr Abdelraouf, Kyungtae Han, Rohit Gupta, Matthew J. Barth, Guoyuan Wu
Date: 2023-09-10 00:00:00
Tasks: Incremental Learning

Advanced Driver Assistance Systems (ADAS) are increasingly important in improving driving safety and comfort, with Adaptive Cruise Control (ACC) being one of the most widely used. However, pre-defined ACC settings may not always align with driver's preferences and habits, leading to discomfort and potential safety issues. Personalized ACC (P-ACC) has been proposed to address this problem, but most existing research uses historical driving data to imitate behaviors that conform to driver preferences, neglecting real-time driver feedback. To bridge this gap, we propose a cloud-vehicle collaborative P-ACC framework that incorporates driver feedback adaptation in real time. The framework is divided into offline and online parts. The offline component records the driver's naturalistic car-following trajectory and uses inverse reinforcement learning (IRL) to train the model on the cloud. In the online component, driver feedback is used to update the driving gap preference in real time. The model is then retrained on the cloud with driver's takeover trajectories, achieving incremental learning to better match driver's preference. Human-in-the-loop (HuiL) simulation experiments demonstrate that our proposed method significantly reduces driver intervention in automatic control systems by up to 62.8%. By incorporating real-time driver feedback, our approach enhances the comfort and safety of P-ACC, providing a personalized and adaptable driving experience.
Expanding Mars Climate Modeling: Interpretable Machine Learning for Modeling MSL Relative Humidity
Authors: Nour Abdelmoneim, Dattaraj B. Dhuri, Dimitra Atri, Germán Martínez
Date: 2023-09-04 00:00:00
Tasks: Decision Making, Interpretable Machine Learning

For the past several decades, numerous attempts have been made to model the climate of Mars with extensive studies focusing on the planet's dynamics and the understanding of its climate. While physical modeling and data assimilation approaches have made significant progress, uncertainties persist in comprehensively capturing and modeling the complexities of Martian climate. In this work, we propose a novel approach to Martian climate modeling by leveraging machine learning techniques that have shown remarkable success in Earth climate modeling. Our study presents a deep neural network designed to accurately model relative humidity in Gale Crater, as measured by NASA's Mars Science Laboratory ``Curiosity'' rover. By utilizing simulated meteorological variables produced by the Mars Planetary Climate Model, a robust Global Circulation Model, our model accurately predicts relative humidity with a mean error of 3\% and an $R^2$ score of 0.92. Furthermore, we present an approach to predict quantile ranges of relative humidity, catering to applications that require a range of values. To address the challenge of interpretability associated with machine learning models, we utilize an interpretable model architecture and conduct an in-depth analysis of its internal mechanisms and decision making processes. We find that our neural network can effectively model relative humidity at Gale crater using a few meteorological variables, with the monthly mean surface H$_2$O layer, planetary boundary layer height, convective wind speed, and solar zenith angle being the primary contributors to the model predictions. In addition to providing a fast and efficient method to modeling climate variables on Mars, this modeling approach can also be used to expand on current datasets by filling spatial and temporal gaps in observations.
Artificial intelligence based glaucoma and diabetic retinopathy detection using MATLAB — retrained AlexNet convolutional neural network
Authors: Isaac Arias, Paolo Velásquez, Laura Avila, Fanny Laurido, Fernando Villalba, Andrés Tirado, Jonathan Cruz, Diego Almeida
Date: 2023-01-05 00:00:00
Proceeding: F1000Research 2023 1
Tasks: Diabetic Retinopathy Detection, Diagnostic, image-classification, Image Classification, Transfer Learning

Background: Glaucoma and diabetic retinopathy are the leading causes of blindness due to an irreversible damage to the retina which results in vision loss. Early detection of these diseases through regular screening is especially important to prevent progression. The image of retinal fundus is the main evaluating strategy for the glaucoma and diabetic retinopathy detection. Then, automated eye disease detection is an important application of retinal image analysis. Compared with classical diagnostic techniques, image classification by convolutional neural networks (CNN) have the potential for better cost-effective performance.

Methods: In this paper, we propose the use of MATLAB – retrained AlexNet CNN for computerized eye diseases identification, particularly glaucoma and diabetic retinopathy, by employing retinal fundus images. The acquisition of the database was carried out through free access databases and access upon request. A transfer learning technique is used for retraining the AlexNet CNN. Specifically, the model divides the fundus image dataset into training and testing data.

Results: As datasets were added by network training, different values were reported for validation accuracy, false positives and false negatives, precision, and recall. Thus, having NetTransfer I with a validation accuracy value of 94.3% for two classes. NetTransfer II with a validation accuracy value of 91.8% for two classes. NetTransfer III with a validation accuracy value of 89.7% for three classes. Net transfer IV with a validation accuracy value of 93.1% for three classes. Finally, NetTransfer V with a validation accuracy value of 92.1% for three classes.

Conclusions: Re-training of the AlexNet network proved to be a powerful tool to create disease detection systems having high accuracy values and being able to discern between more than two diseases.
A Context-Sensitive Approach to XAI in Music Performance
Authors: Nicola Privato, Jack Armitage
Date: 2023-09-05 00:00:00
Tasks: Explainable artificial intelligence, Explainable Artificial Intelligence (XAI), Position

The rapidly evolving field of Explainable Artificial Intelligence (XAI) has generated significant interest in developing methods to make AI systems more transparent and understandable. However, the problem of explainability cannot be exhaustively solved in the abstract, as there is no single approach that can be universally applied to generate adequate explanations for any given AI system, and this is especially true in the arts. In this position paper, we propose an Explanatory Pragmatism (EP) framework for XAI in music performance, emphasising the importance of context and audience in the development of explainability requirements. By tailoring explanations to specific audiences and continuously refining them based on feedback, EP offers a promising direction for enhancing the transparency and interpretability of AI systems in broad artistic applications and more specifically to music performance.
The Moral Machine Experiment on Large Language Models
Authors: Kazuhiro Takemoto
Date: 2023-09-12 00:00:00
Tasks: Autonomous Driving, Decision Making

As large language models (LLMs) become more deeply integrated into various sectors, understanding how they make moral judgments has become crucial, particularly in the realm of autonomous driving. This study utilized the Moral Machine framework to investigate the ethical decision-making tendencies of prominent LLMs, including GPT-3.5, GPT-4, PaLM 2, and Llama 2, comparing their responses to human preferences. While LLMs' and humans' preferences such as prioritizing humans over pets and favoring saving more lives are broadly aligned, PaLM 2 and Llama 2, especially, evidence distinct deviations. Additionally, despite the qualitative similarities between the LLM and human preferences, there are significant quantitative disparities, suggesting that LLMs might lean toward more uncompromising decisions, compared to the milder inclinations of humans. These insights elucidate the ethical frameworks of LLMs and their potential implications for autonomous driving.
ADC/DAC-Free Analog Acceleration of Deep Neural Networks with Frequency Transformation
Authors: Nastaran Darabi, Maeesha Binte Hashem, Hongyi Pan, Ahmet Cetin, Wilfred Gomes, Amit Ranjan Trivedi
Date: 2023-09-04 00:00:00
Tasks: Computational Efficiency, Model Compression

The edge processing of deep neural networks (DNNs) is becoming increasingly important due to its ability to extract valuable information directly at the data source to minimize latency and energy consumption. Frequency-domain model compression, such as with the Walsh-Hadamard transform (WHT), has been identified as an efficient alternative. However, the benefits of frequency-domain processing are often offset by the increased multiply-accumulate (MAC) operations required. This paper proposes a novel approach to an energy-efficient acceleration of frequency-domain neural networks by utilizing analog-domain frequency-based tensor transformations. Our approach offers unique opportunities to enhance computational efficiency, resulting in several high-level advantages, including array micro-architecture with parallelism, ADC/DAC-free analog computations, and increased output sparsity. Our approach achieves more compact cells by eliminating the need for trainable parameters in the transformation matrix. Moreover, our novel array micro-architecture enables adaptive stitching of cells column-wise and row-wise, thereby facilitating perfect parallelism in computations. Additionally, our scheme enables ADC/DAC-free computations by training against highly quantized matrix-vector products, leveraging the parameter-free nature of matrix multiplications. Another crucial aspect of our design is its ability to handle signed-bit processing for frequency-based transformations. This leads to increased output sparsity and reduced digitization workload. On a 16$\times$16 crossbars, for 8-bit input processing, the proposed approach achieves the energy efficiency of 1602 tera operations per second per Watt (TOPS/W) without early termination strategy and 5311 TOPS/W with early termination strategy at VDD = 0.8 V.
A Circuit Domain Generalization Framework for Efficient Logic Synthesis in Chip Design
Authors: Zhihai Wang, Lei Chen, Jie Wang, Xing Li, Yinqi Bai, Xijun Li, Mingxuan Yuan, Jianye Hao, Yongdong Zhang, Feng Wu
Date: 2023-08-22 00:00:00
Tasks: Domain Generalization

Logic Synthesis (LS) plays a vital role in chip design -- a cornerstone of the semiconductor industry. A key task in LS is to transform circuits -- modeled by directed acyclic graphs (DAGs) -- into simplified circuits with equivalent functionalities. To tackle this task, many LS operators apply transformations to subgraphs -- rooted at each node on an input DAG -- sequentially. However, we found that a large number of transformations are ineffective, which makes applying these operators highly time-consuming. In particular, we notice that the runtime of the Resub and Mfs2 operators often dominates the overall runtime of LS optimization processes. To address this challenge, we propose a novel data-driven LS operator paradigm, namely PruneX, to reduce ineffective transformations. The major challenge of developing PruneX is to learn models that well generalize to unseen circuits, i.e., the out-of-distribution (OOD) generalization problem. Thus, the major technical contribution of PruneX is the novel circuit domain generalization framework, which learns domain-invariant representations based on the transformation-invariant domain-knowledge. To the best of our knowledge, PruneX is the first approach to tackle the OOD problem in LS operators. We integrate PruneX with the aforementioned Resub and Mfs2 operators. Experiments demonstrate that PruneX significantly improves their efficiency while keeping comparable optimization performance on industrial and very large-scale circuits, achieving up to $3.1\times$ faster runtime.
AI driven B-cell Immunotherapy Design
Authors: Bruna Moreira da Silva, David B. Ascher, Nicholas Geard, Douglas E. V. Pires
Date: 2023-09-03 00:00:00
Tasks: Prediction, Specificity

Antibodies, a prominent class of approved biologics, play a crucial role in detecting foreign antigens. The effectiveness of antigen neutralisation and elimination hinges upon the strength, sensitivity, and specificity of the paratope-epitope interaction, which demands resource-intensive experimental techniques for characterisation. In recent years, artificial intelligence and machine learning methods have made significant strides, revolutionising the prediction of protein structures and their complexes. The past decade has also witnessed the evolution of computational approaches aiming to support immunotherapy design. This review focuses on the progress of machine learning-based tools and their frameworks in the domain of B-cell immunotherapy design, encompassing linear and conformational epitope prediction, paratope prediction, and antibody design. We mapped the most commonly used data sources, evaluation metrics, and method availability and thoroughly assessed their significance and limitations, discussing the main challenges ahead.