pwc-archive/papers-with-abstracts huggingface dataset:
DLCRec: A Novel Approach for Managing Diversity in LLM-Based Recommender Systems
Authors: Jiaju Chen, Chongming Gao, Shuai Yuan, Shuchang Liu, Qingpeng Cai, Peng Jiang
Date: 2024-08-22 00:00:00
Tasks: Data Augmentation, Diversity, Recommendation Systems

The integration of Large Language Models (LLMs) into recommender systems has led to substantial performance improvements. However, this often comes at the cost of diminished recommendation diversity, which can negatively impact user satisfaction. To address this issue, controllable recommendation has emerged as a promising approach, allowing users to specify their preferences and receive recommendations that meet their diverse needs. Despite its potential, existing controllable recommender systems frequently rely on simplistic mechanisms, such as a single prompt, to regulate diversity-an approach that falls short of capturing the full complexity of user preferences. In response to these limitations, we propose DLCRec, a novel framework designed to enable fine-grained control over diversity in LLM-based recommendations. Unlike traditional methods, DLCRec adopts a fine-grained task decomposition strategy, breaking down the recommendation process into three sequential sub-tasks: genre prediction, genre filling, and item prediction. These sub-tasks are trained independently and inferred sequentially according to user-defined control numbers, ensuring more precise control over diversity. Furthermore, the scarcity and uneven distribution of diversity-related user behavior data pose significant challenges for fine-tuning. To overcome these obstacles, we introduce two data augmentation techniques that enhance the model's robustness to noisy and out-of-distribution data. These techniques expose the model to a broader range of patterns, improving its adaptability in generating recommendations with varying levels of diversity. Our extensive empirical evaluation demonstrates that DLCRec not only provides precise control over diversity but also outperforms state-of-the-art baselines across multiple recommendation scenarios.
Dynamic Product Image Generation and Recommendation at Scale for Personalized E-commerce
Authors: Ádám Tibor Czapp, Mátyás Jani, Bálint Domián, Balázs Hidasi
Date: 2024-08-22 00:00:00
Tasks: Image Generation, Multi-Armed Bandits

Coupling latent diffusion based image generation with contextual bandits enables the creation of eye-catching personalized product images at scale that was previously either impossible or too expensive. In this paper we showcase how we utilized these technologies to increase user engagement with recommendations in online retargeting campaigns for e-commerce.
Hardware Acceleration for Knowledge Graph Processing: Challenges & Recent Developments
Authors: Maciej Besta, Robert Gerstenberger, Patrick Iff, Pournima Sonawane, Juan Gómez Luna, Raghavendra Kanakagiri, Rui Min, Grzegorz Kwaśniewski, Onur Mutlu, Torsten Hoefler, Raja Appuswamy, Aidan O Mahony
Date: 2024-08-22 00:00:00
Tasks: Knowledge Graphs, Systematic Literature Review

Knowledge graphs (KGs) have achieved significant attention in recent years, particularly in the area of the Semantic Web as well as gaining popularity in other application domains such as data mining and search engines. Simultaneously, there has been enormous progress in the development of different types of heterogeneous hardware, impacting the way KGs are processed. The aim of this paper is to provide a systematic literature review of knowledge graph hardware acceleration. For this, we present a classification of the primary areas in knowledge graph technology that harnesses different hardware units for accelerating certain knowledge graph functionalities. We then extensively describe respective works, focusing on how KG related schemes harness modern hardware accelerators. Based on our review, we identify various research gaps and future exploratory directions that are anticipated to be of significant value both for academics and industry practitioners.
Behavior Pattern Mining-based Multi-Behavior Recommendation
Authors: Haojie Li, Zhiyong Cheng, Xu Yu, Jinhuan Liu, Guanfeng Liu, Junwei Du
Date: 2024-08-22 00:00:00
Tasks: Graph Neural Network, Recommendation Systems

Multi-behavior recommendation systems enhance effectiveness by leveraging auxiliary behaviors (such as page views and favorites) to address the limitations of traditional models that depend solely on sparse target behaviors like purchases. Existing approaches to multi-behavior recommendations typically follow one of two strategies: some derive initial node representations from individual behavior subgraphs before integrating them for a comprehensive profile, while others interpret multi-behavior data as a heterogeneous graph, applying graph neural networks to achieve a unified node representation. However, these methods do not adequately explore the intricate patterns of behavior among users and items. To bridge this gap, we introduce a novel algorithm called Behavior Pattern mining-based Multi-behavior Recommendation (BPMR). Our method extensively investigates the diverse interaction patterns between users and items, utilizing these patterns as features for making recommendations. We employ a Bayesian approach to streamline the recommendation process, effectively circumventing the challenges posed by graph neural network algorithms, such as the inability to accurately capture user preferences due to over-smoothing. Our experimental evaluation on three real-world datasets demonstrates that BPMR significantly outperforms existing state-of-the-art algorithms, showing an average improvement of 268.29% in Recall@10 and 248.02% in NDCG@10 metrics. The code of our BPMR is openly accessible for use and further research at https://github.com/rookitkitlee/BPMR.
What are the limits of cross-lingual dense passage retrieval for low-resource languages?
Authors: Jie Wu, Zhaochun Ren, Suzan Verberne
Date: 2024-08-21 00:00:00
Tasks: Answer Generation, Language Modeling, Language Modelling, Passage Retrieval, Question Answering, Retrieval

In this paper, we analyze the capabilities of the multi-lingual Dense Passage Retriever (mDPR) for extremely low-resource languages. In the Cross-lingual Open-Retrieval Answer Generation (CORA) pipeline, mDPR achieves success on multilingual open QA benchmarks across 26 languages, of which 9 were unseen during training. These results are promising for Question Answering (QA) for low-resource languages. We focus on two extremely low-resource languages for which mDPR performs poorly: Amharic and Khmer. We collect and curate datasets to train mDPR models using Translation Language Modeling (TLM) and question--passage alignment. We also investigate the effect of our extension on the language distribution in the retrieval results. Our results on the MKQA and AmQA datasets show that language alignment brings improvements to mDPR for the low-resource languages, but the improvements are modest and the results remain low. We conclude that fulfilling CORA's promise to enable multilingual open QA in extremely low-resource settings is challenging because the model, the data, and the evaluation approach are intertwined. Hence, all three need attention in follow-up work. We release our code for reproducibility and future work: https://anonymous.4open.science/r/Question-Answering-for-Low-Resource-Languages-B13C/
Chain-of-Exemplar: Enhancing Distractor Generation for Multimodal Educational Question Generation
Authors: Haohao Luo, Yang Deng, Ying Shen, See-Kiong Ng, Tat-Seng Chua
Date: 2024-08-16 00:00:00
Proceeding: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics 2024 8
Tasks: Distractor Generation, Multiple-choice, Question Generation, Question-Generation, Retrieval

Multiple-choice questions (MCQs) are important in enhancing concept learning and student engagement for educational purposes. Despite the multimodal nature of educational content, current methods focus mainly on text-based inputs and often neglect the integration of visual information. In this work, we study the problem of multimodal educational question generation, which aims at generating subject-specific educational questions with plausible yet incorrect distractors based on multimodal educational content. To tackle this problem, we introduce a novel framework, named Chain-of-Exemplar (CoE), which utilizes multimodal large language models (MLLMs) with Chain-of-Thought reasoning to improve the generation of challenging distractors. Furthermore, CoE leverages three-stage contextualized exemplar retrieval to retrieve exemplary questions as guides for generating more subject-specific educational questions. Experimental results on the ScienceQA benchmark demonstrate the superiority of CoE in both question generation and distractor generation over existing methods across various subjects and educational levels.
Unifying Text, Tables, and Images for Multimodal Question Answering
Authors: Haohao Luo, Ying Shen, Yang Deng
Date: 2023-12-10 00:00:00
Proceeding: Findings of the Association for Computational Linguistics: EMNLP 2023 12
Tasks: Image Captioning, Question Answering, Text Generation

Multimodal question answering (MMQA), which aims to derive the answer from multiple knowledge modalities (e.g., text, tables, and images), has received increasing attention due to its board applications. Current approaches to MMQA often rely on single-modal or bi-modal QA models, which limits their ability to effectively integrate information across all modalities and leverage the power of pre-trained language models. To address these limitations, we propose a novel framework called UniMMQA, which unifies three different input modalities into a text-to-text format by employing position-enhanced table linearization and diversified image captioning techniques. Additionally, we enhance cross-modal reasoning by incorporating a multimodal rationale generator, which produces textual descriptions of cross-modal relations for adaptation into the text-to-text generation process. Experimental results on three MMQA benchmark datasets show the superiority of UniMMQA in both supervised and unsupervised settings.
Transformer-Based Approaches for Automatic Music Transcription
Authors: Christos Zonios
Date: 2021-02-12 00:00:00
Proceeding: Conference 2021 2
Tasks: Language Modelling, Music Transcription, speech-recognition, Speech Recognition, Transfer Learning

Automatic Music Transcription (AMT) is the process of extracting information
from audio into some form of music notation. In polyphonic music, this is a very
hard problem for computers to solve as it requires significant prior knowledge and
understanding of music language and the audio is subject to a multitude of variations
in frequencies depending on many factors such as instrument materials, tuning, player
performance, recording equipment and others.
Transformers are self-supervised models that have recently showed great promise
as they use self-attention in order to learn contextual representations from unlabeled
data. They have surpassed state of the art (SOTA) performance in various Speech
Recognition (SR), Natural Language Processing (NLP) and Computer Vision tasks.
In this work, we examine transformer-based approaches for performing AMT on
piano recordings by learning audio and music language representations. Specifically,
we look at the popular SR model wav2vec2 as a solution to the former and the NLP
model BERT in order to perform Music Language Modelling (MusicLM).
We propose a new pre-training approach for MusicLM transformers based on
an appropriately defined transcription error correction task. In addition, three novel
models for AMT are proposed and studied that appropriately integrate wav2vec2 and
BERT transformers at various stages.
We conclude that a wav2vec2 encoder model pre-trained on speech audio is not
able to surpass SOTA models using mel-scale spectrograms and convolutional network
encoders without significant conditioning on music audio.
viii
We show that a BERT transformer pre-trained on natural language has transfer
learning potential for MusicLM. We also examine the robustness of such a transformer
for performing MusicLM, and find that we are able to achieve interesting results when
doing Masked MusicLM and when replacing Recurrent Neural Networks with pretrained transformers in SOTA models for AMT.
From Pixels to Prices with ViTMAE: Integrating Real Estate Images through Masked Autoencoder Vision Transformers (ViTMAE) with Conventional Real Estate Data for Enhanced Automated Valuation
Authors: Ekblad Voltaire, Fanny
Date: 2024-04-20 00:00:00
Proceeding: journal 2024 4

The integration of Vision Transformers (ViTs) using Masked Autoencoder pre-training (ViTMAE) into real estate valuation is investigated in this Master’s thesis, addressing the challenge of effectively analyzing visual information from real estate images. This integration aims to enhance the accuracy and efficiency of valuation, a task traditionally dependent on realtor expertise. The research involved developing a model that combines ViTMAE-extracted visual features from real estate images with traditional property data. Focusing on residential properties in Sweden, the study utilized a dataset of images and metadata from online real estate listings. An adapted ViTMAE model, accessed via the Hugging Face library, was trained on the dataset for feature extraction, which was then integrated with metadata to create a comprehensive multimodal valuation model. Results indicate that including ViTMAE-extracted image features improves prediction accuracy in real estate valuation models. The multimodal approach, merging visual and traditional metadata, improved accuracy over metadata-only models. This thesis contributes to real estate valuation by showcasing the potential of advanced image processing techniques in enhancing valuation models. It lays the groundwork for future research in more refined holistic valuation models, incorporating a wider range of factors beyond visual data.
FibroVit—Vision transformer-based framework for detection and classification of pulmonary fibrosis from chest CT images
Date: 2023-11-08 00:00:00
Tasks: Diagnostic, Specificity

Pulmonary Fibrosis (PF) is an immedicable respiratory condition distinguished by permanent fibrotic alterations in the pulmonary tissue for which there is no cure. Hence, it is crucial to diagnose PF swiftly and precisely. The existing research on deep learning-based pulmonary fibrosis detection methods has limitations, including dataset sample sizes and a lack of standardization in data preprocessing and evaluation metrics. This study presents a comparative analysis of four vision transformers regarding their efficacy in accurately detecting and classifying patients with Pulmonary Fibrosis and their ability to localize abnormalities within Images obtained from Computerized Tomography (CT) scans. The dataset consisted of 13,486 samples selected out of 24647 from the Pulmonary Fibrosis dataset, which included both PF-positive CT and normal images that underwent preprocessing. The preprocessed images were divided into three sets: the training set, which accounted for 80% of the total pictures; the validation set, which comprised 10%; and the test set, which also consisted of 10%. The vision transformer models, including ViT, MobileViT2, ViTMSN, and BEiT were subjected to training and validation procedures, during which hyperparameters like the learning rate and batch size were fine-tuned. The overall performance of the optimized architectures has been assessed using various performance metrics to showcase the consistent performance of the fine-tuned model. Regarding performance, ViT has shown superior performance in validation and testing accuracy and loss minimization, specifically for CT images when trained at a single epoch with a tuned learning rate of 0.0001. The results were as follows: validation accuracy of 99.85%, testing accuracy of 100%, training loss of 0.0075, and validation loss of 0.0047. The experimental evaluation of the independently collected data gives empirical evidence that the optimized Vision Transformer (ViT) architecture exhibited superior performance compared to all other optimized architectures. It achieved a flawless score of 1.0 in various standard performance metrics, including Sensitivity, Specificity, Accuracy, F1-score, Precision, Recall, Mathew Correlation Coefficient (MCC), Precision-Recall Area under the Curve (AUC PR), Receiver Operating Characteristic and Area Under the Curve (ROC-AUC). Therefore, the optimized Vision Transformer (ViT) functions as a reliable diagnostic tool for the automated categorization of individuals with pulmonary fibrosis (PF) using chest computed
The Mechanics of Conceptual Interpretation in GPT Models: Interpretative Insights
Authors: Nura Aljaafari, Danilo S. Carvalho, André Freitas
Date: 2024-08-05 00:00:00
Tasks: knowledge editing, Reverse Dictionary

Locating and editing knowledge in large language models (LLMs) is crucial for enhancing their accuracy, safety, and inference rationale. We introduce ``concept editing'', an innovative variation of knowledge editing that uncovers conceptualisation mechanisms within these models. Using the reverse dictionary task, inference tracing, and input abstraction, we analyse the Multi-Layer Perceptron (MLP), Multi-Head Attention (MHA), and hidden state components of transformer models. Our results reveal distinct patterns: MLP layers employ key-value retrieval mechanism and context-dependent processing, which are highly associated with relative input tokens. MHA layers demonstrate a distributed nature with significant higher-level activations, suggesting sophisticated semantic integration. Hidden states emphasise the importance of the last token and top layers in the inference process. We observe evidence of gradual information building and distributed representation. These observations elucidate how transformer models process semantic information, paving the way for targeted interventions and improved interpretability techniques. Our work highlights the complex, layered nature of semantic processing in LLMs and the challenges of isolating and modifying specific concepts within these models.
Evaluating the Validity of Word-level Adversarial Attacks with Large Language Models
Authors: Huichi Zhou, Zhaoyang Wang, Hongtao Wang, Dongping Chen, Wenhan Mu, Fangyuan Zhang
Date: 2024-08-15 00:00:00
Proceeding: Findings of the Association for Computational Linguistics ACL 2024 8
Tasks: Adversarial Attack, Language Modeling, Language Modelling, Large Language Model, valid

Deep neural networks exhibit vulnerability to word-level adversarial attacks in natural language processing. Most of these attack methods adopt synonymous substitutions to perturb original samples for crafting adversarial examples while attempting to maintain semantic consistency with the originals. Some of them claim that they could achieve over 90% attack success rate, thereby raising serious safety concerns. However, our investigation reveals that many purportedly successful adversarial examples are actually invalid due to significant changes in semantic meanings compared to their originals. Even when equipped with semantic constraints such as BERTScore, existing attack methods can generate up to 87.9% invalid adversarial examples. Building on this insight, we first curate a 13K dataset for adversarial validity evaluation with the help of GPT-4. Then, an open-source large language model is fine-tuned to offer an interpretable validity score for assessing the semantic consistency between original and adversarial examples. Finally, this validity score can serve as a guide for existing adversarial attack methods to generate valid adversarial examples. Comprehensive experiments demonstrate the effectiveness of our method in evaluating and refining the quality of adversarial examples.
MASNet: A Robust Deep Marine Animal Segmentation Network
Authors: Zhenqi Fu, Ruizhe Chen, Yue Huang, En Cheng, Xinghao Ding, Kai-Kuang Ma
Date: 2023-05-01 00:00:00
Proceeding: IEEE Journal of Oceanic Engineering 2023 5
Tasks: Data Augmentation, Image Segmentation, Marine Animal Segmentation, Segmentation, Semantic Segmentation

Marine animal studies are of great importance to human beings and instrumental to many research areas. How to identify such animals through image processing is a challenging task that leads to marine animal segmentation (MAS). Although deep neural networks have been widely applied for object segmentation, few of them consider the complex imaging condition in the water and the camouflage property of marine animals. To this end, a robust deep marine animal segmentation network is proposed in this article. Specifically, we design a new data augmentation strategy to randomly change the degradation and camouflage attributes of the original objects. With the augmentations, a fusion-based deep neural network constructed in a Siamese manner is trained to learn the shared semantic representations. Moreover, we construct a new large-scale real-world MAS data set for conducting extensive experiments. It consists of over 3000 images with various underwater scenes and objects. Each image is annotated with an object-level mask and assigned to a category. Extensive experimental results show that our method significantly outperforms 12 state-of-the-art methods both qualitatively and quantitatively.
A Novel Signal Detection Method for Photon-Counting Communications with Nonlinear Distortion Effects
Authors: Chen Wang, Zhiyong Xu, Jingyuan Wang, Jianhua Li, Weifeng Mou, Huatao Zhu, Jiyong Zhao, Yang Su, Yimin Wang, Ailin Qi
Date: 2024-08-20 00:00:00
Tasks: Blocking

This paper proposes a method for estimating and detecting optical signals in practical photon-counting receivers. There are two important aspects of non-perfect photon-counting receivers, namely, (i) dead time which results in blocking loss, and (ii) non-photon-number-resolving, which leads to counting loss during the gate-ON interval. These factors introduce nonlinear distortion to the detected photon counts. The detected photon counts depend not only on the optical intensity but also on the signal waveform, and obey a Poisson binomial process. Using the discrete Fourier transform characteristic function (DFT-CF) method, we derive the probability mass function (PMF) of the detected photon counts. Furthermore, unlike conventional methods that assume an ideal rectangle wave, we propose a novel signal estimation and decision method applicable to arbitrary waveform. We demonstrate that the proposed method achieves superior error performance compared to conventional methods. The proposed algorithm has the potential to become an essential signal processing tool for photon-counting receivers.
A General-Purpose Device for Interaction with LLMs
Authors: Jiajun Xu, Qun Wang, Yuhang Cao, Baitao Zeng, Sicheng Liu
Date: 2024-08-02 00:00:00

This paper investigates integrating large language models (LLMs) with advanced hardware, focusing on developing a general-purpose device designed for enhanced interaction with LLMs. Initially, we analyze the current landscape, where virtual assistants and LLMs are reshaping human-technology interactions, highlighting pivotal advancements and setting the stage for a new era of intelligent hardware. Despite substantial progress in LLM technology, a significant gap exists in hardware development, particularly concerning scalability, efficiency, affordability, and multimodal capabilities. This disparity presents both challenges and opportunities, underscoring the need for hardware that is not only powerful but also versatile and capable of managing the sophisticated demands of modern computation. Our proposed device addresses these needs by emphasizing scalability, multimodal data processing, enhanced user interaction, and privacy considerations, offering a comprehensive platform for LLM integration in various applications.