pwc-archive/papers-with-abstracts huggingface dataset:
HumSum: A Personalized Lecture Summarization Tool for Humanities Students Using LLMs
Authors: Zahra Kolagar, Alessandra Zarcone
Date: 2024-03-17 00:00:00
Proceeding: Proceedings of the 1st Workshop on Personalization of Generative AI Systems (PERSONALIZE 2024) 2024 3

Generative AI systems aim to create customizable content for their users, with a subsequent surge in demand for adaptable tools that can create personalized experiences. This paper presents HumSum, a web-based tool tailored for humanities students to effectively summarize their lecture transcripts and to personalize the summaries to their specific needs. We first conducted a survey driven by different potential scenarios to collect user preferences to guide the implementation of this tool. Utilizing Streamlit, we crafted the user interface, while Langchain’s Map Reduce function facilitated the summarization process for extensive lectures using OpenAI’s GPT-4 model. HumSum is an intuitive tool serving various summarization needs, infusing personalization into the tool’s functionality without necessitating the collection of personal user data.
Aligning Uncertainty: Leveraging LLMs to Analyze Uncertainty Transfer in Text Summarization
Authors: Zahra Kolagar, Alessandra Zarcone
Date: 2024-03-17 00:00:00
Proceeding: Proceedings of the 1st Workshop on Uncertainty-Aware NLP (UncertaiNLP 2024) 2024 3
Tasks: Articles, Text Summarization

Automatically generated summaries can be evaluated along different dimensions, one being how faithfully the uncertainty from the source text is conveyed in the summary. We present a study on uncertainty alignment in automatic summarization, starting from a two-tier lexical and semantic categorization of linguistic expression of uncertainty, which we used to annotate source texts and automatically generate summaries. We collected a diverse dataset including news articles and personal blogs and generated summaries using GPT-4. Source texts and summaries were annotated based on our two-tier taxonomy using a markup language. The automatic annotation was refined and validated by subsequent iterations based on expert input. We propose a method to evaluate the fidelity of uncertainty transfer in text summarization. The method capitalizes on a small amount of expert annotations and on the capabilities of Large language models (LLMs) to evaluate how the uncertainty of the source text aligns with the uncertainty expressions in the summary.
GiCCS: A German in-Context Conversational Similarity Benchmark
Authors: Shima Asaadi, Zahra Kolagar, Alina Liebel, Alessandra Zarcone
Date: 2022-12-16 00:00:00
Proceeding: Proceedings of the 2nd Workshop on Natural Language Generation, Evaluation, and Metrics (GEM) 2022 12
Tasks: Benchmarking, Semantic Textual Similarity, STS

The Semantic textual similarity (STS) task is commonly used to evaluate the semantic representations that language models (LMs) learn from texts, under the assumption that good-quality representations will yield accurate similarity estimates. When it comes to estimating the similarity of two utterances in a dialogue, however, the conversational context plays a particularly important role. We argue for the need of benchmarks specifically created using conversational data in order to evaluate conversational LMs in the STS task. We introduce GiCCS, a first conversational STS evaluation benchmark for German. We collected the similarity annotations for GiCCS using best-worst scaling and presenting the target items in context, in order to obtain highly-reliable context-dependent similarity scores. We present benchmarking experiments for evaluating LMs on capturing the similarity of utterances. Results suggest that pretraining LMs on conversational data and providing conversational context can be useful for capturing similarity of utterances in dialogues. GiCCS will be publicly available to encourage benchmarking of conversational LMs.
Let’s Write Privacy: Survey and Experiment Design on Textual Privacy in Conversation
Authors: Franka Bayer, Zahra Kolagar, Darina Gold
Date: 2023-11-17 00:00:00
Proceeding: Mensch und Computer 2023 - Workshopband, MCI-WS11: 9. Usable Security und Privacy Workshop 2023 11

This research article emphasizes the significance of privacy in textual conversational data, specifically in the context of interactions between virtual assistants and users. While previous studies have primarily focused on named entities containing personally identifiable information, this research focuses on quasi-identifiers.´Due to the limited availability of conversational data in privacyinducing domains like healthcare, police force, and finance, the study employs a discrete prompting technique to engage large language models in generating conversations between hypothetical users and virtual assistants. To facilitate this process, this research introduces a straightforward interface. Additionally, a card-sorting experiment design is presented to involve participants in matching the conversations with persona cards representing hypothetical users. Subsequently, participants are asked to identify the information that aided them in making the appropriate matches. The experiment aims to outline the elements within textual data that can expose the identities of individuals engaged in conversations.
MMT-Bench: A Comprehensive Multimodal Benchmark for Evaluating Large Vision-Language Models Towards Multitask AGI
Authors: Kaining Ying, Fanqing Meng, Jin Wang, Zhiqian Li, Han Lin, Yue Yang, Hao Zhang, Wenbo Zhang, Yuqi Lin, Shuo Liu, Jiayi Lei, Quanfeng Lu, Runjian Chen, Peng Xu, Renrui Zhang, Haozhe Zhang, Peng Gao, Yali Wang, Yu Qiao, Ping Luo, Kaipeng Zhang, Wenqi Shao
Date: 2024-04-24 00:00:00

Large Vision-Language Models (LVLMs) show significant strides in general-purpose multimodal applications such as visual dialogue and embodied navigation. However, existing multimodal evaluation benchmarks cover a limited number of multimodal tasks testing rudimentary capabilities, falling short in tracking LVLM development. In this study, we present MMT-Bench, a comprehensive benchmark designed to assess LVLMs across massive multimodal tasks requiring expert knowledge and deliberate visual recognition, localization, reasoning, and planning. MMT-Bench comprises $31,325$ meticulously curated multi-choice visual questions from various multimodal scenarios such as vehicle driving and embodied navigation, covering $32$ core meta-tasks and $162$ subtasks in multimodal understanding. Due to its extensive task coverage, MMT-Bench enables the evaluation of LVLMs using a task map, facilitating the discovery of in- and out-of-domain tasks. Evaluation results involving $30$ LVLMs such as the proprietary GPT-4V, GeminiProVision, and open-sourced InternVL-Chat, underscore the significant challenges posed by MMT-Bench. We anticipate that MMT-Bench will inspire the community to develop next-generation multimodal foundation models aimed at achieving general-purpose multimodal intelligence.
Metric3Dv2: A Versatile Monocular Geometric Foundation Model for Zero-shot Metric Depth and Surface Normal Estimation
Authors: Mu Hu, Wei Yin, Chi Zhang, Zhipeng Cai, Xiaoxiao Long, Kaixuan Wang, Hao Chen, Gang Yu, Chunhua Shen, Shaojie Shen
Date: 2024-03-22 00:00:00
Proceeding: Under review for Transaction 2024 4
Tasks: Depth Estimation, Surface Normal Estimation, Zero-shot Generalization

We introduce Metric3D v2, a geometric foundation model for zero-shot metric depth and surface normal estimation from a single image, which is crucial for metric 3D recovery. While depth and normal are geometrically related and highly complimentary, they present distinct challenges. SoTA monocular depth methods achieve zero-shot generalization by learning affine-invariant depths, which cannot recover real-world metrics. Meanwhile, SoTA normal estimation methods have limited zero-shot performance due to the lack of large-scale labeled data. To tackle these issues, we propose solutions for both metric depth estimation and surface normal estimation. For metric depth estimation, we show that the key to a zero-shot single-view model lies in resolving the metric ambiguity from various camera models and large-scale data training. We propose a canonical camera space transformation module, which explicitly addresses the ambiguity problem and can be effortlessly plugged into existing monocular models. For surface normal estimation, we propose a joint depth-normal optimization module to distill diverse data knowledge from metric depth, enabling normal estimators to learn beyond normal labels. Equipped with these modules, our depth-normal models can be stably trained with over 16 million of images from thousands of camera models with different-type annotations, resulting in zero-shot generalization to in-the-wild images with unseen camera settings. Our method enables the accurate recovery of metric 3D structures on randomly collected internet images, paving the way for plausible single-image metrology. Our project page is at https://JUGGHM.github.io/Metric3Dv2.
Removal of speckle noises from ultrasound images using five different deep learning networks
Authors: Karaoğlu, O., Bilge, H. Ş., & Uluer, İ
Date: 2021-06-16 00:00:00
Proceeding: Engineering Science and Technology an International Journal 2021 6
Tasks: Deep Learning, Denoising, Image Denoising, Image Enhancement, Image Restoration, SSIM

Image enhancement methods are applied to medical images to reduce the noise that they contain. There are many academic studies in the literature using classical image enhancement methods. Ultrasound imaging is a medical imaging method that is used for the diagnosis of diseases. In this study, speckle noises with Rayleigh distribution at four different noise levels (σ = 0.10, 0.25, 0.50, 0.75) are added to ultrasound images of the brachial plexus nerve region. Five different deep learning networks (Dilated Convolution Autoencoder Denoising Network/Di-Conv-AE-Net, Denoising U-Shaped Net/D-U-Net, BatchRenormalization U-Net/Br-U-Net, Generative Adversarial Denoising Network/DGan-Net, and CNN Residual Network/DeRNet) are used for reducing the speckle noises of the ultrasound images. The performances of the deep networks are compared with block-matching and 3D filtering (BM3D), which is one of the most preferred classical image enhancement algorithms; with classical filters including Bilateral, Frost, Kuan, Lee, Mean, and Median Filters; and with deep learning networks including Learning Pixel-Distribution Prior with Wider Convolution for Image Denoising (WIN5-RB), Denoising Prior Driven Deep Neural Network for Image Restoration (DPDNN), and Fingerprint Image Denoising and Inpainting Using M-Net Based Convolutional Neural Networks (FPD-M-Net). Network performance is evaluated according to peak signal-to-noise ratio (PSNR), structural similarity index (SSIM), and runtime criteria and the proposed deep learning networks are shown to outperform the other networks.
Protein language models are performant in structure-free virtual screening
Authors: Hilbert Lam, Guan Jia Sheng, Ong Xing Er, Robbe Pincket, Mu Yuguang
Date: 2024-04-20 00:00:00
Proceeding: bioRxiv 2024 4
Tasks: Drug Design, Molecular Docking

Hitherto virtual screening has been typically performed using a structure-based drug design paradigm. Such methods typically require the use of molecular docking on high-resolution three-dimensional structures of a target protein - a computationally-intensive and time consuming exercise. This work demonstrates that by employing protein language models and molecular graphs as inputs to a novel graph-to-transformer cross-attention mechanism, a screening power comparable to state-of-the-art structure-based models can be achieved. The implications thereof include highly expedited virtual screening due to the greatly reduced compute required to run this model, and the ability to perform early stages of computer-aided drug design in the complete absence of 3D protein structure.
CharacterFactory: Sampling Consistent Characters with GANs for Diffusion Models
Authors: Qinghe Wang, Baolu Li, Xiaomin Li, Bing Cao, Liqian Ma, Huchuan Lu, Xu Jia
Date: 2024-04-24 00:00:00
Tasks: Consistent Character Generation, Word Embeddings

Recent advances in text-to-image models have opened new frontiers in human-centric generation. However, these models cannot be directly employed to generate images with consistent newly coined identities. In this work, we propose CharacterFactory, a framework that allows sampling new characters with consistent identities in the latent space of GANs for diffusion models. More specifically, we consider the word embeddings of celeb names as ground truths for the identity-consistent generation task and train a GAN model to learn the mapping from a latent space to the celeb embedding space. In addition, we design a context-consistent loss to ensure that the generated identity embeddings can produce identity-consistent images in various contexts. Remarkably, the whole model only takes 10 minutes for training, and can sample infinite characters end-to-end during inference. Extensive experiments demonstrate excellent performance of the proposed CharacterFactory on character creation in terms of identity consistency and editability. Furthermore, the generated characters can be seamlessly combined with the off-the-shelf image/video/3D diffusion models. We believe that the proposed CharacterFactory is an important step for identity-consistent character generation. Project page is available at: https://qinghew.github.io/CharacterFactory/.
Learning Latent Partial Matchings with Gumbel-IPF Networks
Authors: Hedda Cohen Indelman, Tamir Hazan
Date: 2024-04-03 00:00:00
Proceeding: Proceedings of the 26th International Conference on Artificial Intelligence and Statistics (AISTATS) 2024 4
Tasks: Graph Matching

Learning to match discrete objects has been a central task in machine learning, often facilitated by a continuous relaxation of the matching structure. However, practical problems entail partial matchings due to missing correspondences, which pose difficulties for the one-to-one matching learning techniques that dominate the state-of-the-art. This paper introduces Gumbel-IPF networks for learning latent partial matchings. At the core of our method is the differentiable Iterative Proportional Fitting (IPF) procedure that disproportionally projects onto the transportation polytope of target marginals. Our theoretical framework also allows drawing samples from the temperature-dependent partial matching distribution. We investigate the properties of common-practice relaxations through the lens of biproportional fitting and introduce a new metric, the empirical prediction shift. Our method’s advantages are demonstrated in experimental results on the semantic keypoints partial matching task on the Pascal VOC, IMC-PT-SparseGM, and CUB2001 datasets.
Joint Training on Multiple Datasets With Inconsistent Labeling Criteria for Facial Expression Recognition
Authors: Chengyan Yu, Dong Zhang, Wei Zou, Ming Li
Date: 2024-04-02 00:00:00
Proceeding: IEEE Transactions on Affective Computing 2024 4
Tasks: Facial Expression Recognition, Facial Expression Recognition (FER), Multi-Task Learning

One potential way to enhance the performance of facial expression recognition (FER) is to augment the training set by increasing the number of samples. By incorporating multiple FER datasets, deep learning models can extract more discriminative features. However, the inconsistent labeling criteria and subjective biases found in annotated FER datasets can significantly hinder the recognition accuracy of deep learning models when handling mixed datasets. Effectively perform joint training on multiple datasets remains a challenging task. In this study, we propose a joint training method for training an FER model using multiple FER datasets. Our method consists of four steps: (1) selecting a subset from the additional dataset, (2) generating pseudo-continuous labels for the target dataset, (3) refining the labels of different datasets using continuous label mapping and discrete label relabeling according to the labeling criteria of the target dataset, and (4) jointly training the model using multi-task learning. We conduct joint training experiments on two popular in-the-wild FER benchmark databases, RAF-DB and CAER-S, while utilizing the AffectNet dataset as an additional dataset. The experimental results demonstrate that our proposed method outperforms the direct merging of different FER datasets into a single training set and achieves state-of-the-art performance on RAF-DB and CAER-S with accuracies of 92.24% and 94.57%, respectively.
Single Stage Adaptive Multi-Attention Network for Image Restoration
Authors: Anas Zafar, Danyal Aftab, Rizwan Qureshi, Xinqi Fan, Pingjun Chen, Jia Wu, Hazrat Ali, Shah Nawaz, Sheheryar Khan, Mubarak Shah
Date: 2024-04-10 00:00:00
Proceeding: IEEE Transactions on Image Processing 2024 4
Tasks: Deblurring, Denoising, Image Deblurring, Image Denoising, Image Restoration

Recently attention-based networks have been successful for image restoration tasks. However, existing methods are either computationally expensive or have limited receptive fields, adding constraints to the model. They are also less resilient in spatial and contextual aspects and lack pixel-to-pixel correspondence, which may degrade feature representations. In this paper, we propose a novel and computationally efficient architecture Single Stage Adaptive Multi-Attention Network (SSAMAN) for image restoration tasks, particularly for image denoising and image deblurring. SSAMAN efficiently addresses computational challenges and expands receptive fields, enhancing robustness in spatial and contextual feature representation. Its Adaptive Multi-Attention Module (AMAM), which consists of Adaptive Pixel Attention Branch (APAB) and an Adaptive Channel Attention Branch (ACAB), uniquely integrates channel and pixel-wise dimensions, significantly improving sensitivity to edges, shapes, and textures. We perform extensive experiments and ablation studies to validate the performance of SSAMAN. Our model shows state-of-the-art results on various benchmarks, for example, on image denoising tasks, SSAMAN achieves a notable 40.08 dB PSNR on SIDD dataset, outperforming Restormer by 0.06 dB PSNR, with 41.02% less computational cost, and achieves a 40.05 dB PSNR on the DND dataset. For image deblurring, SSAMAN achieves 33.53 dB PSNR on GoPro dataset. Code and models are available at Github.
MaskChanger: A Transformer-Based Model Tailoring Change Detection with Mask Classification
Authors: Mohammad Ebrahimzadeh, Mohammad Taghi Manzuri
Date: 2024-03-06 00:00:00
Proceeding: 13th Iranian/3rd International Machine Vision and Image Processing Conference (MVIP) 2024 3
Tasks: Building change detection for remote sensing images, Change Detection, Decoder

Change detection in multi-temporal remote sensing data enables crucial urban analysis and environmental monitoring applications. However, complex factors like illumination variance and occlusion make robust automated change interpretation challenging. We propose MaskChanger - a novel deep learning paradigm tailored for satellite image change detection. Our method adapts the segmentation-specialized Mask2Former architecture by incorporating Siamese networks to extract features separately from bi-temporal images, while retaining the original mask transformer decoder. To our knowledge, this is the first study in which change detection is converted from the existing per-pixel classification approach into a mask classification approach. Evaluated on the LEVIR-CD benchmark of over 600 very high-resolution image pairs exhibiting real-world rural and urban changes, MaskChanger achieves Fl-Score of 91.96%, outperforming prior transformer-based change detection approaches.
Few-shot Calligraphy Style Learning
Authors: Fangda Chen, Jiacheng Nie, Lichuan Jiang, Zhuoer Zeng
Date: 2024-04-26 00:00:00

We introduced "Presidifussion," a novel approach to learning and replicating the unique style of calligraphy of President Xu, using a pretrained diffusion model adapted through a two-stage training process. Initially, our model is pretrained on a diverse dataset containing works from various calligraphers. This is followed by fine-tuning on a smaller, specialized dataset of President Xu's calligraphy, comprising just under 200 images. Our method introduces innovative techniques of font image conditioning and stroke information conditioning, enabling the model to capture the intricate structural elements of Chinese characters. The effectiveness of our approach is demonstrated through a comparison with traditional methods like zi2zi and CalliGAN, with our model achieving comparable performance using significantly smaller datasets and reduced computational resources. This work not only presents a breakthrough in the digital preservation of calligraphic art but also sets a new standard for data-efficient generative modeling in the domain of cultural heritage digitization.
Large Language Model Agent as a Mechanical Designer
Authors: Yayati Jadhav, Amir Barati Farimani
Date: 2024-04-26 00:00:00
Tasks: Language Modeling, Language Modelling, Large Language Model, model, Navigate

Conventional mechanical design follows an iterative process in which initial concepts are refined through cycles of expert assessment and resource-intensive Finite Element Method (FEM) analysis to meet performance goals. While machine learning models have been developed to assist in parts of this process, they typically require large datasets, extensive training, and are often tailored to specific tasks, limiting their generalizability. To address these limitations, we propose a framework that leverages a pretrained Large Language Model (LLM) in conjunction with an FEM module to autonomously generate, evaluate, and refine structural designs based on performance specifications and numerical feedback. The LLM operates without domain-specific fine-tuning, using general reasoning to propose design candidates, interpret FEM-derived performance metrics, and apply structurally sound modifications. Using 2D truss structures as a testbed, we show that the LLM can effectively navigate highly discrete and multi-faceted design spaces, balance competing objectives, and identify convergence when further optimization yields diminishing returns. Compared to Non-dominated Sorting Genetic Algorithm II (NSGA-II), our method achieves faster convergence and fewer FEM evaluations. Experiments with varying temperature settings (0.5, 1.0, 1.2) and model sizes (GPT-4.1 and GPT-4.1-mini) indicate that smaller models yield higher constraint satisfaction with fewer steps, while lower temperatures enhance design consistency. These results establish LLMs as a promising new class of reasoning-based, natural language-driven optimizers for autonomous design and iterative structural refinement.