pwc-archive/papers-with-abstracts huggingface dataset:
Tuning-less Object Naming with a Foundation Model
Authors: Andrej Lucny, Pavel Petrovic
Date: 2023-11-03 00:00:00
Tasks: model, Object

We implement a real-time object naming system that enables learning a set of named entities never seen. Our approach employs an existing foundation model that we consider ready to see anything before starting. It turns seen images into relatively small feature vectors that we associate with index to a gradually built vocabulary without any training of fine-tuning of the model. Our contribution is using the association mechanism known from transformers as attention. It has features that support generalization from irrelevant information for distinguishing the entities and potentially enable associating with much more than indices to vocabulary. As a result, the system can work in a one-shot manner and correctly name objects named in different contents. We also outline implementation details of the system modules integrated by a blackboard architecture. Finally, we investigate the system's quality, mainly how many objects it can handle in this way.
Investigating Deep-Learning NLP for Automating the Extraction of Oncology Efficacy Endpoints from Scientific Literature
Authors: Aline Gendrin-Brokmann, Eden Harrison, Julianne Noveras, Leonidas Souliotis, Harris Vince, Ines Smit, Francisco Costa, David Milward, Sashka Dimitrievska, Paul Metcalfe, Emilie Louvet
Date: 2023-11-03 00:00:00
Tasks: Benchmarking

Benchmarking drug efficacy is a critical step in clinical trial design and planning. The challenge is that much of the data on efficacy endpoints is stored in scientific papers in free text form, so extraction of such data is currently a largely manual task. Our objective is to automate this task as much as possible. In this study we have developed and optimised a framework to extract efficacy endpoints from text in scientific papers, using a machine learning approach. Our machine learning model predicts 25 classes associated with efficacy endpoints and leads to high F1 scores (harmonic mean of precision and recall) of 96.4% on the test set, and 93.9% and 93.7% on two case studies. These methods were evaluated against - and showed strong agreement with - subject matter experts and show significant promise in the future of automating the extraction of clinical endpoints from free text. Clinical information extraction from text data is currently a laborious manual task which scales poorly and is prone to human error. Demonstrating the ability to extract efficacy endpoints automatically shows great promise for accelerating clinical trial design moving forwards.
More Robots are Coming: Large Multimodal Models (ChatGPT) can Solve Visually Diverse Images of Parsons Problems
Authors: Irene Hou, Owen Man, Sophie Mettille, Sebastian Gutierrez, Kenneth Angelikas, Stephen MacNeil
Date: 2023-11-03 00:00:00
Tasks: Multiple-choice

The advent of large language models is reshaping computing education. Recent research has demonstrated that these models can produce better explanations than students, answer multiple-choice questions at or above the class average, and generate code that can pass automated tests in introductory courses. These capabilities have prompted instructors to rapidly adapt their courses and assessment methods to accommodate changes in learning objectives and the potential for academic integrity violations. While some scholars have advocated for the integration of visual problems as a safeguard against the capabilities of language models, new multimodal language models now have vision and language capabilities that may allow them to analyze and solve visual problems. In this paper, we evaluate the performance of two large multimodal models on visual assignments, with a specific focus on Parsons problems presented across diverse visual representations. Our results show that GPT-4V solved 96.7\% of these visual problems, struggling minimally with a single Parsons problem. Conversely, Bard performed poorly by only solving 69.2\% of problems, struggling with common issues like hallucinations and refusals. These findings suggest that merely transitioning to visual programming problems might not be a panacea to issues of academic integrity in the generative AI era.
Contextualizing the Limits of Model & Evaluation Dataset Curation on Semantic Similarity Classification Tasks
Authors: Daniel Theron
Date: 2023-11-03 00:00:00
Tasks: Classification, Friction, Semantic Similarity, Semantic Textual Similarity

This paper demonstrates how the limitations of pre-trained models and open evaluation datasets factor into assessing the performance of binary semantic similarity classification tasks. As (1) end-user-facing documentation around the curation of these datasets and pre-trained model training regimes is often not easily accessible and (2) given the lower friction and higher demand to quickly deploy such systems in real-world contexts, our study reinforces prior work showing performance disparities across datasets, embedding techniques and distance metrics, while highlighting the importance of understanding how data is collected, curated and analyzed in semantic similarity classification.
Non-Hierarchical Transformers for Pedestrian Segmentation
Authors: Amani Kiruga, Xi Peng
Date: 2023-07-11 00:00:00
Tasks: Instance Segmentation, Segmentation, Semantic Segmentation

We propose a methodology to address the challenge of instance segmentation in autonomous systems, specifically targeting accessibility and inclusivity. Our approach utilizes a non-hierarchical Vision Transformer variant, EVA-02, combined with a Cascade Mask R-CNN mask head. Through fine-tuning on the AVA instance segmentation challenge dataset, we achieved a promising mean Average Precision (mAP) of 52.68\% on the test set. Our results demonstrate the efficacy of ViT-based architectures in enhancing vision capabilities and accommodating the unique needs of individuals with disabilities.
REGIS: Refining Generated Videos via Iterative Stylistic Redesigning
Authors: Jason Abohwo
Date: 2023-11-03 00:00:00
Proceeding: Research Square Preprint 2023 11
Tasks: Text-to-Video Generation, Video Generation

In recent years, generative models have made impressive advancements towards realistic output; in particular, models working in the modalities of text and audio have reached a level of quality at which generated text or audio cannot be easily distinguished from real text or audio Despite these revolutionary advancements, the synthesis of realistic and temporally consistent videos is still in its inception. In this paper, we introduce a novel approach to the creation of realistic videos that focuses on improving the generated video in the latter steps of a video generation process. Specifically, we propose a framework for the iterative refinement of generated videos through repeated passes through a neural network trained to model the spatio-temporal dependencies found in real videos. Through our experiments, we demonstrate that our proposed approach significantly improves upon the generations of text-to-video models and achieves state-of-the-art results of the UCF-101 benchmark; removing the spatio-temporal artifacts and noise that make synthetic videos distinguishable from real videos. In addition, we discuss the ways in which one might augment this framework to achieve better performance.
P-Flow: A Fast and Data-Efficient Zero-Shot TTS through Speech Prompting
Authors: Sungwon Kim ~Sungwon_Kim2, Kevin J. Shih, Rohan Badlani, Joao Felipe Santos, Evelina Bakhturina, Mikyas T. Desta, Rafael Valle, Sungroh Yoon, Bryan Catanzaro
Date: 2023-09-22 00:00:00
Proceeding: NeurIPS 2023 9
Tasks: Decoder, Speech Synthesis

While recent large-scale neural codec language models have shown significant improvement in zero-shot TTS by training on thousands of hours of data, they suffer from drawbacks such as a lack of robustness, slow sampling speed similar to previous autoregressive TTS methods, and reliance on pre-trained neural codec representations. Our work proposes P-Flow, a fast and data-efficient zero-shot TTS model that uses speech prompts for speaker adaptation. P-Flow comprises a speech-prompted text encoder for speaker adaptation and a flow matching generative decoder for high-quality and fast speech synthesis. Our speech-prompted text encoder uses speech prompts and text input to generate speaker-conditional text representation. The flow matching generative decoder uses the speaker-conditional output to synthesize high-quality personalized speech significantly faster than in real-time. Unlike the neural codec language models, we specifically train P-Flow on LibriTTS dataset using a continuous mel-representation. Through our training method using continuous speech prompts, P-Flow matches the speaker similarity performance of the large-scale zero-shot TTS models with two orders of magnitude less training data and has more than 20
 faster sampling speed. Our results show that P-Flow has better pronunciation and is preferred in human likeness and speaker similarity to its recent state-of-the-art counterparts, thus defining P-Flow as an attractive and desirable alternative.
Adaptive Active Contour Model for Brain Tumor Segmentation
Authors: Gunjan Naik, Aditya Abhyankar, Bhushan Garware, Shubhangi kelkar
Date: 2022-07-22 00:00:00
Proceeding: International Journal of Computer Vision and Image Processing (IJCVIP) 12(1) 2022 7
Tasks: Brain Tumor Segmentation, Edge Detection, Image Segmentation, model, Segmentation, Tumor Segmentation

For accurately diagnosing the severity of brain tumors in MRI images, Glioma segmentation is a significant step. The Glioma segmentation is due to noise and weak edges of organs in medical images. The geodesic active contour model (GACM) is a standard method for the segmentation of complex organ structures based on edge maps. The GACM performs poorly due to this noise and weak edges. So, the authors propose a method that uses adaptive kernels instead of a constant kernel for creating strong edge maps for GACM. The kernels used in phase congruency are Log Gabor kernels, which resemble similar anisotropic properties like Gabor kernels. They have replaced these with adaptive kernels. This adaptive kernel-based phase congruency provides a robust edge map to be used in GACM. Experimentation shows that when compared with state-of-the-art edge detection techniques, adaptive kernels enhance the weak as well as strong edges and improve the overall performance. Moreover, the proposed methodology substantially requires fewer parameters compared to existing ACM methods.
Transformer Based Punctuation Restoration for Turkish
Authors: Uygar Kurt, Aykut Çayır
Date: 2023-09-15 00:00:00
Proceeding: 8th International Conference on Computer Science and Engineering (UBMK) 2023 9
Tasks: Automatic Speech Recognition, Automatic Speech Recognition (ASR), Punctuation Restoration, speech-recognition, Speech Recognition

Mobile devices and social media platforms make communication faster than humans have had before, thanks to the technologies such as automatic speech recognition(ASR). However, the speed in text-based communication methods leads to several mistakes that could be solved. The two well-known mistakes are grammatical errors and forgotten punctuation usage. The punctuation restoration task is inherited from the automatic speech recognition domain. Understanding and restoring correct places of punctuation are challenging problems in speech recognition. However, no datasets exist to train a punctuation restoration model for the Turkish language. This paper focuses on restoring punctuations in Turkish texts and introduces a new Turkish dataset for punctuation restoration. Three transformer models: BERT, ELECTRA, and ConvBERT, are fine-tuned and tested on the newly created dataset for three distinct labels: PERIOD, COMMA, and QUESTION MARK. Benchmark results in the paper are reported regarding precision, recall, and F1 score due to imbalanced class distribution. Although each model shows similar performance scores, ELECTRA reaches 83.9% F1 score overall.
A Modern Turkish Poet: Fine-Tuned GPT-2
Authors: Uygar Kurt, Aykut Çayır
Date: 2023-09-15 00:00:00
Proceeding: 8th International Conference on Computer Science and Engineering (UBMK) 2023 9
Tasks: Text Generation

Generative tasks are getting more realistic thanks to the improvements in deep learning. Text generation is getting increasingly important as LLMs (large language models) get more advanced. Even though ChatGPT gave rise to many computer-generated literary works, it has several limitations, such as not being open-sourced and can't be fine-tuned. Because other LLMs are understudied in Turkish, they usually can't be used efficiently to generate literary work. In this paper, we fine-tuned GPT-2 [1] models on sixty Turkish poem categories and trained five models that convey different emotions and write on different topics. The results are gathered in a 70-page Turkish poem book named “Gerçekligin İçinde” that contains 50 poems in total, 10 poems for each chapter. The book got published by Amazon [2], Goodreads [3] and Google Books [4].
Comparing Models of Rapidly Rotating Relativistic Stars Constructed by Two Numerical Methods
Authors: Nikolaos Stergioulas, John L. Friedman
Date: 1994-11-08 00:00:00

We present the first direct comparison of codes based on two different numerical methods for constructing rapidly rotating relativistic stars. A code based on the Komatsu-Eriguchi-Hachisu (KEH) method (Komatsu et al. 1989), written by Stergioulas, is compared to the Butterworth-Ipser code (BI), as modified by Friedman, Ipser and Parker. We compare models obtained by each method and evaluate the accuracy and efficiency of the two codes. The agreement is surprisingly good. A relatively large discrepancy recently reported (Eriguchi et al. 1994) is found to arise from the use of two different versions of the equation of state. We find, for a given equation of state, that equilibrium models with maximum values of mass, baryon mass, and angular momentum are (generically) all distinct and either all unstable to collapse or are all stable. Our implementation of the KEH method will be available as a public domain program for interested users.
Exploring the potential of channel interactions for image restoration
Authors: Yuning Cui, Alois Knoll
Date: 2023-12-20 00:00:00
Proceeding: Knowledge-Based Systems 2023 12
Tasks: Deblurring, Image Deblurring, Image Defocus Deblurring, Image Dehazing, Image Restoration, Rain Removal, Representation Learning, Single Image Deraining, Single Image Desnowing

Image restoration aims to reconstruct a clear image from a degraded observation. Convolutional neural networks have achieved promising performance on this task. The usage of Transformer has recently made significant advancements in state-of-the-art performance by modeling long-range dependencies. However, these deep architectures primarily concentrate on enhancing representation learning for the spatial dimension, neglecting the significance of channel interactions. In this paper, we explore the potential of channel interactions for restoring images through our proposal of a dual-domain channel attention mechanism. To be specific, channel attention in the spatial domain allows each channel to amass valuable signals from adjacent channels under the guidance of learned dynamic weights. In order to effectively exploit the significant difference in infrequency between degraded and clean image pairs, we develop the implicit frequency domain channel attention to facilitate the integration of information from different frequencies. Extensive experiments demonstrate that the proposed network, dubbed ChaIR, achieves state-of-the-art performance on 13 benchmark datasets for five image restoration tasks, including image dehazing, image motion/defocus deblurring, image desnowing, and image deraining.
Layer-wise Auto-Weighting for Non-Stationary Test-Time Adaptation
Authors: Junyoung Park, Jin Kim, Hyeongjun Kwon, Ilhoon Yoon, Kwanghoon Sohn
Date: 2023-11-10 00:00:00
Tasks: Test-time Adaptation

Given the inevitability of domain shifts during inference in real-world applications, test-time adaptation (TTA) is essential for model adaptation after deployment. However, the real-world scenario of continuously changing target distributions presents challenges including catastrophic forgetting and error accumulation. Existing TTA methods for non-stationary domain shifts, while effective, incur excessive computational load, making them impractical for on-device settings. In this paper, we introduce a layer-wise auto-weighting algorithm for continual and gradual TTA that autonomously identifies layers for preservation or concentrated adaptation. By leveraging the Fisher Information Matrix (FIM), we first design the learning weight to selectively focus on layers associated with log-likelihood changes while preserving unrelated ones. Then, we further propose an exponential min-max scaler to make certain layers nearly frozen while mitigating outliers. This minimizes forgetting and error accumulation, leading to efficient adaptation to non-stationary target distribution. Experiments on CIFAR-10C, CIFAR-100C, and ImageNet-C show our method outperforms conventional continual and gradual TTA approaches while significantly reducing computational load, highlighting the importance of FIM-based learning weight in adapting to continuously or gradually shifting target domains.
Tamil-Llama: A New Tamil Language Model Based on Llama 2
Authors: Abhinand Balachandran
Date: 2023-11-10 00:00:00
Tasks: Language Modeling, Language Modelling, Text Generation

Language modeling has witnessed remarkable advancements in recent years, with Large Language Models (LLMs) like ChatGPT setting unparalleled benchmarks in human-like text generation. However, a prevailing limitation is the underrepresentation of languages like Tamil in these cutting-edge models, leading to suboptimal performance in diverse linguistic contexts. This paper addresses this lacuna, enhancing the open-source LLaMA model with an addition of 16,000 Tamil tokens, aiming to achieve superior text generation and comprehension in the Tamil language. We strategically employ the LoRA methodology for efficient model training on a comprehensive Tamil corpus, ensuring computational feasibility and model robustness. Moreover, we introduce a Tamil-translated version of the Alpaca dataset and a subset of the OpenOrca dataset tailored for instruction fine-tuning. Our results showcase significant performance improvements in Tamil text generation, with potential implications for the broader landscape of LLMs in Indian languages. We further underscore our commitment to open research by making our models, datasets, and code publicly accessible, fostering further innovations in language modeling.
Watermarking Vision-Language Pre-trained Models for Multi-modal Embedding as a Service
Authors: Yuanmin Tang, Jing Yu, Keke Gai, Xiangyan Qu, Yue Hu, Gang Xiong, Qi Wu
Date: 2023-11-10 00:00:00
Tasks: Model extraction

Recent advances in vision-language pre-trained models (VLPs) have significantly increased visual understanding and cross-modal analysis capabilities. Companies have emerged to provide multi-modal Embedding as a Service (EaaS) based on VLPs (e.g., CLIP-based VLPs), which cost a large amount of training data and resources for high-performance service. However, existing studies indicate that EaaS is vulnerable to model extraction attacks that induce great loss for the owners of VLPs. Protecting the intellectual property and commercial ownership of VLPs is increasingly crucial yet challenging. A major solution of watermarking model for EaaS implants a backdoor in the model by inserting verifiable trigger embeddings into texts, but it is only applicable for large language models and is unrealistic due to data and model privacy. In this paper, we propose a safe and robust backdoor-based embedding watermarking method for VLPs called VLPMarker. VLPMarker utilizes embedding orthogonal transformation to effectively inject triggers into the VLPs without interfering with the model parameters, which achieves high-quality copyright verification and minimal impact on model performance. To enhance the watermark robustness, we further propose a collaborative copyright verification strategy based on both backdoor trigger and embedding distribution, enhancing resilience against various attacks. We increase the watermark practicality via an out-of-distribution trigger selection approach, removing access to the model training data and thus making it possible for many real-world scenarios. Our extensive experiments on various datasets indicate that the proposed watermarking approach is effective and safe for verifying the copyright of VLPs for multi-modal EaaS and robust against model extraction attacks. Our code is available at https://github.com/Pter61/vlpmarker.
Parameter-Efficient Orthogonal Finetuning via Butterfly Factorization
Authors: Weiyang Liu, Zeju Qiu, Yao Feng, Yuliang Xiu, Yuxuan Xue, Longhui Yu, Haiwen Feng, Zhen Liu, Juyeon Heo, Songyou Peng, Yandong Wen, Michael J. Black, Adrian Weller, Bernhard Schölkopf
Date: 2023-11-10 00:00:00

Large foundation models are becoming ubiquitous, but training them from scratch is prohibitively expensive. Thus, efficiently adapting these powerful models to downstream tasks is increasingly important. In this paper, we study a principled finetuning paradigm -- Orthogonal Finetuning (OFT) -- for downstream task adaptation. Despite demonstrating good generalizability, OFT still uses a fairly large number of trainable parameters due to the high dimensionality of orthogonal matrices. To address this, we start by examining OFT from an information transmission perspective, and then identify a few key desiderata that enable better parameter-efficiency. Inspired by how the Cooley-Tukey fast Fourier transform algorithm enables efficient information transmission, we propose an efficient orthogonal parameterization using butterfly structures. We apply this parameterization to OFT, creating a novel parameter-efficient finetuning method, called Orthogonal Butterfly (BOFT). By subsuming OFT as a special case, BOFT introduces a generalized orthogonal finetuning framework. Finally, we conduct an extensive empirical study of adapting large vision transformers, large language models, and text-to-image diffusion models to various downstream tasks in vision and language.
Data Contamination Quiz: A Tool to Detect and Estimate Contamination in Large Language Models
Authors: Shahriar Golchin, Mihai Surdeanu
Date: 2023-11-10 00:00:00
Tasks: GSM8K, Memorization, Multiple-choice, Sentence

We propose the Data Contamination Quiz (DCQ), a simple and effective approach to detect data contamination in large language models (LLMs) and estimate the amount of it. Specifically, we frame data contamination detection as a series of multiple-choice questions and devise a quiz format wherein three perturbed versions of each subsampled instance from a specific dataset partition (e.g., GSM8k test set) are created. These changes only include word-level perturbations. The generated perturbations, along with the original dataset instance, form the options in the DCQ, with an extra option accommodating the possibility of selecting none of the provided options. Given that the only distinguishing signal among the options is the exact wording with respect to the original dataset instance, an LLM, when tasked with identifying the original dataset instance, gravitates towards selecting the original one if it has been exposed to it in its pre-training phase -- a trait intrinsic to LLMs. While accounting for positional biases in LLMs, the quiz performance reveals the contamination level for the model being examined with the dataset partition to which the quiz pertains. Applied to various datasets with GPT-4 and GPT-3.5, our findings -- while fully lacking access to pre-training data and model parameters -- suggest that DCQ achieves state-of-the-art results and uncovers greater contamination/memorization levels compared to existing methods and proficiently bypasses more safety filters, especially those set to avoid generating copyrighted contents.