pwc-archive/papers-with-abstracts huggingface dataset:
CLIP-Branches: Interactive Fine-Tuning for Text-Image Retrieval
Authors: Christian Lülf, Denis Mayr Lima Martins, Marcos Antonio Vaz Salles, Yongluan Zhou, Fabian Gieseke
Date: 2024-06-19 00:00:00
Tasks: Image Retrieval, Information Retrieval, Retrieval

The advent of text-image models, most notably CLIP, has significantly transformed the landscape of information retrieval. These models enable the fusion of various modalities, such as text and images. One significant outcome of CLIP is its capability to allow users to search for images using text as a query, as well as vice versa. This is achieved via a joint embedding of images and text data that can, for instance, be used to search for similar items. Despite efficient query processing techniques such as approximate nearest neighbor search, the results may lack precision and completeness. We introduce CLIP-Branches, a novel text-image search engine built upon the CLIP architecture. Our approach enhances traditional text-image search engines by incorporating an interactive fine-tuning phase, which allows the user to further concretize the search query by iteratively defining positive and negative examples. Our framework involves training a classification model given the additional user feedback and essentially outputs all positively classified instances of the entire data catalog. By building upon recent techniques, this inference phase, however, is not implemented by scanning the entire data catalog, but by employing efficient index structures pre-built for the data. Our results show that the fine-tuned results can improve the initial search outputs in terms of relevance and accuracy while maintaining swift response times
GLiNER multi-task: Generalist Lightweight Model for Various Information Extraction Tasks
Authors: Ihor Stepanov, Mykhailo Shtopko
Date: 2024-06-14 00:00:00
Tasks: named-entity-recognition, Named Entity Recognition, NER, Question Answering, Relation Extraction, Self-Learning

Information extraction tasks require both accurate, efficient, and generalisable models. Classical supervised deep learning approaches can achieve the required performance, but they need large datasets and are limited in their ability to adapt to different tasks. On the other hand, large language models (LLMs) demonstrate good generalization, meaning that they can adapt to many different tasks based on user requests. However, LLMs are computationally expensive and tend to fail to generate structured outputs. In this article, we will introduce a new kind of GLiNER model that can be used for various information extraction tasks while being a small encoder model. Our model achieved SoTA performance on zero-shot NER benchmarks and leading performance on question-answering, summarization and relation extraction tasks. Additionally, in this article, we will cover experimental results on self-learning approaches for named entity recognition using GLiNER models.
Lip Graph Assisted Audio-Visual Speech Recognition Using Bidirectional Synchronous Fusion
Authors: Hong Liu, Zhan Chen, Bing Yang
Date: 2020-10-25 00:00:00
Proceeding: Interspeech 2020 10
Tasks: Audio-Visual Speech Recognition, Landmark-based Lipreading, speech-recognition, Speech Recognition, Visual Speech Recognition

Current studies have shown that extracting representative visual features and efficiently fusing audio and visual modalities are vital for audio-visual speech recognition (AVSR), but these are still challenging. To this end, we propose a lip graph assisted AVSR method with bidirectional synchronous fusion. First, a hybrid visual stream combines the image branch and graph branch to capture discriminative visual features. Specially, the lip graph exploits the natural and dynamic connections between the lip key points to model the lip shape, and the temporal evolution of the lip graph is captured by the graph convolutional networks followed by bidirectional gated recurrent units. Second, the hybrid visual stream is combined with the audio stream by an attention-based bidirectional synchronous fusion which allows bidirectional information interaction to resolve the asynchrony between the two modalities during fusion. The experimental results on LRW-BBC dataset show that our method outperforms the end-to-end AVSR baseline method in both clean and noisy conditions.
Another Point of View on Visual Speech Recognition
Authors: Baptiste Pouthier, Laurent Pilati, Giacomo Valenti, Charles Bouveyron, Frederic Precioso
Date: 2023-08-20 00:00:00
Proceeding: Interspeech 2023 8
Tasks: Landmark-based Lipreading, speech-recognition, Speech Recognition, Visual Speech Recognition

Standard Visual Speech Recognition (VSR) systems directly process images as input features without any apriori link between raw pixel data and facial traits. Pixel information is smartly sieved when facial landmarks are extracted from pictures and repurposed as graph nodes. Their evolution through time is thus modeled by a Graph Convolutional Network. However, with graph-based VSR being in its infancy, the selection of points and their correlation are still ill-defined and often bound to aprioristic knowledge and handcrafted techniques. In this paper, we investigate the graph approach for VSR and its ability to learn the correlation between points beyond the mouth region. We also study the different contributions that each facial region brings to the system accuracy, proving that more scattered but better connected graphs can be both computationally light and accurate.
Adaptive Semantic-Spatio-Temporal Graph Convolutional Network for Lip Reading
Authors: Changchong Sheng, Xinzhong Zhu, Huiying Xu, Matti Pietikäinen, Li Liu
Date: 2021-08-16 00:00:00
Proceeding: IEEE Transactions on Multimedia 2021 8
Tasks: Landmark-based Lipreading, Lip Reading, Optical Flow Estimation

The goal of this work is to recognize words, phrases, and sentences being spoken by a talking face without given the audio. Current deep learning approaches for lip reading focus on exploring the appearance and optical flow information of videos. However, these methods do not fully exploit the characteristics of lip motion. In addition to appearance and optical flow, the mouth contour deformation usually conveys significant information that is complementary to others. However, the modeling of dynamic mouth contour has received little attention than that of appearance and optical flow. In this work, we propose a novel model of dynamic mouth contours called Adaptive Semantic-Spatio-Temporal Graph Convolution Network (ASST-GCN), to go beyond previous methods by automatically learning both the spatial and temporal information from videos. To combine the complementary information from appearance and mouth contour, a two-stream visual front-end network is proposed. Experimental results demonstrate that the proposed method significantly outperforms the state-of-the-art lip reading methods on several large-scale lip reading benchmarks.
CNO-LSTM: A Chaotic Neural Oscillatory Long Short-Term Memory Model for Text Classification
Authors: Nuobei Shi, Zhuohui Chen, Ling Chen, Raymond S. T. Lee
Date: 2022-12-12 00:00:00
Proceeding: IEEE Access 2022 12
Tasks: Classification, GPU, Language Modeling, Language Modelling, Multi-Label Classification, MUlTI-LABEL-ClASSIFICATION, text-classification, Text Classification

Long Short-Term Memory (LSTM) networks are unique to exercise data in its memory cell with long-term memory as Natural Language Processing (NLP) tasks have inklings of intensive time and computational power due to their complex structures like magnitude language model Transformer required to pre-train and learn billions of data performing different NLP tasks. In this paper, a dynamic chaotic model is proposed for the objective of transforming neurons states in network with neural dynamic characteristics by restructuring LSTM as Chaotic Neural Oscillatory-Long-Short Term Memory (CNO-LSTM), where neurons in LSTM memory cells are weighed in substitutes by oscillatory neurons to speed up computational training of language model and improve text classification accuracy for real-world applications. From the implementation perspective, five popular datasets of general text classification including binary, multi classification and multi-label classification are used to compare with mainstream baseline models on NLP tasks. Results showed that the performance of CNO-LSTM, a simplified model structure and oscillatory neurons state in exercising different types of text classification tasks are above baseline models in terms of evaluation index such as Accuracy, Precision, Recall and F1. The main contributions are time reduction and improved accuracy. It achieved approximately 46.76% of the highest reduction training time and 2.55% accuracy compared with vanilla LSTM model. Further, it achieved approximately 35.86% in time reduction compared with attention model without oscillatory indicating that the model restructure has reduced GPU dependency to improve training accuracy.
ReLU-Oscillator: Chaotic VGG10 Model for Real-Time Neural Style Transfer on Painting Authentication
Authors: Nuobei Shi, Zhuohui Chen, Ling Chen, Raymond S. T. Lee
Date: 2024-06-12 00:00:00
Proceeding: Expert Systems with Applications 2024 6
Tasks: Style Transfer

Neural Style Transfer (NST) has exerted algorithms to generate animation images in computer vision for decades. The Convolution Neural Network (CNN) applied to image content and styles in the NST has improved the extraction of functionalities and the calculation of the convergence speed to recognize and generate high quality structure images, but unpredictable loss elements are inadequate to iterate human learning ability of unique artists’ paintings or styles. This paper offers a chaotic VGG10 NST model based on CNN, ReLU and Lee-Oscillator. The proposed ReLU-Oscillator dynamically relies on activation functions in a chaotic state that dynamically improves high-frequency iterative training of high-quality image and high-speed time optimization. In addition, the best parameters recognizing and determining a personalized painting style from the search for ReLU-Oscillator by corresponding to a set of parameters from proposed Optimal Oscillator Parameter Search Algorithm. Experimental results showed that the stylized image generated by the Chaotic VGG 10 model with high-frequency oscillation succeeded in reducing the training time in magnitude models with the smallest Params and FLOPs in model performance and image quality with the lowest content loss for preserving semantic information and moderate style loss for style similarity balanced with comparison to 8 state-of-the-art models in visual perception evaluation. Chaotic NST has a unique identification for each artist supplemented with a set of oscillator parameters to evaluate the loss performance based on the relative error between the famous painting and its imitation, indicating that the authentication of paintings can be detected through specific ReLU-Oscillator parameters for each stylization, estimated from the loss value performance.
UDBRNet: A novel uncertainty driven boundary refined network for organ at risk segmentation
Authors: Riad Hassan, M. Rubaiyat Hossain Mondal, Sheikh Iqbal Ahamed
Date: 2024-06-17 00:00:00
Proceeding: PLOS ONE 2024 6
Tasks: Organ Segmentation, Segmentation

Organ segmentation has become a preliminary task for computer-aided intervention, diagnosis, radiation therapy, and critical robotic surgery. Automatic organ segmentation from medical images is a challenging task due to the inconsistent shape and size of different organs. Besides this, low contrast at the edges of organs due to similar types of tissue confuses the network’s ability to segment the contour of organs properly. In this paper, we propose a novel convolution neural network based uncertainty-driven boundary-refined segmentation network (UDBRNet) that segments the organs from CT images. The CT images are segmented first and produce multiple segmentation masks from multi-line segmentation decoder. Uncertain regions are identified from multiple masks and the boundaries of the organs are refined based on uncertainty data. Our method achieves remarkable performance, boasting dice accuracies of 0.80, 0.95, 0.92, and 0.94 for Esophagus, Heart, Trachea, and Aorta respectively on the SegThor dataset, and 0.71, 0.89, 0.85, 0.97, and 0.97 for Esophagus, Spinal Cord, Heart, Left-Lung, and Right-Lung respectively on the LCTSC dataset. These results demonstrate the superiority of our uncertainty-driven boundary refinement technique over state-of-the-art segmentation networks such as UNet, Attention UNet, FC-denseNet, BASNet, UNet++, R2UNet, TransUNet, and DS-TransUNet. UDBRNet presents a promising network for more precise organ segmentation, particularly in challenging, uncertain conditions. The source code of our proposed method will be available at https://github.com/riadhassan/UDBRNet
SuperGLEBer: German Language Understanding Evaluation Benchmark
Authors: Jan Pfister, Andreas Hotho
Date: 2024-06-20 00:00:00
Proceeding: NAACL 2024 6
Tasks: Document Classification, Natural Language Understanding, Question Answering, Sentence, Sentence Similarity

We assemble a broad Natural Language Understanding benchmark suite for the German language and consequently evaluate a wide array of existing German-capable models in order to create a better understanding of the current state of German LLMs. Our benchmark consists of 29 different tasks ranging over different types such as document classification, sequence tagging, sentence similarity, and question answering, on which we evaluate 10 different German-pretrained models, thereby charting the landscape of German LLMs. In our comprehensive evaluation we find that encoder models are a good choice for most tasks, but also that the largest encoder model does not necessarily perform best for all tasks. We make our benchmark suite and a leaderboard publically available at https://supergleber.professor-x.de and encourage the community to contribute new tasks and evaluate more models on it (https://github.com/LSX-UniWue/SuperGLEBer).
Automatic segmentation of the femur and tibia bones from X-ray images based on pure dilated residual U-Net
Authors: Weihao Shen, Wenbo Xu, Hongyang Zhang, Zexin Sun, Jianxiong Ma, Xinlong Ma, Shoujun Zhou, Shijie Guo, Yuanquan Wang
Date: 2021-12-01 00:00:00
Proceeding: American Institute of Mathematical Sciences 2021 12
Tasks: Segmentation

X-ray images of the lower limb bone are the most commonly used imaging modality for clinical studies, and segmentation of the femur and tibia in an X-ray image is helpful for many medical studies such as diagnosis, surgery and treatment. In this paper, we propose a new approach based on pure dilated residual U-Net for the segmentation of the femur and tibia bones. The proposed approach employs dilated convolution completely to increase the receptive field, in this way, we can make full use of the advantages of dilated convolution. We conducted experiments and evaluations on datasets provided by Tianjin hospital. Comparison with the classical U-net and FusionNet, our method has fewer parameters, higher accuracy, and converges more rapidly, which means the high performance of the proposed method.
Automated identification of toxigenic cyanobacterial genera for water quality control purposes
Authors: Iman Kianian, MohammadSadeq Mottaqi, Fatemeh Mohammadipanah, Hedieh Sajedi
Date: 2024-06-01 00:00:00
Proceeding: Journal of Environmental Management 2024 6

Cyanobacteria are the dominating microorganisms in aquatic environments, posing significant risks to public health due to toxin production in drinking water reservoirs. Traditional water quality assessments for abundance of the toxigenic genera in water samples are both time-consuming and error-prone, highlighting the urgent need for a fast and accurate automated approach. This study addresses this gap by introducing a novel public dataset, TCB-DS (Toxigenic Cyanobacteria Dataset), comprising 2593 microscopic images of 10 toxigenic cyanobacterial genera and subsequently, an automated system to identify these genera which can be divided into two parts. Initially, a feature extractor Convolutional Neural Network (CNN) model was employed, with MobileNet emerging as the optimal choice after comparing it with various other popular architectures such as MobileNetV2, VGG, etc. Secondly, to perform classification algorithms on the extracted features of the first section, multiple approaches were tested and the experimental results indicate that a Fully Connected Neural Network (FCNN) had the optimal performance with weighted accuracy and f1-score of 94.79% and 94.91%, respectively. The highest macro accuracy and f1-score were 90.17% and 87.64% which were acquired using MobileNetV2 as the feature extractor and FCNN as the classifier. These results demonstrate that the proposed approach can be employed as an automated screening tool for identifying toxigenic Cyanobacteria with practical implications for water quality control replacing the traditional estimation given by the lab operator following microscopic observations. The dataset and code of this paper are publicly available at https://github.com/iman2693/CTCB.
Automatic Transcription of Handwritten Old Occitan Language
Authors: Esteban Garces Arias, Vallari Pai, Matthias Schöffel, Christian Heumann, and Matthias Aßenmacher
Date: 2023-12-06 00:00:00
Proceeding: EMNLP 2023 12
Tasks: Data Augmentation, Decoder, Handwritten Text Recognition, HTR

While existing neural network-based approaches have shown promising results in Handwritten Text Recognition (HTR) for high-resource languages and standardized/machine-written text, their application to low-resource languages often presents challenges, resulting in reduced effectiveness. In this paper, we propose an innovative HTR approach that leverages the Transformer architecture for recognizing handwritten Old Occitan language. Given the limited availability of data, which comprises only word pairs of graphical variants and lemmas, we develop and rely on elaborate data augmentation techniques for both text and image data. Our model combines a custom-trained Swin image encoder with a BERT text decoder, which we pre-train using a large-scale augmented synthetic data set and fine-tune on the small human-labeled data set. Experimental results reveal that our approach surpasses the performance of current state-of-the-art models for Old Occitan HTR, including open-source Transformer-based models such as a fine-tuned TrOCR and commercial applications like Google Cloud Vision. To nurture further research and development, we make our models, data sets, and code publicly available.
Tender: Accelerating Large Language Models via Tensor Decomposition and Runtime Requantization
Authors: Jungi Lee, Wonbeom Lee, Jaewoong Sim
Date: 2024-06-16 00:00:00
Tasks: Quantization, Tensor Decomposition

Large language models (LLMs) demonstrate outstanding performance in various tasks in machine learning and have thus become one of the most important workloads in today's computing landscape. However, deploying LLM inference poses challenges due to the high compute and memory requirements stemming from the enormous model size and the difficulty of running it in the integer pipelines. In this paper, we present Tender, an algorithm-hardware co-design solution that enables efficient deployment of LLM inference at low precision. Based on our analysis of outlier values in LLMs, we propose a decomposed quantization technique in which the scale factors of decomposed matrices are powers of two apart. The proposed scheme allows us to avoid explicit requantization (i.e., dequantization/quantization) when accumulating the partial sums from the decomposed matrices, with a minimal extension to the commodity tensor compute hardware. Our evaluation shows that Tender achieves higher accuracy and inference performance compared to the state-of-the-art methods while also being significantly less intrusive to the existing accelerators.
Adaptive Curves for Optimally Efficient Market Making
Authors: Viraj Nadkarni, Sanjeev Kulkarni, Pramod Viswanath
Date: 2024-06-19 00:00:00

Automated Market Makers (AMMs) are essential in Decentralized Finance (DeFi) as they match liquidity supply with demand. They function through liquidity providers (LPs) who deposit assets into liquidity pools. However, the asset trading prices in these pools often trail behind those in more dynamic, centralized exchanges, leading to potential arbitrage losses for LPs. This issue is tackled by adapting market maker bonding curves to trader behavior, based on the classical market microstructure model of Glosten and Milgrom. Our approach ensures a zero-profit condition for the market maker's prices. We derive the differential equation that an optimal adaptive curve should follow to minimize arbitrage losses while remaining competitive. Solutions to this optimality equation are obtained for standard Gaussian and Lognormal price models using Kalman filtering. A key feature of our method is its ability to estimate the external market price without relying on price or loss oracles. We also provide an equivalent differential equation for the implied dynamics of canonical static bonding curves and establish conditions for their optimality. Our algorithms demonstrate robustness to changing market conditions and adversarial perturbations, and we offer an on-chain implementation using Uniswap v4 alongside off-chain AI co-processors.
New Reservoir Computing Kernel Based on Chaotic Chua Circuit and Investigating Application to Post-Quantum Cryptography
Authors: Matthew John Cossins, Sendy Phang
Date: 2024-06-18 00:00:00

The aim of this project was to develop a new Reservoir Computer implementation, based on a chaotic Chua circuit. In addition to suitable classification and regression benchmarks, the Reservoir Computer was applied to Post-Quantum Cryptography, with its suitability for this application investigated and assessed. The cryptographic algorithm utilised was the Learning with Errors problem, for both encryption and decryption. To achieve this, the Chua circuit was characterised, in simulation, and by physical circuit testing. The Reservoir Computer was designed and implemented using the results of the characterisation. As part of this development, noise was considered and mitigated. The benchmarks demonstrate that the Reservoir Computer can achieve current literature benchmarks with low error. However, the results with Learning with Errors suggest that a Chua-based Reservoir Computer is not sufficiently complex to tackle the high non-linearity in Post-Quantum Cryptography. Future work would involve researching the use of different combinations of multiple Chua Reservoir Computers in larger neural network architectures. Such architectures may produce the required high-dimensional behaviour to achieve the Learning with Errors problem. This project is believed to be only the second instance of a Chua-based Reservoir Computer in academia, and it is the first to be applied to challenging real-world tasks such as Post-Quantum Cryptography. It is also original by its investigation of hitherto unexplored parameters, and their impact on performance. It demonstrates a proof-of-concept for a mass-producible, inexpensive, low-power consumption hardware neural network. It also enables the next stages in research to occur, paving the road for using Chua-based Reservoir Computers across various applications.