pwc-archive/papers-with-abstracts huggingface dataset:
FedFA: Federated Learning with Feature Anchors to Align Features and Classifiers for Heterogeneous Data
Authors: Tailin Zhou, Jun Zhang, Danny H. K. Tsang
Date: 2022-11-17 00:00:00
Tasks: Classifier calibration, Federated Learning, Image Classification

Federated learning allows multiple clients to collaboratively train a model without exchanging their data, thus preserving data privacy. Unfortunately, it suffers significant performance degradation due to heterogeneous data at clients. Common solutions involve designing an auxiliary loss to regularize weight divergence or feature inconsistency during local training. However, we discover that these approaches fall short of the expected performance because they ignore the existence of a vicious cycle between feature inconsistency and classifier divergence across clients. This vicious cycle causes client models to be updated in inconsistent feature spaces with more diverged classifiers. To break the vicious cycle, we propose a novel framework named Federated learning with Feature Anchors (FedFA). FedFA utilizes feature anchors to align features and calibrate classifiers across clients simultaneously. This enables client models to be updated in a shared feature space with consistent classifiers during local training. Theoretically, we analyze the non-convex convergence rate of FedFA. We also demonstrate that the integration of feature alignment and classifier calibration in FedFA brings a virtuous cycle between feature and classifier updates, which breaks the vicious cycle existing in current approaches. Extensive experiments show that FedFA significantly outperforms existing approaches on various classification datasets under label distribution skew and feature distribution skew.
The Effectiveness of Bidirectional Generative Patent Language Models
Authors: Jieh-Sheng Lee
Date: 2022-09-04 00:00:00
Tasks: Text Generation

Generative patent language models can assist humans to write patent text more effectively. The question is how to measure effectiveness from a human-centric perspective and how to improve effectiveness. In this manuscript, a simplified design of the autocomplete function is proposed to increase effectiveness by more than 10%. With the new design, the effectiveness of autocomplete can reach more than 60%, which means that more than 60% of keystrokes can be saved by autocomplete. Since writing patent text does not necessarily start from the beginning to the end, a question is whether the generative model can assist a user no matter where to start writing. To answer the question, the generative models in this manuscript are pre-trained with training data in both directions. The generative models become bidirectional. Since text generation is bidirectional, the calculation of autocomplete effectiveness can be bidirectional and starts from anywhere in the text. After thorough experiments, a key finding is that the autocomplete effectiveness of a model for the same text remains similar no matter where the calculation starts. The finding indicates that such bidirectional models can assist a user at a similar level, no matter where the user starts to write.
Design Considerations For Hypothesis Rejection Modules In Spoken Language Understanding Systems
Authors: Aman Alok, Rahul Gupta, Shankar Ananthakrishnan
Date: 2022-10-31 00:00:00
Tasks: Spoken Language Understanding

Spoken Language Understanding (SLU) systems typically consist of a set of machine learning models that operate in conjunction to produce an SLU hypothesis. The generated hypothesis is then sent to downstream components for further action. However, it is desirable to discard an incorrect hypothesis before sending it downstream. In this work, we present two designs for SLU hypothesis rejection modules: (i) scheme R1 that performs rejection on domain specific SLU hypothesis and, (ii) scheme R2 that performs rejection on hypothesis generated from the overall SLU system. Hypothesis rejection modules in both schemes reject/accept a hypothesis based on features drawn from the utterance directed to the SLU system, the associated SLU hypothesis and SLU confidence score. Our experiments suggest that both the schemes yield similar results (scheme R1: 2.5% FRR @ 4.5% FAR, scheme R2: 2.5% FRR @ 4.6% FAR), with the best performing systems using all the available features. We argue that while either of the rejection schemes can be chosen over the other, they carry some inherent differences which need to be considered while making this choice. Additionally, we incorporate ASR features in the rejection module (obtaining an 1.9% FRR @ 3.8% FAR) and analyze the improvements.
Numerical Optimizations for Weighted Low-rank Estimation on Language Model
Authors: Ting Hua, Yen-Chang Hsu, Felicity Wang, Qian Lou, Yilin Shen, Hongxia Jin
Date: 2022-11-02 00:00:00
Tasks: Language Modeling, Language Modelling

Singular value decomposition (SVD) is one of the most popular compression methods that approximate a target matrix with smaller matrices. However, standard SVD treats the parameters within the matrix with equal importance, which is a simple but unrealistic assumption. The parameters of a trained neural network model may affect task performance unevenly, which suggests non-equal importance among the parameters. Compared to SVD, the decomposition method aware of parameter importance is the more practical choice in real cases. Unlike standard SVD, weighted value decomposition is a non-convex optimization problem that lacks a closed-form solution. We systematically investigated multiple optimization strategies to tackle the problem and examined our method by compressing Transformer-based language models. Further, we designed a metric to predict when the SVD may introduce a significant performance drop, for which our method can be a rescue strategy. The extensive evaluations demonstrate that our method can perform better than current SOTA methods in compressing Transformer-based language models.
Federated Multilingual Models for Medical Transcript Analysis
Authors: Andre Manoel, Mirian Hipolito Garcia, Tal Baumel, Shize Su, Jialei Chen, Dan Miller, Danny Karmon, Robert Sim, Dimitrios Dimitriadis
Date: 2022-11-04 00:00:00
Tasks: Federated Learning

Federated Learning (FL) is a novel machine learning approach that allows the model trainer to access more data samples, by training the model across multiple decentralized data sources, while data access constraints are in place. Such trained models can achieve significantly higher performance beyond what can be done when trained on a single data source. As part of FL's promises, none of the training data is ever transmitted to any central location, ensuring that sensitive data remains local and private. These characteristics make FL perfectly suited for large-scale applications in healthcare, where a variety of compliance constraints restrict how data may be handled, processed, and stored. Despite the apparent benefits of federated learning, the heterogeneity in the local data distributions pose significant challenges, and such challenges are even more pronounced in the case of multilingual data providers. In this paper we present a federated learning system for training a large-scale multi-lingual model suitable for fine-tuning on downstream tasks such as medical entity tagging. Our work represents one of the first such production-scale systems, capable of training across multiple highly heterogeneous data providers, and achieving levels of accuracy that could not be otherwise achieved by using central training with public data. Finally, we show that the global model performance can be further improved by a training step performed locally.
Generative Adversarial Training Can Improve Neural Language Models
Authors: Sajad Movahedi, Azadeh Shakery
Date: 2022-11-02 00:00:00
Tasks: Language Modeling, Language Modelling

While deep learning in the form of recurrent neural networks (RNNs) has caused a significant improvement in neural language modeling, the fact that they are extremely prone to overfitting is still a mainly unresolved issue. In this paper we propose a regularization method based on generative adversarial networks (GANs) and adversarial training (AT), that can prevent overfitting in neural language models. Unlike common adversarial training methods such as the fast gradient sign method (FGSM) that require a second back-propagation through time, and therefore effectively require at least twice the amount of time for regular training, the overhead of our method does not exceed more than 20% of the training of the baselines.
Behavior Score-Embedded Brain Encoder Network for Improved Classification of Alzheimer Disease Using Resting State fMRI
Authors: Wan-Ting Hsieh, Jeremy Lefort-Besnard, Hao-Chun Yang, Li-Wei Kuo, Chi-Chun Lee
Date: 2022-11-04 00:00:00
Tasks: Classification

The ability to accurately detect onset of dementia is important in the treatment of the disease. Clinically, the diagnosis of Alzheimer Disease (AD) and Mild Cognitive Impairment (MCI) patients are based on an integrated assessment of psychological tests and brain imaging such as positron emission tomography (PET) and anatomical magnetic resonance imaging (MRI). In this work using two different datasets, we propose a behavior score-embedded encoder network (BSEN) that integrates regularly adminstrated psychological tests information into the encoding procedure of representing subject's restingstate fMRI data for automatic classification tasks. BSEN is based on a 3D convolutional autoencoder structure with contrastive loss jointly optimized using behavior scores from MiniMental State Examination (MMSE) and Clinical Dementia Rating (CDR). Our proposed classification framework of using BSEN achieved an overall recognition accuracy of 59.44% (3-class classification: AD, MCI and Healthy Control), and we further extracted the most discriminative regions between healthy control (HC) and AD patients.
Dimensionality Reduction and Classification feature using Mutual Information applied to Hyperspectral Images : A Filter strategy based algorithm
Authors: Elkebir Sarhrouni, Ahmed Hammouch, Driss Aboutajdine
Date: 2012-09-28 00:00:00
Tasks: Dimensionality Reduction

Hyperspectral images (HIS) classification is a high technical remote sensing tool. The goal is to reproduce a thematic map that will be compared with a reference ground truth map (GT), constructed by expecting the region. The HIS contains more than a hundred bidirectional measures, called bands (or simply images), of the same region. They are taken at juxtaposed frequencies. Unfortunately, some bands contain redundant information, others are affected by the noise, and the high dimensionality of features made the accuracy of classification lower. The problematic is how to find the good bands to classify the pixels of regions. Some methods use Mutual Information (MI) and threshold, to select relevant bands, without treatment of redundancy. Others control and eliminate redundancy by selecting the band top ranking the MI, and if its neighbors have sensibly the same MI with the GT, they will be considered redundant and so discarded. This is the most inconvenient of this method, because this avoids the advantage of hyperspectral images: some precious information can be discarded. In this paper we'll accept the useful redundancy. A band contains useful redundancy if it contributes to produce an estimated reference map that has higher MI with the GT.nTo control redundancy, we introduce a complementary threshold added to last value of MI. This process is a Filter strategy; it gets a better performance of classification accuracy and not expensive, but less preferment than Wrapper strategy.
Extending Logic Explained Networks to Text Classification
Authors: Rishabh Jain, Gabriele Ciravegna, Pietro Barbiero, Francesco Giannini, Davide Buffelli, Pietro Lio
Date: 2022-11-04 00:00:00
Tasks: Classification, Sensitivity, text-classification, Text Classification

Recently, Logic Explained Networks (LENs) have been proposed as explainable-by-design neural models providing logic explanations for their predictions. However, these models have only been applied to vision and tabular data, and they mostly favour the generation of global explanations, while local ones tend to be noisy and verbose. For these reasons, we propose LENp, improving local explanations by perturbing input words, and we test it on text classification. Our results show that (i) LENp provides better local explanations than LIME in terms of sensitivity and faithfulness, and (ii) logic explanations are more useful and user-friendly than feature scoring provided by LIME as attested by a human survey.
The Exponentiated Gumbel Type-2 Distribution: Properties and Application
Authors: Okorie, I. E. Akpanta, A. C. Ohakwe, J.
Date: 2016-07-10 00:00:00
Proceeding: International Journal of Mathematics and Mathematical Sciences 2016 7
Tasks: Vocal Bursts Type Prediction

We introduce a generalized version of the standard Gumble type-2 distribution. The new lifetime distribution is called the ExponentiatedGumbel (EG) type-2 distribution. The EG type-2 distribution has three nested submodels, namely, theGumbel type2 distribution, the Exponentiated Fr´echet (EF) distribution, and the Fr´echet distribution. Some statistical and reliability properties of the new distribution were given and the method of maximum likelihood estimates was proposed for estimating the model parameters. The usefulness and flexibility ofthe ExponentiatedGumbel (EG) type-2 distributionwere illustrated with a real lifetime data set. Results based on the log-likelihood and information statistics values showed that the EG type-2 distribution provides a better fit to the data than the other competing distributions. Also, the consistency of the parameters of the new distribution was demonstrated through a simulation study. The EG type-2 distribution is therefore recommended for effective modelling of lifetime data.
Exponentiated transformation of Gumbel Type-II distribution for modeling COVID-19 data
Authors: Sindhu, Tabassum Naz Shafiq, Anum Al-Mdallal, Qasem M.
Date: 2020-10-21 00:00:00
Proceeding: Alexandria Engineering Journal 2020 10

Abstract The aim of this study is to analyze the number of deaths due to COVID-19 for Europe and China. For this purpose, we proposed a novel three parametric model named as Exponentiated transformation of Gumbel Type-II (ETGT-II) for modeling the two data sets of death cases due to COVID-19. Specific statistical attributes are derived and analyzed along with moments and associated measures, moments generating functions, uncertainty measures, complete/incomplete moments, survival function, quantile function and hazard function, etc. Additionally, model parameters are estimated by utilizing maximum likelihood method and Bayesian paradigm. To examine efficiency of the ETGT-II model a simulation analysis is performed. Finally, using the data sets of death cases of COVID-19 of Europe and China to show adaptability of suggested model. The results reveal that it may fit better than other well-known models.
Spatial variability in the community structure of microphytoplankton along the southeastern Arabian sea during an unusually warm winter monsoon season
Authors: Twinkle Sathish
Date: 2022-01-15 00:00:00
Proceeding: Regional Studies in Marine Science 2022 1

The microphytoplankton community structure and composition were investigated along the South Eastern Arabian Sea (SEAS) during the late winter monsoon season (January 2020). A total of 186 species of phytoplankton were identified, with cyanobacteria Trichodesmium erythraeum (72%) being the most dominant, followed by diatoms (11%) and dinoflagellates (2.7%). Major blooms of T. erythraeum were observed towards the southern region of SEAS characterised by high SST (28.75 ± 0.7 °C), high surface salinity (34.6 ± 1.2 psu) and low wind speed (3.6 m s−1). The T. erythraeum bloom areas generally observed high nitrate and low silicate and phosphate concentrations with high chlorophyll a concentration (8.96 ± 2.92 mg m−3). In non-bloom regions, centric diatoms like Chaetoceros spp., Bacteriastrum spp. and Leptocylindrus sp. dominated. The abundance of diatoms observed a decreasing trend towards the southern extend of SEAS. The remote sensing data derived for the area observed sea surface temperatures are about 1.0 °C higher than the climatological mean of January. The higher temperatures (29 °C–32 °C) are clearly visible in the Indian coastal region from Kanyakumari to the Gujarat coast, suggesting unusual warming in the eastern coast of the Arabian Sea in January 2020. This warming which was more prominent towards the southern extend of SEAS, might have augmented the proliferation of T. erythraeum along the surface waters of SEAS.
An Efficient Short-Time Discrete Cosine Transform and Attentive MultiResUNet Framework for Music Source Separation
Authors: T. Sgouros, A. Bousis, N. Mitianoudis
Date: 2022-11-14 00:00:00
Proceeding: IEEE Access 2022 11
Tasks: Music Source Separation

The music source separation problem, where the task at hand is to estimate the audio components that are present in a mixture, has been at the centre of research activity for a long time. In more recent frameworks, the problem is tackled by creating deep learning models, which attempt to extract information from each component by using Short-Time Fourier Transform (STFT) spectrograms as input. Most approaches assume that one source is present at each time-frequency point, which allows to allocate this point from the mixture to the desired source. Since this assumption is strong and is reported not to hold in practice, there is a problem that arises from the use of the magnitude of the STFT as input to these networks, which is the absence of the Fourier phase information during the separated source reconstruction. The recovery of the Fourier phase information is neither easily tractable, nor computationally efficient to estimate. In this paper, we propose a novel Attentive MultiResUNet architecture, that uses real-valued Short-Time Discrete Cosine Transform data as inputs. This step avoids the phase recovery problem, by estimating the appropriate values within the network itself, rather than employing complex estimation or post-processing algorithms. The proposed novel network features a U-Net type structure with residual skip connections and an attention mechanism that correlates the skip connection and the decoder output at the previous level. The proposed network is used for the first time in source separation and is more computationally efficient than state-of-the-art separation networks and features favourable performance compared to the state-of-the-art with a fraction of the computational cost.
Language Model Based Chinese Handwriting Address Recognition
Authors: Chieh-Jen Wang, Yung-Ping Tien, Yun-Wei Hung
Date: 2022-11-01 00:00:00
Proceeding: ROCLING 2022 11
Tasks: Handwriting Recognition, Language Modeling, Language Modelling, model

Chinese handwritten address recognition of consignment note is an important challenge of smart logistics automation. Chinese handwritten characters detection and recognition is the key technology for this application. Since the writing mode of handwritten characters is more complex and diverse than printed characters, it is easy misjudgment for recognition. Moreover, the address text occupies a small proportion in the image of the consignment note and arranged closely, which is easy to cause difficulties in detection. Therefore, how to detect the address text on the consignment note accurately is a focus of this paper. The consignment note address automatic detection and recognition system proposed in this paper detects and recognizes address characters, reduces the probability of misjudgment of Chinese handwriting recognition through language model, and improves the accuracy.
Chinese Movie Dialogue Question Answering Dataset
Authors: Shang-Bao Luo, Cheng-Chung Fan, Kuan-Yu Chen, Yu Tsao, Hsin-Min Wang, Keh-Yih Su
Date: 2022-11-01 00:00:00
Proceeding: ROCLING 2022 11
Tasks: Information Retrieval, Question Answering, Retrieval

This paper constructs a Chinese dialogue-based information-seeking question answering dataset CMDQA, which is mainly applied to the scenario of getting Chinese movie related information. It contains 10K QA dialogs (40K turns in total). All questions and background documents are compiled from the Wikipedia via an Internet crawler. The answers to the questions are obtained via extracting the corresponding answer spans within the related text passage. In CMDQA, in addition to searching related documents, pronouns are also added to the question to better mimic the real dialog scenario. This dataset can test the individual performance of the information retrieval, the question answering and the question re-writing modules. This paper also provides a baseline system and shows its performance on this dataset. The experiments elucidate that it still has a big gap to catch the human performance. This dataset thus provides enough challenge for the researcher to conduct related research.
Unsupervised Text Summarization of Long Documents using Dependency-based Noun Phrases and Contextual Order Arrangement
Authors: Yen-Hao Huang, Hsiao-Yen Lan, Yi-Shin Chen
Date: 2022-11-01 00:00:00
Proceeding: ROCLING 2022 11
Tasks: Extractive Summarization, Semantic Similarity, Semantic Textual Similarity, Sentence, Text Summarization, Unsupervised Extractive Summarization, Unsupervised Text Summarization

Unsupervised extractive summarization has recently gained importance since it does not require labeled data. Among unsupervised methods, graph-based approaches have achieved outstanding results. These methods represent each document by a graph, with sentences as nodes and word-level similarity among sentences as edges. Common words can easily lead to a strong connection between sentence nodes. Thus, sentences with many common words can be misinterpreted as salient sentences for a summary. This work addresses the common word issue with a phrase-level graph that (1) focuses on the noun phrases of a document based on grammar dependencies and (2) initializes edge weights by term-frequency within the target document and inverse document frequency over the entire corpus. The importance scores of noun phrases extracted from the graph are then used to select the most salient sentences. To preserve summary coherence, the order of the selected sentences is re-arranged by a flow-aware orderBERT. The results reveal that our unsupervised framework outperformed other extractive methods on ROUGE as well as two human evaluations for semantic similarity and summary coherence.
Enhancing Chinese Multi-Label Text Classification Performance with Response-based Knowledge Distillation
Authors: Szu-Chi Huang, Cheng-Fu Cao, Po-Hsun Liao, Lung-Hao Lee, Po-Lei Lee, Kuo-Kai Shyu
Date: 2022-11-01 00:00:00
Proceeding: ROCLING 2022 11
Tasks: Knowledge Distillation, Multi Label Text Classification, Multi-Label Text Classification, text-classification, Text Classification

It’s difficult to optimize individual label performance of multi-label text classification, especially in those imbalanced data containing long-tailed labels. Therefore, this study proposes a response-based knowledge distillation mechanism comprising a teacher model that optimizes binary classifiers of the corresponding labels and a student model that is a standalone multi-label classifier learning from distilled knowledge passed by the teacher model. A total of 2,724 Chinese healthcare texts were collected and manually annotated across nine defined labels, resulting in 8731 labels, each containing an average of 3.2 labels. We used 5-fold cross-validation to compare the performance of several multi-label models, including TextRNN, TextCNN, HAN, and GRU-att. Experimental results indicate that using the proposed knowledge distillation mechanism effectively improved the performance no matter which model was used, about 2-3% of micro-F1, 4-6% of macro-F1, 3-4% of weighted-F1 and 1-2% of subset accuracy for performance enhancement.