Alternative non- BERT model choices for the textual classification in low-resource languages and environments (2022)

Abstract: Natural Language Processing (NLP) tasks in non-dominant and low-resource languages have not experienced significant progress. Although pre-trained BERT models are available, GPU-dependency, large memory requirement, and data scarcity often limit their applicability. As a solution, this paper proposes a fusion chain architecture comprised of one or more layers of CNN, LSTM, and BiLSTM and identifies precise configuration and chain length. The study shows that a simpler, CPU-trainable non-BERT fusion CNN + BiLSTM + CNN is sufficient to surpass the textual classification performance of the BERT-related models in resource-limited languages and environments. The fusion architecture competitively approaches the state-of-the-art accuracy in several Bengali NLP tasks and a six-class emotion detection task for a newly developed Bengali dataset. Interestingly, the performance of the identified fusion model, for instance, CNN + BiLSTM + CNN, also holds for other lowresource languages and environments. Efficacy study shows that the CNN + BiLSTM + CNN model outperforms BERT implementation for Vietnamese languages and performs almost equally in English NLP tasks experiencing artificial data scarcity. For the GLUE benchmark and other datasets such as Emotion, IMDB, and Intent classification, the CNN + BiLSTM + CNN model often surpasses or competes with BERT-base, TinyBERT, DistilBERT, and mBERT. Besides, a position-sensitive selfattention layer role further improves the fusion models’ performance in the Bengali emotion classification. The models are also compressible to as low as ≈ 5× smaller through pruning and retraining, making them more viable for resource-constrained environments. Together, this study may help NLP practitioners and serve as a blueprint for NLP model choices in textual classification for low-resource languages and environments.

## introduction many developed nations are now considering deep learning approaches for tackling textual toxicity in social media. but countries lacking substantial socio-economic capacity and technological infrastructures are lagging. the current trend of nlp research evolves mainly around a few dominant languages, leaving nlp research for many low-resource languages unattended or less explored @xcite . the nlp tasks in lowresource languages generally suffer from exceptionally scarce resources, ranging from lack of annotated data to insufficient computational facilities. in contrast, most nlp breakthroughs that achieve high accuracy are computationally intensive, making it more challenging for societies suffering from inadequate technological infrastructures. for instance, while the bidirectional transformer bert has about 340 millions parameters @xcite , a more advanced model @xcite , has about 170 billions parameters, requiring extensive gpu/tpu support and memory storage that may be unaffordable for low-resource societies. as a result, low-resource languages and environments are frequently left out with little attention from the nlp community @xcite . further complicating matters, the serverless free deployment of deep learning models, as commonly done using amazon web services (aws) and google cloud platform (gcp), is restrictive for larger model size @xcite . also, latency increases with increasing memory requirement and model size, suggesting memoryintensive device gpu/tpu for faster inference and response. these additional financial costs limit access to bert models for nlp community works in resource-constrained environments @xcite . one intriguing question thus arises: could computationally less-expensive non-bert models reduce gp/tpu dependency and associated fi-nancial cost without affecting the classification accuracy for textual classification in a low-resource context? the multilingual-bert (mbert) @xcite and its reduced versions @xcite , other compressed bert modifications, such as tinybert @xcite , mobilebert @xcite , are a few viable models proposed for many languages and contexts, including the low-resource ones. nevertheless, these models require additional fine-tuning and training for target-specific nlp tasks, requiring gpu/tpu support even in a resource-constrained context. also, size of these models may not be optimal for deployment in low-end devices. so, textual classification in many non-dominant languages remains rudimentary, leaving the communities unequipped against the increasing toxicity and abusive comments on social platforms. besides, many textual classification tasks do not require a rigorous use of linguistic semantics. so, models that are structured well against the semantics, for instance, the bert models, may not always be the most optimal choice in nlp tasks less dependent on language semantics. thus, a viable trade-off between the deployability, scarce resources, and dnn models' accuracy in nlp tasks for low-resource languages and environments needs unraveling. as a solution, this study integrates local and global dependencies in sentences by bringing alternative dnn models into a hybrid model structure, namely the fusion chain models. subsequently, a rigorous architecture search identifies deployable dnn models for low-resource languages, with an improved understanding on a few intriguing questions such as: we identify that classification accuracy is sensitive to fusion chain length, beyond which classification accuracy deteriorates considerably. subsequent exploration of the identified fusion models reveals a position-sensitive performance of the self-attention layer for the newly annotated six-class bengali emotion dataset. ## related works previous works attempted alternative deep learning models in nlp tasks for low-resource languages and environments. for instance, using a teacherstudent framework, the bert distillation with simpler models such as cbow + ffn and bilstm as the student models for the limited availability of labeled data @xcite . while such models are more deployable in low-end devices, the training still relies on a memory-hungry and costly setup requiring gpu/tpu as well as large unlabelled data for student model training. alternative approaches consider freezing the bert-layer outcomes by assessing their roles in the classification process @xcite , requiring gpu/tpu support to train. also, the sequence of frozen layers may vary across alternative datasets, and hence, the accuracy for a particular set of frozen layers becomes context-dependent. instead, we investigate if a simple, cpu-trainable cnn and rnn fusion layer stack can achieve textual classification accuracy in nlp tasks where syntactical knowledge is less influential than the keywords or sentiment-based phrases. to find out such alternative non-bert models, we propose fusionchain architecture comprising one or more cnn and rnn layers and perform a rigorous network architecture search (nas). interestingly, the nas process identifies a few optimal candidate mod-els capable of achieving accuracy comparable to the baseline models, as elaborated further in the subsequent sections. the emergence of more advanced deep neural networks capable of learning the word orders and information dependency in sentences replaces the classical machine learning models @xcite in many nlp tasks. precisely, the neural network models of the form of rnn (lstm, bilstm) or cnn independently, or in combination with a pre-trained word embedding facility such as word2vec @xcite , fasttext @xcite , have become the standard alternatives. for instance, dynamic cnn architecture (dcnn) performs semantic modeling to identify words' short and long-range relations in sentences @xcite . whereas the cnn-based models are good at local and positioninvariant feature extraction, the lstm/bilstm models explicitly treat sentences as a sequence of words and capture sentence-level (for instance, syntactical @xcite ) dependencies. also, a few alternative attempts integrate local and global textual dependencies using cnn and rnn architectures (known as hybrid models) to improve accuracy of textual classification reviewed thoroughly in @xcite . intriguingly, the hybrid models also appear promising for target-specific sequential analysis, as evident from quantifying the function of specific dna sequences @xcite . named entity recognition (ner) tasks also employ a hybrid approach by merging bilstm and cnn models @xcite . one of the initial works leveraging the advantages of both cnn and rnn architectures for textual classification is the convolutional-lstm (c-lstm). precisely, in c-lstm, n-gram features extracted by a cnn layer are fed to the lstm layer for learning the intrasentence sequential dependence of words @xcite . authors in @xcite also tried a hybrid model with lstm outputs fed to a cnn layer in document modeling. alternative models include an attention mechanism with either cnn or rnn architecture to optimize textual classification performance further. for instance, attention-based bidirectional long short-term memory networks (att-blstm) capture the position variant semantic information from the sentences @xcite . another study implements an attention-based convolutional neural net-work (abcnn) to model a pair of sentences @xcite . however, most of the studied hybrid models are single and two-layer models and did not explore the relevance of a larger stacking depth in textual classification tasks. the optimal fusion length and the order of the layers are still debatable and context-dependent. besides, these cpuimplementable models facilitate the exploration and deployment of dnn models in low-resourced environments devoid of adequate advanced computing devices and facilities. ## results and discussion optimal fusion chain length of fusion models: the nas process identifies (see fig. 2a , b , c ) that stacking unlimited dnn layers do not improve performance of the fusion models. instead, the accuracy and ld of the textual classification deteriorate after the chain length attains an optimal value. interestingly, chain-structure of length three or fewer layers yield the optimal performance (shown in fig. 2a , b, c) irrespective of the fact whether fusion models start with any of the cnn, lstm, bilstm layers. the nas considers three fusion chains: a comparison between the competing models for our newly developed corpus of emotion classification reveals that accuracy deteriorates as the chain length goes beyond three. as it appeared, the accuracy gradually reduces to lower values as the length increases beyond three (shown in fig. 2a , b , c ). among the models with a chain length of three or less, a model with a chain length of three is the smallest in ld values among the three allowed chains. a fusion chain that starts with a cnn layer attains the lowest validation loss and is explored further in subsequent analysis by replacing the lstm layer with a bilstm layer. glue benchmark with artificial data scarcity: the glue benchmark datasets have different sentence classification tasks. the performance evaluation of cnn + bilstm + cnn for all the categories has been done by assuming an artificial data scarcity. precisely, the artificial scarcity considers only 250 samples from each class. as reported, the proposed cnn + bilstm + cnn model frequently outperforms baseline @xcite . in another dataset of bengali hate speech detection @xcite , the fusion model with self-attention cnn + attn. + bilstm + cnn outperforms all the previous dnn and ml implementations, as evident from as appeared in position-sensitive self-attention role of fusion models in new bengali corpus: an attention layer may aid in capturing the necessary information for a sequence to sequence model. we also investigated how adding a self-attention layer to the fusion model affects the accuracy of the the newly developed 6-class bengali emotion corpus. however, an immediate question arises-what the optimal position of the attention layer be within a fusion chain. to answer this, we execute four different experiments, utilizing a self-attention layer in four alternative places: between the embedding and the first cnn layer, between the first cnn layer and the first lstm layer, between the first lstm layer and the second cnn layer, and between the second cnn layer and the final output 3a ). besides, the fusion models decrease ld in all the artificially produced scarce cases studied. a close comparison (fig. 3b ) shows that the ld of mbert (blue line) remains way above the lds reported by the fusion models. for the 25% case, the ld value is doubled for mbert, indicating an advantage of fusion models in low-resource contexts. fusion models are computationally less expensive: along with other factors, the computational cost of an nlp model also depends on its size and the flops count. a comparison of these metrics between the baseline models and the fusion models exhibits that fusion models are more advantageous for a small number of annotated samples (shown in ## conclusion generally, the rnn and cnn models are computationally less intensive but compromise accuracy in textual classification. in contrast, bert-variants and other advanced transformer-based implementations demonstrate improved performance but are computationally intensive. this study analyzed a few low-resource textual classification contexts to identify cpu-trainable and comparatively smaller deployable dnn models sufficiently accurate in textual classification tasks. these identified lessintensive dnn fusion models attained accuracy that frequently surpasses bert performance in low-resource contexts. interestingly, the efficacy of cnn + bilstm + cnn remains equally applicable in other alternative languages, tasks. this study also demonstrates that the fusion models are all cpu-trainable, making them easily accessible for communities suffering from an infrastructural deficiency. moreover, low-resource languages always suffer from smaller corpus, infrequent research initiatives, and a lack of intensive computational facilities. these hinder the potential deployment of dnn models to monitor toxic and abusive elements in the ever-increasing social media platforms. because of its relatively small size and acceptable classification accuracy, the fusion models are a suitable alternative to computationally intensive bert variants for deployment in low-end devices. further improvement of the fusion models may consider a multichannel word-embedding technique, equipping the models better for out of vocabulary words now common in the era of social media platforms, pos-tagging to exploit the key phrases of the sentiment better. such extensions, alone or in a cohort, can improve the fusion models to tackle the long-term dependencies analysis by forming phrases from the dependent and related words in longer sentences. overall, this work provides sufficiently accurate, computationally less intensive cpu-trainable dnn models for nlp tasks for low-resource languages and may serve as the blueprint to identify the deployable nlp models for low-resource languages and environments.