Empathetic BERT 2 BERT Conversational Model: Learning A rabic Language Generation with Little Data (2021)

Abstract: Enabling empathetic behavior in Arabic dialogue agents is an important aspect of building human-like conversational models. While Arabic Natural Language Processing has seen significant advances in Natural Language Understanding (NLU) with language models such as AraBERT, Natural Language Generation (NLG) remains a challenge. The shortcomings of NLG encoder-decoder models are primarily due to the lack of Arabic datasets suitable to train NLG models such as conversational agents. To overcome this issue, we propose a transformer-based encoder-decoder initialized with AraBERT parameters. By initializing the weights of the encoder and decoder with AraBERT pre-trained weights, our model was able to leverage knowledge transfer and boost performance in response generation. To enable empathy in our conversational model, we train it using the ArabicEmpatheticDialogues dataset and achieve high performance in empathetic response generation. Specifically, our model achieved a low perplexity value of 17.0 and an increase in 5 BLEU points compared to the previous state-of-the-art model. Also, our proposed model was rated highly by 85 human evaluators, validating its high capability in exhibiting empathy while generating relevant and fluent responses in open-domain settings.

## introduction conversational models with empathetic responding capabilities are crucial in making human-machine interactions closer to human-human interactions, as they can lead to increased engagement, more trust, and reduced frustration @xcite . these characteristics are highly desirable in open-domain conversational models as they can boost user satisfaction and make chatbots look less boorish. while empathy can be attributed to a range of behaviors, it can be generally described as the innate human capacity of relating to another person's feelings and making sense of their emotional state @xcite ). an important factor towards developing human-like dialogue agents is enabling their empathetic capability @xcite . to this end, there has been a significant interest in developing empathetic conversational models @xcite @xcite . these models infer the emotions of a human user and provide a suitable empathetic response. the desired behavior of an empathetic conversational agent is illustrated in figure 1 , where the empathetic agent recognizes that the user is feeling proud and, thus, generates an empathetic response that congratulates the user with enthusiasm. recent work in open-domain empathetic conversational models have adopted neural-based sequence generation approaches @xcite . these approaches are based on encoderdecoder neural network architectures such as sequence-to-sequence (seq2seq) recurrent neural network models @xcite or transformers @xcite . despite the significant work done in this direction, the focus so far has been mostly on the english language with fewer efforts being directed towards low-resource languages, such as arabic. the first dataset for arabic utterances and empathetic responses was recently introduced by @xcite , where a bidirectional long short-term memory (bi-lstm) seq2seq model was trained on the dataset. however, the model proposed by @xcite delivered suboptimal performance due to the limited size of the dataset. the additional challenges in developing neural-based empathetic conversational models for arabic is the lack of open-domain conversational data that can be used for pre-training @xcite , and thus no availability of pre-trained conversational models that can be used directly for fine-tuning @xcite . to address the challenges of small dataset size and lack of conversational resources, in terms of datasets and pre-trained models, we propose a transformer-based encoder-decoder model initialized with arabert (antoun et al., 2020) pretrained weights. our work extends the english bert2bert architecture @xcite to arabic response generation. we fine-tune our proposed model on the limited-sized dataset of empathetic responses in arabic @xcite . by using the pre-trained weights of the arabert language model to initialize the encoder and decoder, our proposed bert2bert model is expected to leverage knowledge transfer and show enhanced performance in empathetic response generation compared to the baseline bi-lstm model proposed by @xcite . the rest of this paper is organized as follows: section 2 reviews the recent literature on empathetic conversational models in both english and arabic. our proposed bert2bert approach for empathetic response generation is presented in section 3, including the dataset and pre-processing steps. section 4 analyzes the performance of our model and compares its results to several benchmark models. concluding remarks and future directions are presented in section 5. ## experiments & results we evaluate the proposed bert2bert model in comparison to three benchmark models. we conduct numerical as well as human evaluation of the different conversational models. ## conclusion in this paper, we address the limitation in resources for arabic conversational systems, in particular, empathetic conversations. unlike the english language which has seen great advancements in language generation models due to large corpora and million parameter pre-trained models like gpt, arabic is considered a low-resource language with limited availability of conversational datasets and pre-trained models for response generation. we propose an empathetic bert2bert, a transformer-based model, of which the encoder and decoder are warm-started using arabert pre-trained parameters and fine-tuned for arabic empathetic response generation using the arabicem-patheticdialogues dataset. by adopting this transfer learning strategy, the proposed bert2bert model was able to address the challenges of building an open-domain neural-based empathetic conversational model for a low resource language such as arabic. bert2bert achieved significant performance improvements in comparison to three benchmark models, a baseline seq2seq bi-lstm model, a seq2seq bi-lstm model with prepended supervised information about the emotion label during the training process, and a transformer-based encoder-decoder that is not initialized with pretrained weights. the proposed bert2bert model achieved a low ppl value of 17.0, a bleu score of 5.58, and was rated highly by human evaluators with a score of 4.3/5.0, reflecting its ability to generate empathetic, relevant, and fluent responses. hence, our results show the ability to develop high-performing conversational models in low resource settings by adopting the bert2bert strategy. despite its high performance in empathetic response generation, bert2bert showed a limitation in its ability to handle regular chit-chat conversations with neutral emotional states. to this end, future directions include the development of a strategy that improves the model's ability to determine when an empathetic response is suitable and when it is not.