Neural Networks approaches focused on F rench Spoken Language Understanding: application to the MEDIA Evaluation Task (2020)

Abstract: In this paper, we present a study on a French Spoken Language Understanding (SLU) task: the MEDIA task. Many works and studies have been proposed for many tasks, but most of them are focused on English language and tasks. The exploration of a richer language like French within the framework of a SLU task implies to recent approaches to handle this difficulty. Since the MEDIA task seems to be one of the most difficult, according several previous studies, we propose to explore Neural Networks approaches focusing of three aspects: firstly, the Neural Network inputs and more specifically the word embeddings; secondly, we compared French version of BERT against the best setup through different ways; Finally, the comparison against State-of-the-Art approaches. Results show that the word embeddings trained on a small corpus need to be updated during SLU model training. Furthermore, the French BERT fine-tuned approaches outperform the classical Neural Network Architectures and achieves state of the art results. However, the contextual embeddings extracted from one of the French BERT approaches achieve comparable results in comparison to word embedding, when integrated into the proposed neural architecture.

## introduction spoken language understanding (slu) module is a key component for a spoken language dialogue system. it consists on semantically analyse user queries and identifies text spans that mention semantic concepts. slu task can fall into three sub-tasks: domain classification, intent classification, and slotfilling @xcite . the latter is the task that interests us in this study. over the past five years, the studies developed for slu task are based on neural network architectures @xcite @xcite @xcite @xcite . recent approaches take benefit from contextual or language model embeddings such as bert @xcite . @xcite investigated the transfer ability of a pre-trained bert representation for english slu tasks. but, as far as we know, there are no such studies on a french slu task. following @xcite 's study, many avenues can be explored. in their study, the word embeddings have been frozen during training (are not updated), since @xcite show that finetuned word embeddings show very similar performance and provide comparable results. however, the evaluation of whether updating the embeddings during slu model training improves or not the results, for slu task, is less studied. in addition, their slu model is fed only with word embeddings, and they did not use any additional features, thus there are some rows for improvements. finally, @xcite benchmarked several slu tasks and proposed a difficulty hierarchy in which the media evaluation @xcite seems to be the most difficult slu task. contributions: this study focuses on a french slu task: the media evaluation, in which we firstly propose the evaluation of whether updating the word embeddings during training can improve the results, according to several scenarios. secondly, we propose to use a bilstm-cnn architecture @xcite that integrates character embeddings as additional features, using a convolution layer. finally, we propose to evaluate the performance of bert approaches against the bilstm-cnn architecture and state-of-the-art on the media task @xcite through different ways: i) we propose to fine-tune bert on slu task using two french models: camembert @xcite and flaubert @xcite . ii) based on the results of i) we propose to integrate the extracted bert contextual embeddings to the bilstm-cnn architecture and compare it to word embeddings. ## slu model descriptions this section describe the slu models used in this study. the first two models are based on bilstm and its update: the bilstm-cnn. the neuronlp2 implementation foot_0 was used for both bilstm implementations. the third model is based on the bert models. bilstm (bidirectional long short-term memory) architecture has been proven to be relevant to model output dependencies on slu tasks @xcite . to further improve the performance of our slu model, we propose to use a bilstm-cnn (convolutional neural network) architecture @xcite that integrates character embeddings using a convolution layer, in addition to the word embeddings. finally, we propose to fine-tune bert @xcite on slu task using two french models: camembert @xcite and flaubert @xcite . the camembert model is trained on the french part of the oscar corpus @xcite composed of 138gb of raw text, and flaubert @xcite on 71gb of heterogeneous french corpora. ## experiments in this section, we present the experiments we performed using our approaches and their setup foot_1 . for both bilstm and bilstm-cnn, we made some hyper-parameters tuning by varying the number of layers @xmath0, 2, 3}, the size of the bilstm hidden layers @xmath1, 256, 512} and the batch size @xmath2, 32, 64}. for bilstm-cnn, in addition to the other parameters, the window size is set to 3 and the number of filters (dimension of character embeddings) is set to @xmath3, 50, 100}. ## conclusions and future work the paper presented a study focuses on french spoken language understanding (slu) task using the media corpus. first we proposed the evaluation of whether updating the word embeddings during training improves or not the results, according to several scenarios.second, we proposed to use a bilstm-cnn architecture that integrates character embeddings as additional features. last, we proposed to evaluate the performance of bert approaches on the media task through different ways. experimental results show, that the word embeddings needed to be updated during slu model training are the ones trained on small corpus like media. however, it is better for word embeddings trained on huge and out-of-domain to be frozen, since those word embeddings have captured enough general semantic and syntactic characteristics relevant to slu task. more, the word embeddings trained on wiki and wiki+media achieve comparable results. this shows that, we don't need to use both a task-dependent corpus and another out-of-domain corpus to train the word embeddings. in addition, we observed the usefulness of character embeddings when added as additional features. regarding the evaluation of the performance of bert approaches, the fune-tuning of camembert and flaubert base models show that the best results are achieved using camembert base model trained on ccnet data. it yields to 29.35% of relative improvement in terms of cer reduction in comparison to the baseline (7.56 vs 10.7). finally, the integration of the extracted camembert's contextual embeddings to the bilstm and bilstm-cnn architectures reveal that contextual embeddings achieves competitive results in comparison to cbow word embeddings whatever the architecture, and confirm the importance of character embeddings. for future work, we propose to evaluate the performance of bert's contextual embeddings extracted from different encoder's layers, and to make in-depth error analysis for the different systems.