A R es N et-50-Based Convolutional Neural Network Model for Language ID Identification from Speech Recordings (2021)

Abstract: This paper describes the model built for the SIGTYP 2021 Shared Task aimed at identifying 18 typologically different languages from speech recordings. Mel-frequency cepstral coefficients derived from audio files are transformed into spectrograms, which are then fed into a ResNet-50-based CNN architecture. The final model achieved validation and test accuracies of 0.73 and 0.53, respectively.

## introduction in the sigtyp 2021 shared task, participants are asked to predict language ids from speech recordings. the novelty of this shared task consists in (i) the variety of the languages involved, which comprises very different language genera/families (see indeed, many linguistics-related shared tasks seem to focus on a restricted number of related languages (often indo-european ones) and model their spellings. 1 in particular, this latter feature poses a number of theoretical and practical challenges, especially when some language comparison is involved, as in typological studies. writing systems, as is known, can highly diverge in what they represent, even when they are segmental scripts (not to mention that a language can be encoded in different writing systems, like, for example, kabyle). if we consider the languages in the shared task dataset, it would be very hard to find a meaningful way to compare, for example, the javanese writing system with the portuguese one: the former could be written in the scriptio continua of its traditional script, foot_1 while the latter's alphabetical script distinguishes space-delimited tokens (mostly corresponding to morphosyntactic words). interestingly enough, it is no less challenging to compare word-based scripts, in that there is no single definition of graphemic (let alone morphosyntactic) word across languages, and even within the same writing system, inconsistencies are not uncommon. the use of language recordings instead of written documents should therefore ensure a more direct and consistent encoding of languages. recordings also allow us to capture intonation structure, which is usually absent (or represented in a minimal form) in writing systems, despite its crucial role in conveying information (see @xcite and, more in general, information structure studies). on the downside, speech recordings are sensitive to idiolect variances, which a statistical model should however be able to properly address by not overfitting the training data. this is even more relevant for the sigtyp 2021 shared task, in that its goal is to train a model being able to generalize to recordings of not only different people, but also very different genres/content. in the following sections, i present the model i built to tackle the multiclass classification task at hand. in section 2, the training and validation sets are described. section 3 details the training phase of a number of models, including the resnet-50based cnn one, which i chose to participate in the sigtyp 2021 shared task. section 4 summarizes the results of the resnet-50-based cnn model, while section 5 contains some concluding remarks. ## the training and validation sets the training and validation sets are released by the organizers of the shared task as npy files containing mel-frequency cepstral coefficients (mfccs) computed from audio files. the training set consists of 72,000 readings of the new testament (each of them usually corresponding to a verse), while the validation set consists of 8,000 instances from different sources. ## results and discussion the resnet-50-based model provides good training and validation accuracy scores (0.98 and 0.73, respectively). importantly, both accuracy scores grow during training, and both loss scores get smaller and smaller. in figure 6 , the algorithm seems to have converged. however, the final accuracy score (0.53) calculated on the test set released by the organizers seems to suggest that some overfitting has occurred. the confusion matrices (appendix c and d), the heatmaps (appendix e and f), as well as the tables containing precision, recall, and f1 scores ( notably, the rows for eastern bru and vlax romani are not available in the heatmaps (appendix e and f) because the languages are absent in both the validation and test sets. tweaking the hyperparameters and especially experimenting with deeper resnet architectures could probably lead to an improvement of the results. to tackle the task with relatively simple rnn and cnn architectures were unsuccessful. resnet-50, however, proved to offer a robust architecture to train linguistic data for language id prediction. the task at hand was challenging because the training data differ considerably from the validation data, and therefore any model needs strong ability to generalize. the resnet-50-based cnn model proposed in this article shows good validation and test accuracies (0.73 and 0.53, respectively). notably, sudanese is very often misclassified as javanese.