Speedy Gonzales: A Collection of Fast Task-Specific Models for S panish (2024)

Abstract: Large language models (LLM) are now a very common and successful path to approach language and retrieval tasks. While these LLM achieve surprisingly good results it is a challenge to use them on more constrained resources. Techniques to compress these LLM into smaller and faster models have emerged for English or Multilingual settings, but it is still a challenge for other languages. In fact, Spanish is the second language with most native speakers but lacks of these kind of resources. In this work, we evaluate all the models publicly available for Spanish on a set of 6 tasks and then, by leveraging on Knowledge Distillation, we present Speedy Gonzales, a collection of inference-efficient task-specific language models based on the ALBERT architecture. All of our models (fine-tuned and distilled) are publicly available on: https://huggingface.co/dccuchile.

## introduction the utilization of learned dense representations of text is nowadays a common and successful approach for different kind of information retrieval (ir) tasks @xcite . these learned representations are usually obtained by training a language model using large collections of texts from the web. two key aspects to watch to make the most of these models are size and speed of them. the size of these models has grown overtime and now very large language models (llm) are common, with models that range from hundred of millions to billions of parameters. these pretrained models are not only heavy on memory requirements but also on the operations they do on every inference, which is a bottleneck when trying to deploy these models for tasks that are expected to be fast such as question answering or semantic search. these llms are usually trained on english by big technology companies using web-scale datasets and substantial computational resources. prominent examples include the well-known gpt-3 model @xcite . for languages other than english the available models are typically variants of bert @xcite , roberta @xcite or albert @xcite . in the case of spanish, which is one of the five most spoken languages in the world and the second with most native speakers, the available models range from 5m to 335m of parameters. in figure 1 we showed how different spanish pre-trained models compare in terms of model size (number of parameters) and inference speed (macs). despite the remarkable performance of these llms across a range of tasks, it remains a challenge to utilize them effectively in computing environments that are constrained by limited resources, such as web or mobile applications. new techniques to address this problem have emerged for english @xcite @xcite @xcite or multilingual @xcite models. these typically leverage on different kinds of knowledge distillation @xcite to compress the results of a large and performant model into another one which is typically lighter and more inference efficient. for other languages this is still an open challenge, where we lack from this kind of resources. in this work we try to close this gap with new resources (inference-efficient models) for the spanish language. our contributions are the following: albert models, achieving more lighter and inference efficient models, while retaining most of the task performance of the bigger counterparts. ## related work transformers, introduced by @xcite have become the default architecture for textrelated tasks. transformer encoders like bert @xcite , roberta @xcite or albert @xcite are some of the most popular, by its ability to encode complex relations on texts by training on large collections of texts, with the training task consisting of corrupt some parts of a text sequence and train a model to reconstruct the correct sequence. while models with billions of parameters have become common for english language @xcite , it is not the case for most other languages, which are typically restricted to hundreds of millions of parameters. for spanish language, which is one of the most spoken languages in the world, the models available follow the bert @xcite , roberta @xcite or albert @xcite architecture and are described in further detail in section 4.2. several ways to compress these models have been proposed through the years. the most com-mon ones are quantization @xcite , pruning @xcite and knowledge distillation @xcite . network quantization compresses the original network by reducing the number of bits required to represent each weight, resulting in a lighter model. in the case of bert, examples of these kinds of methods are ternarybert @xcite and binarybert @xcite where they were able to reduce the weight size to 2 and 1 bit respectively, while maintaining most of the original bert performance. the technique of pruning aims to reduce the number of connections (weights) in a neural network, which results in a reduction of the model size and also a very sparse pattern of the weights. @xcite showed that in most feed-forward neural networks it is possible to find a subnetwork that achieves similar or better accuracy. in knowledge distillation (kd) @xcite the knowledge learned by a big and strong model, the teacher model, is transferred to a lighter model, the student model, by forcing this student to mimic the teacher. multiples ways of knowledge distillation have been proposed @xcite . @xcite uses kd to transfer the knowledge from bert to lighter rnns. @xcite proposes pre-training compact bert models and then using task-specific kd to achieve better results. sanh et al. (2019) introduces a taskagnostic scheme where kd is used on the pretraining task. wang et al. (2020) and jiao et al. (2020) proposed different methods exclusive for 2 in line with common practices, the framework includes both a distillation loss between the teacher and student models and a cross-entropy loss between the gold labels and the student's predictions, as indicated by the dashed line. transformers, to directly distill the knowledge from the self-attention layers of the teacher model to the student model. our work is similar to @xcite by proposing the use of compact transformers but we use the albert architecture instead of the bert one. we also use the idea from @xcite of reusing the layers of a pre-trained model, instead of random initializing a new one. differently from that work, that has to choose which layers to reuse, we only adjust the number of layers (and thus, the inference speed) since all the albert layers are shared. another difference with those two works is that in our work we skip pre-training (or kd on the pre-training task) and directly apply kd on the task-specific phase. ## methodology in pursuit of our goal to have efficient models for spanish in various tasks, we employ the method of knowledge distillation. this method will be further elaborated in the subsequent section. ## evaluating spanish pre-trained language models in order to achieve our goal of have efficient models for spanish in a variety of tasks we first define a set of tasks to evaluate those models. these tasks are the same evaluated by @xcite and are described in section 4.1. we then define a set of possible teacher models, in particular, we wanted to try every model that was pre-trained on general domain spanish text and is publicly available, therefore we exclude rigoberta @xcite , which is a deberta @xcite model for spanish that is not public and robertuito @xcite which is a roberta-like model for spanish that was trained on twitter datasets and should be better suited for social media related tasks. all considered models are described in section 4.2. after evaluating all models on each task, we selected the model with lowest validation loss as the teacher model for the task. the list of selected models can be found in appendix a. ## results ## conclusion and future work in this work, we introduce speedy gonzales, a novel resource for the spanish nlp and ir communities comprising a collection of computationally efficient language models trained on six tasks and eight datasets. by applying the knowledge distillation technique, our models achieve comparable performance to state-of-the-art models, while showing faster inference speeds. the full collection of models, including our proposed models and all the teacher models fine-tuned on the tasks considered, are made publicly available for further research. we believe that the availability of these models and the expansion of the knowledge distillation method to additional tasks will drive the widespread utilization of large language models in the spanish speaking community, particularly for individuals and organizations seeking to tackle crucial information retrieval challenges, such as question answering, text similarity and semantic search, in both academic and industrial settings. potential directions for future research include exploring the use of multiple teachers in the distillation process and developing metrics to formally evaluate the balance between inference speed and task performance. in proceedings of the 2019 conference of the north american chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers), pages 1298-1308, minneapolis, minnesota. association for computational linguistics.