Is Encoder-Decoder Transformer the Shiny Hammer? (2022)

Abstract: We present an approach to multi-class classification using an encoder-decoder transformer model. We trained a network to identify French varieties using the same scripts we use to train an encoder-decoder machine translation model. With some slight modification to the data preparation and inference parameters, we showed that the same tools used for machine translation can be easily re-used to achieve competitive performance for classification. On the French Dialectal Identification (FDI) task, we scored 32.4 on weighted F1, but this is far from a simple naive bayes classifier that outperforms a neural encoder-decoder model at 41.27 weighted F1.

## introduction sometimes you just kinda wanna sometimes one might find more appealing to reuse the same code, scripts and infrastructure that already serve an nlp product for another purpose. in this case, an eco-system of tools is already available to train machine translation models and serve the model with a restful api, then we need some language identification tools. then, one might think, then you think, technically, an auto-regressive encoderdecoder model that produces a single token at inference is sort of like a classifier. recent works had validated the thought @xcite @xcite , most notably the "don't classify, translate!" (dct) idea simply re-used an encoderdecoder machine translation models as a hierarchical classifier to categorize e-commerce products. to test the dct model for language identification, we evaluated the approach on the french cross-domain dialect identification (fdi) dataset @xcite while participating in a vardial shared task . 1 an example of the input and output of the fdi data looks as follows: ## motivation our initial thought was to use the least effort in script changes to train a machine translation model to a multi-class classification one. being frugal, the secondary objective is to ensure that we do not spend more than a day's worth of gpu hours. intuitively, we need the decoder to produce only one token that marks the class label, so we shouldn't be needing heavy machinery (i.e. deep layers) in the decoder. previous works @xcite have also shown that offsetting decoder layers with more encoder layers could improve inference latency. also, when training encoder-decoder models on small datasets, deep decoder layers might be an overkill. therefore, we decided to re-use a "mini" transformer @xcite with 6 encoder, 2 decoder layers trained with the marian nmt toolkit @xcite ). foot_1 ## tl;dr (experimental setup) we trained an encoder-decoder machine translation model using the marian nmt framework with the following hyperparameters: the modified script with the above hyperparameter used to train the model is available on https://github.com/alvations/ myth/blob/master/train-esther.sh . we refer to this model as dct mini for the rest of the paper. ## analysis figure 1 and 2 presents the confusion matrices for the dct mini and dct micro models. figure 2: confusion matrix for dct micro for both models, we observe that the: ## related work while generic language identification seemed solved @xcite @xcite , distinguishing language varieties which are often lower resourced remains a challenge @xcite @xcite . hence, the language varieties identification task is a staple of the evaluation campaigns hosted by the vardial workshops @xcite @xcite @xcite . across the many evaluation campaigns, probabilistic models like naive bayes have often ranked top on the leaderboard @xcite @xcite . ## conclusion in this paper, we have described our experiments to reuse encoder-decoder transformer models as a classifier based on the "don't classify, translate" idea. evaluating on the french dialect identification (fdi) dataset, we found that a simple naive bayes model works better than the 6 layers encoderdecoder models and a really small neural model worked even worse. and now, some concluding remarks: the encoder-decoder transformer is a shiny hammer that works fairly well for many nlp/mt tasks. but note, the 'your miles may vary' (ymmv) caution. also, as a sanity check, a simple non-neural approach is a good baseline.