One Size Does Not Fit All: Finding the Optimal Subword Sizes for F ast T ext Models across Languages (2021)

Abstract: Unsupervised representation learning of words from large multilingual corpora is useful for downstream tasks such as word sense disambiguation, semantic text similarity, and information retrieval. The representation precision of log-bilinear fastText models is mostly due to their use of subword information. In previous work, the optimization of fastText’s subword sizes has not been fully explored, and non-English fastText models were trained using subword sizes optimized for English and German word analogy tasks. In our work, we find the optimal subword sizes on the English, German, Czech, Italian, Spanish, French, Hindi, Turkish, and Russian word analogy tasks. We then propose a simple n-gram coverage model and we show that it predicts better-than-default subword sizes on the Spanish, French, Hindi, Turkish, and Russian word analogy tasks. We show that the optimization of fastText’s subword sizes matters and results in a 14% improvement on the Czech word analogy task. We also show that expensive parameter optimization can be replaced by a simple n-gram coverage model that consistently improves the accuracy of fastText models on the word analogy tasks by up to 3% compared to the default subword sizes, and that it is within 1% accuracy of the optimal subword sizes.

## methods in this section, we describe our methods and propose our 𝑛-gram coverage model, which can be used to suggest subword sizes for a fasttext model without expensive parameter optimization. ## results in this section, we show and discuss the optimal subword sizes, accuracies, and 𝑛-gram coverages on the english, german, czech, italian, spanish, french, hindi, turkish, and russian word analogy tasks. we also show that the 𝑛-gram coverage can be used to suggest subword sizes that are close to the optimal subword sizes. ## conclusion subword sizes have a profound impact on the accuracy of fasttext language models and their word embeddings. however, they are expensive to optimize on large corpora. in this work, we showed the optimal subword sizes for czech, italian, spanish, french, hindi, turkish, and russian fasttext language models, we confirmed prior optimal subword sizes reported for english and german, and we showed that the optimization of subword sizes improves the accuracy of fasttext on word analogy tasks by up to 14% compared to the default subword sizes. our optimal subword sizes can be used in applications of fasttext as the new default. furthermore, we proposed a cheap and simple 𝑛-gram coverage model that consistently improves the accuracy of fasttext models on the word analogy tasks by up to 3% compared to the default subword sizes, and that it is within 1% accuracy of the optimal subword sizes on average. subword sizes suggested by our 𝑛-gram coverage model can be used in applications of fasttext as the new default for under-resourced languages, where the optimal subword sizes are unknown.