Language identification is the task of identifying the language a given document is written in. This paper describes a detailed examination of what models perform best under different conditions, based on experiments across three separate datasets and a range of tokenisation strategies. We demonstrate that the task becomes increasingly difficult as we increase the number of languages, reduce the amount of training data and reduce the length of documents. We also show that it is possible to perform language identification without having to perform explicit character encoding detection.
REF analyzed the influence of number of discriminated languages, the amount of training data and the length of documents on the accuracy of document language detection.
Language Identification: The Long and the Short of the Matter
Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics
Computer Science