We present a simple method for language independent and task independent text categorization learning, based on character-level n-gram language models. Our approach uses simple information theoretic principles and achieves effective performance across a variety of languages and tasks without requiring feature selection or extensive pre-processing. To demonstrate the language and task independence of the proposed technique, we present experimental results on several languages-Greek, English, Chinese and Japanese-in several text categorization problems-language identification, authorship attribution, text genre classification, and topic detection. Our experimental results show that the simple approach achieves state of the art performance in each case.
For instance, REF used simple n-gram language modeling analyses on well-defined corpora in various languages (Greek, Japanese, Chinese, and English) to classify texts based on languages, authorship, genres, and topics.
Language And Task Independent Text Categorization With Simple Language Models
Human Language Technology Conference And Meeting Of The North American Association For Computational Linguistics
Computer Science