Twitter provides access to large volumes of data in real time, but is notoriously noisy, hampering its utility for NLP. In this paper, we target out-of-vocabulary words in short text messages and propose a method for identifying and normalising ill-formed words. Our method uses a classifier to detect ill-formed words, and generates correction candidates based on morphophonemic similarity. Both word similarity and context are then exploited to select the most probable correction candidate for the word. The proposed method doesn't require any annotations, and achieves state-of-the-art performance over an SMS corpus and a novel dataset based on Twitter.
Others used a classifier to detect misspelled words, and generates correction possibilities based on morphophonemic similarity REF . But these methods need large-scale of annotated training data, which limits their adaptability to new domains and languages.
Lexical Normalisation of Short Text Messages: Makn Sens a #twitter
Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies
Computer Science