This paper presents a simple but effective approach to unknown word processing in Japanese morphological analysis, which handles 1) unknown words that are derived from words in a pre-defined lexicon and 2) unknown onomatopoeias. Our approach leverages derivation rules and onomatopoeia patterns, and correctly recognizes certain types of unknown words. Experiments revealed that our approach recognized about 4,500 unknown words in 100,000 Web sentences with only 80 harmful side effects and a 6% loss in speed.
Sasano et al. REF introduced a variety of derivational patterns into Japanese morphological analysis to handle unknown words that are derived from words in a predefined lexicon.
A Simple Approach to Unknown Word Processing in Japanese Morphological Analysis
IJCNLP
Computer Science