A probabilistic model for stemmer generation
In this paper we will present a language-independent probabilistic model which can automatically generate stemmers. Stemmers can improve the retrieval effectiveness of information retrieval systems, however the designing and the implementation of stemmers requires a laborious amount of effort due to the fact that documents and queries are often written or spoken in several different languages. The probabilistic model proposed in this paper aims at the development of sternmers used for several languages. The proposed model describes the mutual reinforcement relationship between stems and derivations and then provides a probabilistic interpretation. A series of experiments shows that the sternmers generated by the probabilistic model are as effective as the ones based on linguistic knowledge. (C) 2004 Elsevier Ltd. All rights reserved.
stemming mutual reinforcement multi-lingual information retrieval web information retrieval