We study distributional similarity measures for the purpose of improving probability estimation for unseen cooccurrences. Our contributions are three-fold: an empirical comparison of a broad range of measures; a classification of similarity functions based on the information that they incorporate; and the introduction of a novel function that is superior at evaluating potential proxy distributions. P(v[n) of an unseen word pair (n, v) drawn from some finite set N x V. Two state-of-the-art technologies are Katz's (1987) backoff method and Jelinek and Mercer's (1980) interpolation method. Both use P(v) to estimate P(v[n) when (n, v) is unseen, essentially ignoring the identity of n. An alternative approach is distance-weighted averaging, which arrives at an estimate for unseen cooccurrences by combining estimates for 25 cooccurrences involving similar words: 1 /P(v[n) ----~-~mES(n) sim(n, m)P(v[m) ~-]mES(n) sim(n, m)
Mapping distributional similarity, REF opts for similar word co-occurrence probability estimation improvement.
Measures Of Distributional Similarity
Annual Meeting Of The Association For Computational Linguistics
Computer Science