Constructing n-gram rules for natural language models through exploring the limitation of the Zipf-Mandelbrot law
The Zipf-Mandelbrot law is widely used to model a power-law distribution on ranked data. One of the best known applications of the Zipf-Mandelbrot law is in the area of linguistic analysis of the distribution of words ranked by their frequency in a text corpus. By exploring known limitations of the Zipf-Mandelbrot law in modeling the actual linguistic data from different domains in both printed media and online content, a new algorithm is developed to effectively construct n-gram rules for building natural language (NL) models required for a human-to-computer interface. The construction of statistically-oriented n-gram rules is based on a new computing algorithm that identifies the area of divergence between Zipf-Mandelbrot curve and the actual frequency distribution of the ranked n-gram text tokens extracted from a large text corpus derived from the online electronic programming guide (EPG) for television shows and movies. Two empirical experiments were carried out to evaluate the EPG-specific language models created using the new algorithm in the context of NL-based information retrieval systems. The experimental results show the effectiveness of the algorithm for developing low-complexity concept models with high coverage for the user's language models associated with both typed and spoken queries when interacting with a NL-based EPG search interface.
zipf-mandelbrot law natural language processing n-gram statistical language models quantitative linguistics