Auto-induced semantic classes
Advanced computer dialogue agents contain a natural language understanding component that requires knowledge of semantic classes and concepts. These are frequently generated manually for new tasks. We avoid this time-consuming procedure by using a two-step unsupervised clustering process. First, a semantic generalizer automatically induces semantic classes using training data from well-studied applications (domains) for which large transcribed corpora of humanhuman dialogues are available. Candidate word pairs are grouped into similar semantic groups according to the similarity of their lexical bigram contexts. We show that the proposed algorithms for automatically inducing semantic classes perform very well for typical spoken dialogue applications. We exceed 90% precision for the first 100 cluster assignments for narrowly defined tasks such as a movie information task. For a heterogeneous task such as the text-based WSJ, a precision of only 24% is increased to 73% by including context thresholding and part-of-speech tagging. Second, we determine the degree of domain independence for each class by using concept comparison and projection metrics to rank order semantic classes by degree of domain independence.
computer dialogue agent natural language understanding unsupervised clustering lexical bigram context spoken dialogue domain independence