We present a new keyword extraction algorithm that applies to a single document without using a corpus. Frequent terms are extracted first, then a set of cooccurrence between each term and the frequent terms, i.e., occurrences in the same sentences, is generated. Co-occurrence distribution shows importance of a term in the document as follows. If probability distribution of co-occurrence between term a and the frequent terms is biased to a particular subset of frequent terms, then term a is likely to be a keyword. The degree of biases of distribution is measured by the χ 2 -measure. Our algorithm shows comparable performance to tfidf without using a corpus.
Matsuo, Ishizuka REF proposed a method called keyword extraction algorithm that applies to a single document without using a corpus.
Keyword extraction from a single document using word co-occurrence statistical information
International Journal on Artificial Intelligence Tools
Computer Science