While there has been a lot of work on finding frequent itemsets in transaction data streams, none of these solve the problem of finding similar pairs according to standard similarity measures. This paper is a first attempt at dealing with this, arguably more important, problem. We start out with a negative result that also explains the lack of theoretical upper bounds on the space usage of data mining algorithms for finding frequent itemsets: Any algorithm that (even only approximately and with a chance of error) finds the most frequent k-itemset must use space Ω(min{mb, n k , (mb/ϕ) k }) bits, where mb is the number of items in the stream so far, n is the number of distinct items and ϕ is a support threshold. To achieve any non-trivial space upper bound we must thus abandon a worst-case assumption on the data stream. We work under the model that the transactions come in random order, and show that surprisingly, not only is small-space similarity mining possible for the most common similarity measures, but the mining accuracy improves with the length of the stream for any fixed support threshold.
Campagna and Pagh REF make the arguably weaker, but still questionable, assumption that the order of the transactions in the stream is random.
On Finding Similar Items in a Stream of Transactions
2010 IEEE International Conference on Data Mining Workshops
2010 IEEE International Conference on Data Mining Workshops
Computer Science Mathematics