waashk/dblp huggingface dataset:
a rule evaluation support method with learning models based on objective rule evaluation indexes. in this paper, we present a novel rule evaluation support method for post-processing of mined results with rule evaluation models based on objective indexes. post-processing of mined results is one of the key issues to make a data mining process successfully. however, it is difficult for human experts to evaluate many thousands of rules from a large dataset with noises completely. to reduce the costs of rule evaluation procedures, we have developed the rule evaluation support method with rule evaluation models, which are obtained with objective rule evaluation indexes and evaluations of a human expert for each rule. since the method is needed more accurate rule evaluation models, we have compared learning algorithms to construct rule evaluation models with the actual meningitis data mining result and actual rule sets from uci datasets. then we show the availability of our adaptive rule evaluation support method.
mining high utility itemsets. traditional association rule mining algorithms onlygenerate a large number of highly frequent rules, butthese rules do not provide useful answers for what thehigh utility rules are. in this work, we develop a novelidea of top-k objective-directed data mining, which focuseson mining the top-k high utility closed patterns thatdirectly support a given business objective. to associationmining, we add the concept of utility to capture highly desirablestatistical patterns and present a level-wise item-setmining algorithm. with both positive and negativeutilities, the anti-monotone pruning strategy in apriorialgorithm no longer holds. in response, we develop a newpruning strategy based on utilities that allow pruning oflow utility itemsets to be done by means of a weaker butanti-monotonic condition. our experimental results showthat our algorithm does not require a user specifiedminimum utility and hence is effective in practice.
investigative profiling with computer forensic log data and association rules. investigative profiling is an important activity in computerforensics that can narrow the search for one or morecomputer perpetrators. data mining is a technique that hasproduced good results in providing insight into large volumesof data. this paper describes how the associationrule data mining technique may be employed to generateprofiles from log data and the methodology used for the interpretationof the resulting rule sets. the process relies onbackground knowledge in the form of concept hierarchiesand beliefs, commonly available from, or attainable by, thecomputer forensic investigative team. results obtained withthe profiling system has identified irregularities in computerlogs.
distributed web mining using bayesian networks from multiple data streams. we present a collective approach to mine bayesian net-works from distributed heterogenous web-log data streams. in this approach we first learn a local bayesian network at each site using the local data. then each site identifies the observations that are most likely to be evidence of coupling between local and non-local variables and transmits asub-set of these observations to a central site. another bayesian network is learnt at the central site using the data transmittedfrom the local site. the local and central bayesian networks are combined to obtain a collective bayesian net-work, that models the entire data. we applied this techniqueto mine multiple data streams where data centralization is difficult because of large response time and scalability issues.experimental results and theoretical justification that demonstrate the feasibility of our approach are presented.
handling generalized cost functions in the partitioning optimization problem through sequential binary programming. this paper proposes a framework for cost-sensitive classification under a generalized cost function. by combining decision trees with sequential binary programming, we can handle unequal misclassification costs, constrained classification, and complex objective functions that other methods cannot. our approach has two main contributions. first, it provides a new method for cost-sensitive classification that outperforms a traditional, accuracy-based method and some current cost-sensitive approaches. second, and more important, our approach can handle a generalized cost function, instead of the simpler misclassification cost matrix to which other approaches are limited.
learning with progressive transductive support vector machine. support vector machine (svm) is a new learningmethod developed in recent years based on thefoundations of statistical learning theory. by taking atransductive approach instead of an inductive one insupport vector classifiers, the test set can be used as anadditional source of information about margins. intuitively,we would expect transductive learning to yieldimprovements when the training sets are small or whenthere is a significant deviation between the training andworking set subsamples of the total population. in thispaper, a progressive transductive support vector machineis addressed to extend joachims' transductive svm tohandle different class distributions. it solves the problemof having to estimate the ratio of positive/negativeexamples from the working set. the experimental resultsshow that the algorithm is very promising.
online hierarchical clustering in a data warehouse environment. many important industrial applications rely on data mining methods to uncover patterns and trends in large data warehouse environments. since a data warehouse is typically updated periodically in a batch mode, the mined patterns have to be updated as well. this requires not only accuracy from data mining methods but also fast availability of up-to-date knowledge, particularly in the presence of a heavy update load. to cope with this problem, we propose the use of online data mining algorithms which permanently store the discovered knowledge in suitable data structures and enable an efficient adaptation of these structures after insertions and deletions on the raw data. in this paper, we demonstrate how hierarchical clustering methods can be reformulated as online algorithms based on the hierarchical clustering method optics, using a density estimator for data grouping. we also discuss how this algorithmic schema can be specialized for efficient online single-link clustering. a broad experimental evaluation demonstrates that the efficiency is superior with significant speed-up factors even for large bulk insertions and deletions.
sequential pattern mining in multiple streams. in this paper, we deal with mining sequential patterns in multiple data streams. building on a state-of-the-art sequential pattern mining algorithm prefixspan for mining transaction databases, we propose mile¹, an efficient algorithm to facilitate the mining process. mile recursively utilizes the knowledge of existing patterns to avoid redundant data scanning, and can therefore effectively speed up the new patterns' discovery process. another unique feature of mile is that it can incorporate some prior knowledge of the data distribution in data streams into the mining process to further improve the performance. extensive empirical results show thatmile is significantly faster than prefixspan. as mile consumes more memory than prefixspan, we also present a solution to balance the memory usage and time efficiency in memory constrained environments.
emailsift: email classification based on structure and content. in this paper we propose a novel approach that uses structure as well as the content of emails in a folder for email classification. our approach is based on the premise that representative — common and recurring — structures/patterns can be extracted from a pre-classified email folder and the same can be used effectively for classifying incoming emails. a number of factors that influence representative structure extraction and the classification are analyzed conceptually and validated experimentally. in our approach, the notion of inexact graph match is leveraged for deriving structures that provide coverage for characterizing folder contents. extensive experimentation validate the selection of parameters and the effectiveness of our approach for email classification.
incremental mining of frequent xml query pattern. recently, the discovering of frequent xml query patterns gains its focus due to its many applications in xml data management, and several algorithms have been proposed to discover frequent query patterns using the frequent structure mining techniques. in this paper we consider the problem of incremental mining of frequent xml query patterns. we propose a novel method to minimize the i/o and computation requirements for handling incremental updates.
mining chains of relations. traditional data mining applications consider the problem of mining a single relation between two attributes. for example, in a scientific bibliography database, authors are related to papers, and we may be interested in discovering association rules between authors. however, in real life, we often have multiple attributes related though chains of relations. for example, authors write papers, and papers concern one or more topics. mining such relational chains poses additional challenges. in this paper we consider the following problem: given a chain of two relationsr₁(a, p) and r₂(p, t) we want to find selectors for the objects in t such that the projected relation between a and p satisfies a specific property. the motivation for our approach is that a given property might not hold on the whole dataset, but it might hold when projecting the data on a selector set. we discuss various algorithms and we examine the conditions under which the apriori technique can be used. we experimentally demonstrate the effectiveness of our methods.
a hypergraph based clustering algorithm for spatial data sets. clustering is a discovery process in data mining an can be used to group together the objects of a database into meaningful subclasses which serve as the foundation for other data analysis techniques.in this paper, we focus on dealing with a set of spatial data. for the spatial data, the clustering problem becomes that of finding the densely populate regions of the space and thus grouping these regions into clusters such that the intracluster similarity is maximized and theintercluster similarity is minimized. we develop a novel hierarchical clustering algorithm that uses a hypergraph to represent a set of spatial data. this hypergraph is initially constructed from the delaunay triangulation graph of the data set and can correctly capture the relationships among sets of data points. two phases are developed for the proposed clustering algorithm to find the clusters in the data set.we evaluate our hierarchical clustering algorithm with some spatial data sets in which contain clusters of different sizes, shapes, densities, and noise. experimental results on these data sets are very encouraging.
an empirical bayes approach to detect anomalies in dynamic multidimensional arrays. we consider the problem of detecting anomalies in data that arise as multidimensional arrays with each dimension corresponding to the levels of a categorical variable. in typical data mining applications, the number of cells in such arrays are usually large. our primary focus is detecting anomalies by comparing information at the current time to historical data. naive approaches advocated in the process control literature do not work well in this scenario due to the multiple testing problem - performing multiple statistical tests on the same data produce excessive number of false positives. we use an empirical bayes method which works by fitting a two component gaussian mixture to deviations at current time. the approach is scalable to problems that involve monitoring massive number of cells and fast enough to be potentially useful in many streaming scenarios. we show the superiority of the method relative to a naive "per component error rate" procedure through simulation. a novel feature of our technique is the ability to suppress deviations that are merely the consequence of sharp changes in the marginal distributions. this research was motivated by the need to extract critical application information and business intelligence from the daily logs that accompany large-scale spoken dialog systems deployed by at&t. we illustrate our method on one such system.
spam filtering using a markov random field model with variable weighting schemas. in this paper we present a markov random field model based approach to filter spam. our approach examines the importance of the neighborhood relationship (mrf cliques) among words in an email message for the purpose of spam classification. we propose and test several different theoretical bases for weighting schemes among corresponding neighborhood windows. our results demonstrate that unexpected side effects depending on the neighborhood window size may have larger accuracy impact than the neighborhood relationship effects of the markov random field.
on effective conceptual indexing and similarity search in text data. similarity search in text has proven to be an interesting problem from the qualitative perspective because of inherent redundancies and ambiguities in textual descriptions. the methods used in search engines in order to retrieve documents most similar to user-defined sets of keywords are not applicable to targets which are medium to large size documents, because of even greater noise effects stemming from the presence of a large number of words unrelated to the overall topic in the document. the inverted representation is the dominant method for indexing text, but it is not as suitable for document-to-document similarity search, as for short user-queries. one way of improving the quality of similarity search is latent semantic indexing (lsi), which maps the documents from the original set of words to a concept space. u fortunately, lsi maps the data into a domain in which it is not possible to provide effectiveindexing techniques. in this paper, we investigate new ways of providing conceptual search among documents bycreating a representation in terms of conceptual word-chains. this technique also allows effective indexing techniques so that similarity queries ca be performed on large collectionsof documents by accessing a small amount of data. we demonstrate that our scheme outperforms standard textual similarity search o the inverted representation both in terms of quality a d search efficiency.
moment: maintaining closed frequent itemsets over a stream sliding window. this paper considers the problem of mining closed frequent itemsets over a sliding window using limited memory space. we design a synopsis data structure to monitor transactions in the sliding window so that we can output the current closed frequent itemsets at any time. due to time and memory constraints, the synopsis data structure cannot monitor all possible itemsets. however, monitoring only frequent itemsets will make it impossible to detect new itemsets when they become frequent. in this paper, we introduce a compact data structure, the closed enumeration tree (cet), to maintain a dynamically selected set of itemsets over a sliding-window. the selected itemsets consist of a boundary between closed frequent itemsets and the rest of the itemsets. concept drifts in a data stream are reflected by boundary movements in the cet. in other words, a status change of any itemset (e.g., from non-frequent to frequent) must occur through the boundary. because the boundary is relatively stable, the cost of mining closed frequent itemsets over a sliding window is dramatically reduced to that of mining transactions that can possibly cause boundary movements in the cet. our experiments show that our algorithm performs much better than previous approaches.
a preference model for structured supervised learning tasks. the preference model introduced in this paper gives a natural framework and a principled solution for a broad class of supervised learning problems with structured predictions, such as predicting orders (label and instance ranking), and predicting rates (classification and ordinal regression). we show how all these problems can be cast as linear problems in an augmented space, and we propose an on-line method to efficiently solve them. experiments on an ordinal regression task confirm the effectiveness of the approach.
indexing and mining free trees. tree structures are used extensively in domains such ascomputational biology, pattern recognition, computer networks,and so on. in this paper, we present an indexing techniquefor free trees and apply this indexing technique to theproblem of mining frequent subtrees. we first define a novelrepresentation, the canonical form, for rooted trees and extendthe definition to free trees. we also introduce anotherconcept, the canonical string, as a simpler representationfor free trees in their canonical forms. we then apply ourtree indexing technique to the frequent subtree mining problemand present freetreeminer, a computationally efficientalgorithm that discovers all frequently occurring subtreesin a database of free trees. we study the performance andthe scalability of our algorithms through extensive experimentsbased on both synthetic data and datasets from tworeal applications: a dataset of chemical compounds and adataset of internet multicast trees.
integrating fuzziness into olap for multidimensional fuzzy association rules mining. this paper contributes to the ongoing research onmultidimensional online association rules mining byproposing a general architecture that utilizes a fuzzy datacube for knowledge discovery. three different methods areintroduced to mine fuzzy association rules in the constructedfuzzy data cube, namely single dimension, multidimensionaland hybrid association rules mining. experimental resultsobtained for each of the three methods on the adult data ofthe united states census in 2000 show their effectiveness andapplicability.
efficient determination of dynamic split points in a decision tree. we consider the problem of choosing split points forcontinuous predictor variables in a decision tree. previousapproaches to this problem typically either (1) discretize the continuous predictor values prior to learning or (2) apply a dynamic method that considers all possible split points for each potential split. in this paper, we describe anumber of alternative approaches that generate a smallnumber of candidate split points dynamically with littleoverhead. we argue that these approaches are preferable to pre-discretization, and provide experimental evidence that they yield probabilistic decision trees with the same prediction accuracy as the traditional dynamic approach.furthermore, because the time to grow a decision tree isproportional to the number of split points evaluated, our approach is significantly faster than the traditional dynamic approach.
using emerging patterns and decision trees in rare-class classification. the problem of classifying rarely occurring cases is faced in many real life applications. the scarcity of the rare cases makes it difficult to classify them correctly using traditional classifiers. in this paper, we propose a new approach to use emerging patterns (eps) and decision trees (dts) in rare-class classification (epdt). eps are those itemsets whose supports in one class are significantly higher than their supports in the other classes. epdt employs the power of eps to improve the quality of rare-case classification. to achieve this aim, we first introduce the idea of generating new non-existing rare-class instances, and then we over-sample the most important rare-class instances. our experiments show that epdt outperforms many classification methods.
efficient splitting rules based on the probabilities of pre-assigned intervals. this paper describes new methods for classification in orderto find an optimal tree. unlike the current splitting rules that areprovided by searching all threshold values, this paper proposes thesplitting rules that are based on the probabilities of pre-assignedintervals.
discovering similar patterns for characterising time series in a medical domain. in this article, we describe the process of discovering similar patterns in time series and creating reference models for population groups in a medical domain, and particularly in the field of physiotherapy, using data mining techniques on a set of isokinetic data. the discovered knowledge was evaluated against the expertise of a physician specialized in isokinetic techniques, and applied in the i4 (intelligent interpretation of isokinetic information) project developed in conjunction with the spanish national center for sports research and sciences for muscular diagnosis and rehabilitation, injury prevention, training evaluation and planning, etc., of elite athletes and ordinary people.
an adaptive learning approach for noisy data streams. two critical challenges typically associated with mining data streams are concept drift and data contamination. to address these challenges, we seek learning techniques and models that are robust to noise and can adapt to changes in timely fashion. we approach the stream-mining problem using a statistical estimation framework, and propose a fast and robust discriminative model for learning noisy data streams. we build an ensemble of classifiers to achieve timely adaptation by weighting classifiers in a way that maximizes the likelihood of the data. we further employ robust statistical techniques to alleviate the problem of noise sensitivity. experimental results on both synthetic and real-life data sets demonstrate the effectiveness of this new model learning approach.
discovery of functional relationships in multi-relational data using inductive logic programming. ilp systems have been largely applied to datamining classification tasks with a considerable success. the use of ilp systems in regression tasks has been far less successful. current systems have very limited numerical reasoning capabilities, which limits the application of ilp to discovery of functional relationships of numeric nature. this paper proposes improvements in numerical reasoning capabilities of ilp systems for dealing with regression tasks. it proposes the use of statistical-based techniques like model validation and model selection to improve noise handling and it introduces a new search stopping criterium based on the pac method to evaluate learning performance. we have found these extensions essential to improve on results over machine learning and statistical-based algorithms used in the empirical evaluation study.
towards automatic generation of query taxonomy: a hierarchical query clustering approach. previous works on automatic query clustering most generatea flat, un-nested partition of query terms. in this work,we are pursuing to organize query terms into a hierarchicalstructure and construct a query taxonomy in an automaticway. the proposed approach is designed based on a hierarchicalagglomerative clustering algorithm to hierarchicallygroup similar queries and generate the cluster hierarchiesby a novel cluster partition technique. the search processesof real-world search engines are combined to obtain highlyranked web documents as the feature source for each queryterm. preliminary experiments show that the proposed approachis effective to obtain thesaurus information for queryterms, and is also feasible to construct a query taxonomywhich provides a basis for in-depth analysis of users' searchinterests and domain-specific vocabulary on a larger scale.