waashk/dblp huggingface dataset:
efficient multidimensional quantitative hypotheses generation. finding local interrelations (hypotheses) among attributeswithin very large databases of high dimensionalityis an acute problem for many databases and data miningapplications. these include, dependency modeling, clusteringlarge databases, correlation and link analysis.traditional statistical methods are concerned with the corroborationof (a set of) hypotheses on a given body ofdata. testing all of the hypotheses that can be generatedfrom a database with millions of records and dozens offields is clearly infeasible. generating, on the other hand,a set of the most "promising" hypotheses (to be corroborated)requires much intuition and ingenuity.in this paper we present an efficient method for rankingthe multidimensional hypotheses using image processingof data visualization. in the heart of the method lies theuse of visualization techniques and image processing ideasto rank subsets of attributes according to the relation betweenthem in the databases. some of the scalability issuesare solved by concise generalized histograms and by usingan efficient on-line computation of clustering around amedian with only five additional memory words. in additionto presenting our algorithmic methodology, we demonstrateits efficiency and performance by applying it to realcensus data sets, as well as synthetic data sets.
evolutionary time series segmentation for stock data mining. stock data in the form of multiple time series aredifficult to process, analyze and mine. however, when theycan be transformed into meaningful symbols like technicalpatterns, it becomes an easier task. most recent work ontime series queries only concentrates on how to identify agiven pattern from a time series. researchers do notconsider the problem of identifying a suitable set of timepoints for segmenting the time series in accordance with agiven set of pattern templates (e.g., a set of technicalpatterns for stock analysis). on the other hand, using fixedlength segmentation is a primitive approach to thisproblem; hence, a dynamic approach (with highcontrollability) is preferred so that the time series can besegmented flexibly and effectively according to the needs ofthe users and the applications. in view of the facts that sucha segmentation problem is an optimization problem andevolutionary computation is an appropriate tool to solve it,we propose an evolutionary time series segmentationalgorithm. this approach allows a sizeable set of stockpatterns to be generated for mining or query. in addition,defining the similarity between time series (or time seriessegments) is of fundamental importance in fitnesscomputation. by identifying the perceptually importantpoints directly from the time domain, time series segmentsand templates of different lengths can be compared andintuitive pattern matching can be carried out in an effectiveand efficient manner. encouraging experimental results arereported from tests that segment the time series of selectedhong kong stocks.
analyzing high-dimensional data by subspace validity. we are proposing a novel method that makes it possibleto analyze high dimensional data with arbitrary shapedprojected clusters and high noise levels. at the core of ourmethod lies the idea of subspace validity. we map the datain a way that allows us to test the quality of subspaces usingstatistical tests. experimental results, both on synthetic andreal data sets, demonstrate the potential of our method.
scalable multi-relational association mining. we propose the new radar technique for multi-relational data mining. this permits the mining of very large collections and provides a new technique for discovering multi-relational associations. results show that radar is reliable and scalable for mining a large yeast homology collection, and that it does not have the main-memory scalability constraints of the farmer and warmr tools.
objective and subjective algorithms for grouping association rules. we propose two algorithms for grouping and summarizingassociation rules. the first algorithm recursively groupsrules according to the structure of the rules and generatesa tree of clusters as a result. the second algorithm groupsthe rules according to the semantic distance between therules by making use of an autometically tagged semantictree-structured network of items. we provide a case study inwhich the proposed algorithms are evaluated. the resultsshow that our grouping methods are effective and producegood grouping results.
fast pnn-based clustering using k-nearest neighbor graph. search for nearest neighbor is the main source ofcomputation in most clustering algorithms. we proposethe use of nearest neighbor graph for reducing thenumber of candidates. the number of distancecalculations per search can be reduced from o(n) to o(k)where n is the number of clusters, and k is the number ofneighbors in the graph. we apply the proposed schemewithin agglomerative clustering algorithm known as thepnn algorithm.
using functional pca for cardiac motion exploration. principal component analysis (pca) [14, 6] is a maintool in multivariate data analysis. its paradigms are alsoused in the karhunen-loeve decomposition [5], a standardtool in image processing. extensions of pca to the frameworkof functional data have been proposed. the analy-sisprovided by the functional pca seems to be a powerfultool to find principal sources of variability in curves or images,but it fails in providing us with easy interpretationsin the case of multifunctional data. guide lines aiming atspot information from the outputs of pca applied to functionalswith values in space of continuous functions upona bounded domain are proposed. an application to cardiacmotion analysis illustrates the complexity of the multi-functionalframework and the results provided by functionalpca. the acm portal is published by the association for computing machinery. copyright © 2010 acm, inc. terms of usage privacy policy code of ethics contact us useful downloads: adobe acrobat quicktime windows media player real player
knowledge discovery from diagrammatically represented data. knowledge discover from diagrammatic data can be facilitated by a language that permits queries on such data.such a language (diagrammatic sql) is being developed to expedite the development of an autonomous artificially intelligent agent with a capacity to deal with diagrammatic information.this language is described and examples of how it can be used to facilitatediagrammatic data mining are detaled
an evaluation of approaches to classification rule selection. in this paper a number of classification rule evaluation measures are considered. in particular the authors review the use of a variety of selection techniques used to order classification rules contained in a classifier, and a number of mechanisms used to classify unseen data. the authors demonstrate that rule ordering founded on the size of antecedent works well given certain conditions.
integrating e-commerce and data mining: architecture and challenges. we show that the e-commerce domain can provide all the right ingredients for successful data mining. we describe an integrate architecture for supporting this integration. thearchitecture can dramatically reduce the pre-processing, cleaning, and data understanding effort often documented to take 80%of the time in knowledge discovery projects. we emphasize the need for data collection at the application server layer (not the web server)in order to support logging of data and metadata that is essential to the discovery process. we describe the datatransformation bridges require from the transaction processing systems an customer event streams (e.g.,clickstreams) to the data warehouse. we detail the mining workbench, which needs to provide multiple views of the data through reporting, data mining algorithms, visualization, and olap. we conclude with a set of challenges.
obtaining best parameter values for accurate classification. in this paper we examine the effect that the choice of support and confidence thresholds has on the accuracy of classifiers obtained by classification association rule mining. we show that accuracy can almost always be improved by a suitable choice of threshold values, and we describe a method for finding the best values. we present results that demonstrate this approach can obtain higher accuracy without the need for coverage analysis of the training data. keywords: classification, association rule mining.
text document categorization by term association. a good text classifier is a classifier that efficiently categorizeslarge sets of text documents in a reasonable timeframe and with an acceptable accuracy, and that providesclassification rules that are human readable for possiblefine-tuning. if the training of the classifier is also quick,this could become in some application domains a good assetfor the classifier. many techniques and algorithms forautomatic text categorization have been devised. accordingto published literature, some are more accurate than others,and some provide more interpretable classification modelsthan others. however, none can combine all the beneficialproperties enumerated above. in this paper, we present anovel approach for automatic text categorization that borrowsfrom market basket analysis techniques using associationrule mining in the data-mining field. we focus on twomajor problems: (1) finding the best term association rulesin a textual database by generating and pruning; and (2)using the rules to build a text classifier. our text categorizationmethod proves to be efficient and effective, and experimentson well-known collections show that the classifierperforms well. in addition, training as well as classificationare both fast and the generated rules are human readable.
t-trees, vertical partitioning and distributed association rule mining. in this paper we consider a technique (data-vp) fordistributed (and parallel) association rule mining thatmakes use of a vertical partitioning technique to distributethe input data amongst processors. the proposed verticalpartitioning is facilitated by a novel compressed set enumerationtree data structure (the t-tree), and an associatedmining algorithm (apriori-t), that allows for computationallyeffective distributed/parallel arm when compared withexisting approaches.
efficient subsequence matching in time series databases under time and amplitude transformations. subsequence matching in large time series databases hasattracted a lot of interest and many methods have been proposedthat cope with this problem in an adequate extend.however, locating subsequence matches of arbitrary length,under time and amplitude transformations, has received farless attention and is still an open problem. in this paperwe present an efficient algorithm for variable-length subsequencematching under transformations that guaranteesno false dismissals. further, this algorithm uses a novelsimilarity criterion for determining similarity under amplitudetransformations in a most efficient way. finally, ouralgorithm has been tested in various experiments on realdata, resulting in a running time improvement of one orderof magnitude compared to the naive approach.
unsupervised segmentation of categorical time series into episodes. this paper describes an unsupervised algorithm forsegmenting categorical time series into episodes. thevoting-experts algorithm first collects statistics aboutthe frequency and boundary entropy of ngrams, then passesa window over the series and has two "expert methods" decidewhere in the window boundaries should be drawn. thealgorithm successfully segments text into words in four languages.the algorithm also segments time series of robotsensor data into subsequences that represent episodes inthe life of the robot. we claim that voting-expertsfinds meaningful episodes in categorical time series becauseit exploits two statistical characteristics of meaningfulepisodes.
attribute measurement policies for time and cost sensitive classification. attribute measurement is an important component of classification algorithms, which could limit their applicability in realtime settings. the time taken to assign a value to an unknown attribute may reduce the overall utility of the final result. we identify three different costs that must be considered, including a time sensitive utility function. we model this attribute measurement problem as a markov decision process (mdp), and build a policy to control this process using ao* heuristic search. the results offer a cost-effective approach to attribute measurement and classification for a variety of realtime applications.
speed-up iterative frequent itemset mining with constraint changes. mining of frequent itemsets is a fundamental datamining task. past research has proposed many efficientalgorithms for the purpose. recent work also highlightedthe importance of using constraints to focus the miningprocess to mine only those relevant itemsets. in practice,data mining is often an interactive and iterative process.the user typically changes constraints and runs the miningalgorithm many times before satisfied with the finalresults. this interactive process is very time consuming.existing mining algorithms are unable to take advantageof this iterative process to use previous mining results tospeed up the current mining process. this results inenormous waste in time and in computation. in this paper,we propose an efficient technique to utilize previousmining results to improve the efficiency of current miningwhen constraints are changed. we first introduce theconcept of tree boundary to summarize the usefulinformation available from previous mining. we then showthat the tree boundary provides an effective and efficientframework for the new mining. the proposed techniquehas been implemented in the contexts of two existingfrequent itemset mining algorithms, fp-tree and treeprojection. experiment results on both synthetic and real-lifedatasets show that the proposed approach achievesdramatic saving in computation.
online algorithms for mining semi-structured data stream. in this paper, we study an online data mining problemfrom streams of semi-structured data such as xml data.modeling semi-structured data and patterns as labeled orderedtrees, we present an online algorithm streamt thatreceives fragments of an unseen possibly infinite semi-structureddata in the document order through a datastream, and can return the current set of frequent patternsimmediately on request at any time. a crucial part of our algorithmis the incremental maintenance of the occurrencesof possibly frequent patterns using a tree sweeping technique.we give modifications of the algorithm to other on-linemining model. we present theoretical and empiricalanalyses to evaluate the performance of the algorithm.
mining frequent closed patterns in microarray data. microarray data typically contains a large number of columns and a small number of rows, which poses a great challenge for existing frequent (closed) pattern mining algorithms that discover patterns in item enumeration space. in this paper, we propose two new algorithms that explore the row enumeration space to mine frequent closed patterns. several experiments on real-life gene expression data show that the new algorithms are faster than existing algorithms, including closet, charm, closet+ and carpenter.
detection of significant sets of episodes in event sequences. we present a method for a reliable detection of "unusual" sets of episodes in the form of many pattern sequences, scanned simultaneously for an occurrence as a subsequence in a large event stream within a window of size w. we also investigate the important special case of all permutations of the same sequence, which models the situation where the order of events in an episode does not matter, e.g., when events correspond to purchased market basket items. in order to build a reliable monitoring system we compare obtained measurements to a reference model which in our case is a probabilistic model (bernoulli or markov). we first present a precise analysis that leads to a construction of a threshold. the difficulties of carrying out a probabilistic analysis for an arbitrary set of patterns, stems from the possible simultaneous occurrence of many members of the set as subsequences in the same window, the fact that the different patterns typically do have common symbols or common subsequences or possibly common prefixes, and that they may have different lengths. we also report on extensive experimental results, carried out on the wal-mart transactions database, that show a remarkable agreement with our theoretical analysis. this paper is an extension of our previous work in [reliable detection of episodes in event sequences] where we laid out foundation for the problem of the reliable detection of an "unusual" episodes, but did not consider more than one episode scanned simultaneously for an occurrence.
optimal projections of high dimensional data. in this paper, we compare two artificial neuralnetwork algorithms for performing exploratoryprojection pursuit, a statistical technique forinvestigating data by projecting it onto lower dimensionalmanifolds. the neural networks are extensions of anetwork which performs principal component analysis.we illustrate the technique on artificial data beforeapplying it to real data.
blocking anonymity threats raised by frequent itemset mining. in this paper we study when the disclosure of datamining results represents, per se, a threat to the anonymity of the individuals recorded in the analyzed database. the novelty of our approach is that we focus on an objective definition of privacy compliance of patterns without any reference to a preconceived knowledge of what is sensitive and what is not, on the basis of the rather intuitive and realistic constraint that the anonymity of individuals should be guaranteed. in particular, the problem addressed here arises from the possibility of inferring from the output of frequent itemset mining (i.e., a set of itemsets with support larger than a threshold ó), the existence of patterns with very low support (smaller than an anonymity threshold k)[3]. in the following we develop a simple methodology to block such inference opportunities by introducing distortion on the dangerous patterns.
identifying markov blankets with decision tree induction. the markov blanket of a target variable is theminimum conditioning set of variables that makes thetarget independent of all other variables. markovblankets inform feature selection, aid in causal discoveryand serve as a basis for scalable methods of constructingbayesian networks. this paper applies decision treeinduction to the task of markov blanket identification.notably, we compare (a) c5.0, a widely used algorithmfor decision rule induction, (b) c5c, which post-processesc5.0's rule set to retain the most frequentlyreferenced variables and (c) pc, a standard method forbayesian network induction. c5c performs as well as orbetter than c5.0 and pc across a number of data sets.our modest variation of an inexpensive, accurate, off-the-shelfinduction engine mitigates the need for specializedprocedures, and establishes baseline performance againstwhich specialized algorithms can be compared.
classification with degree of membership: a fuzzy approach. algorithms adopt either a decision tree based approach or an approach that requires users to provide some user-specifiedthresholds to guide the search for interesting rules. in this paper, we propose a new approach based on the use of an objective interestingness measure todistinguish interesting rules from uninteresting ones. using linguistic terms to represent the revealed regularities and exceptions, this approach s especially useful when the discovered rules are presented to human experts for examination because of the affinity with thehuman knowledge representation. the use of fuzzy technique allows the predict on of attribute values to be associated with degree of membership. our approach s, therefore, able to deal with the cases that an object can belong to more than one class. for example, a person can suffer from cold and fever to certain extent at the same time. furthermore, our approach is more resilient to noise and missing data values because of the use of fuzzy technique. to evaluate the performance of our approach, we tested it using several real-life databases. the experimental results show that it can be very effective at data mining tasks. in fact, when compared to popular data mining algorithms, our approach can be better ableto uncover useful rules hidden in databases.
on feature selection through clustering. we study an algorithm for feature selection that clusters attributes using a special metric and then makes use of the dendrogram of the resulting cluster hierarchy to choose the most relevant attributes. the main interest of our technique resides in the improved understanding of the structure of the analyzed data and of the relative importance of the attributes for the selection process.
adaptive clustering: obtaining better clusters using feedback and past experience. adaptive clustering uses external feedback to improve cluster quality; past experience serves to speed up execution time. an adaptive clustering environment is proposed that uses q-learning to learn the reward values of successive data clusterings. adaptive clustering supports the reuse of clusterings by memorizing what worked well in the past. it has the capability of exploring multiple paths in parallel when searching for good clusters. in a case study, we apply adaptive clustering to instance-based learning relying on a distance function modification approach. a distance function adaptation scheme that uses external feedback is proposed and compared with other distance function learning approaches. experimental results indicate that the use of adaptive clustering leads to significant improvements of instance-based learning techniques, such as k-nearest neighbor classifiers. moreover, as a by-product a new instance-based learning technique is introduced that classifies examples by solely using cluster representatives; this technique shows high promise in our experimental evaluation.
heuristic optimization for decentralized frequent itemset counting. the choices for mining of decentralized data are numerous, and we have developed techniques to enumerate andoptimize decentralized frequent itemset counting. in thispaper, we introduce our heuristic approach to improve theperformance of such techniques developed in ways similarto query processing in database systems. we also describeempirical results that validate our heuristic techniques.
a fast algorithm for computing hypergraph transversals and its application in mining emerging patterns. computing the minimal transversals of a hypergraph isan important problem in computer science that has significantapplications in data mining. in this paper, we present anew algorithm for computing hypergraph transversals andhighlight their close connection to an important class ofpatterns known as emerging patterns. we evaluate our techniqueon a number of large datasets and show that it out-performsprevious approaches by a factor of 9-29 times.
generating an informative cover for association rules. mining association rules may generate a large numbersof rules making the results hard to analyze manually.pasquier et al. have discussed the generation of guigues-duquenne-luxenburger basis (gd-l basis). using a similarapproach, we introduce a new rule of inference anddefine the notion of association rules cover as a minimalset of rules that are non-redundant with respect to this newrule of inference. our experimental results (obtained usingboth synthetic and real data sets) show that our coversare smaller than the gd-l basis and they are computed intime that is comparable to the classic apriori algorithm forgenerating rules.