waashk/dblp huggingface dataset:
mining similar temporal patterns in long time-series data and its application to medicine. data mining in time-series medical databases has beenreceiving considerable attention since it provides a way ofrevealing useful information hidden in the database; forexample relationships between temporal course of examinationresults and onset time of diseases. this paperpresents a new method for finding similar patterns in temporalsequences. the method is a hybridization of phase-constraintmultiscale matching and rough clustering. multiscalematching enables us cross-scale comparison of thesequences, namely, it enable us to compare temporal patternsby partially changing observation scales. rough clusteringenable us to construct interpretable clusters of thesequences even if their similarities are given as relativesimilarities. we combine these methods and cluster the sequencesaccording to multiscale similarity of patterns. experimentalresults on the chronic hepatitis dataset showedthat clusters demonstrating interesting temporal patternswere successfully discovered.
text mining for a clear picture of defect reports: a praxis report. we applied the text mining categorization technology,in the publicly available, ibm enterprise informationportal v8.1 to more than 15,000 customer reported,product problem records. we used a proven softwarequality category set to categorize these problem recordsinto different areas of interest. our intent was to developa clear picture of potential areas for quality improvementin each of the software products reviewed, and to providethis information to development's management.the paper presents the benefits that can be gained fromcategorizing problem records, as well as the limitations.
wavelet based uxo detection. the detection and classification of unexploded ordnance(uxo) is considered a multi-dimensional pattern recognitionproblem. standard techniques in solving multi-dimensionaldetection and classification problems involveusing large sets of templates or libraries. this paper showsthat by using wavelet transformation a single library willallow a particular class of ordnance to be classified over arange of depths.
effective and efficient distributed model-based clustering. in many companies data is distributed among several sites, i.e. each site generates its own data and manages its own data repository. analyzing and mining these distributed sources requires distributed data mining techniques to find global patterns representing the complete information. the transmission of the entire local data set is often unacceptable because of performance considerations, privacy and security aspects, and bandwidth constraints. traditional data mining algorithms, demanding access to complete data, are not appropriate for distributed applications. thus, there is a need for distributed data mining algorithms in order to analyze and discover new knowledge in distributed environments. one of the most important data mining tasks is clustering which aims at detecting groups of similar data objects. in this paper, we propose a distributed model-based clustering algorithm that uses em for detecting local models in terms of mixtures of gaussian distributions. we propose an efficient and effective algorithm for deriving and merging these local gaussian distributions to generate a meaningful global model. in a broad experimental evaluation we show that our framework is scalable in a highly distributed environment.
ensemble modeling through multiplicative adjustment of class probability. we develop a new concept for aggregating items of evidencefor class probability estimation. in naïve bayes, eachfeature contributes an independent multiplicative factor tothe estimated class probability. we modify this model to includean exponent in each factor in order to introduce fea-tureimportance. these exponents are chosen to maximizethe accuracy of estimated class probabilities on the trainingdata. for naïve bayes, this modification accomplishes morethan what feature selection can. more generally, since theindividual features can be the outputs of separate probabilitymodels, this yields a new ensemble modeling approach,which we call apm (adjusted probability model), alongwith a regularized version called apmr.
a generic framework for efficient subspace clustering of high-dimensional data. subspace clustering has been investigated extensively since traditional clustering algorithms often fail to detect meaningful clusters in high-dimensional data spaces. many recently proposed subspace clustering methods suffer from two severe problems: first, the algorithms typically scale exponentially with the data dimensionality and/or the subspace dimensionality of the clusters. second, for performance reasons, many algorithms use a global density threshold for clustering, which is quite questionable since clusters in subspaces of significantly different dimensionality will most likely exhibt significantly varying densities. in this paper, we propose a generic framework to overcome these limitations. our framework is based on an efficient filter-refinement architecture that scales at most quadratic w.r.t. the data dimensionality and the dimensionality of the subspace clusters. it can be applied to any clustering notions including notions that are based on a local density threshold. a broad experimental evaluation on synthetic and real-world data empirically shows that our method achieves a significant gain of runtime and quality in comparison to state-of-the-art subspace clustering algorithms.
mining coverage-based fuzzy rules by evolutional computation. in this paper, we propose a novel mining approach based on the genetic process and an evaluation mechanism to automatically construct an effective fuzzy rule base. the proposed approach consists of three phases: fuzzy-rule generating, fuzzy-rule encoding and fuzzy-ruleevolution. in the fuzzy-rule generating phase, a number of fuzzy rules are randomly generated. in the fuzzy-rule encoding phase, all the rules generated are translated into fixed-length bit strings to form an initial population. in the fuzzy-rule evolution phase, genetic operations andcredit assignment are applied at the rule level. the proposed mining approach chooses good individuals in the population for mating, gradually creating better offspring fuzzy rules. a concise and compact fuzzy rule base is thus constructed effectively without human expertintervention.
hierarchical density-based clustering of uncertain data. the hierarchical density-based clustering algorithm optics has proven to help the user to get an overview over large data sets. when using optics for analyzing uncertain data which naturally occur in many emerging application areas, e.g. location based services, or sensor databases, the similarity between uncertain objects has to be expressed by one numerical distance value. based on such single-valued distance functions optics, like other standard data mining algorithms, can work without any changes. in this paper, we propose to express the similarity between two fuzzy objects by distance probability functions which assign a probability value to each possible distance value. contrary to the traditional approach, we do not extract aggregated values from the fuzzy distance functions but enhance optics so that it can exploit the full information provided by these functions. the resulting algorithm foptics helps the user to get an overview over a large set of fuzzy objects.
effectiveness of information extraction, multi-relational, and semi-supervised learning for predicting functional properties of genes. we focus on the problem of predicting functional propertiesof the proteins corresponding to genes in the yeastgenome. our goal is to study the effectiveness of approachesthat utilize all data sources that are availablein this problem setting, including unlabeled and relationaldata, and abstracts of research papers. we study transductionand co-training for using unlabeled data. we investigatea propositionalization approach which uses relationalgene interaction data. we study the benefit of informationextraction for utilizing a collection of scientific abstracts.the studied tasks are kdd cup tasks of 2001 and 2002.the solutions which we describe achieved the highest scorefor task 2 in 2001, the fourth rank for task 3 in 2001, thehighest score for one of the two subtasks and the third placefor the overall task 2 in 2002.
ontologies improve text document clustering. text document clustering plays an important role in providingintuitive navigation and browsing mechanisms by organizinglarge sets of documents into a small number ofmeaningful clusters. the bag of words representation usedfor these clustering methods is often unsatisfactory as it ignoresrelationships between important terms that do not co-occurliterally. in order to deal with the problem, we integratecore ontologies as background knowledge into theprocess of clustering text documents. our experimentalevaluations compare clustering techniques based on pre-categorizationsof texts from reuters newsfeeds and on asmaller domain of an elearning course about java. in theexperiments, improvements of results by background knowledgecompared to a baseline without background knowledgecan be shown in many interesting combinations.
concise representation of frequent patterns based on disjunction-free generators. many data mining problems require the discover of frequent patterns in order to be solved.frequent itemsets are useful in the discover of association rules, episode rules, sequential patterns and clusters. the number of frequent itemsets is usually huge. therefore, it is important to work out concise representations of frequent itemsets. in the paper, we describe three basic loassless representations of frequent patters in a uniform wayand offer a new lossless representation of frequent patterns based on disjunction-free generators. the new representation is more concisethan two of the basic representations and more efficiently computablethan the third representation. we propose an algorithm for the determining the new representation.
evolutionary algorithms for clustering gene-expression data. this work deals with the problem of automatically finding optimal partitions in bioinformatics datasets. we propose incremental improvements for a clustering genetic algorithm (cga), culminating in the evolutionary algorithm for clustering (eac). the cga and its modified versions are evaluated in five gene-expression datasets, showing that the proposed eac is a promising tool for clustering gene-expression data.
probabilistic noise identification and data cleaning. real world data is never as perfect as we would like itto be and can often suffer from corruptions that may impactinterpretations of the data, models created from thedata, and decisions made based on the data.one approachto this problem is to identify and remove records that containcorruptions.unfortunately, if only certain fields in arecord have been corrupted then usable, uncorrupted datawill be lost.in this paper we present lens, an approach foridentifying corrupted fields and using the remaining non-corruptedfields for subsequent modeling and analysis.ourapproach uses the data to learn a probabilistic model containingthree components: a generative model of the cleanrecords, a generative model of the noise values, and a probabilisticmodel of the corruption process.we provide an algorithmfor the unsupervised discovery of such models andempirically evaluate both its performance at detecting corruptedfields and, as one example application, the resultingimprovement this gives to a classifier.
the hybrid poisson aspect model for personalized shopping recommendation. predicting an individual customer's likelihood of purchasinga specific item forms the basis of many marketingactivities, such as personalized shopping recommendation.collaborative filtering and association rule miningcan be applied to this problem, but in retail supermarkets,the problem becomes particularly challenging because ofthe sparsity and skewness of transaction data. this paperpresents hypam(hybrid poisson aspect model), a newprobabilistic graphical model that combines a poisson mixturewith a latent aspect class model to model customers'shopping behavior. we empirically compare hypam withtwo well-known recommenders, grouplens (a correlation-basedmethod), and ibm smartpad (association rules andcosine similarity). experimental results show that hypamoutperforms the other recommenders by a large margin fortwo real-world retail supermarkets, ranking most of actualpurchases in the top ten percent of the most likely purchaseditems. we also present a new visualization method, rankplot, to evaluate the quality of recommendations.
tractable group detection on large link data sets. discovering underlying structure from co-occurrencedata is an important task in a variety of fields, including:insurance, intelligence, criminal investigation, epidemiology,human resources, and marketing.previously kubicaet. al. presented the group detection algorithm (gda) - analgorithm for finding underlying groupings of entities fromco-occurrence data.this algorithm is based on a probabilisticgenerative model and produces coherent groups thatare consistent with prior knowledge.unfortunately, the optimizationused in gda is slow, potentially making it infeasiblefor many large data sets.to this end, we present k-groups - an algorithm that uses an approach similar tothat of k-means to significantly acclerate the discovery ofgroups while retaining gda's probabilistic model.we comparethe performance of gda and k-groups on a variety ofdata, showing that k-groups' sacrifice in solution quality issignificantly offset by its increase in speed.
mining ratio rules via principal sparse non-negative matrix factorization. association rules are traditionally designed to capture statistical relationship among itemsets in a given database. to additionally capture the quantitative association knowledge, f.korn et al recently proposed a paradigm named ratio rules for quantifiable data mining. however, their approach is mainly based on principle component analysis (pca) and as a result, it cannot guarantee that the ratio coefficient is non-negative. this may lead to serious problems in the rules' application. in this paper, we propose a new method, called principal sparse non-negative matrix factorization (psnmf), for learning the associations between itemsets in the form of ratio rules. in addition, we provide a support measurement to weigh the importance of each rule for the entire dataset.
transduction and typicalness for quality assessment of individual classifications in machine learning and data mining. in the past machine learning algorithms have been successfully used in many problems, and are emerging as valuable data analysis tools. however, their serious practical use is affected by the fact, that more often than not, they cannot produce reliable and unbiased assessments of their predictions' quality. in last years, several approaches for estimating reliability or confidence of individual classifiers have emerged, many of them building upon the algorithmic theory of randomness, such as (historically ordered) transduction-based confidence estimation, typicalness-based confidence estimation, and transductive reliability estimation. unfortunately, they all have weaknesses: either they are tightly bound with particular learning algorithms, or the interpretation of reliability estimations is not always consistent with statistical confidence levels. in the paper we propose a joint approach that compensates the mentioned weaknesses by integrating typicalness-based confidence estimation and transductive reliability estimation into joint confidence machine. the resulting confidence machine produces confidence values in the statistical sense (e.g., a confidence level of 95% means that in 95% the predicted class is also a true class), as well as provides us with a general principle that is independent of to the particular underlying classifier we perform a series of tests with several different machine learning algorithms in several problem domains. we compare our results with that of a proprietary tcm-nn method as well as with kernel density estimation. we show that the proposed method significantly outperforms density estimation methods, and how it may be used to improve their performance.
efficient mining of frequent subgraphs in the presence of isomorphism. frequent subgraph mining is an active research topic inthe data mining community. a graph is a general modelto represent data and has been used in many domains likecheminformatics and bioinformatics. mining patterns fromgraph databases is challenging since graph related operations,such as subgraph testing, generally have higher timecomplexity than the corresponding operations on itemsets,sequences, and trees, which have been studied extensively.in this paper, we propose a novel frequent subgraph miningalgorithm: ffsm, which employs a vertical search schemewithin an algebraic graph framework we have developedto reduce the number of redundant candidates proposed.our empirical study on synthetic and real datasets demonstratesthat ffsm achieves a substantial performance gainover the current start-of-the-art subgraph mining algorithmgspan.
semi-supervised clustering with metric learning using relative comparisons. semi-supervised clustering algorithms partition a given data set using limited supervision from the user. in this paper, we propose a clustering algorithmthat uses supervision in terms of relative comparisons, viz., is closer to than to . the success of a clustering algorithm also depends on the kind of dissimilarity measure. the proposed clustering algorithm learns the underlying dissimilarity measure while finding compact clusters in the given data set. through our experimental studies on high-dimensional textual data sets, we demonstrate that the proposed algorithm achieves higher accuracy than the algorithms using pairwise constraints for supervision.
discovery of interesting association rules from livelink web log data. we present our experience in mining web usage patternsfrom a large collection of livelink log data. livelink is aweb-based product of open text, which provides automaticmanagement and retrieval of different types of informationobjects over an intranet or extranet. we report our experiencein preprocessing raw log data and post-processing themining results for finding interesting rules. in particular,we compare and evaluate a number of rule interestingnessmeasures and find that two of the measures that have notbeen used in association rule learning work very well.
mining the smallest association rule set for predictions. mining transaction databases for association rules usually generates a large number of rules, most of which are unnecessary when used for subsequent prediction. in this paper we define a rule set for a given transaction database that is much smaller than the association rule set but makes the same predictions as the association rule set by the confidence priority. we call this subset the informative rule set. the informative rule set is not constrained to particular target items; and it is smaller than the non-redundant association rule set. we present an algorithm to directly generate the informative rule set, i.e., without generating all frequentitemsets first, and that accesses the database less often than other unconstrained direct methods. we show experimentally that the informative rule set is much smaller than boththe association rule set and the non-redundant association rule set, and that it can be generated more efficiently.
mining temporal patterns without predefined time windows. this paper proposes algorithms for discovering temporal patterns without predefined time windows.the problem of discovering temporal patterns is divided into two sub-tasks: (1) using "cheap statistics" for dependence testing and candidates removal (2) identifying the temporal relationships between dependent event types.the dependence problem is formulated as the problem of comparing two probability distributions and is solved using a technique reminiscent of the distance methods used in spatial point process, while the latter problem is solved using an approach based on chi-squared tests.experiments are conducted to evalaute the effectiveness and scalability of the proposed methods.
mass spectrum labeling: theory and practice. we introduce the problem of labeling a particle's mass spectrum with the substances it contains, and develop several formal representations of the problem, taking into account practical complications such as unknown compounds and noise. this task is currently a bottle-neck in analyzing data from a new generation of instruments for real-time environmental monitoring.
on learning asymmetric dissimilarity measures. many practical applications require that distance measures to be asymmetric and context-sensitive. we introduce context-sensitive learnable asymmetric dissimilarity (clad) measures, which are defined to be a weighted sum of a fixed number of dissimilarity measures where the associated weights depend on the point from which the dissimilarity is measured. the parameters used in defining the measure capture the global relationships among the features. we provide an algorithm to learn the dissimilarity measure automatically from a set of user specified comparisons in the form "x is closer to y than to z," and study its performance. the experimental results show that the proposed algorithm outperforms other approaches due to the context sensitive nature of the clad measures.
partial ensemble classifiers selection for better ranking. ranking is an important task in data mining and knowledge discovery. we propose a novel approach called pecs algorithm to improve the overall ranking performance of a given ensemble. we formally analyse the sufficient and necessary condition under whichpecs algorithm can effectively improve ensemble ranking performance. the experiments with real-world data sets show that this new approach achieves significant improvements in ranking over the original bagging and adaboost ensembles.
a personalized music filtering system based on melody style classification. with the growth of digital music, the personalized musicfiltering system is helpful for users. melody style is one ofthe music features to represent user's music preference. inthis paper, we present a personalized content-based musicfiltering system to support music recommendation based onuser's preference of melody style. we propose the multitypemelody style classification approach to recommend themusic objects. the system learns the user preference bymining the melody patterns from the music access behaviorof the user. a two-way melody preference classifier istherefore constructed for each user. music recommendationis made through this melody preference classifier.performance evaluation shows that the filtering effect of theproposed approach meets user's preference.
comparing naive bayes, decision trees, and svm with auc and accuracy. predictive accuracy has often been used as the mainand often only evaluation criterion for the predictive performanceof classification or data mining algorithms. inrecent years, the area under the roc (receiver operatingcharacteristics) curve, or simply auc, has been proposedas an alternative single-number measure for evaluating performanceof learning algorithms. in our previous work, weproved that auc is, in general, a better measure (definedprecisely) than accuracy. many popular data mining algorithmsshould then be re-evaluated in terms of auc. forexample, it is well accepted that naive bayes and decisiontrees are very similar in accuracy. how do they compare inauc? also, how does the recently developed svm (supportvector machine) compare to traditional learning algorithmsin accuracy and auc? we will answer these questions inthis paper. our conclusions will provide important guide-linesin data mining applications on real-world datasets.