waashk/dblp huggingface dataset:
mining frequent itemsets in distributed and dynamic databases. traditional methods for frequent itemset mining typicallyassume that data is centralized and static. such methods imposeexcessive communication overhead when data is distributed,and they waste computational resources when datais dynamic. in this paper we present what we believe to bethe first unified approach that overcomes these assumptions.our approach makes use of parallel and incremental techniquesto generate frequent itemsets in the presence of dataupdates without examining the entire database, and imposesminimal communication overhead when mining distributeddatabases. further, our approach is able to generate bothlocal and global frequent itemsets. this ability permits ourapproach to identify high-contrast frequent itemsets, whichallows one to examine how the data is skewed over differentsites.
interpretations of association rules by granular computing. this paper presents interpretations for associationrules. it first introduces pawlak's method, and thecorresponding algorithm of finding decision rules (a kindof association rules). it then uses extended random sets topresent a new algorithm of finding interesting rules. itproves that the new algorithm is faster than pawlak'salgorithm. the extended random sets are easily to includemore than one criterion for determining interesting rules.they also provide two measures for dealing withuncertainties in association rules.
metric rule generation with septic shock patient data. in this contribution we present an application of metric rule generation in the domain of medical research. we consider intensive car unit patients developing a septic shockduring their stay at the hospital. to analyse the patient data, rule generation is embedded in a medical data mining cycle. for rule generation, we improve an architecture basedon a growing trapezoidal basis function network.
using discriminant analysis for multi-class classification. discriminant analysis is known to learn discriminativefeature transformations. this paper studies its use in multi-classclassification problems. the performance is tested ona large collection of benchmark datasets.
intersection based generalization rules for the analysis of symbolic septic shock patient data. in intensive care units much data is irregularly recorded.here, we consider the analysis of symbolic septic shock patientdata. we show that it could be worth consideringthe generalization paradigm (individual cases generalizedto more general rules) instead of the association paradigm(combining single attributes) when considering very individualcases (e.g. patients) and when expecting longer rulesthan shorter ones. we present an algorithm for rule generationand classification based on heuristically generatedset-based intersections. we demonstrate the usefulness ofour algorithm by analysing our septic shock patient data.
applications of data mining in hydrology. long-term range streamflow forecast plays an invaluable role in water resources planning andmanagement. in this study, the potential applicability and limitations of the time series forecasting approach using neural network with the multiresolution learning paradigm (nnmlp) are investigated. the predictedlongterm range streamflows using the nnmlp are compared with the observations. the results show that the time series forecasting approach of nnmlp has good predicting skill. the nnmlp requires only historicalstreamflow information. the time series forecasting approach of nnmlp has great potential for being used alone in regions with limited available information, and for being combined with other approaches to improve long-term range streamflow forecasts.
lpminer: an algorithm for finding frequent itemsets using length-decreasing support constraint. over the years, a variety of algorithms or finding frequentitemsets in very large transaction databases have been developed. the key feature in most to these algorithms is that they use a constant support constraint to control the inherently exponential complexity of the problem. in general, itemsets that contain only a few items will tend to be interesting if they have a high support, whereas long itemsets can still be interesting even if their support is relatively small. ideally, we desire to have an algorithm that finds all the frequent itemsets whose support decreases as a function of their length. in this paper we present an algorithm called lpminer, that finds all itemsets that satisfy a length-decreasing support constraint. our experimental evaluation shows that lpminer is up to two orders of magnitude faster than the fp-growth algorithm or finding itemsets at a constant support constraint, and that its runtime increasesgradually as the average length of the transactions (and the discovered itemsets) increases.
attribute (feature) completion - the theory of attributes from data mining prospect. a "correct" selection of attributes (features) is vital indata mining. as a first step, this paper constructs all possibleattributes of a given relation. the results are basedon the observations that each relation is isomorphic to aunique abstract relation, called canonical model. the completeset of attributes of the canonical model is, then, constructed.any attribute of a relation can be interpreted (viaisomorphism) from such a complete set.
slpminer: an algorithm for finding frequent sequential patterns using length-decreasing support constraint. over the years, a variety of algorithms for finding frequentsequential patterns in very large sequential databaseshave been developed. the key feature in most of these algorithmsis that they use a constant support constraint tocontrol the inherently exponential complexity of the problem.in general, patterns that contain only a few items willtend to be interesting if they have a high support, whereaslong patterns can still be interesting even if their supportis relatively small. ideally, we desire to have an algorithmthat finds all the frequent patterns whose support decreasesas a function of their length. in this paper we present an algorithmcalled slpminer, that finds all sequential patternsthat satisfy a length-decreasing support constraint. our experimentalevaluation shows that slpminer achieves up totwo orders of magnitude of speedup by effectively exploitingthe length-decreasing support constraint, and that itsruntime increases gradually as the average length of the sequences(and the discovered frequent patterns) increases.
mining associations by linear inequalities. the main theorem is: generalized associations of a relational table can be found by a finite set of linear inequalities within polynomial time. it is derived from the following three results, which were established in icdm0'02 and are re-developed here. they are (1) isomorphic theorem: isomorphic relations have isomorphic patterns. such an isomorphism classifies relational tables into isomorphic classes. (2) a variant of the classical bitmaps indexes uniquely exists in each isomorphic class. we take it as the canonical model of the class. (3) all possible attributes/features can be generated by a generalized procedure of the classical aog (attribute oriented generalization). then, (4) the main theorem for canonical model is established. by isomorphism theorem, we had the final result (5).
process diagnosis via electrical-wafer-sorting maps classification. the commonality analysis is a proven tool for fault detection in semiconductor manufacturing. this methodology extracts subsets of production lots from all the available data. then, data mining techniques are used only on the selected data. this approach loses part of the available information and does not discriminate among the lots. the new methodology performance the automatic classificationof the electrical wafer test maps in order to identify the classes of failure present in the production lots. subsequently, the proposed procedure uses the process history of each wafer to create a list of the root cause candidates. this methodology is the core of the software tool acid which is currently used for process diagnosis at the agrate site of the st microelectronics. a real analysis is presented.
unsupervised link discovery in multi-relational data via rarity analysis. a significant portion of knowledge discovery and datamining research focuses on finding patterns of interest indata. once a pattern is found, it can be used to recognizesatisfying instances. the new area of link discoveryrequires a complementary approach, since patterns ofinterest might not yet be known or might have too fewexamples to be learnable. this paper presents anunsupervised link discovery method aimed at discoveringunusual, interestingly linked entities in multi-relationaldatasets. various notions of rarity are introduced tomeasure the "interestingness" of sets of paths andentities. these measurements have been implemented andapplied to a real-world bibliographic dataset where theygive very promising results.
finding representative set from massive data. in the information age, data is pervasive. in some applications, data explosion is a significant phenomenon. the massive data volume poses challenges to both human users and computers. in this project, we propose a new model for identifying representative set from a large database. a representative set is a special subset of the original dataset, which has three main characteristics: it is significantly smaller in size compared to the original dataset. it captures the most information from the original dataset compared to other subsets of the same size. it has low redundancy among the representatives it contains. we use information-theoretic measures such as mutual information and relative entropy to measure the representativeness of the representative set. we first design a greedy algorithm and then present a heuristic algorithm that delivers much better performance. we run experiments on two real datasets and evaluate the effectiveness of our representative set in terms of coverage and accuracy. the experiments show that our representative set attains expected characteristics and captures information more efficiently.
improving the reliability of decision tree and naive bayes learners. the c4.5 decision tree and naive bayes learners are known to produce unreliable probability forecasts. we have used simple binning and laplace transform techniques to improve the reliability of these learners and compare their effectiveness with that of the newly developed venn probability machine (vpm) meta-learner. we assess improvements in reliability using loss functions, receiver operator characteristic (roc) curves and empirical reliability curves (erc). the vpm outperforms the simple techniques to improve reliability, although at the cost of increased computational intensity and slight increase in error rate. these trade-offs are discussed.
mmss: multi-modal story-oriented video summarization. we propose multi-modal story-oriented video summarization (mmss) which, unlike previous works that use fine-tuned, domain-specific heuristics, provides a domain-independent, graph-based framework. mmss uncovers correlation between information of different modalities which gives meaningful story-oriented news video summaries. mmss can also be applied for video retrieval, giving performance that matches the best traditional retrieval techniques (okapi and lsi), with no fine-tuned heuristics such as tf/idf.
mining optimal actions for profitable crm. data mining has been applied to crm (customer relationshipmanagement) in many industries witha limitedsuccess.most data mining tools can only discover customer modelsor profiles (such as customers who are likely attritors andcustomers who are loyal), but not actions that would improvecustomer relationship (such as changing attritors toloyal customers). we describe a novel algorithm that suggestsactions to change customers from an undesired status(such as attritors) to a desired one (such as loyal). our algorithmtakes into account the cost of actions, and further,it attempts to maximize the expected net profit. to our bestknowledge, no data mining algorithms or tools today can accomplishthis important task in crm. the algorithm is implemented,with many advanced features, in a specializedand highly effective data mining software called proactivesolution.
parameter-free spatial data mining using mdl. consider spatial data consisting of a set of binary features taking values over a collection of spatial extents (grid cells). we propose a method that simultaneously finds spatial correlation and feature co-occurrence patterns, without any parameters. in particular, we employ the minimum description length (mdl) principle coupled with a natural way of compressing regions. this defines what "good" means: a feature co-occurrence pattern is good, if it helps us better compress the set of locations for these features. conversely, a spatial correlation is good, if it helps us better compress the set of features in the corresponding region. our approach is scalable for large datasets (both number of locations and of features). we evaluate our method on both real and synthetic datasets.
predicting software escalations with maximum roi. enterprise software venders often have to release software products before all reported defects are corrected, and a small number of these reported defects will be escalated by customers whose businesses are seriously impacted. escalated defects must be quickly resolved at a high cost by the software vendors. the total costs can be even greater, including loss of reputation, satisfaction, loyalty, and repeat revenue. in this paper, we develop an escalation prediction (ep) system to mine historic defect report data and predict the escalation risk of current defect reports for maximum roi (return on investment). more specifically, we first describe a simple and general framework to convert the maximum roi problem to cost-sensitive learning. we then apply and compare several best-known cost-sensitive learning approaches for ep. the ep system has produced promising results, and has been deployed in the product group of an enterprise software vendor. conclusions drawn from this study also provide guidelines for mining imbalanced datasets and cost-sensitive learning.
discovering frequent arrangements of temporal intervals. in this paper we study a new problem in temporal pattern mining: discovering frequent arrangements of temporal intervals. we assume that the database consists of sequences of events, where an event occurs during a time-interval. the goal is to mine arrangements of event intervals that appear frequently in the database. there are many applications where these type of patterns can be useful, including data network, scientific, and financial applications. efficient methods to find frequent arrangements of temporal intervals using both breadth first and depth first search techniques are described. the performance of the proposed algorithms is evaluated and compared with other approaches on real datasets (american sign language streams and network data) and large synthetic datasets.
improving text classification using local latent semantic indexing. latent semantic indexing (lsi) has been shown to be extremely useful in information retrieval, but it is not an optimal representation for text classification. it always drops the text classification performance when being applied to the whole training set (global lsi) because this completely unsupervised method ignores class discrimination while only concentrating on representation. some local lsi methods have been proposed to improve the classification by utilizing class discrimination information. however, their performance improvements over original term vectors are still very limited. in this paper, we propose a new local lsi method called "local relevancy weighted lsi" to improve text classification by performing a separate single value decomposition (svd) on the transformed local region of each class. experimental results show that our method is much better than global lsi and traditional local lsi methods on classification within a much smaller lsi dimension.
inference of protein-protein interactions by unlikely profile pair. we note that a set of statistically "unusual" protein-profilepairs in experimentally determined database ofprotein-protein interactions can typify protein-proteininteractions, and propose a novel method calledpicupp that sifts such protein-profile pairs using astatistical simulation. it is demonstrated that unusualpfam and interpro profile pairs can be extracted fromthe dip database using a bootstrapping approach. weparticularly illustrate that such protein-profile pairs canbe used for predicting putative pairs of interactingproteins. their prediction accuracies are around 86%and 90% when interpro and pfam profiles are used,respectively at 75% confidence level.
building text classifiers using positive and unlabeled examples. this paper studies the problem of building text classifiersusing positive and unlabeled examples. the key feature ofthis problem is that there is no negative example forlearning. recently, a few techniques for solving thisproblem were proposed in the literature. these techniquesare based on the same idea, which builds a classifier intwo steps. each existing technique uses a different methodfor each step. in this paper, we first introduce some newmethods for the two steps, and perform a comprehensiveevaluation of all possible combinations of methods of thetwo steps. we then propose a more principled approachto solving the problem based on a biased formulation ofsvm, and show experimentally that it is more accuratethan the existing techniques.
efficient nonlinear dimension reduction for clustered data using kernel functions. in this paper, we propose a nonlinear feature extractionmethod which is based on centroids and kernel functions.the dimension reducing nonlinear transformation isobtained by implicitly mapping the input data into a featurespace using a kernel function, and then finding a linearmapping based on an orthonormal basis of centroids in thefeature space that maximally separates the between-classrelationship. the proposed method utilizes an efficient algorithmto compute an orthonormal basis of centroids in thefeature space transformed by a kernel function and achievesdramatic computational savings. the experimental resultsdemonstrate that our method is capable of extracting non-linearfeatures effectively so that competitive performanceof classification can be obtained in the reduced dimensionalspace.
analyzing the interestingness of association rules from the temporal dimension. rule discovery is one of the central tasks of data mining. existing research has produced many algorithms for the purpose. these algorithms, however, often generate too manyrules. in the past few years, rule interestingness techniques were proposed to help the user find interesting rules. these techniques typically employ the dataset as a whole to mine rules, and then filter and/or rank the discovered rules in various ways. in this paper, we argue that this is insufficient. these techniques are unable to answer a question that is of criticalimportance to the application of rules, i.e., can the rules be trusted? in practice, the users are always concerned with the question. they want to know whether the rules indeed represent some true and stable (or reliable)underlying relationships in the domain. if a rule is not stable, does it show any systematic pattern such as a trend? before any rule can be used, these questions must be answered. this paper proposes a technique to use statistical methods to analyze rules from the temporal dimension to answer these questions. experimental results show that the proposed technique is very effective.
a comparative study of linear and nonlinear feature extraction methods. this paper presents theoretical relationships among several generalized lda algorithms and proposes computationally efficient approaches for them utilizing the relationships. generalized lda algorithms are extended nonlinearly by kernel methods resulting in nonlinear discriminant analysis. performances and computational complexities of these linear and nonlinear discriminant analysis algorithms are compared.
mining approximate frequent itemsets from noisy data. frequent itemset mining is a popular and important first step in analyzing data sets across a broad range of applications. the traditional, "exact" approach for finding frequent itemsets requires that every item in the itemset occurs in each supporting transaction. however, real data is typically subject to noise, and in the presence of such noise, traditional itemset mining may fail to detect relevant itemsets, particularly those large itemsets that are more vulnerable to noise. in this paper we propose approximate frequent itemsets (afi), as a noise-tolerant itemset model. in addition to the usual requirement for sufficiently many supporting transactions, the afi model places constraints on the fraction of errors permitted in each item column and the fraction of errors permitted in a supporting transaction. taken together, these constraints winnow out the approximate itemsets that exhibit systematic errors. in the context of a simple noise model, we demonstrate that afi is better at recovering underlying data patterns, while identifying fewer spurious patterns than either the exact frequent itemset approach or the existing error tolerant itemset approach of yang et al. [11].
a new implementation technique for fast spectral based document retrieval systems. the traditional methods of spectral text retrieval(fds,cds) create an index of spatial data and convert thedata to its spectral form at query time. we present a newmethod of implementing and querying an index containingspectral data which will conserve the high precision performanceof the spectral methods, reduce the time needed toresolve the query, and maintain an acceptable size for theindex. this is done by taking advantage of the propertiesof the discrete cosine transform and by applying ideas fromvector space document ranking methods.
revealing true subspace clusters in high dimensions. subspace clustering is one of the best approaches for discovering meaningful clusters in high dimensional space. one cluster in high dimensional space may be transcribed into multiple distinct maximal clusters by projecting onto different subspaces. a direct consequence of clustering independently in each subspace is an overwhelmingly large set of overlapping clusters which may be significantly similar. to reveal the true underlying clusters, we propose a similarity measurement of the overlapping clusters. we adopt the model of gaussian tailed hyper-rectangles to capture the distribution of any subspace cluster. a set of experiments on a synthetic dataset demonstrates the effectiveness of our approach. application to real gene expression data also reveals impressive meta-clusters expected by biologists.
hybrid pre-query term expansion using latent semantic analysis. latent semantic retrieval methods (unlike vector space methods) take the document and query vectors and map them into a topic space to cluster related terms and documents. this produces a more precise retrieval but also a long query time. we present a new method of document retrieval which allows us to process the latent semantic information into a hybrid latent semantic-vector space query mapping. this mapping automatically expands the users query based on the latent semantic information in the document set. this expanded query is processed using a fast vector space method. since we have the latent semantic data in a mapping, we are able to store and retrieve vector information in the same fast manner that the vector space method offers. multiple mappings are combined to produce hybrid latent semantic retrieval which provide precision results 5% greater than the vector space method and fast query times.