waashk/dblp huggingface dataset:
progressive and interactive analysis of event data using event miner. exploring large data sets typically involves activities that iteratebetween data selection and data analysis, in which insights obtainedfrom analysis result in new data selection. further, data analysis needs touse a combination of analysis techniques: data summarization, mining algorithmsand visualization. this interweaving of functions arises both fromthe semantics of what the analyst hopes to achieve and from scalability requirementsfor dealing with large data volumes. we refer to such a processas a progressive analysis. herein is described a tool, event miner, that integratesdata selection, mining and visualization for progressive analysis oftemporal, categorical data. we discuss a data model and architecture. weillustrate how our tool can be used for complex mining tasks such as findingpatterns not occurring on monday. further, we discuss the novel visualizationemployed, such as visualizing categorical data and the results of datamining. also, we discuss the extension of the existing mining frameworkneeded to mine temporal events with multiple attributes. throughout, weillustrate the capabilities of event miner by applying it to event data fromlarge computer networks.
structure search and stability enhancement of bayesian networks. learning bayesian network structure from large-scale datasets, without any expert-specified ordering of variables, remainsa difficult problem. we propose systematic improvements toautomatically learn bayesian network structure from data. (1)we propose a linear parent search method to generate candidategraph. (2) we propose a comprehensive approach to eliminatecycles using minimal likelihood loss, a short cycle first heuristic,and a cut-edge repairing. (3) we propose structure perturbationto assess the stability of the network and a stability-improvementmethod to refine the network structure. the algorithms are easyto implement and efficient for large networks. experimental resultson two data sets show that our new approach outperformsexisting methods.
finding constrained frequent episodes using minimal occurrences. recurrent combinations of events within an event sequence, known as episodes, oftenreveal useful information. most of the proposed episode mining algorithms adopt an apriori-like approach that generates candidates and then calculates their support levels. obviously, such an approach is computationally expensive. moreover, those algorithms are capable ofhandling only a limited range of constraints. in this paper, we introduce two miningalgorithms - episode prefix tree (ept) and position pairs set (pps) - based on a prefix-growth approach to overcome the above limitations. both algorithms push constraints systematically into the mining process. performance study shows that the proposed algorithms run considerably faster than minepi.
a transaction-based neighbourhood-driven approach to quantifying interestingness of association rules. in this paper, we present a data-driven approach for ranking association rules (ars) based on interestingness. the occurrence of unrelated or weakly related item-pairs in an ar is interesting. in the retail market-basket context, items may be related through various relationships arising due to mutual interaction, 'substitutability' and 'complementarity.' item-relatedness is a composite of these relationships. we introduce three relatedness measures for capturing relatedness between item-pairs. these measures use the concept of function embedding to appropriately weigh the relatedness contributions due tocomplementarity and substitutability between items. we propose an interestingness coefficient by combining the three relatedness measures. we compare this with two objective measures of interestingness and show the intuitiveness of the proposed interestingness coefficient.
an adaptive density-based clustering algorithm for spatial database with noise. clustering spatial data has various applications. several clustering algorithms have been proposed to cluster objects in spatial databases. spatial object distribution has significant effect on the results of clustering. few of current algorithms consider the distribution of objects while processing clusters. in this paper, we propose an adaptive density-based clustering algorithm, adbc, which uses a novel adaptive strategy for neighbor selection based on spatial object distribution to improve clustering accuracy. we perform a series of experiments on simulated data sets and real data sets. a comparison with dbscan and optics shows the superiority of our new approach.
exploiting unlabeled data for improving accuracy of predictive data mining. predictive data mining typically relies on labeled datawithout exploiting a much larger amount of availableunlabeled data. the goal of this paper is to show thatusing unlabeled data can be beneficial in a range ofimportant prediction problems and therefore should be anintegral part of the learning process. given an unlabeleddataset representative of the underlying distribution and ak-class labeled sample that might be biased, ourapproach is to learn k contrast classifiers each trained todiscriminate a certain class of labeled data from theunlabeled population. we illustrate that contrastclassifiers can be useful in one-class classification, outlierdetection, density estimation, and learning from biaseddata. the advantages of the proposed approach aredemonstrated by an extensive evaluation on synthetic datafollowed by real-life bioinformatics applications for (1)ranking pubmed articles by their relevance to proteindisorder and (2) cost-effective enlargement of adisordered protein database.
visually mining web user clickpaths. as powerful as clickpath mining methods can be, theyoften lead to huge incomprehensible and non-interestingresult sets. our clickpath mining practice at msn wasfaced with challenges of keeping analysts closer to thedata exploration process, revealing powerful insight fromclickpath mining that business owners can directly actupon. these challenges stressed the importance of aninteractive and visual representation of clickpath miningresults. most products today that can perform clickpathvisualization do so by presenting massive cross-weavingweb graphs. we present a new type of clickpathvisualization which focuses only on clickpaths of interest,simplifying the visualization space while still retaining thesame degree of mineable knowledge in the data. we alsodescribe visualization techniques we have used toenhance the detection of interesting clickpath patternsfrom data, and provide a real-life case study that hasbenefited from the use of our implemented clickpathvisualizer pave.
farm: a framework for exploring mining spaces with multiple attributes. mining for frequent itemsets typically involves a preprocessing step in which data with multiple attributes are grouped into transactions, and item are defined based on attribute values. we have observed that such fixed attribute mining can severely constrain the pattern that are discovered. herein, we introduce mining paces, a new framework for mining multi-attribute data that include the discovery of transaction and item definition (with the exploitation of taxonomies and functional dependenciesif they are available).we prove that special downward closure properties (or anti-monotonic property) hold for mining paces, aresult that allows us to construct efficient algorithms for mining pattern without the constraint of fixed attribute mining. we apply our algorithm to real world data collected from a production computer network. the result how that by exploiting the special kind of downward closure in mining paces, execution times for mining can be reduced by a factor of three to four.
learning rules for anomaly detection of hostile network traffic. we introduce an algorithm called lerad that learnsrules for finding rare events in nominal time-series datawith long range dependencies. we use lerad to findanomalies in network packets and tcp sessions to detectnovel intrusions. we evaluated lerad on the 1999darpa/lincoln laboratory intrusion detection evaluationdata set and on traffic collected in a universitydepartmental server environment.
user-directed exploration of mining space with multiple attributes. there has been a growing interest in mining frequentitemsets in relational data with multiple attributes. a keystep in this approach is to select a set of attributes thatgroup data into transactions and a separate set of attributesthat labels data into items. unsupervised and unrestrictedmining, however, is stymied by the combinatorial complexityand the quantity of patterns as the number of attributesgrows. in this paper, we focus on leveraging the semanticsof the underlying data for mining frequent itemsets. forinstance, there are usually taxonomies in the data schemaand functional dependencies among the attributes. domainknowledge and user preferences often have the potentialto significantly reduce the exponentially growing miningspace. these observations motivate the design of a user-directeddata mining framework that allows such domainknowledge to guide the mining process and control the miningstrategy. we show examples of tremendous reductionin computation by using domain knowledge in mining relationaldata with multiple attributes.
an algorithm for the exact computation of the centroid of higher dimensional polyhedra and its application to kernel machines. the support vector machine (svm) solution correspondsto the centre of the largest sphere inscribed in versionspace. alternative approaches like bayesian pointmachines (bpm) and analytic centre machines have suggestedthat the generalization performance can be furtherenhanced by considering other possible centres of versionspace like the centroid (centre of mass) or the analytic centre.we present an algorithm to compute exactly the centroidof higher dimensional polyhedra, then derive approximationalgorithms to build a new learning machine whoseperformance is comparable to bpm. we also show that forregular kernel matrices (gaussian kernels for example), thesvm solution can be obtained by solving a linear system ofequalities.
the representative basis for association rules. we define the concept of the representative basic for interesting association rules, and an inference system which is purely qualitative. the representative basis is unique, and minimal with respect to (wrt) the inference system. on the representative basis, the inference system is correct and complete. experimental results show that the number of rule in the representative basis is significantly reduced wrt the number of rules generated by other existing approaches.
probabilistic user behavior models. we present a mixture model based approach for learningindividualized behavior models for the web users. weinvestigate the use of maximum entropy and markov mixturemodels for generating probabilistic behavior models.we first build a global behavior model for the entire populationand then personalize this global model for the existingusers by assigning each user individual componentweights for the mixture model. we then use these individualweights to group the users into behavior model clusters.we show that the clusters generated in this manner areinterpretable and able to represent dominant behavior patterns.we conduct offline experiments on around two monthsworth of data from citeseer, an online digital library forcomputer science research papers currently storing morethan 470,000 documents. we show that both maximum entropyand markov based personal user behavior modelsare strong predictive models. we also show that maximumentropy based mixture model outperforms markov mixturemodels in recognizing complex user behavior patterns.
regulatory element discovery using tree-structured models. computational discovery of transcriptional regulatoryregions in dna sequences provides an efficient way tobroaden our understanding of how cellular processes arecontrolled. in this paper, we formulate the regulatoryelement discovery problem in the regression frameworkwith regulatory regions treated as predictor variables andgene expression levels as responses. we use regressiontree models to identify structural relationships betweenpredictors and responses. the regression treemethodology is extended to handle multiple responsesfrom different experiments by modifying the split function.we apply this method to two data sets of the yeastsaccharomyces cerevisiae. the method successfullyidentifies most of regulatory motifs that are known tocontrol gene transcription under the given experimentalconditions. our method also suggests several putativemotifs that can present novel regulatory motifs.
spatial clustering of chimpanzee locations for neighborhood identification. since 1960, the chimpanzees (pan troglodytes) of gombe national park, tanzania, have been studied by behavioral ecologists, including jane goodall. data have been collected for more than 40 years and are being analyzed by researchers in order to increase our understanding of the social structure of chimpanzees. in this paper, we consider the following question of interest to behavioral ecologists — "does clustering exist among female chimpanzees in terms of their spatial locations?" the analysis of this question will help behavioral ecologists to learn about the space use and the social interactions between female chimpanzees. the data collected for this analysis are marked spatial point patterns over the park. current spatial clustering methods lack the ability to handle such marked point patterns directly. this paper presents a novel application of spatial point pattern analysis and data mining techniques to the ecological problem of clustering female chimpanzees. we found that ripley's k-function provides a powerful statistical tool for evaluating clustering behavior among spatial point patterns. we then proposed two clustering approaches for marked point patterns using the k-function. experimental results using the proposed clustering methods provide significant insight into the dynamics of female chimpanzee space use and into the overall social stucture of the species. in addition, the proposed methods can be extended to also include temporal information.
privacy-preserving collaborative filtering using randomized perturbation techniques. collaborative filtering (cf) techniques are becomingincreasingly popular with the evolution of the internet. toconduct collaborative filtering, data from customers areneeded. however, collecting high quality data from customersis not an easy task because many customers areso concerned about their privacy that they might decide togive false information. we propose a randomized perturbation(rp) technique to protect users' privacy while stillproducing accurate recommendations.
estimation of false negatives in classification. in many classification problems such as spam detection and network intrusion, a large number of unlabeled test instances are predicted negative by the classifier. however, the high costsas well as time constraints on an expert's time prevent further analysis of the "predicted false" class instances in order to segregate the false negatives from the true negatives. a systematic method is thus required to obtain an estimate of the number of false negatives. a capture-recapture based method can be used to obtain an ml-estimate of false negatives when two or more independent classifiers are available. in the case for which independence does not hold, we can apply log-linear models to obtain an estimate of false negatives. however, as shown in this paper, lesser the dependencies among the classifiers, better is the estimate obtained for false negatives. thus, ideally independent classifiers should be used to estimate the false negatives in an unlabeled dataset. experimental results on the spam dataset from the uci machine learning repository are presented.
statistical relational learning for document mining. a major obstacle to fully integrated deployment of manydata mining algorithms is the assumption that data sitsin a single table, even though most real-world databaseshave complex relational structures. we propose an integratedapproach to statistical modeling from relationaldatabases. we structure the search space based on "refinementgraphs", which are widely used in inductive logic programmingfor learning logic descriptions. the use of statisticsallows us to extend the search space to include richerset of features, including many which are not boolean.search and model selection are integrated into a single process,allowing information criteria native to the statisticalmodel, for example logistic regression, to make feature selectiondecisions in a step-wise manner. we present experimentalresults for the task of predicting where scientific paperswill be published based on relational data taken fromciteseer. our approach results in classification accuraciessuperior to those achieved when using classical "flat" features.the resulting classifier can be used to recommendwhere to publish articles.
a new algorithm for finding minimal sample uniques for use in statistical disclosure assessment. we present suda2, a recursive algorithm for finding minimal sample uniques (msus). suda2 uses a novel method for representing the search space formsus and new observations about the properties ofmsus to prune and traverse this space. experimental comparisons with previous work demonstrate that suda2 is not only several orders of magnitude faster but is also capable of identifying the boundaries of the search space, enabling datasets of larger numbers of columns than before to be addressed.
a fast algorithm to cluster high dimensional basket data. clustering is a data mining problem that has received significant attention by the database community. data set size, dimensionality and sparsity have been identified as aspectsthat make clustering more difficult. this work introduces a fast algorithm to cluster large binary data sets where data points have high dimensionality and most o their coordinates are zero. this is the case with basket data transactions containing items, that can be represented as sparse binary vectors with very high dimensionality. an experimental section shows performance, advantages and limitations of the proposed approach.
aligning boundary in kernel space for learning imbalanced dataset. an imbalanced training dataset poses serious problem for many real-world supervised learning tasks. in this paper, we propose a kernel-boundary-alignment algorithm, which considers training-data imbalance as prior information to augment svms to improve class-prediction accuracy. using a simple example, we first show that svms can suffer from high incidences of false negatives when the training instances of the target class are heavily outnumbered by the training instances of a non-target class. the remedy we propose is to adjust the class boundary by modifying the kernel matrix, according to the imbalanced data distribution. through theoretical analysis backed by empirical study, we show that our kernel-boundary-alignment algorithm works effectively on several datasets.
learning automatic acquisition of subcategorization frames using bayesian inference and support vector machines. learning bayesian belief network (bbn) from corpora and support vector machines (svm) have been applied to the automatic acquisition of verb subcategorization frames for modern greek.we are incorporating minimal linguistic resources, i.e. basic morphological tagging and phrase chunking, to demonstrate that verb subcategorization, which is of great significance for developing robust natural language human computer interaction systems, could be achieved using large corpora, without having any general-purpose syntactic parser at all.
svm and graphical algorithms: a cooperative approach. we present a cooperative approach using both support vector machine (svm) algorithms and visualization methods. svm are widely used today and often give high quality results, but they are used as "black-box" (it is very difficult to explain the obtained results) and cannot treat easily very large datasets. we have developed graphical methods to help the user to evaluate and explain the svm results. the first method is a graphical representation of the separating frontier quality, it is then linked with other visualization tools to help the user explaining svm results. the information provided by these graphical methods is also used for svm parameter tuning, they are then used together with automatic algorithms to deal with very large datasets on standard computers. we present an evaluation of our approach with the uci and the kent ridge bio-medical data sets.
zigzag: a new algorithm for mining large inclusion dependencies in database. in the relational model, inclusion dependencies (inds)convey many information on data semantics. they generalizeforeign keys, which are very popular constraints inpractice. however, one seldom knows the set of satisfiedinds in a database. the ind discovery problem in existingdatabases can be formulated as a data-mining problem.we underline in this article that the exploration of ind expressionsfrom most general (smallest) inds to most specific(largest) inds does not succeed whenever large indshave to be discovered. to cope with this problem, we introducea new algorithm, called zigzag , which combinesthe strength of levelwise algorithms (to find out some smallestinds) with an optimistic criteria to jump more or lessto largest inds. preliminary tests, on synthetic databases,are presented and commented on. it is worth noting that themain result of this paper is general enough to be appliedto other data-mining problems, such as maximal frequentitemsets mining.
semantic role parsing: adding semantic structure to unstructured text. there is a ever-growing need to add structure in the formof semantic markup to the huge amounts of unstructured textdata now available. we present the technique of shallow semanticparsing, the process of assigning a simple who didwhat to whom, etc., structure to sentences in text, as auseful tool in achieving this goal. we formulate the semanticparsing problem as a classification problem using supportvector machines. using a hand-labeled training setand a set of features drawn from earlier work together withsome feature enhancements, we demonstrate a system thatperforms better than all other published results on shallowsemantic parsing.
alternate representation of distance matrices for characterization of protein structure. the most suitable method for the automated classification of protein structures remains an open problem in computational biology. in order to classify a protein structure with any accuracy, an effective representation must be chosen. here we present two methods of representing protein structure. one involves representing the distances between the cá atoms of a protein as a two-dimensional matrix and creating a model of the resulting surface with zernike polynomials. the second uses a wavelet-based approach. we convert the distances between a protein's cα atoms into a one-dimensional signal which is then decomposed using a discrete wavelet transformation. using the zernike co-efficients and the approximation coefficients of the wavelet decomposition as feature vectors, we test the effectiveness of our representation with two different classifiers on a dataset of more than 600 proteins taken from the 27 most-populated scop folds. we find that the wavelet decomposition greatly outperforms the zernike model.with the wavelet representation, we achieve an accuracy of approximately 56%, roughly 12% higher than results reported on a similar, but less-challenging dataset. in addition, we can couple our structure-based feature vectors with several sequence-based properties to increase accuracy another 5-7%. finally, we use a multi-stage classification strategy on the combined features to increase performance to 78%, an improvement in accuracy of more than 15-20% and 34% over the highest reported sequence-based and structure-based classification results, respectively.