waashk/dblp huggingface dataset:
mining the network value of customers. one of the major applications of data mining is in helping companies determine which potential customers to market to. if the expected profit from a customer is greater than the cost of marketing to her, the marketing action for that customer is executed. so far, work in this area has considered only the intrinsic value of the customer (i.e, the expected profit from sales to her). we propose to model also the customer's network value: the expected profit from sales to other customers she may influence to buy, the customers those may influence, and so on recursively. instead of viewing a market as a set of independent entities, we view it as a social network and model it as a markov random field. we show the advantages of this approach using a social network mined from a collaborative filtering database. marketing that exploits the network value of customers---also known as viral marketing---can be extremely effective, but is still a black art. our work can be viewed as a step towards providing a more solid foundation for it, taking advantage of the availability of large relevant databases.
an approach to spacecraft anomaly detection problem using kernel feature space. development of advanced anomaly detection and failure diagnosis technologies for spacecraft is a quite significant issue in the space industry, because the space environment is harsh, distant and uncertain. while several modern approaches based on qualitative reasoning, expert systems, and probabilistic reasoning have been developed recently for this purpose, any of them has a common difficulty in obtaining accurate and complete a priori knowledge on the space systems from human experts. a reasonable alternative to this conventional anomaly detection method is to reuse a vast amount of telemetry data which is multi-dimensional time-series continuously produced from a number of system components in the spacecraft.this paper proposes a novel "knowledge-free" anomaly detection method for spacecraft based on kernel feature space and directional distribution, which constructs a system behavior model from the past normal telemetry data from a set of telemetry data in normal operation and monitors the current system status by checking incoming data with the model.in this method, we regard anomaly phenomena as unexpected changes of causal associations in the spacecraft system, and hypothesize that the significant causal associations inside the system will appear in the form of principal component directions in a high-dimensional non-linear feature space which is constructed by a kernel function and a set of data.we have confirmed the effectiveness of the proposed anomaly detection method by applying it to the telemetry data obtained from a simulator of an orbital transfer vehicle designed to make a rendezvous maneuver with the international space station.
mining the internet: the eighth wonder of the world. the internet takes behavioral consumer research to a new level by providing the ability to passively and continuously monitor the complete online behavior of millions of consumers in an opt-in, privacy protected manner. imagine the analytical possibilities if every site visited, every page viewed, content seen, transaction conducted ..... all of this granularity in behavior --- was continuously captured with explicit consumer permission for millions of consumers and privacy was protected. what unique insights could one gain into consumers' behavior, their interests, passions and lifestyles? what behavior could be predicted? what commercial applications would be possible. the acm portal is published by the association for computing machinery. copyright © 2010 acm, inc. terms of usage privacy policy code of ethics contact us useful downloads: adobe acrobat quicktime windows media player real player
early detection of insider trading in option markets. "inside information" comes in many forms: knowledge of a corporate takeover, a terrorist attack, unexpectedly poor earnings, the fda's acceptance of a new drug, etc. anyone who knows some piece of soon-to-break news possesses inside information. historically, insider trading has been detected after the news is public, but this is often too late: fraud has been perpetrated, innocent investors have been disadvantaged, or terrorist acts have been carried out. this paper explores early detection of insider trading - detection before the news breaks. data mining holds great promise for this emerging application, but the problem also poses significant challenges. we present the specific problem of insider trading in option markets, compare decision tree, logistic regression, and neural net results to results from an expert model, and discuss insights that knowledge discovery techniques shed upon this problem.
proximal support vector machine classifiers. instead of a standard support vector machine (svm) that classifies points by assigning them to one of two disjoint half-spaces, points are classified by assigning them to the closest of two parallel planes (in input or feature space) that are pushed apart as far as possible. this formulation, which can also be interpreted as regularized least squares and considered in the much more general context of regularized networks [8, 9], leads to an extremely fast and simple algorithm for generating a linear or nonlinear classifier that merely requires the solution of a single system of linear equations. in contrast, standard svms solve a quadratic or a linear program that require considerably longer computational time. computational results on publicly available datasets indicate that the proposed proximal svm classifier has comparable test set correctness to that of standard svm classifiers, but with considerably faster computational time that can be an order of magnitude faster. the linear proximal svm can easily handle large datasets as indicated by the classification of a 2 million point 10-attribute set in 20.8 seconds. all computational results are based on 6 lines of matlab code.
rule extraction from linear support vector machines. we describe an algorithm for converting linear support vector machines and any other arbitrary hyperplane-based linear classifiers into a set of non-overlapping rules that, unlike the original classifier, can be easily interpreted by humans. each iteration of the rule extraction algorithm is formulated as a constrained optimization problem that is computationally inexpensive to solve. we discuss various properties of the algorithm and provide proof of convergence for two different optimization criteria we demonstrate the performance and the speed of the algorithm on linear classifiers learned from real-world datasets, including a medical dataset on detection of lung cancer from medical images. the ability to convert svm's and other "black-box" classifiers into a set of human-understandable rules, is critical not only for physician acceptance, but also to reducing the regulatory barrier for medical-decision support systems based on such classifiers.
applications of sampling and fractional factorial designs to model-free data squashing. the concept of "data squashing" was introduced by dumouchel et al [4] as a method of summarizing massive data sets that preserves statistical relationships among variables. the idea is to create a smaller data set that allows statistical modeling to take place using in-memory algorithms, and to preserve the modeling results more accurately than would a same-size random sample from the massive data set. this research attempts to avoid several limitations of previous approaches to data squashing. our method avoids the curse of dimensionality by a double use of principal components transformations that makes computing time linear in the number of cases and quadratic in the number of variables. categorical and continuous variables are smoothly integrated. because the binning is based on principal components, which are uncorrelated, we can use fractional factorial designs that sample less than one point per bin. we also investigate various weighting schemes for the squashed sample to see whether matching moments or matching subregion data counts is more effective. finally, previous work required the specification of a statistical model, either to perform the squashing algorithm or to compare the worth of different squashing methods. our approach to evaluation is model free and does not even require the specification of variables as responses or predictors. instead, we develop a chi-squared like measure of accuracy to compare the closeness of various discrete densities (the squashed data sets) to the discrete massive data set.
empirical bayes screening for multi-item associations. this paper considers the framework of the so-called "market basket problem", in which a database of transactions is mined for the occurrence of unusually frequent item sets. in our case, "unusually frequent" involves estimates of the frequency of each item set divided by a baseline frequency computed as if items occurred independently. the focus is on obtaining reliable estimates of this measure of interestingness for all item sets, even item sets with relatively low frequencies. for example, in a medical database of patient histories, unusual item sets including the item "patient death" (or other serious adverse event) might hopefully be flagged with as few as 5 or 10 occurrences of the item set, it being unacceptable to require that item sets occur in as many as 0.1% of millions of patient reports before the data mining algorithm detects a signal. similar considerations apply in fraud detection applications. thus we abandon the requirement that interesting item sets must contain a relatively large fixed minimal support, and adopt a criterion based on the results of fitting an empirical bayes model to the item set counts. the model allows us to define a 95% bayesian lower confidence limit for the "interestingness" measure of every item set, whereupon the item sets can be ranked according to their empirical bayes confidence limits. for item sets of size j > 2, we also distinguish between multi-item associations that can be explained by the observed j(j-1)/2 pairwise associations, and item sets that are significantly more frequent than their pairwise associations would suggest. such item sets can uncover complex or synergistic mechanisms generating multi-item associations. this methodology has been applied within the u.s. food and drug administration (fda) to databases of adverse drug reaction reports and within at&t to customer international calling histories. we also present graphical techniques for exploring and understanding the modeling results.
efficient closed pattern mining in the presence of tough block constraints. various constrained frequent pattern mining problem formulations and associated algorithms have been developed that enable the user to specify various itemset-based constraints that better capture the underlying application requirements and characteristics. in this paper we introduce a new class of block constraints that determine the significance of an itemset pattern by considering the dense block that is formed by the pattern's items and its associated set of transactions. block constraints provide a natural framework by which a number of important problems can be specified and make it possible to solve numerous problems on binary and real-valued datasets. however, developing computationally efficient algorithms to find these block constraints poses a number of challenges as unlike the different itemset-based constraints studied earlier, these block constraints are tough as they are neither anti-monotone, monotone, nor convertible. to overcome this problem, we introduce a new class of pruning methods that significantly reduce the overall search space and present a computationally efficient and scalable algorithm called cbminer to find the closed itemsets that satisfy the block constraints.
symp: an efficient clustering approach to identify clusters of arbitrary shapes in large data sets. we propose a new clustering algorithm, called symp, which is based on synchronization of pulse-coupled oscillators. symp represents each data point by an integrate-and-fire oscillator and uses the relative similarity between the points to model the interaction between the oscillators. symp is robust to noise and outliers, determines the number of clusters in an unsupervised manner, identifies clusters of arbitrary shapes, and can handle very large data sets. the robustness of symp is an intrinsic property of the synchronization mechanism. to determine the optimum number of clusters, symp uses a dynamic resolution parameter. to identify clusters of various shapes, symp models each cluster by multiple gaussian components. the number of components is automatically determined using a dynamic intra-cluster resolution parameter. clusters with simple shapes would be modeled by few components while clusters with more complex shapes would require a larger number of components. the scalable version of symp uses an efficient incremental approach that requires a simple pass through the data set. the proposed clustering approach is empirically evaluated with several synthetic and real data sets, and its performance is compared with cure.
data mining: are we there yet? data mining started its move out of the statistics and machine learning ghettos and into the mainstream almost 10 years ago. with great fanfare and a large influx of venture capital, data mining was going to change the very nature of business. yet data mining products have had relatively modest success in the marketplace. the reasons include limitations and misplaced emphasis in the products' features and functions, unrealistic expectations set by messages from the data mining community, and a lack of readiness by many prospective users. this session will look at where vendors have succeeded and failed with their products, what expectations users should have, and suggestions for achieving the potential of this exciting and valuable technology.
sewep: using site semantics and a taxonomy to enhance the web personalization process. web personalization is the process of customizing a web site to the needs of each specific user or set of users, taking advantage of the knowledge acquired through the analysis of the user's navigational behavior. integrating usage data with content, structure or user profile data enhances the results of the personalization process. in this paper, we present sewep, a system that makes use of both the usage logs and the semantics of a web site's content in order to personalize it. web content is semantically annotated using a conceptual hierarchy (taxonomy). we introduce c-logs, an extended form of web usage logs that encapsulates knowledge derived from the link semantics. c-logs are used as input to the web usage mining process, resulting in a broader yet semantically focused set of recommendations.
inverted matrix: efficient discovery of frequent items in large datasets in the context of interactive mining. existing association rule mining algorithms suffer from many problems when mining massive transactional datasets. one major problem is the high memory dependency: either the gigantic data structure built is assumed to fit in main memory, or the recursive mining process is too voracious in memory resources. another major impediment is the repetitive and interactive nature of any knowledge discovery process. to tune parameters, many runs of the same algorithms are necessary leading to the building of these huge data structures time and again. this paper proposes a new disk-based association rule mining algorithm called inverted matrix, which achieves its efficiency by applying three new ideas. first, transactional data is converted into a new database layout called inverted matrix that prevents multiple scanning of the database during the mining phase, in which finding frequent patterns could be achieved in less than a full scan with random access. second, for each frequent item, a relatively small independent tree is built summarizing co-occurrences. finally, a simple and non-recursive mining process reduces the memory requirements as minimum candidacy generation and counting is needed. experimental studies reveal that our inverted matrix approach outperform fp-tree especially in mining very large transactional databases with a very large number of unique items. our random access disk-based approach is particularly advantageous in a repetitive and interactive setting.
from run-time behavior to usage scenarios: an interaction-pattern mining approach. a key challenge facing it organizations today is their evolution towards adopting e-business practices that gives rise to the need for reengineering their underlying software systems. any reengineering effort has to be aware of the functional requirements of the subject system, in order not to violate the integrity of its intended uses. however, as software systems get regularly maintained throughout their lifecycle, the documentation of their requirements often become obsolete or get lost. to address this problem of "software requirements loss", we have developed an interaction-pattern mining method for the recovery of functional requirements as usage scenarios. our method analyzes traces of the run-time system-user interaction to discover frequently recurring patterns; these patterns correspond to the functionality currently exercised by the system users, represented as usage scenarios. the discovered scenarios provide the basis for reengineering the software system into web-accessible components, each one supporting one of the discovered scenarios. in this paper, we describe ipm2, our interaction-pattern discovery algorithm, we illustrate it with a case study from a real application and we give an overview of the reengineering process in the context of which it is employed.
accurate decision trees for mining high-speed data streams. in this paper we study the problem of constructing accurate decision tree models from data streams. data streams are incremental tasks that require incremental, online, and any-time learning algorithms. one of the most successful algorithms for mining data streams is vfdt. in this paper we extend the vfdt system in two directions: the ability to deal with continuous data and the use of more powerful classification techniques at tree leaves. the proposed system, vfdtc, can incorporate and classify new information online, with a single scan of the data, in time constant per example. the most relevant property of our system is the ability to obtain a performance similar to a standard decision tree algorithm even for medium size datasets. this is relevant due to the any-time property. we study the behaviour of vfdtc in different problems and demonstrate its utility in large and medium data sets. under a bias-variance analysis we observe that vfdtc in comparison to c4.5 is able to reduce the variance component.
magical thinking in data mining: lessons from coil challenge 2000. coil challenge 2000 was a supervised learning contest that attracted 43 entries. the authors of 29 entries later wrote explanations of their work. this paper discusses these reports and reaches three main conclusions. first, naive bayesian classifiers remain competitive in practice: they were used by both the winning entry and the next best entry. second, identifying feature interactions correctly is important for maximizing predictive accuracy: this was the difference between the winning classifier and all others. third and most important, too many researchers and practitioners in data mining do not appreciate properly the issue of statistical significance and the danger of overfitting. given a dataset such as the one for the coil contest, it is pointless to apply a very complicated learning algorithm, or to perform a very time-consuming model search. in either ease, one is likely to overfit the training data and to fool oneself in estimating predictive accuracy and in discovering useful correlations.
discovering significant opsm subspace clusters in massive gene expression data. order-preserving submatrixes (opsms) have been accepted as a biologically meaningful subspace cluster model, capturing the general tendency of gene expressions across a subset of conditions. in an opsm, the expression levels of all genes induce the same linear ordering of the conditions. opsm mining is reducible to a special case of the sequential pattern mining problem, in which a pattern and its supporting sequences uniquely specify an opsm cluster. those small twig clusters, specified by long patterns with naturally low support, incur explosive computational costs and would be completely pruned off by most existing methods for massive datasets containing thousands of conditions and hundreds of thousands of genes, which are common in today's gene expression analysis. however, it is in particular interest of biologists to reveal such small groups of genes that are tightly coregulated under many conditions, and some pathways or processes might require only two genes to act in concert. in this paper, we introduce the kiwi mining framework for massive datasets, that exploits two parameters k and w to provide a biased testing on a bounded number of candidates, substantially reducing the search space and problem scale, targeting on highly promising seeds that lead to significant clusters and twig clusters. extensive biological and computational evaluations on real datasets demonstrate that kiwi can effectively mine biologically meaningful opsm subspace clusters with good efficiency and scalability.
consistent bipartite graph co-partitioning for star-structured high-order heterogeneous data co-clustering. heterogeneous data co-clustering has attracted more and more attention in recent years due to its high impact on various applications. while the co-clustering algorithms for two types of heterogeneous data (denoted by pair-wise co-clustering), such as documents and terms, have been well studied in the literature, the work on more types of heterogeneous data (denoted by high-order co-clustering) is still very limited. as an attempt in this direction, in this paper, we worked on a specific case of high-order co-clustering in which there is a central type of objects that connects the other types so as to form a star structure of the inter-relationships. actually, this case could be a very good abstract for many real-world applications, such as the co-clustering of categories, documents and terms in text mining. in our philosophy, we treated such kind of problems as the fusion of multiple pair-wise co-clustering sub-problems with the constraint of the star structure. accordingly, we proposed the concept of consistent bipartite graph co-partitioning, and developed an algorithm based on semi-definite programming (sdp) for efficient computation of the clustering results. experiments on toy problems and real data both verified the effectiveness of our proposed method.