waashk/dblp huggingface dataset:
interactive exploration of coherent patterns in time-series gene expression data. discovering coherent gene expression patterns in time-series gene expression data is an important task in bioinformatics research and biomedical applications. in this paper, we propose an interactive exploration framework for mining coherent expression patterns in time-series gene expression data. we develop a novel tool, coherent pattern index graph, to give users highly confident indications of the existences of coherent patterns. to derive a coherent pattern index graph, we devise an attraction tree structure to record the genes in the data set and summarize the information needed for the interactive exploration. we present fast and scalable algorithms to construct attraction trees and coherent pattern index graphs from gene expression data sets. we conduct an extensive performance study on some real data sets to verify our design. the experimental results strongly show that our approach is more effective than the state-of-the-art methods in mining real gene expression data, and is scalable in mining large data sets.
efficient decision tree construction on streaming data. decision tree construction is a well studied problem in data mining. recently, there has been much interest in mining streaming data. domingos and hulten have presented a one-pass algorithm for decision tree construction. their work uses hoeffding inequality to achieve a probabilistic bound on the accuracy of the tree constructed.in this paper, we revisit this problem. we make the following two contributions: 1) we present a numerical interval pruning (nip) approach for efficiently processing numerical attributes. our results show an average of 39% reduction in execution times. 2) we exploit the properties of the gain function entropy (and gini) to reduce the sample size required for obtaining a given bound on the accuracy. our experimental results show a 37% reduction in the number of data instances required.
similarity measure based on partial information of time series. similarity measure of time series is an important subroutine in many kdd applications. previous similarity models mainly focus on the prominent series behaviors by considering the whole information of time series. in this paper, we address the problem: which portion of information is more suitable for similarity measure for the data collected from a certain field. we propose a model for the retrieval and representation of the partial information in time series data, and a methodology for evaluating the similarity measurements based on partial information. the methodology is to retrieve various portions of information from the raw data and represent it in a concise form, then cluster the time series using the partial information and evaluate the similarity measurements through comparing the results with a standard classification. experiments on data set from stock market give some interesting observations and justify the usefulness of our approach.
simultaneous optimization of complex mining tasks with a knowledgeable cache. with an increasing use of data mining tools and techniques, we envision that a knowledge discovery and data mining system (kddms) will have to support and optimize for the following scenarios: 1) sequence of queries: a user may analyze one or more datasets by issuing a sequence of related complex mining queries, and 2) multiple simultaneous queries: several users may be analyzing a set of datasets concurrently, and may issue related complex queries.this paper presents a systematic mechanism to optimize for the above cases, targeting the class of mining queries involving frequent pattern mining on one or multiple datasets. we present a system architecture and propose new algorithms to simultaneously optimize multiple such queries and use a knowledgeable cache to store and utilize the past query results. we have implemented and evaluated our system with both real and synthetic datasets. our experimental results show that our techniques can achieve a speedup of up to a factor of 9, compared with the systems which do not support caching or optimize for multiple queries.
mining top-n local outliers in large databases. outlier detection is an important task in data mining with numerous applications, including credit card fraud detection, video surveillance, etc. a recent work on outlier detection has introduced a novel notion of local outlier in which the degree to which an object is outlying is dependent on the density of its local neighborhood, and each object can be assigned a local outlier factor (lof) which represents the likelihood of that object being an outlier. although the concept of local outliers is a useful one, the computation of lof values for every data objects requires a large number of &kgr;-nearest neighbors searches and can be computationally expensive. since most objects are usually not outliers, it is useful to provide users with the option of finding only n most outstanding local outliers, i.e., the top-n data objects which are most likely to be local outliers according to their lofs. however, if the pruning is not done carefully, finding top-n outliers could result in the same amount of computation as finding lof for all objects. in this paper, we propose a novel method to efficiently find the top-n local outliers in large databases. the concept of "micro-cluster" is introduced to compress the data. an efficient micro-cluster-based local outlier mining algorithm is designed based on this concept. as our algorithm can be adversely affected by the overlapping in the micro-clusters, we proposed a meaningful cut-plane solution for overlapping data. the formal analysis and experiments show that this method can achieve good performance in finding the most outstanding local outliers.
discovering frequent topological structures from graph datasets. the problem of finding frequent patterns from graph-based datasets is an important one that finds applications in drug discovery, protein structure analysis, xml querying, and social network analysis among others. in this paper we propose a framework to mine frequent large-scale structures, formally defined as frequent topological structures, from graph datasets. key elements of our framework include, fast algorithms for discovering frequent topological patterns based on the well known notion of a topological minor, algorithms for specifying and pushing constraints deep into the mining process for discovering constrained topological patterns, and mechanisms for specifying approximate matches when discovering frequent topological patterns in noisy datasets. we demonstrate the viability and scalability of the proposed algorithms on real and synthetic datasets and also discuss the use of the framework to discover meaningful topological structures from protein structure data.
web usage mining based on probabilistic latent semantic analysis. the primary goal of web usage mining is the discovery of patterns in the navigational behavior of web users. standard approaches, such as clustering of user sessions and discovering association rules or frequent navigational paths, do not generally provide the ability to automatically characterize or quantify the unobservable factors that lead to common navigational patterns. it is, therefore, necessary to develop techniques that can automatically discover hidden semantic relationships among users as well as between users and web objects. probabilistic latent semantic analysis (plsa) is particularly useful in this context, since it can uncover latent semantic associations among users and pages based on the co-occurrence patterns of these pages in user sessions. in this paper, we develop a unified framework for the discovery and analysis of web navigational patterns based on plsa. we show the flexibility of this framework in characterizing various relationships among users and web objects. since these relationships are measured in terms of probabilities, we are able to use probabilistic inference to perform a variety of analysis tasks such as user segmentation, page classification, as well as predictive tasks such as collaborative recommendations. we demonstrate the effectiveness of our approach through experiments performed on real-world data sets.
a maximum entropy web recommendation system: combining collaborative and content features. web users display their preferences implicitly by navigating through a sequence of pages or by providing numeric ratings to some items. web usage mining techniques are used to extract useful knowledge about user interests from such data. the discovered user models are then used for a variety of applications such as personalized recommendations. web site content or semantic features of objects provide another source of knowledge for deciphering users' needs or interests. we propose a novel web recommendation system in which collaborative features such as navigation or rating data as well as the content features accessed by the users are seamlessly integrated under the maximum entropy principle. both the discovered user patterns and the semantic relationships among web objects are represented as sets of constraints that are integrated to fit the model. in the case of content features, we use a new approach based on latent dirichlet allocation (lda) to discover the hidden semantic relationships among items and derive constraints used in the model. experiments on real web site usage data sets show that this approach can achieve better recommendation accuracy, when compared to systems using only usage information. the integration of semantic information also allows for better interpretation of the generated recommendations.
optimizing search engines using clickthrough data. this paper presents an approach to automatically optimizing the retrieval quality of search engines using clickthrough data. intuitively, a good information retrieval system should present relevant documents high in the ranking, with less relevant documents following below. while previous approaches to learning retrieval functions from examples exist, they typically require training data generated from relevance judgments by experts. this makes them difficult and expensive to apply. the goal of this paper is to develop a method that utilizes clickthrough data for training, namely the query-log of the search engine in connection with the log of links the users clicked on in the presented ranking. such clickthrough data is available in abundance and can be recorded at very low cost. taking a support vector machine (svm) approach, this paper presents a method for learning retrieval functions. from a theoretical perspective, this method is shown to be well-founded in a risk minimization framework. furthermore, it is shown to be feasible even for large sets of queries and features. the theoretical results are verified in a controlled experiment. it shows that the method can effectively adapt the retrieval function of a meta-search engine to a particular group of users, outperforming google in terms of retrieval quality after only a couple of hundred training examples.
training linear svms in linear time. linear support vector machines (svms) have become one of the most prominent machine learning techniques for high-dimensional sparse data commonly encountered in applications like text classification, word-sense disambiguation, and drug design. these applications involve a large number of examples n as well as a large number of features n, while each example has only s << n non-zero features. this paper presents a cutting plane algorithm for training linear svms that provably has training time 0(s,n) for classification problems and o(sn log (n))for ordinal regression problems. the algorithm is based on an alternative, but equivalent formulation of the svm optimization problem. empirically, the cutting-plane algorithm is several orders of magnitude faster than decomposition methods like svm light for large datasets.
introducing perpetual analytics. common strategies to liberate an organization's information assets for situational awareness frequently rely on infrastructure components such as data integration, enterprise search, federation, data warehousing, and so on. and while these traditional platforms enable analysts to get better and faster answers to their queries, the next big advance will change this paradigm. users cannot be expected to formulate and ask every smart question every day. and to escape this impractical and un-scalable model, the new paradigm will involve technologies where "the data finds the data" and "relevance finds the user." perpetual analytics describes a class of application whereby enterprise context is assembled, in real-time, on data streams as fast as operational systems record observations. context construction is a "data finds the data" activity which enables events of interest to be streamed to subscribers. in this talk, i will talk at some depth about the dynamics of such systems including scalability and sustainability.
predicting rare classes: can boosting make any weak learner strong? boosting is a strong ensemble-based learning algorithm with the promise of iteratively improving the classification accuracy using any base learner, as long as it satisfies the condition of yielding weighted accuracy > 0.5. in this paper, we analyze boosting with respect to this basic condition on the base learner, to see if boosting ensures prediction of rarely occurring events with high recall and precision. first we show that a base learner can satisfy the required condition even for poor recall or precision levels, especially for very rare classes. furthermore, we show that the intelligent weight updating mechanism in boosting, even in its strong cost-sensitive form, does not prevent cases where the base learner always achieves high precision but poor recall or high recall but poor precision, when mapped to the original distribution. in either of these cases, we show that the voting mechanism of boosting falls to achieve good overall recall and precision for the ensemble. in effect, our analysis indicates that one cannot be blind to the base learner performance, and just rely on the boosting mechanism to take care of its weakness. we validate our arguments empirically on variety of real and synthetic rare class problems. in particular, using adacost as the boosting algorithm, and variations of pnrule and ripper as the base learners, we show that if algorithm a achieves better recall-precision balance than algorithm b, then using a as the base learner in adacost yields significantly better performance than using b as the base learner.
a bag of paths model for measuring structural similarity in web documents. structural information (such as layout and look-and-feel) has been extensively used in the literatuce for extraction of interesting or relevant data, efficient storage, and query optimization. traditionally, tree models (such as dom trees) have been used to represent structural information, especially in the case of html and xml documents. however, computation of structural similarity between documents based on the tree model is computationally expensive. in this paper, we propose an alternative scheme for representing the structural information of documents based on the paths contained in the corresponding tree model. since the model includes partial information about parents, children and siblings, it allows us to define a new family of meaningful (and at the same time computationally simple) structural similarity measures. our experimental results based on the sigmod xml data set as well as html document collections from ibm.com, dell.com, and amazon.com show that the representation is powerful enough to produce good clusters of structurally similar pages.
mining intrusion detection alarms for actionable knowledge. in response to attacks against enterprise networks, administrators increasingly deploy intrusion detection systems. these systems monitor hosts, networks, and other resources for signs of security violations. the use of intrusion detection has given rise to another difficult problem, namely the handling of a generally large number of alarms. in this paper, we mine historical alarms to learn how future alarms can be handled more efficiently. first, we investigate episode rules with respect to their suitability in this approach. we report the difficulties encountered and the unexpected insights gained. in addition, we introduce a new conceptual clustering technique, and use it in extensive experiments with real-world data to show that intrusion detection alarms can be handled efficiently by using previously mined knowledge.
capital one's statistical problems: our top ten list. capital one is a highly quantitatively driven diversified financial services firm. as such, we make broad and deep use of the entire repertory of highly quantitative techniques. this talk will present our top ten statistical problems. indeed, one of them has, as a sub point, the data mining dimension, but it will likely be useful for data miners to see how their research needs to complement, and fit into, the entire range of hard statistical issues.
a system for automated mapping of bill-of-materials part numbers. part numbers are widely used within an enterprise throughout the manufacturing process. the point of entry of such part numbers into this process is normally via a bill of materials, or bom, sent by a contact manufacturer or supplier. each line of the bom provides information about one part such as the supplier part number, the bom receiver's corresponding internal part number, an unstructured textual part description, the supplier name, etc. however, in a substantial number of cases, the bom receiver's internal part number is absent. hence, before this part can be incorporated into the receiver's manufacturing process, it has to be mapped to an internal part (of the bom receiver) based on the information of the part in the bom. historically, this mapping process has been done manually which is a highly time-consuming, labor intensive and error-prone process. this paper describes a system for automating the mapping of bom part numbers. the system uses a two step modeling and mapping approach. first, the system uses historical bom data, receiver's part specifications data and receiver's part taxonomic data along with domain knowledge to automatically learn classification models for mapping a given bom part description to successively lower levels of the receiver's part taxonomy to reduce the set of potential internal parts to which the bom part could map to. then, information about various part parameters is extracted from the bom part description and compared to the specifications data of the potential internal parts to choose the final mapped internal part. mappings done by the system are very accurate, and the system is currently being deployed within ibm for mapping boms received by the corporate procurement/manufacturing divisions.
data mining in the chemical industry. in this paper we describe the experience of introducing data mining to a large chemical manufacturing company. the multi-national nature of doing business with multiple business units, presents a unique opportunity for the deployment of data mining. while each business unit has its own objectives and challenges, which may be at odds with those of other units, they also share many common interests and resources. in this environment, data mining can be used to identify potential value-creating opportunities, through large site integration of multiple assets and synergies from the use of common assets, such as site-wide manufacturing facilities, and world-wide supply-chain, purchasing and other shared services. however, issues arise, on one hand from overly complex systems, and on the other hand, from the danger of reaching sub-optimal solutions, if a big enough picture is not considered when executing projects. the company-wide initiative and use of six sigma at all levels of the company provided a fertile ground for making the case for data mining and facilitating its acceptance. the six sigma mindset of measuring the performance of processes and analyzing data promotes data-based decision making, therefore making data mining a natural extension of this methodology. we will describe the approach for launching a data mining capability within this framework, the strategy for securing upper management support, drawing from internal modeling, statistical, and other communities, and from external consultants and universities. lessons learned from industrial case studies, enterprise-wide tool evaluation and peer benchmarking will be discussed.
generalized clustering, supervised learning, and data assignment. clustering algorithms have become increasingly important in handling and analyzing data. considerable work has been done in devising effective but increasingly specific clustering algorithms. in contrast, we have developed a generalized framework that accommodates diverse clustering algorithms in a systematic way. this framework views clustering as a general process of iterative optimization that includes modules for supervised learning and instance assignment. the framework has also suggested several novel clustering methods. in this paper, we investigate experimentally the efficacy of these algorithms and test some hypotheses about the relation between such unsupervised techniques and the supervised methods embedded in them.
nantonac collaborative filtering: recommendation based on order responses. a recommender system suggests the items expected to be preferred by the users. recommender systems use collaborative filtering to recommend items by summarizing the preferences of people who have tendencies similar to the user preference. traditionally, the degree of preference is represented by a scale, for example, one that ranges from one to five. this type of measuring technique is called the semantic differential (sd) method. web adopted the ranking method, however, rather than the sd method, since the sd method is intrinsically not suited for representing individual preferences. in the ranking method, the preferences are represented by orders, which are sorted item sequences according to the users' preferences. we here propose some methods to recommed items based on these order responses, and carry out the comparison experiments of these methods.
visualizing multi-dimensional clusters, trends, and outliers using star coordinates. interactive visualizations are effective tools in mining scientific, engineering, and business data to support decision-making activities. star coordinates is proposed as a new multi-dimensional visualization technique, which supports various interactions to stimulate visual thinking in early stages of knowledge discovery process. in star coordinates, coordinate axes are arranged on a two-dimensional surface, where each axis shares the same origin point. each multi-dimensional data element is represented by a point, where each attribute of the data contributes to its location through uniform encoding. interaction features of star coordinates provide users the ability to apply various transformations dynamically, integrate and separate dimensions, analyze correlations of multiple dimensions, view clusters, trends, and outliers in the distribution of data, and query points based on data ranges. our experience with star coordinates shows that it is particularly useful for the discovery of hierarchical clusters, and analysis of multiple factors providing insight in various real datasets including telecommunications churn.
when do data mining results violate privacy? privacy-preserving data mining has concentrated on obtaining valid results when the input data is private. an extreme example is secure multiparty computation-based methods, where only the results are revealed. however, this still leaves a potential privacy breach: do the results themselves violate privacy? this paper explores this issue, developing a framework under which this question can be addressed. metrics are proposed, along with analysis that those metrics are consistent in the face of apparent problems.