waashk/dblp huggingface dataset:
applying data mining in investigating money laundering crimes. in this paper, we study the problem of applying data mining to facilitate the investigation of money laundering crimes (mlcs). we have identified a new paradigm of problems --- that of automatic community generation based on uni-party data, the data in which there is no direct or explicit link information available. consequently, we have proposed a new methodology for link discovery based on correlation analysis (ldca). we have used mlc group model generation as an exemplary application of this problem paradigm, and have focused on this application to develop a specific method of automatic mlc group model generation based on timeline analysis using the ldca methodology, called coral. a prototype of coral method has been implemented, and preliminary testing and evaluations based on a real mlc case data are reported. the contributions of this work are: (1) identification of the uni-party data community generation problem paradigm, (2) proposal of a new methodology ldca to solve for problems in this paradigm, (3) formulation of the mlc group model generation problem as an example of this paradigm, (4) application of the ldca methodology in developing a specific solution (coral) to the mlc group model generation problem, and (5) development, evaluation, and testing of the coral prototype in a real mlc case data.
predicting prostate cancer recurrence via maximizing the concordance index. in order to effectively use machine learning algorithms, e.g., neural networks, for the analysis of survival data, the correct treatment of censored data is crucial. the concordance index (ci) is a typical metric for quantifying the predictive ability of a survival model. we propose a new algorithm that directly uses the ci as the objective function to train a model, which predicts whether an event will eventually occur or not. directly optimizing the ci allows the model to make complete use of the information from both censored and non-censored observations. in particular, we approximate the ci via a differentiable function so that gradient-based methods can be used to train the model. we applied the new algorithm to predict the eventual recurrence of prostate cancer following radical prostatectomy. compared with the traditional cox proportional hazards model and several other algorithms based on neural networks and support vector machines, our algorithm achieves a significant improvement in being able to identify high-risk and low-risk groups of patients.
a new scheme on privacy-preserving data classification. we address privacy-preserving classification problem in a distributed system. randomization has been the approach proposed to preserve privacy in such scenario. however, this approach is now proven to be insecure as it has been discovered that some privacy intrusion techniques can be used to reconstruct private information from the randomized data tuples. we introduce an algebraic-technique-based scheme. compared to the randomization approach, our new scheme can build classifiers more accurately but disclose less private information. furthermore, our new scheme can be readily integrated as a middleware with existing systems.
immc: incremental maximum margin criterion. subspace learning approaches have attracted much attention in academia recently. however, the classical batch algorithms no longer satisfy the applications on streaming data or large-scale data. to meet this desirability, incremental principal component analysis (ipca) algorithm has been well established, but it is an unsupervised subspace learning approach and is not optimal for general classification tasks, such as face recognition and web document categorization. in this paper, we propose an incremental supervised subspace learning algorithm, called incremental maximum margin criterion (immc), to infer an adaptive subspace by optimizing the maximum margin criterion. we also present the proof for convergence of the proposed algorithm. experimental results on both synthetic dataset and real world datasets show that immc converges to the similar subspace as that of batch approach.
a data mining approach to modeling relationships among categories in image collection. this paper proposes a data mining approach to modeling relationships among categories in image collection. in our approach, with image feature grouping, a visual dictionary is created for color, texture, and shape feature attributes respectively. labeling each training image with the keywords in the visual dictionary, a classification tree is built. based on the statistical properties of the feature space we define a structure, called α-semantics graph, to discover the hidden semantic relationships among the semantic categories embodied in the image collection. with the α-semantics graph, each semantic category is modeled as a unique fuzzy set to explicitly address the semantic uncertainty and semantic overlap among the categories in the feature space. the model is utilized in the semantics-intensive image retrieval application. an algorithm using the classification accuracy measures is developed to combine the built classification tree with the fuzzy set modeling method to deliver semantically relevant image retrieval for a given query image. the experimental evaluations have demonstrated that the proposed approach models the semantic relationships effectively and the image retrieval prototype system utilizing the derived model is promising both in effectiveness and efficiency.
the complexity of mining maximal frequent itemsets and maximal frequent patterns. mining maximal frequent itemsets is one of the most fundamental problems in data mining. in this paper we study the complexity-theoretic aspects of maximal frequent itemset mining, from the perspective of counting the number of solutions. we present the first formal proof that the problem of counting the number of distinct maximal frequent itemsets in a database of transactions, given an arbitrary support threshold, is #p-complete, thereby providing strong theoretical evidence that the problem of mining maximal frequent itemsets is np-hard. this result is of particular interest since the associated decision problem of checking the existence of a maximal frequent itemset is in p.we also extend our complexity analysis to other similar data mining problems dealing with complex data structures, such as sequences, trees, and graphs, which have attracted intensive research interests in recent years. normally, in these problems a partial order among frequent patterns can be defined in such a way as to preserve the downward closure property, with maximal frequent patterns being those without any successor with respect to this partial order. we investigate several variants of these mining problems in which the patterns of interest are subsequences, subtrees, or subgraphs, and show that the associated problems of counting the number of maximal frequent patterns are all either #p-complete or #p-hard.
building connected neighborhood graphs for isometric data embedding. neighborhood graph construction is usually the first step in algorithms for isometric data embedding and manifold learning that cope with the problem of projecting high dimensional data to a low space. this paper begins by explaining the algorithmic fundamentals of techniques for isometric data embedding and derives a general classification of these techniques. we will see that the nearest neighbor approaches commonly used to construct neighborhood graphs do not guarantee connectedness of the constructed neighborhood graphs and, consequently, may cause an algorithm fail to project data to a single low dimensional coordinate system. in this paper, we review three existing methods to construct k-edge-connected neighborhood graphs and propose a new method to construct k-connected neighborhood graphs. these methods are applicable to a wide range of data including data distributed among clusters. their features are discussed and compared through experiments.
event detection from evolution of click-through data. previous efforts on event detection from the web have focused primarily on web content and structure data ignoring the rich collection of web log data. in this paper, we propose the first approach to detect events from the click-through data, which is the log data of web search engines. the intuition behind event detection from click-through data is that such data is often event-driven and each event can be represented as a set ofquery-page pairs that are not only semantically similar but also have similar evolution pattern over time. given the click-through data, in our proposed approach, we first segment it into a sequence of bipartite graphs based on theuser-defined time granularity. next, the sequence of bipartite graphs is represented as a vector-based graph, which records the semantic and evolutionary relationships between queries and pages. after that, the vector-based graph is transformed into its dual graph, where each node is a query-page pair that will be used to represent real world events. then, the problem of event detection is equivalent to the problem of clustering the dual graph of the vector-based graph. the clustering process is based on a two-phase graph cut algorithm. in the first phase, query-page pairs are clustered based on thesemantic-based similarity such that each cluster in the result corresponds to a specific topic. in the second phase, query-page pairs related to the same topic are further clustered based on the evolution pattern-based similarity such that each cluster is expected to represent a specific event under the specific topic. experiments with real click-through data collected from a commercial web search engine show that the proposed approach produces high quality results.
efficient discovery of error-tolerant frequent itemsets in high dimensions. we present a generalization of frequent itemsets allowing for the notion of errors in the itemset definition. we motivate the problem and present an efficient algorithm that identifies error-tolerant frequent clusters of items in transactional data (customer-purchase data, web browsing data, text, etc.). the algorithm exploits sparseness of the underlying data to find large groups of items that are correlated over database records (rows). the notion of transaction coverage allows us to extend the algorithm and view it as a fast clustering algorithm for discovering segments of similar transactions in binary sparse data. we evaluate the new algorithm on three real-world applications: clustering high-dimensional data, query selectivity estimation and collaborative filtering. results show that the algorithm consistently uncovers structure in large sparse databases that other traditional clustering algorithms fail to find.
opportunity map: identifying causes of failure - a deployed data mining system. in this paper, we report a deployed data mining application system for motorola. originally, its intended use was for identifying causes of cellular phone failures, but it has been found to be useful for many other engineering data sets as well. for this report, the case study is a dataset containing cellular phone call records. this data set is like any dataset used in classification applications, i.e., with a set of attributes which can be continuous or discrete, and a discrete class attribute. in our application, the classes are normally ended calls, calls which failed to setup, and calls which failed while in progress. however, the task is not to predict any failure, but to identify possible causes that resulted in failures. then, engineering efforts may focus on improvements that can be made to the phones. in the course of the project, various classification techniques, e.g., decision trees, naïve bayesian classification and svm were tried. however, the results were unsatisfactory. after several demonstrations and interaction with domain experts, we finally designed and implemented an effective approach to perform the task. the final system is based on class association rules, general impressions and visualization. the system has been deployed and is in regular use at motorola. in this paper, we first describe our experiences with some existing classification systems and discuss why they are not suitable for the task. we then present our techniques. as an illustration, we show several visualization screens in the case study, which reveal some important knowledge. due to confidentiality, we will not give specifics but only present a general discussion about the results.
clope: a fast and effective clustering algorithm for transactional data. this paper studies the problem of categorical data clustering, especially for transactional data characterized by high dimensionality and large volume. starting from a heuristic method of increasing the height-to-width ratio of the cluster histogram, we develop a novel algorithm -- clope, which is very fast and scalable, while being quite effective. we demonstrate the performance of our algorithm on two real world datasets, and compare clope with the state-of-art algorithms.
v-miner: using enhanced parallel coordinates to mine product design and test data. analyzing data to find trends, correlations, and stable patterns is an important task in many industrial applications. this paper proposes a new technique based on parallel coordinate visualization. previous work on parallel coordinate methods has shown that they are effective only when variables that are correlated and/or show similar patterns are displayed adjacently. although current parallel coordinate tools allow the user to manually rearrange the order of variables, this process is very time-consuming when the number of variables is large. automated assistance is required. this paper introduces an edit-distance based technique to rearrange variables so that interesting change patterns can be easily detected visually. the visual miner (v-miner) software includes both automated methods for visualizing common patterns and a query tool that enables the user to describe specific target patterns to be mined or displayed by the system. in addition, the system can filter data according to rules sets imported from other data mining tools. this feature was found very helpful in practice, because it enables decision makers to visually identify interesting rules and data segments for further analysis or data mining. this paper begins with an introduction to the proposed techniques and the v-miner system. next, a case study illustrates how v-miner has been used at motorola to guide product design and test decisions.
learning to predict train wheel failures. this paper describes a successful but challenging application of data mining in the railway industry. the objective is to optimize maintenance and operation of trains through prognostics of wheel failures. in addition to reducing maintenance costs, the proposed technology will help improve railway safety and augment throughput. building on established techniques from data mining and machine learning, we present a methodology to learn models to predict train wheel failures from readily available operational and maintenance data. this methodology addresses various data mining tasks such as automatic labeling, feature extraction, model building, model fusion, and evaluation. after a detailed description of the methodology, we report results from large-scale experiments. these results clearly show the great potential of this innovative application of data mining in the railway industry.
blosom: a framework for mining arbitrary boolean expressions. we introduce a novel framework, called blosom, for mining (frequent) boolean expressions over binary-valued datasets. we organize the space of boolean expressions into four categories: pure conjunctions, pure disjunctions, conjunction of disjunctions, and disjunction of conjunctions. we focus on mining the simplest expressions the minimal generators for each class. we also propose a closure operator for each class that yields closed boolean expressions. blosom efficiently mines frequent boolean expressions by utilizing a number of methodical pruning techniques. experiments showcase the behavior of blosom, and an application study on a real dataset is also given.
2pxminer: an efficient two pass mining of frequent xml query patterns. caching the results of frequent query patterns can improve the performance of query evaluation. this paper describes a 2-pass mining algorithm called 2pxminer to discover frequent xml query patterns. we design 3 data structures to expedite the mining process. experiments results indicate that 2pxminer is both efficient and scalable.
real world performance of association rule algorithms. this study compares five well-known association rule algorithms using three real-world datasets and an artificial dataset. the experimental results confirm the performance improvements previously claimed by the authors on the artificial data, but some of these gains do not carry over to the real datasets, indicating overfitting of the algorithms to the ibm artificial dataset. more importantly, we found that the choice of algorithm only matters at support levels that generate more rules than would be useful in practice. for support levels that generate less than 1,000,000 rules, which is much more than humans can handle and is sufficient for prediction purposes where data is loaded into ram, apriori finishes processing in less than 10 minutes. on our datasets, we observed super-exponential growth in the number of rules. on one of our datasets, a 0.02% change in the support increased the number of rules from less than a million to over a billion, implying that outside a very narrow range of support values, the choice of algorithm is irrelevant.
a dea approach for model combination. this paper proposes a novel data envelopment analysis (dea) based approach for model combination. we first prove that for the 2-class classification problems dea models identify the same convex hull as the popular roc analysis used for model combination. for general k-class classifiers, we then develop a dea-based method to combine multiple classifiers. experiments show that the method outperforms other benchmark methods and suggest that dea can be a promising tool for model combination.
a generalized framework for mining spatio-temporal patterns in scientific data. in this paper, we present a general framework to discover spatial associations and spatio-temporal episodes for scientific datasets. in contrast to previous work in this area, features are modeled as geometric objects rather than points. we define multiple distance metrics that take into account objects' extent and thus are more robust in capturing the influence of an object on other objects in spatial neighborhood. we have developed algorithms to discover four different types of spatial object interaction (association) patterns. we also extend our approach to accommodate temporal information and propose a simple algorithm to derive spatio-temporal episodes. we show that such episodes can be used to reason about critical events. we evaluate our framework on real datasets to demonstrate its efficacy. the datasets originate from two different areas: computational molecular dynamics and computational fluid flow. we present results highlighting the importance of the identified patterns and episodes by using knowledge from the underlying domains. we also show that the proposed algorithms scale linearly with respect to the dataset size.
mining asynchronous periodic patterns in time series data. periodicy detection in time series data is a challenging problem of great importance in many applications. most previous work focused on mining synchronous periodic patterns and did not recognize the misaligned presence of a pattern due to the intervention of random noise. in this paper, we propose a more flexible model of asynchronous periodic pattern that may be present only within a subsequence and whose occurrences may be shifted due to disturbance. two parameters min_rep and max_dis are employed to specify the minimum number of repetitions that is required within each segment of nondisrupted pattern occurrences and the maximum allowed disturbance between any two successive valid segments. upon satisfying these two requirements, the longest valid subsequence of a pattern is returned. a two-phase algorithm is devised to first generate potential periods by distance-based pruning followed by an iterative procedure to derive and validate candidate patterns and locate the longest valid subsequence. we also show that this algorithm cannot only provide linear time complexity with respect to the length of the sequence but also achieve space efficiency.
infominer: mining surprising periodic patterns. in this paper, we focus on mining surprising periodic patterns in a sequence of events. in many applications, e.g., computational biology, an infrequent pattern is still considered very significant if its actual occurrence frequency exceeds the prior expectation by a large margin. the traditional metric, such as support, is not necessarily the ideal model to measure this kind of surprising patterns because it treats all patterns equally in the sense that every occurrence carries the same weight towards the assessment of the significance of a pattern regardless of the probability of occurrence. a more suitable measurement, information, is introduced to naturally value the degree of surprise of each occurrence of a pattern as a continuous and monotonically decreasing function of its probability of occurrence. this would allow patterns with vastly different occurrence probabilities to be handled seamlessly. as the accumulated degree of surprise of all repetitions of a pattern, the concept of information gain is proposed to measure the overall degree of surprise of the pattern within a data sequence. the bounded information gain property is identified to tackle the predicament caused by the violation of the downward closure property by the information gain measure and in turn provides an efficient solution to this problem. empirical tests demonstrate the efficiency and the usefulness of the proposed model.
streaming feature selection using alpha-investing. in streaming feature selection (sfs), new features are sequentially considered for addition to a predictive model. when the space of potential features is large, sfs offers many advantages over traditional feature selection methods, which assume that all features are known in advance. features can be generated dynamically, focusing the search for new features on promising subspaces, and overfitting can be controlled by dynamically adjusting the threshold for adding features to the model. we describe α-investing, an adaptive complexity penalty method for sfs which dynamically adjusts the threshold on the error reduction required for adding a new feature. α-investing gives false discovery rate-style guarantees against overfitting. it differs from standard penalty methods such as aic, bic or ric, which always drastically over- or under-fit in the limit of infinite numbers of non-predictive features. empirical results show that sfs is competitive with much more compute-intensive feature selection methods such as stepwise regression, and allows feature selection on problems with over a million potential features.