waashk/dblp huggingface dataset:
data quality through knowledge engineering. traditionally, data quality programs have acted as a preprocessing stage to make data suitable for a data mining or analysis operation. recently, data quality concepts have been applied to databases that support business operations such as provisioning and billing. incorporating business rules that drive operations and their associated data processes is critically important to the success of such projects. however, there are many practical complications. for example, documentation on business rules is often meager. rules change frequently. domain knowledge is often fragmented across experts, and those experts do not always agree. typically, rules have to be gathered from subject matter experts iteratively, and are discovered out of logical or procedural sequence, like a jigsaw puzzle. our approach is to impement business rules as constraints on data in a classical expert system formalism sometimes called production rules. our system works by allowing good data to pass through a system of constraints unchecked. bad data violate constraints and are flagged, and then fed back after correction. constraints are added incrementally as better understanding of the business rules is gained. we include a real-life case study.
data mining in metric space: an empirical analysis of supervised learning performance criteria. many criteria can be used to evaluate the performance of supervised learning. different criteria are appropriate in different settings, and it is not always clear which criteria to use. a further complication is that learning methods that perform well on one criterion may not perform well on other criteria. for example, svms and boosting are designed to optimize accuracy, whereas neural nets typically optimize squared error or cross entropy. we conducted an empirical study using a variety of learning methods (svms, neural nets, k-nearest neighbor, bagged and boosted trees, and boosted stumps) to compare nine boolean classification performance metrics: accuracy, lift, f-score, area under the roc curve, average precision, precision/recall break-even point, squared error, cross entropy, and probability calibration. multidimensional scaling (mds) shows that these metrics span a low dimensional manifold. the three metrics that are appropriate when predictions are interpreted as probabilities: squared error, cross entropy, and calibration, lay in one part of metric space far away from metrics that depend on the relative order of the predicted values: roc area, average precision, break-even point, and lift. in between them fall two metrics that depend on comparing predictions to a threshold: accuracy and f-score. as expected, maximum margin methods such as svms and boosted trees have excellent performance on metrics like accuracy, but perform poorly on probability metrics such as squared error. what was not expected was that the margin methods have excellent performance on ordering metrics such as roc area and average precision. we introduce a new metric, sar, that combines squared error, accuracy, and roc area into one metric. mds and correlation analysis shows that sar is centrally located and correlates well with other metrics, suggesting that it is a good general purpose metric to use when more specific criteria are not known.
single-pass online learning: performance, voting schemes and online feature selection. to learn concepts over massive data streams, it is essential to design inference and learning methods that operate in real time with limited memory. online learning methods such as perceptron or winnow are naturally suited to stream processing; however, in practice multiple passes over the same training data are required to achieve accuracy comparable to state-of-the-art batch learners. in the current work we address the problem of training an on-line learner with a single passover the data. we evaluate several existing methods, and also propose a new modification of margin balanced winnow, which has performance comparable to linear svm. we also explore the effect of averaging, a.k.a. voting, on online learning. finally, we describe how the new modified margin balanced winnow algorithm can be naturally adapted to perform feature selection. this scheme performs comparably to widely-used batch feature selection methods like information gain or chi-square, with the advantage of being able to select features on-the-fly. taken together, these techniques allow single-pass online learning to be competitive with batch techniques, and still maintain the advantages of on-line learning.
extracting semantics from data cubes using cube transversals and closures. in this paper we propose a lattice-based approach intended for extracting semantics from datacubes: borders of version spaces for supervised classification, closed cube lattice to summarize the semantics of datacubes w.r.t. count, sum, and covering graph of the quotient cube as a visualization tool of minimal multidimensional associations. with this intention, we introduce two novel concepts: the cube transversals and the cube closures over the cube lattice of a categorical database relation. we propose a levelwise merging algorithm for mining minimal cube transversals with a single database scan. we introduce the cube connection, show that it is a galois connection and derive a closure operator over the cube lattice. using cube transversals and closures, we define a new characterization of boundary sets which provide a condensed representation of version spaces used to enhance supervised classification. the algorithm designed for computing such borders improves the complexity of previous proposals. we also introduce the concept of closed cube lattice and show that it is isomorph to on one hand the galois lattice and on the other hand the quotient cube w.r.t. count, sum. proposed in [16], the quotient cube is a succinct summary of a datacube preserving the rollup/drilldown semantics. we show that the quotient cube w.r.t. count, sum and the closed cube lattice have a similar expression power but the latter has the smallest possible size. finally we focus on the multidimensional association issue and introduce the covering graph of the quotient cube which provides the user with a visualization tool of minimal multidimensional associations.
onboard classifiers for science event detection on a remote sensing spacecraft. typically, data collected by a spacecraft is downlinked to earth and preprocessed before any analysis is performed. we have developed classifiers that can be used onboard a spacecraft to identify high priority data for downlink to earth, providing a method for maximizing the use of a potentially bandwidth limited downlink channel. onboard analysis can also enable rapid reaction to dynamic events, such as flooding, volcanic eruptions or sea ice break-up.four classifiers were developed to identify cryosphere events using hyperspectral images. these classifiers include a manually constructed classifier, a support vector machine (svm), a decision tree and a classifier derived by searching over combinations of thresholded band ratios. each of the classifiers was designed to run in the computationally constrained operating environment of the spacecraft. a set of scenes was hand-labeled to provide training and testing data. performance results on the test data indicate that the svm and manual classifiers outperformed the decision tree and band-ratio classifiers with the svm yielding slightly better classifications than the manual classifier.the manual and svm classifiers have been uploaded to the eo-1 spacecraft and have been running onboard the spacecraft for over a year. results of the onboard analysis are used by the autonomous sciencecraft experiment (ase) of nasa's new millennium program onboard eo-1 to automatically target the spacecraft to collect follow-on imagery. the software demonstrates the potential for future deep space missions to use onboard decision making to capture short-lived science events.
estimating business targets. determining and setting maximal revenue expectations or other business performance targets---whether it is for regional company divisions or individual customers---can have profound financial implications. operational techniques are changed, staffing levels are altered and management attention is re-focused---all in the name of expectations. in practice these expectations are often derived in an ad hoc manner. to address this unsupervised task, we combine nearest neighbor methods and classical statistical methods and derive a new solution to the classical econometric task of frontier analysis. we apply our methodology to two real world business problems in verizon, a major telecommunications provider in the united states, more specifically in the print yellow page division verizon information services: (1) identifying under marketed customers for targeted upselling campaigns and focused sales attention, and (2) benchmarking regional directory divisions to incent performance improvements. our analysis uncovers some commercially useful aspects of these domains and by conservative estimates can increase revenue by several million dollars in each domain.
a general approach to incorporate data quality matrices into data mining algorithms. data quality is a central issue for many information-oriented organizations. recent advances in the data quality field reflect the view that a database is the product of a manufacturing process. while routine errors, such as non-existent zip codes, can be detected and corrected using traditional data cleansing tools, many errors systemic to the manufacturing process cannot be addressed. therefore, the product of the data manufacturing process is an imprecise recording of information about the entities of interest (i.e. customers, transactions or assets). in this way, the database is only one (flawed) version of the entities it is supposed to represent. quality assurance systems such as motorola's six-sigma and other continuous improvement methods document the data manufacturing process's shortcomings. a widespread method of documentation is quality matrices. in this paper, we explore the use of the readily available data quality matrices for the data mining classification task. we first illustrate that if we do not factor in these quality matrices, then our results for prediction are sub-optimal. we then suggest a general-purpose ensemble approach that perturbs the data according to these quality matrices to improve the predictive accuracy and show the improvement is due to a reduction in variance.
data mining challenges in the automotive domain. automotive companies, such as ford motor company, have no shortage of large databases with abundant opportunities for cost reduction and revenue enhancement. the data mining group at ford has worked in the areas of quality, customer satisfaction and warranty analytics for close to ten years. in this time, we have developed a number of methods for building systems to help the business. one area of particular success has been in warranty analysis. while traditional hazard analysis has been applied at ford for a number of years, we have used techniques from other industries (e.g. retail), as well as text mining to view warranty analytics in a new way. however, our success has been tempered by serious challenges particularly in the areas of data understanding, computing meaningful aggregations and implementation. case studies from the automobile industry (warranty, quality, forecasting, etc.) as well as from other industries will be used.
locating secret messages in images. steganography involves hiding messages in innocuous media such as images, while steganalysis is the field of detecting these secret messages. the ultimate goal of steganalysis is two-fold: making a binary classification of a file as stego-bearing or innocent, and secondly, locating the hidden message with an aim to extracting, sterilizing or manipulating it. almost all steganalysis approaches (known as attacks) focus on the first of these two issues. in this paper, we explore the difficult related problem: given that we know an image file contains steganography, locate which pixels contain the message. we treat the hidden message location problem as outlier detection using probability/energy measures of images motivated by the image restoration community. pixels contributing the most to the energy calculations of an image are deemed outliers. typically, of the top third of one percent of most energized pixels (outliers), we find that 87% are stego-bearing in color images and 61% in grayscale images. in all image types only 1% of all pixels are stego-bearing indicating our techniques provides a substantial lift over random guessing.
translation-invariant mixture models for curve clustering. in this paper we present a family of algorithms that can simultaneously align and cluster sets of multidimensional curves defined on a discrete time grid. our approach uses the expectation-maximization (em) algorithm to recover both the mean curve shapes for each cluster, and the most likely shifts, offsets, and cluster memberships for each curve. we demonstrate how bayesian estimation methods can improve the results for small sample sizes by enforcing smoothness in the cluster mean curves. we evaluate the methodology on two real-world data sets, time-course gene expression data and storm trajectory data. experimental results show that models that incorporate curve alignment systematically provide improvements in predictive power and within-cluster variance on test data sets. the proposed approach provides a non-parametric, computationally efficient, and robust methodology for clustering broad classes of curve data.
evolutionary clustering. we consider the problem of clustering data over time. an evolutionary clustering should simultaneously optimize two potentially conflicting criteria: first, the clustering at any point in time should remain faithful to the current data as much as possible; and second, the clustering should not shift dramatically from one timestep to the next. we present a generic framework for this problem, and discuss evolutionary versions of two widely-used clustering algorithms within this framework: k-means and agglomerative hierarchical clustering. we extensively evaluate these algorithms on real data sets and show that our algorithms can simultaneously attain both high accuracy in capturing today's data, and high fidelity in reflecting yesterday's clustering.
pattern discovery in sequences under a markov assumption. in this paper we investigate the general problem of discovering recurrent patterns that are embedded in categorical sequences. an important real-world problem of this nature is motif discovery in dna sequences. we investigate the fundamental aspects of this data mining problem that can make discovery "easy" or "hard." we present a general framework for characterizing learning in this context by deriving the bayes error rate for this problem under a markov assumption. the bayes error framework demonstrates why certain patterns are much harder to discover than others. it also explains the role of different parameters such as pattern length and pattern frequency in sequential discovery. we demonstrate how the bayes error can be used to calibrate existing discovery algorithms, providing a lower bound on achievable performance. we discuss a number of fundamental issues that characterize sequential pattern discovery in this context, present a variety of empirical results to complement and verify the theoretical analysis, and apply our methodology to real-world motif-discovery problems in computational biology.
mining and summarizing customer reviews. merchants selling products on the web often ask their customers to review the products that they have purchased and the associated services. as e-commerce is becoming more and more popular, the number of customer reviews that a product receives grows rapidly. for a popular product, the number of reviews can be in hundreds or even thousands. this makes it difficult for a potential customer to read them to make an informed decision on whether to purchase the product. it also makes it difficult for the manufacturer of the product to keep track and to manage customer opinions. for the manufacturer, there are additional difficulties because many merchant sites may sell the same product and the manufacturer normally produces many kinds of products. in this research, we aim to mine and to summarize all the customer reviews of a product. this summarization task is different from traditional text summarization because we only mine the features of the product on which the customers have expressed their opinions and whether the opinions are positive or negative. we do not summarize the reviews by selecting a subset or rewrite some of the original sentences from the reviews to capture the main points as in the classic text summarization. our task is performed in three steps: (1) mining product features that have been commented on by customers; (2) identifying opinion sentences in each review and deciding whether each opinion sentence is positive or negative; (3) summarizing the results. this paper proposes several novel techniques to perform these tasks. our experimental results using reviews of a number of products sold online demonstrate the effectiveness of the techniques.
fully automatic cross-associations. large, sparse binary matrices arise in numerous data mining applications, such as the analysis of market baskets, web graphs, social networks, co-citations, as well as information retrieval, collaborative filtering, sparse matrix reordering, etc. virtually all popular methods for the analysis of such matrices---e.g., k-means clustering, metis graph partitioning, svd/pca and frequent itemset mining---require the user to specify various parameters, such as the number of clusters, number of principal components, number of partitions, and "support." choosing suitable values for such parameters is a challenging problem.cross-association is a joint decomposition of a binary matrix into disjoint row and column groups such that the rectangular intersections of groups are homogeneous. starting from first principles, we furnish a clear, information-theoretic criterion to choose a good cross-association as well as its parameters, namely, the number of row and column groups. we provide scalable algorithms to approach the optimal. our algorithm is parameter-free, and requires no user intervention. in practice it scales linearly with the problem size, and is thus applicable to very large matrices. finally, we present experiments on multiple synthetic and real-life datasets, where our method gives high-quality, intuitive results.
estimating the global pagerank of web communities. localized search engines are small-scale systems that index a particular community on the web. they offer several benefits over their large-scale counterparts in that they are relatively inexpensive to build, and can provide more precise and complete search capability over their relevant domains. one disadvantage such systems have over large-scale search engines is the lack of global pagerank values. such information is needed to assess the value of pages in the localized search domain within the context of the web as a whole. in this paper, we present well-motivated algorithms to estimate the global pagerank values of a local domain. the algorithms are all highly scalable in that, given a local domain of size n, they use o(n) resources that include computation time, bandwidth, and storage. we test our methods across a variety of localized domains, including site-specific domains and topic-specific domains. we demonstrate that by crawling as few as n or 2n additional pages, our methods can give excellent global pagerank estimates.
a rank sum test method for informative gene discovery. finding informative genes from microarray data is an important research problem in bioinformatics research and applications. most of the existing methods rank features according to their discriminative capability and then find a subset of discriminative genes (usually top k genes). in particular, t-statistic criterion and its variants have been adopted extensively. this kind of methods rely on the statistics principle of t-test, which requires that the data follows a normal distribution. however, according to our investigation, the normality condition often cannot be met in real data sets.to avoid the assumption of the normality condition, in this paper, we propose a rank sum test method for informative gene discovery. the method uses a rank-sum statistic as the ranking criterion. moreover, we propose using the significance level threshold, instead of the number of informative genes, as the parameter. the significance level threshold as a parameter carries the quality specification in statistics. we follow the pitman efficiency theory to show that the rank sum method is more accurate and more robust than the t-statistic method in theory.to verify the effectiveness of the rank sum method, we use support vector machine (svm) to construct classifiers based on the identified informative genes on two well known data sets, namely colon data and leukemia data. the prediction accuracy reaches 96.2% on the colon data and 100% on the leukemia data. the results are clearly better than those from the previous feature ranking methods. by experiments, we also verify that using significance level threshold is more effective than directly specifying an arbitrary k.
finding recent frequent itemsets adaptively over online data streams. a data stream is a massive unbounded sequence of data elements continuously generated at a rapid rate. consequently, the knowledge embedded in a data stream is more likely to be changed as time goes by. identifying the recent change of a data stream, specially for an online data stream, can provide valuable information for the analysis of the data stream. in addition, monitoring the continuous variation of a data stream enables to find the gradual change of embedded knowledge. however, most of mining algorithms over a data stream do not differentiate the information of recently generated transactions from the obsolete information of old transactions which may be no longer useful or possibly invalid at present. this paper proposes a data mining method for finding recent frequent itemsets adaptively over an online data stream. the effect of old transactions on the mining result of the data steam is diminished by decaying the old occurrences of each itemset as time goes by. furthermore, several optimization techniques are devised to minimize processing time as well as main memory usage. finally, the proposed method is analyzed by a series of experiments.
parallel computation of high dimensional robust correlation and covariance matrices. the computation of covariance and correlation matrices are critical to many data mining applications and processes. unfortunately the classical covariance and correlation matrices are very sensitive to outliers. robust methods, such as qc and the maronna method, have been proposed. however, existing algorithms for qc only give acceptable performance when the dimensionality of the matrix is in the hundreds; and the maronna method is rarely used in practice because of its high computational cost.in this paper, we develop parallel algorithms for both qc and the maronna method. we evaluate these parallel algorithms using a real data set of the gene expression of over 6,000 genes, giving rise to a matrix of over 18 million entries. in our experimental evaluation, we explore scalability in dimensionality and in the number of processors. we also compare the parallel behaviours of the two methods. after thorough experimentation, we conclude that for many data mining applications, both qc and maronna are viable options. less robust, but faster, qc is the recommended choice for small parallel platforms. on the other hand, the maronna method is the recommended choice when a high degree of robustness is required, or when the parallel platform features a high number of processors.