waashk/dblp huggingface dataset:
telecommunications strategic marketing - kdd and economic modeling. the italian deregulation process of telecommunications market in the last years has produced a largeeconomic impact since it has altered equilibriums thatwere established for a long time. in this framework, wenotice a strong need for adequate tools to analyze themarket and its trends and, at the same time, a lack ofspecific solutions within the scientific literature, due tothe new technical challenges issued by the problem.in particular, in the context of building a decisionsupport system (dss) for the strategic marketing unit oftelecom italia (ti) we have devised a newmethodology to profitably combine most powerful toolsfrom kdd and economic sciences. we have tested ourapproach by analyzing the residential telecommunicationsmarket demand in italy during the transition from amonopolistic structure to an oligopolistic one.in this paper, we first address the state of the art indss design, then we describe the proposed methodologyand its application in the case study.
ensembles of cascading trees. we introduce a new method, called cs4, to constructcommittees of decision trees for classification. the methodconsiders different top-ranked features as the root nodes ofmember trees. this idea is particularly suitable for dealingwith high-dimensional bio-medical data as top-ranked featuresin this type of data usually possess similar merits forclassification. to make a decision, the committee combinesthe power of individual trees in a weighted manner. unlikebagging or boosting which uses bootstrapped trainingdata, our method builds all the member trees of a committeeusing exactly the same set of training data. we have testedthese ideas on uci data sets as well as recent bio-medicaldata sets of gene expression or proteomic profiles that areusually described by more than 10,000 features. all the experimentalresults show that our method is efficient and thatthe classification performance are superior to c4.5 familyalgorithms.
efficient density-based clustering of complex objects. nowadays data mining in large databases of complex objects from scientific, engineering or multimedia applications is getting more and more important. in many different application domains complex object representations along with complex distance functions are used for measuring the similarity between objects. often not only these complex distance measures are available but also simpler distance functions which can be computed much more efficiently. traditionally, the well known concept of multi-step query processing which is based on exact and lower-bounding approximative distance functions is used independently of data mining algorithms. in this paper, we will demonstrate how the paradigm of multi-step query processing can be integrated into the two density-based clustering algorithms dbscan and optics resulting in a considerable efficiency boost. our approach tries to confine itself to ¿-range queries on the simple distance functions and carries out complex distance computations only at that stage of the clustering algorithm where they are compulsory to compute the correct clustering result. in a broad experimental evaluation based on real-world test data sets, we demonstrate that our approach accelerates the generation of flat and hierarchical density-based clusterings by more than one order of magnitude.
optimized disjunctive association rules via sampling. the problem of finding optimized support associationrules for a single numerical attribute, where the optimizedregion is a union of k disjoint intervals from the range ofthe attribute, is investigated. the first polynomial timealgorithm for the problem of finding such a region maximizingsupport and meeting a minimum cumulative confidencethreshold is given. because the algorithm is notpractical, an ostensibly easier, more constrained versionof the problem is considered. experiments demonstratethat the best extant algorithm for the constrained versionhas significant performance degradation on both a syntheticmodel of patterned data and on real world data sets.running the algorithm on a small random sample is proposedas a means of obtaining near optimal results withhigh probability. theoretical bounds on sufficient samplesize to achieve a given performance level are proved, andrapid convergence on synthetic and real-world data is validatedexperimentally.
matching in frequent tree discovery. various definitions and frameworks for discovering frequent trees in forests have been developed recently. at the heart of these frameworks lies the notion of matching, which determines when a pattern tree matches a tree in a data set. we introduce a novel notion of tree matching for use in frequent tree mining and we show that it generalizes the framework of zaki while still being more specific than that of termier et al. furthermore, we show how zaki's treeminerv algorithm can be adapted towards our notion of tree matching. experiments show the promise of the approach.
warp: time warping for periodicity detection. periodicity mining is used for predicting trends in time series data. periodicity detection is an essential process in periodicity mining to discover potential periodicity rates. existing periodicity detection algorithms do not take into account the presence of noise, which is inevitable in almost every real-world time series data. in this paper, we tackle the problem of periodicity detection in the presence of noise. we propose a new periodicity detection algorithm that deals efficiently with all types of noise. based on time warping, the proposed algorithm warps (extends or shrinks) the time axis at various locations to optimally remove the noise. experimental results show that the proposed algorithm out-performs the existing periodicity detection algorithms in terms of noise resiliency.
segment-based injection attacks against collaborative filtering recommender systems. significant vulnerabilities have recently been identi- fied in collaborative filtering recommender systems. researchers have shown that attackers can manipulate a system's recommendations by injecting biased profiles into it. in this paper, we examine attacks that concentrate on a targeted set of users with similar tastes, biasing the system's responses to these users. we show that such attacks are both pragmatically reasonable and also highly effective against both user-based and item-based algorithms. as a result, an attacker can mount such a "segmented" attack with little knowledge of the specific system being targeted and with strong likelihood of success.
summarization - compressing data into an informative representation. in this paper, we formulate the problem of summarization of a dataset of transactions with categorical attributes as an optimization problem involving two objective functions - compaction gain and information loss. we propose metrics to characterize the output of any summarization algorithm. we investigate two approaches to address this problem. the first approach is an adaptation of clustering and the second approach makes use of frequent itemsets from the association analysis domain. we illustrate one application of summarization in the field of network data where we show how our technique can be effectively used to summarize network traffic into a compact but meaningful representation. specifically, we evaluate our proposed algorithms on the 1998 darpa off-line intrusion detection evaluation data and network data generated by skaion corp for the arda information assurance program.
preprocessing opportunities in optimal numerical range partitioning. we show that only the segment borders have to be taken into account as cut point candidates in searching for theoptimal multisplit of a numerical value range with respect to convex attribute evaluation functions. segment borders can be found efficiently in a linear-time preprocessing step. with training set error, which is not strictly convex, the data can be preprocessed into an even smaller number of cut point candidates, called alternations, when striving for the optimal partition. we show that no segment borders(resp. alternations) can be overlooked with strictly convex functions (resp. training set error) without risking to lose optimality. our experiments show that while in real-world domainssignificant reduction in the number of cut point candidates can be obtained for training set error, the number of segment borders is usually not much lower than that of boundary points.
mining general temporal association rules for items with different exhibition periods. in this paper, we explore a new model of mining generaltemporal association rules from large databases wherethe exhibition periods of the items are allowed to be differentfrom one to another. note that in this new model,the downward closure property which all prior apriori-basedalgorithms relied upon to attain good efficiency isno longer valid. as a result, how to efficiently generatecandidate itemsets form large databases has become themajor challenge. to address this issue, we develop an efficientalgorithm, referred to as algorithm spf (standingfor segmented progressive filter) in this paper. the basicidea behind spf is to first segment the database into sub-databasesin such a way that items in each sub-databasewill have either the common starting time or the commonending time. then, for each sub-database, spf progressivelyfilters candidate 2-itemsets with cumulative filteringthresholds either forward or backward in time. this featureallows spf of adopting the scan reduction techniqueby generating all candidate k-itemsets (k >2) from candidate2-itemsets directly. the experimental results show thatalgorithm spf significantly outperforms other schemeswhich are extended from prior methods in terms of the executiontime and scalability.
an improved categorization of classifier's sensitivity on sample selection bias. a recent paper categorizes classifier learning algorithms according to their sensitivity to a common type of sample selection bias where the chance of an example being selected into the training sample depends on its feature vector x but not (directly) on its class label y. a classifier learner is categorized as "local" if it is insensitive to this type of sample selection bias, otherwise, it is considered "global". in that paper, the true model is not clearly distinguished from the model that the algorithm outputs. in their discussion of bayesian classifiers, logistic regression and hard-margin svms, the true model (or the model that generates the true class label for every example) is implicitly assumed to be contained in the model space of the learner, and the true class probabilities and model estimated class probabilities are assumed to asymptotically converge as the training data set size increases. however, in the discussion of naive bayes, decision trees and soft-margin svms, the model space is assumed not to contain the true model, and these three algorithms are instead argued to be "global learners". we argue that most classifier learners may or may not be affected by sample selection bias; this depends on the dataset as well as the heuristics or inductive bias implied by the learning algorithm and their appropriateness to the particular dataset.
creating ensembles of classifiers. ensembles of classifiers offer promise in increasing overall classification accuracy. the availability of extremely large datasets has opened avenues for application of distributed and/or parallel learning to efficiently learn models of them. in this paper, distributed learningis done by training classifiers on disjoint subsets of the data. we examine a random partitioning method to create disjoint subsets and propose a more intelligent way of partitioning into disjointsubsets using clustering. it was observed that the intelligent method of partitioning generally performs better than random partitioning for our datasets. in both methods a significant gain in accuracy may be obtained by applying bagging to each of the disjoint subsets, creating multiple diverse classifiers. the significance of our finding is that a partition strategy for even small/moderate sized datasets when combined with bagging can yield better performancethan applying a single learner using the entire dataset.
effective estimation of posterior probabilities: explaining the accuracy of randomized decision tree approaches. there has been increasing number of independently proposed randomization methods in different stages of decision tree construction to build multiple trees. randomized decision tree methods have been reported to be significantly more accurate than widely-accepted single decision trees, although the training procedure of some methods incorporates a surprisingly random factor and therefore opposes the generally accepted idea of employing gain functions to choose optimum features at each node and compute a single tree that fits the data. one important question that is not well understood yet is the reason behind the high accuracy. we provide an insight based on posterior probability estimations. we first establish the relationship between effective posterior probability estimation and effective loss reduction. we argue that randomized decision tree methods effectively approximate the true probability distribution using the decision tree hypothesis space. we conduct experiments using both synthetic and real-world datasets under both 0-1 and cost-sensitive loss functions.
making subsequence time series clustering meaningful. recently, the startling claim was made that sequential time series clustering is meaningless. this has important consequences for a significant amount of work in the literature, since such a claim invalidates this work's contribution. in this paper, we show that sequential time series clustering is not meaningless, and that the problem highlighted in these works stem from their use of the euclidean distance metric as the distance measure in the subsequence vector space. as a solution, we consider quite a general class of time series, and propose a regime based on two types of similarity that can exist between subsequence vectors, which give rise naturally to an alternative distance measure to euclidean distance in the subsequence vector space. we show that, using this alternative distance measure, sequential time series clustering can indeed be meaningful. we repeat a key experiment in the work on which the "meaningless" claim was based, and show that our method leads to a successful clustering outcome.
decision tree evolution using limited number of labeled data items from drifting data streams. most previously proposed mining methods on data streams make an unrealistic assumption that "labelled" data stream is readily available and can be mined at anytime. however, in most real-world problems, labelled data streams are rarely immediately available. due to this reason, models are reconstructed only when labelled data become available periodically. this passive stream mining model has several drawbacks. we propose a new concept of demand-driven active data mining. in active mining, the loss of the model is either continuously guessed without using any true class labels or estimated, whenever necessary, from a small number of instances whose actual class labels are verified by paying an affordable cost. when the estimated loss is more than a tolerable threshold, the model evolves by using a small number of instances with verified true class labels. previous work on active mining concentrates on error guess and estimation. in this paper, we discuss several approaches on decision tree evolution.
a computational framework for taxonomic research: diagnosing body shape within fish species complexes. it is estimated that ninety percent of the world's species have yet to be discovered and described. the main reason for the slow pace of new species description is that the science of taxonomy, as traditionally practiced, can be very laborious. to formally describe a new species, taxonomists have to manually gather and analyze data from large numbers of specimens, often from broad geographic areas, and identify the smallest subset of external body characters that uniquely diagnoses the new species as distinct from all its known relatives. in this paper, we use an automated feature selection and classification approach to address the taxonomic impediment in new species discovery. the experiments on a taxonomic problem involving species of suckers in the genus carpiodes demonstrate promising results.
using artificial anomalies to detect unknown and known network intrusions. intrusion detection systems (idss) must be capable of detecting new and unknown attacks, or anomalies. we study the problem of building detection models for both pure anomaly detection and combined misuse and anomaly detection (i.e., detection of both known and unknown intrusions). we show the necessity of artificial anomalies by discussing the failure to use conventional inductive learning methods to detect anomalies. we propose an algorithm to generate artificial anomalies to coerce the inductive learner into discovering an accurate boundary between known classes (normal connections and known intrusions) and anomalies. empirical studies show that our pure anomaly-detection model trained using normal and artificial anomalies is capable of detecting more than 77% of all unknown intrusion classes with more than 50% accuracy per intrusion class. the combined misuse and anomaly-detection models are as accurate as a pure misuse detection model in detecting known intrusions and are capable of detecting at least 50% of unknown intrusion classes with accuracy measurements between 75 and 100% per class.
labeling unclustered categorical data into clusters based on the important attribute values. sampling has been recognized as an important technique to improve the efficiency of clustering. however, with sampling applied, those points which are not sampled will not have their labels. although there is a straightforward approach in the numerical domain, the problem of how to allocate those unlabeled data points into proper clusters remains as a challenging issue in the categorical domain. in this paper, a mechanism named maximal resemblance data labeling (abbreviated as mardl) is proposed to allocate each unlabeled data point into the corresponding appropriate cluster based on the novel categorical clustering representative, namely, node importance representative(abbreviated as nir), which represents clusters by the importance of attribute values. mardl has two advantages: (1) mardl exhibits high execution efficiency; (2) after each unlabeled data is allocated into the proper cluster, mardl preserves clustering characteristics, i.e., high intra-cluster similarity and low inter-cluster similarity. mardl is empirically validated via real and synthetic data sets, and is shown to be not only more efficient than prior methods but also attaining results of better quality.
progressive modeling. presently, inductive learning is still performed in a frustratingbatch process. the user has little interaction withthe system and no control over the final accuracy and trainingtime. if the accuracy of the produced model is too low,all the computing resources are misspent. in this paper, wepropose a progressive modeling framework. in progressivemodeling, the learning algorithm estimates online both theaccuracy of the final model and remaining training time. ifthe estimated accuracy is far below expectation, the usercan terminate training prior to completion without wastingfurther resources. if the user chooses to complete the learningprocess, progressive modeling will compute a modelwith expected accuracy in expected time. we describe oneimplementation of progressive modeling using ensemble ofclassifiers.
icon-based visualization of large high-dimensional datasets. high dimensional data visualization is critical todata analysts since it gives a direct view of originaldata. we present a method to visualize large amount ofhigh dimensional data. we divide dimensions of datainto several groups. then, we use one icon to represent each group, and associate visual properties of eachicon with dimensions in each group. a high dimensional data record will be represented by multiple different types of icons located in the same position. furthermore, we use summary icons to display local detailsof viewer's interests and the whole data set at meantime. we show its effectiveness and efficiency through a case study on a real large data set.
is random model better? on its accuracy and efficiency. inductive learning searches an optimal hypothesis thatminimizes a given loss function. it is usually assumed thatthe simplest hypothesis that fits the data is the best approximateto an optimal hypothesis. since finding the simplesthypothesis is np-hard for most representations, we generallyemploy various heuristics to search its closest match.computing these heuristics incurs significant cost, makinglearning inefficient and unscalable for large dataset. in thesame time, it is still questionable if the simplest hypothesisis indeed the closest approximate to the optimal model.recent success of combining multiple models, such as bagging,boosting and meta-learning, has greatly improved theaccuracy of the simplest hypothesis, providing a strong argumentagainst the optimality of the simplest hypothesis.however, computing these combined hypotheses incurs significantlyhigher cost. in this paper, we first advert that aslong as the error of a hypothesis on each example is withina range dictated by a given loss function, it can still be optimal.contrary to common beliefs, we propose a completelyrandom decision tree algorithm that achieves much higheraccuracy than the single best hypothesis and is comparableto boosted or bagged multiple best hypotheses. the advantageof multiple random tree is its training efficiency aswell as minimal memory requirement.
validating and refining clusters via visual rendering. the automatic clustering algorithms are known towork well in dealing with clusters of regular shapes, e.g.compact spherical/elongated shapes, but may incur highererror rates when dealing with arbitrarily shaped clusters.although some efforts have been devoted to addressingthe problem of skewed datasets, the problem of handlingclusters with irregular shapes is still in its infancy,especially in terms of dimensionality of the datasets andthe precision of the clustering results considered. notsurprisingly, the statistical indices works ineffective invalidating clusters of irregular shapes, too. in this paper,we address the problem of clustering and validatingarbitrarily shaped clusters with a visual framework(vista). the main idea of the vista approach is tocapitalize on the power of visualization and interactivefeedbacks to encourage domain experts to participate inthe clustering revision and clustering validation process.
using rule sets to maximize roc performance. rules are commonly use for classification because they are modular, intelligible and easy to learn. existing work in classification rule learning assumes the goal is to produce categorical classifications to maximize classification accuracy. recent work in machine learning has pointed out the limitations of classification accuracy: when class distributions are skewed, or error costs are unequal, an accuracy maximizing rule set can perform poorly. amore flexible use of a rule set is to produce instance scores indicating the likelihood that an instance belongs to a given class. with such an ability, we can apply rulesets effectively whendistributions are skewed or error costs are unequal. this paper empirically investigates different strategies for evaluating rule sets when the goal is to maximize the scoring (roc)performance.