waashk/dblp huggingface dataset:
detecting patterns of change using enhanced parallel coordinates visualization. analyzing data to find trends, correlations, and stablepatterns is an important problem for many industrialapplications. in this paper, we propose a new techniquebased on parallel coordinates visualization. previous workon parallel coordinates methods has shown that they areeffective only when variables that are correlated and/orshow similar patterns are displayed adjacently. althoughcurrent parallel coordinates tools allow the user tomanually rearrange the order of variables, this process isvery time-consuming when the number of variables islarge. automated assistance is needed. this paperproposes an edit-distance based technique to rearrangevariables so that interesting patterns can be easilydetected. our system, v-miner, includes both automatedmethods for visualizing common patterns and a query toolthat enables the user to describe specific target patterns tobe mined/displayed by the system. following an overviewof the system, a case study is presented to explain howmotorola engineers have used v-miner to identifysignificant patterns in their product test and design data.
frequent-pattern based iterative projected clustering. irrelevant attributes add noise to high dimensional clustersand make traditional clustering techniques inappropriate.projected clustering algorithms have been proposed to findthe clusters in hidden subspaces. we realize the analogy betweenmining frequent itemsets and discovering the relevantsubspace for a given cluster. we propose a methodology forfinding projected clusters by mining frequent itemsets andpresent heuristics that improve its quality. our techniquesare evaluated with synthetic and real data; they are scalableand discover projected clusters accurately.
a visual data mining framework for convenient identification of useful knowledge. data mining algorithms usually generate a large number of rules, which may not always be useful to human users. in this project, we propose a novel visual data-mining framework, called opportunity map, to identify useful and actionable knowledge quickly and easily from the discovered rules. the framework is inspired by the house of quality from quality function deployment (qfd) in quality engineering. it associates discovered rules, related summarized data and data distributions with the application objective using an interactive matrix. combined with drill down visualization, integrated visualization of data distribution bars and rules, visualization of trend behaviors, and comparative analysis, the opportunity map allows users to analyze rules and data at different levels of detail and quickly identify the actionable knowledge and opportunities. the proposed framework represents a systematic and flexible approach to rule analysis. applications of the system to large-scale data sets from our industrial partner have yielded promising results.
demand forecasting by the neural network with discrete fourier transform. this paper proposes a new demand forecastingmethod using the neural network and fourier transform.in this method, time series data of sales resultsconsidered as a combination of frequency aretransformed into several frequency data. they areidentified from objective indexes that consist of productproperties or economic indicators and so forth. thismethod is efficient for demand forecasting aimed at newproducts that have no historical data.
adaptive parallel sentences mining from web bilingual news collection. in this paper a robust, adaptive approach for miningparallel sentences from a bilingual comparable newscollection is described. sentence length models andlexicon-based models are combined under a maximumlikelihood criterion. specific models are proposed to handleinsertions and deletions that are frequent in bilingualdata collected from the web. the proposed approach isadaptive, updating the translation lexicon iteratively usingthe mined parallel data to get better vocabulary coverageand translation probability parameter estimation.experiments are carried out on 10 years of xinhuabilingual news collection. using the mined data, we getsignificant improvement in word-to-word alignment accuracyin machine translation modeling.
a join-less approach for co-location pattern mining: a summary of results. spatial co-location patterns represent the subsets of features whose instances are frequently located together in geographic space. co-location pattern discovery presents challenges since the instances of spatial features are embedded in a continuous space and share a variety of spatial relationships. a large fraction of the computation time is devoted to identifying the instances of co-location patterns. we propose a novel join-less approach for co-location pattern mining, which materializes spatial neighbor relationships with no loss of co-location instances and reduces the computational cost of identifying the instances. the join-less co-location mining algorithm is efficient since it uses an instance-lookup scheme instead of an expensive spatial or instance join operation for identifying co-location instances. the experimental evaluations show the join-less algorithm performs more efficiently than a current join-based algorithm and is scalable in dense spatial datasets.
on active learning for data acquisition. many applications are characterized by having naturallyincomplete data on customers - where data on only somefixed set of local variables is gathered. however, having amore complete picture can help build better models. thenaïve solution to this problem - acquiring complete datafor all customers - is often impractical due to the costs ofdoing so. a possible alternative is to acquire completedata for "some" customers and to use this to improve themodels built. the data acquisition problem is determininghow many, and which, customers to acquire additionaldata from. in this paper we suggest using active learningbased approaches for the data acquisition problem. inparticular, we present initial methods for data acquisitionand evaluate these methods experimentally on web usagedata and uci datasets. results show that the methodsperform well and indicate that active learning basedmethods for data acquisition can be a promising area fordata mining research.
example-based robust outlier detection in high dimensional datasets. detecting outliers is an important problem. most of its applications typically possess high dimensional datasets. in high dimensional space, the data becomes sparse which implies that every object can be regarded as an outlier from the point of view of similarity. furthermore, a fundamental issue is that the notion of which objects are outliers typically varies between users, problem domains or, even, datasets. in this paper, we present a novel robust solution which detects high dimensional outliers based on user examples and tolerates incorrect inputs. it studies the behavior of projections of such a few examples, to discover further objects that are outstanding in the projection where many examples are outlying. our experiments on both real and synthetic datasets demonstrate the ability of the proposed method to detect outliers corresponding to the user examples.
general mc: estimating boundary of positive class from small positive data. single-class classification (scc) seeks to distinguishone class of data from the universal set of multiple classes.we propose a scc method called general mc that estimatesan accurate classification boundary of positive classfrom small positive data using the distribution of unlabeleddata. our theoretical and empirical analyses show that,as long as the distribution of unlabeled data is not highlyskewed in the feature space, general mc significantly outperformsother recent scc methods when the positive dataset is highly under-sampled.
a feature selection framework for text filtering. this paper presents a new framework for local featureselection in text filtering. in this framework, a feature setis constructed per category by first selecting a set of termshighly indicative of membership (positive set) and anotherset of terms highly indicative of non-membership (negativeset), and then combining these two sets. this feature selectionframework not only unifies several standard featureselection methods, but also facilitates the proposal of a newmethod that optimally combines the positive and negativesets. the experimental comparison between the proposedmethod and standard methods was conducted on six featureselection metrics: chi-square, correlation coefficient, oddsratio, gss coefficient and two proposed variants of odds ratioand gss coefficient: or-square and gss-square respectively.the results show that the proposed feature selectionmethod improves text filtering performance.
heterogeneous learner for web page classification. classification of an interesting class of web pages (e.g.,personal homepages, resume pages) has been an interestingproblem. typical machine learning algorithms for thisproblem require two classes of data for training: positiveand negative training examples. however, in applicationto web page classification, gathering an unbiased sampleof negative examples appears to be difficult. we proposea heterogeneous learning framework for classifying webpages, which (1) eliminates the need for negative trainingdata, and (2) increases classification accuracy by using twoheterogeneous learners. our framework uses two heterogeneouslearners - a decision list and a linear separatorwhich complement each other - to eliminate the need fornegative training data in the training phase and to increasethe accuracy in the testing phase. our results show that ourheterogeneous framework achieves high accuracy withoutrequiring negative training data; it enhances the accuracyof linear separators by reducing the errors on "low-margindata". that is, it classifies more accurately while requiringless human efforts in training.
relational peculiarity oriented data mining. peculiarity rules are a new type of interesting rules which can be discovered by searching the relevance among peculiar data. a main task of mining peculiarity rules is the identification of peculiarity. traditional methods of finding peculiar data are attribute-based approaches. this paper extends peculiarity oriented mining to relational peculiarity oriented mining. peculiar data are identified on record level, and peculiar rules are mined and explained in a relational mining framework. the results from preliminary experiments show that relational peculiarity oriented mining is very effective.
scalable construction of topic directory with nonparametric closed termset mining. a topic directory, e.g., yahoo directory, provides a view of a document set at different levelsof abstraction and is ideal for the interactive exploration and visualization of the document set. we present a method that dynamically generates a topic directory from a document set usinga frequent closed termset mining algorithm. our method shows experimental results of equal quality to recent document clustering methods and has additional benefits such as automatic generation of topic labels and determination of a clustering parameter.
the relationships among various nonnegative matrix factorization methods for clustering. the nonnegative matrix factorization (nmf) has been shown recently to be useful for clustering and various extensions and variations of nmf have been proposed recently. despite significant research progress in this area, few attempts have been made to establish the connections between various factorization methods while highlighting their differences. in this paper we aim to provide a comprehensive study on matrix factorization for clustering. in particular, we present an overview and summary on various matrix factorization algorithms and theoretically analyze the relationships among them. experiments are also conducted to empirically evaluate and compare various factorization methods. in addition, our study also answers several previously unaddressed yet important questions for matrix factorizations including the interpretation and normalization of cluster posterior and the benefits and evaluation of simultaneous clustering. we expect our study would provide good insights on matrix factorization research for clustering.
a k-nn associated fuzzy evidential reasoning classifier with adaptive neighbor selection. the paper presents a fuzzy evidential reasoning algorithmin light of the dempster-shafer evidence theory andthe k-nearest neighbor algorithm for pattern classification.given an input pattern to be classified, each of its k nearestneighbors is viewed as an evidence source, in terms ofa fuzzy evidence structure. the distance between the inputpattern and each of its k nearest neighbors is usedfor mass determination while the contextual information ofthe nearest neighbor in the training sample space is formulatedby a fuzzy set in determining a fuzzy focal element.therefore, pooling evidence provided by neighbors is realizedby a fuzzy evidential reasoning, where feature selectionis further considered through ranking and adaptive combinationof neighbors. a fast implementation scheme of thefuzzy evidential reasoning is also developed. experimentalresults of classifying multi-channel remote sensing imageshave shown that the proposed approach outperforms the k-nearestneighbor (k-nn) algorithm [1], the fuzzy k-nearestneighbor (f-knn) algorithm [2], the evidence-theoretic k-nearestneighbor (e-knn) algorithm [3], and the fuzzy ex-tendedversion of e-knn (fe-knn) [4], in terms of theclassification accuracy and insensitivity to the number kof nearest neighbors.
cost-guided class noise handling for effective cost-sensitive learning. recent research in machine learning, data mining and related areas has produced a wide variety of algorithms for cost-sensitive (cs) classification, where instead of maximizing the classification accuracy, minimizing the misclassification cost becomes the objective. however, these methods assume that training sets do not contain significant noise, which is rarely the case in real-world environments. in this paper, we systematically study the impacts of class noise on cs learning, and propose a cost-guided class noise handling algorithm to identify noise for effective cs learning. we call it cost-guided iterative classification filter (cicf), because it seamlessly integrates costs and an existing classification filter for noise identification. instead of putting equal weights to handle noise in all classes in existing efforts, cicf puts more emphasis on expensive classes, which makes it especially successful in dealing with datasets with a large cost-ratio. experimental results and comparative studies from real-world datasets indicate that the existence of noise may seriously corrupt the performance of cs classifiers, and by adopting the proposed cicf algorithm, we can significantly reduce the misclassification cost of a cs classifier in noisy environments.
dynamic classifier selection for effective mining from noisy data streams. recently, mining from data streams has become an important and challenging task for many real-world applications such as credit card fraud protection and sensor networking. one popular solution is to separate stream data into chunks, learn a base classifier from each chunk, and then integrate all base classifiers for effective classification. in this paper, we propose a new dynamic classifier selection (dcs) mechanism to integrate base classifiers for effective mining from data streams. the proposed algorithm dynamically selects a single "best" classifier to classify each test instance at run time. our scheme uses statistical information from attribute values, and uses each attribute to partition the evaluation set into disjoint subsets, followed by a procedure that evaluates the classification accuracy of each base classifier on these subsets. given a test instance, its attribute values determine the subsets that the similar instances in the evaluation set have constructed, and the classifier with the highest classification accuracy on those subsets is selected to classify the test instance. experimental results and comparative studies demonstrate the efficiency and efficacy of our method. such a dcs scheme appears to be promising in mining data streams with dramatic concept drifting or with a significant amount of noise, where the base classifiers are likely conflictive or have low confidence.
efficient text classification by weighted proximal svm. in this paper, we present an algorithm that can classify large-scale text data with high classification quality and fast training speed. our method is based on a novel extension of the proximal svm mode [3]. previous studies on proximal svm have focused on classification for low dimensional data and did not consider the unbalanced data cases. such methods will meet difficulties when classifying unbalanced and high dimensional data sets such as text documents. in this work, we extend the original proximal svm by learning a weight for each training error. we show that the classification algorithm based on this model is capable of handling high dimensional and unbalanced data. in the experiments, we compare our method with the original proximal svm (as a special case of our algorithm) and the standard svm (such as svm light) on the recently published rcv1-v2 dataset. the results show that our proposed method had comparable classification quality with the standard svm. at the same time, both the time and memory consumption of our method are less than that of the standard svm.
ctc - correlating tree patterns for classification. we present ctc, a new approach to structural classification. it uses the predictive power of tree patterns correlating with the class values, combining state-of-the-art tree mining with sophisticated pruning techniques to find the k most discriminative pattern in a dataset. in contrast to existing methods, ctc uses no heuristics and the only parameters to be chosen by the user are the maximum size of the rule set and a single, statistically well founded cut-off value. the experiments show that ctc classifiers achieve good accuracies while the induced models are smaller than those of existing approaches, facilitating comprehensibility.
a pattern decomposition (pd) algorithm for finding all frequent patterns in large datasets. efficient algorithms to mine frequent patterns are crucial to many tasks in data mining. since the apriori algorithm was proposed in 1994, there have been several methods proposed to improve its performance. however, most still adopt its candidate set generation-and-testapproach. we propose a pattern decomposition (pd) algorithm that can significantly reduce the size of the dataset on each pass making it more efficient to mine frequent patterns in a large dataset. the proposed algorithm avoids the costly process of candidate set generation and saves time by reducing dataset. our empirical evaluation shows that the algorithmoutperforms apriori by one order of magnitude and is faster than fp-tree. further, pd is more scalable than both apriori and fp-tree.
smartminer: a depth first algorithm guided by tail information for mining maximal frequent itemsets. maximal frequent itemsets (mfi) are crucial to manytasks in data mining. since the maxminer algorithm firstintroduced enumeration trees for mining mfi in 1998,several methods have been proposed to use depth firstsearch to improve performance. to further improve theperformance of mining mfi, we proposed a techniquethat takes advantage of the information gathered fromprevious steps to discover new mfi. more specifically,our algorithm called smartminer gathers and passes tailinformation and uses a heuristic select function whichuses the tail information to select the next node toexplore. compared with mafia and genmax, smartminergenerates a smaller search tree, requires a smallernumber of support counting, and does not requiresuperset checking. using the datasets mushroom andconnect, our experimental study reveals that smartminergenerates the same mfi as mafia and genmax, but yieldsan order of magnitude improvement in speed.
hierarchical classification by expected utility maximization. hierarchical classification refers to an extension of the standard classification problem, in which labels must be chosen from a class hierarchy. in this paper, we look at hierarchical classification from an information retrieval point of view. more specifically, we consider a scenario in which a user searches a document in a topic hierarchy. this scenario gives rise to the problem of predicting an optimal entry point, that is, a topic node in which the user starts searching. the usefulness of a corresponding prediction strongly depends on the search behavior of the user, which becomes relevant if the document is not immediately found in the predicted node. typically, users tend to browse the hierarchy in a top-down manner, i.e., they look at a few more specific subcategories but usually refuse exploring completely different branches of the search tree. from a classification point of view, this means that a prediction should be evaluated, not solely on the basis of its correctness, but rather by judging its usefulness against the background of the user behavior. the idea of this paper is to formalize hierarchical classification within a decision-theoretic framework which allows for modeling this usefulness in terms of a user-specific utility function. the prediction problem thus becomes a problem of expected utility maximization. apart from its theoretical appeal, we provide first empirical results showing that the approach performs well in practice.
regularized least absolute deviations regression and an efficient algorithm for parameter tuning. linear regression is one of the most important and widely used techniques for data analysis. however, sometimes people are not satisfied with it because of the following two limitations: 1) its results are sensitive to outliers, so when the error terms are not normally distributed, especially when they have heavy-tailed distributions, linear regression often works badly; 2) its estimated coefficients tend to have high variance, although their bias is low. to reduce the influence of outliers, robust regression models were developed. least absolute deviation (lad) regression is one of them. lad minimizes the mean absolute errors, instead of mean squared errors, so its results are more robust. to address the second limitation, shrinkage methods were proposed, which add a penalty on the size of the coefficients. the lasso is one of these methods and it uses the l1-norm penalty, which not only reduces the prediction error and the variance of estimated coefficients, but also provides an automatic feature selection function. in this paper, we propose the regularized least absolute deviation (rlad) regression model, which combines the nice features of the lad and the lasso together. the rlad is a regularization method, whose objective function has the form of "loss + penalty." the "loss" is the sum of the absolute deviations and the "penalty" is the l1-norm of the coefficient vector. furthermore, to facilitate parameter tuning, we develop an efficient algorithm which can solve the entire regularization path in one pass. simulations with various settings are performed to demonstrate its performance. finally, we apply the algorithm to solve the image reconstruction problem and find interesting results.
cominer: an effective algorithm for mining competitors from the web. this paper attempts to accomplish a novel task of mining competitive information with respect to an entity (such as a company, product, person) from the web. an algorithm called "cominer" is proposed, which first extracts a set of comparative candidates of the input entity and then ranks them according to the comparability, and finally extracts the competitive fields. the experimental results show that the proposed algorithm drafts a complete picture of competitive relation of a given entity effectively.