waashk/dblp huggingface dataset:
a self-organizing map with expanding force for data clustering and visualization. the self-organizing map (som) is a powerful tool in theexploratory phase of data mining. however, due to the dimensional conflict, the neighborhood preservation cannotalways lead to perfect topology preservation. in this paper, we establish an expanding som (esom) to detect andpreserve better topology correspondence between the twospaces. our experiment results demonstrate that the esomconstructs better mappings than the classic som in terms ofboth the topological and the quantization errors. furthermore, clustering results generated by the esom are moreaccurate than those by the som.
learning functional dependency networks based on genetic programming. bayesian network (bn) is a powerful network model, which represents a set of variables in the domain and provides the probabilistic relationships among them. but bn can handle discrete values only; it cannot handle continuous, interval and ordinal ones, which must be converted to discrete values and the order information is lost. thus, bn tends to have higher network complexity and lower understandability. in this paper, we present a novel dependency network which can handle discrete, continuous, interval and ordinal values through functions; it has lower network complexity and stronger expressive power; it can represent any kind of relationships; and it can incorporate a-priori knowledge though user-defined functions. we also propose a novel genetic programming (gp) to learn dependency networks. the novel gp does not use any knowledge-guided nor application-oriented operator, thus it is robust and easy to replicate. the experimental results demonstrate that the novel gp can successfully discover the target novel dependency networks, which have the highest accuracy and the lowest network complexity.
instability of classifiers on categorical data. in this paper we study the local behaviour of arbitrary classifiers using the instability of that classifier in a data point. moreover, we introduce two algorithms. the first to find highly unstable points, the second to find islands of stability.
mining patterns of change in remote sensing image databases. remote sensing image databases are the fastest growing archives of spatial information. however, we still have a limited capacity for extracting information from large remote sensing image databases. there are currently very few techniques for image data mining and information extraction in large image data sets, and thus we are failing to exploit our large remote sensing data archives. this paper proposes a methodology to provide guidance for mining remote sensing image databases. the basic idea is to use domain concepts to build generic description of patterns in remote sensing images, and then use structural approaches to identify such patterns in images. we illustrate our proposal with a case study for detecting land use patterns in amazonia from inpe's remote sensing image database.
document clustering and cluster topic extraction in multilingual corpora. a statistics-based approach for clustering documents and for extracting cluster topics is described. relevant (meaningful) expressions (res) automatically extracted from corpora are used as clustering base features. these features are transformed and its number is strongly reduced in order to obtain a small set of document classificationfeatures. this is achieved on the basis of principalcomponents analysis. model-based clustering analysis finds thebest number of clusters. then, the most important res are extracted from each cluster and taken as document cluster topics.
metric incremental clustering of nominal data. we present an algorithm for clustering nominal data that is based on a metric on the set of partitions of a finite set of objects; this metric is defined starting from a lower valuation of the lattice of partitions. the proposed algorithm seeks to determine a clustering partition such that the total distance between this partition and the partitions determined by the attributes of the objects has a local minimum. the resulting clustering is quite stable relative to the ordering of the objects.
pruning social networks using structural properties and descriptive attributes. scale is often an issue with understanding and making sense of large social networks. here we investigate methods for pruning social networks by determining the most relevant relationships. we measure importance in terms of predictive accuracy on a set of target attributes of the social network. our goal is to create a pruned network that models only the most informative affiliations and relationships. we present methods for pruning networks based on both structural properties and descriptive attributes demonstrate it on a network of nasdaq and nyse businesses and on a bibliographic network.
closeminer: discovering frequent closed itemsets using frequent closed tidsets. complete set of itemsets can be grouped into non-overlapping clusters identified by closed tidsets. each cluster has only one closed itemset and is the superset of all itemsets with the same support. number of closed itemsets is identical to the number of clusters. therefore, the problem of discovering closed itemsets can be considered as the problem of clustering the complete set of itemsets by closed tidsets. in this paper, we present closeminer, a new algorithm for discovering all frequent closed itemsets by grouping the complete set of itemsets into non-overlapping clusters identified by closed tidsets. an extensive experimental evaluation on a number of real and synthetic databases shows that closeminer outperforms apriori and charm.
a simple knn algorithm for text categorization. text categoriztion (also called text classification) is the process of identifying the class to which a text document belongs. this paper proposes to use a simple non-weighted feature knn algorithm for text caegoriztion. we propose to use a feature selection method that finds the relevant features for the learning task at hand using feature interaction (based on word interdependencies).
optimizing constraint-based mining by automatically relaxing constraints. in constraint-based mining, the monotone and anti-monotone properties are exploited to reduce the search space. even if a constraint has not such suitable properties, existing algorithms can be re-used thanks to an approximation, called relaxation. in this paper, we automatically compute monotone relaxations of primitive-based constraints. first, we show that the latter are a superclass of combinations of both kinds of monotone constraints. second, we add two operators to detect the properties of monotonicity of such constraints. finally, we define relaxing operators to obtain monotone relaxations of them.
probabilistic principal surfaces for yeast gene microarray data mining. the recent technological advances are producing huge data sets in almost all fields of scientific research, from astronomy to genetics. although each research field often requires ad-hoc, fine tuned, procedures to properly exploit all the available information inherently present in the data, there is an urgent need for a new generation of general computational theories and tools capable to boost most human activities of data analysis. here we propose probabilistic principal surfaces (pps) as an effective high-d data visualization and clustering tool for data mining applications, emphasizing its flexibility and generality of use in data-rich field. in order to better illustrate the potentialities of the method, we also provide a real world case-study by discussing the use of pps for the analysis of yeast gene expression levels from microarray chips.
generalizing the notion of confidence. in this paper, we explore extending association analysis to non-traditional types of patterns and non-binary data by generalizing the notion of confidence. the key idea is to regard confidence as a measure of the extent to which the strength of one association pattern provides information about the strength of another. this approach provides a framework that encompasses the traditional concept of confidence as a special case and can be used as the basis for designing a variety of new confidence measures. besides discussing such confidence measures, we provide examples that illustrate the potential usefulness of a generalized notion of confidence. in particular, we describe an approach to defining confidence for error tolerant itemsets that preserves the interpretation of confidence as a conditional probability and derive a confidence measure for continuous data that agrees with the standard confidence measure when applied to binary transaction data.
measuring real-time predictive models. in this paper we examine the problem of comparing real-time predictive models and propose a number of measures for selecting the best model, based on a combination of accuracy, timeliness, and cost. we apply the measure to the real-time attrition problem.
svm feature selection for classification of spect images of alzheimer's disease using spatial information. alzheimer's disease is the most frequent type of dementia for elderly patients. due to aging populations the occurrence of this disease will increase in the next years. early diagnosis is crucial to be able to develop more powerful treatments. brain perfusion changes can be a marker for alzheimer's disease. in this article we study the use of spect perfusion imaging for the diagnosis of alzheimer's disease differentiating between images from healthy subjects and images from alzheimer's disease patients. our classification approach is based on a linear programming formulation similar to the 1-norm support vector machines. in contrastwith other linear hyperplane-based methods that perform simultaneous feature selection and classification, our proposed formulation incorporates proximity information about the features and generates a classifier that does not just select the most relevant voxels but the most relevant "areas" for classification resulting in more robust classifiersthat are better suitable for interpretation. this approach is compared with the classical fisher linear discriminant (fld) classifier as well as with statistical parametric mapping (spm). we tested our method on data from four european institutions. our method achieved sensitivity of 84.4% at 90.9% specificity, this is considerable better the human experts. our method also outperformed the fld and spm techniques. we conclude that our approach has the potential to be a useful help for clinicians.
evolutionary gabor filter optimization with application to vehicle detection. despite the considerable amount of research work on the applicationof gabor filters in pattern classification, their design and selectionhave been mostly done on a trial and error basis. existing techniques areeither only suitable for a small number of filters or less problem-oriented.a systematic and general evolutionary gabor filter optimization (egfo)approach that yields a more optimal, problem-specific, set of filters is proposedin this study. the egfo approach unifies filter design with filter selectionby integrating genetic algorithms (gas) with an incremental clusteringapproach. specifically, filter design is performed using gas, a globaloptimization approach that encodes the parameters of the gabor filters ina chromosome and uses genetic operators to optimize them. filter selectionis performed by grouping together filters having similar characteristics(i.e., similar parameters) using incremental clustering in the parameterspace. each group of filters is represented by a single filter whose parameterscorrespond to the average parameters of the filters in the group. thisstep eliminates redundant filters, leading to a compact, optimized set of filters.the average filters are evaluated using an application-oriented fitnesscriterion based on support vector machines (svms). to demonstrate theeffectiveness of the proposed framework, we have considered the challengingproblem of vehicle detection from gray-scale images. our experimentalresults illustrate that the set of gabor filters, specifically optimized for theproblem of vehicle detection, yield better performance than using traditionalfilter banks.
on local spatial outliers. we propose a measure, spatial local outlier measure (slom) which captures the local behaviour of datum in their spatial neighborhood. with the help of slom we are able to discern local spatial outliers which are usually missed by global techniques like "three standard deviations away from the mean". furthermore the measure takes into account the local stability around a data point and supresses the reporting of outliers in highly unstable areas, where data is too heterogeneous and the notion of outliers is not meaningful. we prove several properties of slom and report experiments on synthetic and real data sets which show that our approach is novel and scalable to large data sets.
supervised latent semantic indexing for document categorization. latent semantic indexing (lsi) is a successful technology in information retrieval (ir) which attempts to explore the latent semantics implied by a query or a document through representing them in a dimension-reduced space. however, lsi is not optimal for document categorization tasks because it aims to find the most representative features for document representation rather than the most discriminative ones. in this paper, we propose supervised lsi (slsi) which selects the most discriminative basis vectors using the training data iteratively. the extracted vectors are then used to project the documents into a reduced dimensional space for better classification. experimental evaluations show that the slsi approach leads to dramatic dimension reduction while achieving good classification results.
hierarchical text classification and evaluation. hierarchical classification refers to assigning of one or more suitable categories from a hierarchical category space to a document. while previous work in hierarchical classification focused on virtual category trees where documents are assigned only to the leaf categories, we propose atop-down level-based classification method that can classify documents to both leaf and internal categories. as the standard performance measures assume independence between categories, they have not considered the documents incorrectly classified into categories that are similar or not far from the correct ones in the category tree. we therefore propose the category-similarity measures and distance-based measures to consider the degree of misclassification in measuring the classification performance. an experiment has been carried out to measure the performance four proposed hierarchical classification method. the results showed that our method performs well for reuters text collection when enough training documents are given andthe new measures have indeed considered the contributions of misclassified documents.
introducing uncertainty into pattern discovery in temporal event sequences. pattern discovery in temporal event sequences is of greatimportance in many application domains, such as telecommunicationnetwork fault analysis. in reality, not every typeof event has an accurate timestamp. some of them, definedas inaccurate events in this paper, may only have an intervalas possible time of occurrence. the existence of inaccurateevents may cause uncertainty in event ordering. thetraditional support model cannot deal with this uncertainty,which would cause some interesting patterns to be missing.in this paper, a new concept, precise support, is introducedto evaluate the probability of a pattern contained in a sequence.based on this new metric, we define the uncertaintymodel and present an algorithm to discover interesting patternsin the sequence database that has one type of inaccurateevent. in our model, the number of types of inaccurateevents can be extended to k readily, however, at a cost ofincreasing computational complexity.
neighborhood formation and anomaly detection in bipartite graphs. many real applications can be modeled using bipartite graphs, such as users vs. files in a p2p system, traders vs. stocks in a financial trading system, conferences vs. authors in a scientific publication network, and so on. we introduce two operations on bipartite graphs: 1) identifying similar nodes (neighborhood formation), and 2) finding abnormal nodes (anomaly detection). and we propose algorithms to compute the neighborhood for each node using random walk with restarts and graph partitioning; we also propose algorithms to identify abnormal nodes, using neighborhood information. we evaluate the quality of neighborhoods based on semantics of the datasets, and we also measure the performance of the anomaly detection algorithm with manually injected anomalies. both effectiveness and efficiency of the methods are confirmed by experiments on several real datasets.
a border-based approach for hiding sensitive frequent itemsets. sharing data among organizations often leads to mutual benefit. recent technology in data mining has enabled efficientextraction of knowledge from large databases. this, however, increases risks of disclosing the sensitive knowledge when the database is released to other parties. to address this privacy issue, one may sanitize the original database so that the sensitive knowledge is hidden. the challenge is to minimize the side effect on the quality of the sanitized database so that non-sensitive knowledge can still be mined. in this paper, we study such a problem in the context of hiding sensitive frequent itemsets by judiciously modifying the transactions in the database. to preserve the non-sensitive frequent itemsets, we propose a border-based approach to efficiently evaluate the impact of any modification to the database during the hiding process. the quality of database can be well maintained by greedily selecting the modifications with minimal side effect. experiments results are also reported to show the effectiveness of the proposed approach.
detecting interesting exceptions from medical test data with visual summarization. in this paper, we propose a method which visualizes irregularmulti-dimensional time-series data as a sequence ofprobabilistic prototypes for detecting exceptions from medicaltest data. conventional visualization methods often requireiterative analysis and considerable skill thus are nottotally supported by a wide range of medical experts. ourprototypelines displays summarized information based ona probabilistic mixture model by using hue only thus is consideredto exhibit novelty. the effectiveness of the summarizationis pursued mainly through use of a novel informationcriterion. we report our endeavor with chronic hepatitisdata, especially discoveries of interesting exceptions bya non-expert and an untrained expert.
x-mhmm: an efficient algorithm for training mixtures of hmms when the number of mixtures is unknown. in this paper we consider sequence clustering problems and propose an algorithm for the estimation of the number of clusters based on the x-means algorithm. the sequences are modeled using mixtures of hidden markov models. by means of experiments with synthetic data we analyze the proposed algorithm. this algorithm proved to be both computationally efficient and capable of providing accurate estimates of the number of clusters. some results of experiments with real-world web-log data are also given.
a random walk through human associations. letting one's thoughts wander is not simply an arbitrary or rambling process. it can better be described as "associative thinking", where a complex chain of associative thoughts and ideas are linked. it is our contention that this seemingly chaotic process can be modeled by a random walk in a weighted directed graph. furthermore, is it possible to predict mathematically the "steady state" of such a process, to determine where such wandering is leading. the random walk process uses rules of association, defined by the local confidence gain (lcg) interestingness measure. extracted concepts are used as nodes of a directed graph. the associative "forces" between any two concepts (measured by lcg) are used to weigh the edges connecting the nodes that create a graph of associations. it is common, yet not trivial, for people to look for data about a subject without knowing its exact nomenclature (for example, finding the name of a disease just by knowing its symptoms). random walk in association graphs can discover highly informative phrases that can be used for query expansion in a way that better expresses the user's initial search goals. a different usage is to create a user profile representing his current interests. we used a modified version of the turing test to show that the random walk process discovers association rules that conform to a human associations generating process. by constructing the user associations we were able to build a profile representing the user's "line of thoughts". the suggested algorithm can be used in any database and can implement the ranking measures of other association rules.
mining the web to discover the meanings of an ambiguous word. in information retrieval and text mining, informationon word senses is usually taken from dictionaries or lexicaldatabases that have been prepared by lexicographers.in this paper we propose an automatic method for wordsense induction, i.e. for the discovery of a set of sensedescriptors to a given ambiguous word. the approach isbased on the statistics of word co-occurrence as derivedfrom web pages. the underlying assumption is that thesenses of an ambiguous word are best described by termsthat, although bearing a strong association to this word,are mutually exclusive, i.e. whose association strengthwithin the retrieved web pages is as weak as possible.measuring association strength is based upon a novelconfidence gain approach that relates the observed co-occurrencefrequency for two sense descriptor candidatesto an average co-occurrence frequency for pairs of arbitrarywords. the proposed approach is fully unsupervisedand takes into account the contemporary meanings ofwords, as reflected in texts from the internet. our resultsare evaluated using a list of ambiguous words commonlyreferred to in the literature.
bias analysis in text classification for highly skewed data. feature selection is often applied to high-dimensional data as a preprocessing step in text classification. when dealing with highly skewed data, we observe that typical feature selection metrics like information gain or chi-squared are biased toward selecting features for the minor class, and the metric of bi-normal separation can select features for both minor and major classes. in this work, we investigate how these feature selection metrics impact on the performance of frequently used classifiers such as decision trees, na¨ýve bayes, and support vector machines via bias analysis for highly skewed data. three types of biases are metric bias, class bias, and classifier bias. extensive experiments are designed to understand how these biases can be employed in concert and efficiently to achieve good classificationperformance. we report our findings and present recommended approaches to text classification based on bias analysis and the empirical study.