waashk/dblp huggingface dataset:
exploratory mining in cube space. data mining has evolved as a new discipline at the intersection of several existing areas, including database systems, machine learning, optimization, and statistics. an important question is whether the field has matured to the point where it has originated substantial new problems and techniques that distinguish it fromits parent disciplines. in this paper, we discuss a class of new problems and techniques that show great promise for exploratory mining, while synthesizing and generalizing ideas from the parent disciplines. while the class of problems we discuss is broad, there is a common underlying objective-to look beyond a single data mining step (e.g., data summarization or model construction) and address the combined process of data selection and transformation, parameter and algorithm selection, and model construction. the fundamental difficulty lies in the large space of alternative choices at each step, and good solutions must provide a natural framework for managing this complexity. we regard this as a grand challenge for datamining, and see the ideas in this paper as promising initial steps towards a rigorous exploratory framework that supports the entire process.
adaptive blocking: learning to scale up record linkage. many data mining tasks require computing similarity between pairs of objects. pairwise similarity computations are particularly important in record linkage systems, as well as in clustering and schema mapping algorithms. because the number of object pairs grows quadratically with the size of the dataset, computing similarity between all pairs is impractical and becomes prohibitive for large datasets and complex similarity functions. blocking methods alleviate this problem by efficiently selecting approximately similar object pairs for subsequent distance computations, leaving out the remaining pairs as dissimilar. previously proposed blocking methods require manually constructing an index-based similarity function or selecting a set of predicates, followed by hand-tuning of parameters. in this paper, we introduce an adaptive framework for automatically learning blocking functions that are efficient and accurate. we describe two predicate-based formulations of learnable blocking functions and provide learning algorithms for training them. the effectiveness of the proposed techniques is demonstrated on real and simulated datasets, on which they prove to be more accurate than non-adaptive blocking methods.
meta clustering. clustering is ill-defined. unlike supervised learning where labels lead to crisp performance criteria such as accuracy and squared error, clustering quality depends on how the clusters will be used. devising clustering criteria that capture what users need is difficult. most clustering algorithms search for optimal clusterings based on a pre-specified clustering criterion. our approach differs. we search for many alternate clusterings of the data, and then allow users to select the clustering(s) that best fit their needs. meta clustering first finds a variety of clusterings and then clusters this diverse set of clusterings so that users must only examine a small number of qualitatively different clusterings. we present methods for automatically generating a diverse set of alternate clusterings, as well as methods for grouping clusterings into meta clusters. we evaluate meta clustering on four test problems and two case studies. surprisingly, clusterings that would be of most interest to users often are not very compact clusterings.
an experimental investigation of graph kernels on a collaborative recommendation task. this work presents a systematic comparison between seven kernels (or similarity matrices) on a graph, namely the exponential diffusion kernel, the laplacian diffusion kernel, the von neumann kernel, the regularized laplacian kernel, the commute time kernel, and finally the markov diffusion kernel and the cross-entropy diffusion matrix -- both introduced in this paper -- on a collaborative recommendation task involving a database. the database is viewed as a graph where elements are represented as nodes and relations as links between nodes. from this graph, seven kernels are computed, leading to a set of meaningful proximity measures between nodes, allowing to answer questions about the structure of the graph under investigation; in particular, recommend items to users. crossvalidation results indicate that a simple nearest-neighbours rule based on the similarity measure provided by the regularized laplacian, the markov diffusion and the commute time kernels performs best. we therefore recommend the use of the commute time kernel for computing similarities between elements of a database, for two reasons: (1) it has a nice appealing interpretation in terms of random walks and (2) no parameter needs to be adjusted.
adding semantics to email clustering. this paper presents a novel algorithm to cluster emails according to their contents and the sentence styles of their subject lines. in our algorithm, natural language processing techniques and frequent itemset mining techniques are utilized to automatically generate meaningful generalized sentence patterns (gsps) from subjects of emails. then we put forward a novel unsupervised approach which treats gsps as pseudo class labels and conduct email clustering in a supervised manner, although no human labeling is involved. our proposed algorithm is not only expected to improve the clustering performance, it can also provide meaningful descriptions of the resulted clusters by the gsps. experimental results on open dataset (enron email dataset) and a personal email dataset collected by ourselves demonstrate that the proposed algorithm outperforms the k-means algorithm in terms of the popular measurement f1. furthermore, the cluster naming readability is improved by 68.5% on the personal email dataset.
pattern mining in frequent dynamic subgraphs. graph-structured data is becoming increasingly abundant in many application domains. graph mining aims at finding interesting patterns within this data that represent novel knowledge. while current data mining deals with static graphs that do not change over time, coming years will see the advent of an increasing number of time series of graphs. in this article, we investigate how pattern mining on static graphs can be extended to time series of graphs. in particular, we are considering dynamic graphs with edge insertions and edge deletions over time. we define frequency in this setting and provide algorithmic solutions for finding frequent dynamic subgraph patterns. existing subgraph mining algorithms can be easily integrated into our framework to make them handle dynamic graphs. experimental results on real-world data confirm the practical feasibility of our approach.
mining correlation between motifs and gene expression. one of the major challenges in the post-genomic era is to determine all dna-binding transcription factors (tfs) and their regulatory binding sites (motifs) within the genomes. to discover the relationship between the motifs and changes in gene expression, we propose a new algorithm, co-miner (correlation miner). correlation rules are generated based on the expression profiles of genes with significant expression change through the time course of gene expression. thus, we may consider the change in gene expression to be causatively associated with the transcription binding sites in the upstream sequences. in addition, we introduce partition and constraint pushing techniques to improve the performance and demonstrate their effectiveness by our experiments. by applying co-miner to a yeast dataset, the relationships between motifs and gene expression revealed by co-miner are confirmed in the literature.
intelligent icons: integrating lite-weight data mining and visualization into gui operating systems. the vast majority of visualization tools introduced so far are specialized pieces of software that run explicitly on a particular dataset at a particular time for a particular purpose. in this work we introduce a novel framework for allowing visualization to take place in the background of normal day-to-day operation of any gui based operation system. our system works by replacing the standard file icons with automatically created icons that reflect the contents of the files in a principled way. we call such icons intelligent icons. the utility of intelligent icons is further enhanced by arranging them in a way that reflects their similarity/differences. we demonstrate the utility of our approach on diverse applications.
discover bayesian networks from incomplete data using a hybrid evolutionary algorithm. this paper proposes a novel hybrid approach for learning bayesian networks from incomplete data in the presence of missing values, which combines an evolutionary algorithm with the traditional expectation-maximization (em) algorithm. the new algorithm can overcome the problem of getting stuck in sub-optimal solutions which occurs in most existing learning algorithms. the experimental results on the data sets generated from several benchmark networks illustrate that the new algorithm has better performance than some state-of-the-art algorithms. we also apply the approach to a data set of direct marketing and compare the performance of the discovered bayesian networks obtained by the new algorithm with the networks generated by other methods. in the comparison, the bayesian networks learned by the new algorithm outperform other networks.
who thinks who knows who? socio-cognitive analysis of email networks. interpersonal interaction plays an important role in organizational dynamics, and understanding these interaction networks is a key issue for any organization, since these can be tapped to facilitate various organizational processes. however, the approaches of collecting data about them using surveys/interviews are fraught with problems of scalability, logistics and reporting biases, especially since such surveys may be perceived to be intrusive. widespread use of computer networks for organizational communication provides a unique opportunity to overcome these difficulties and automatically map the organizational networks with a high degree of detail and accuracy. this paper describes an effective and scalable approach for modeling organizational networks by tapping into an organization's email communication. the approach models communication between actors as non-stationary bernoulli trials and bayesian inference is used for estimating model parameters over time. this approach is useful for sociocognitive analysis (who knows who knows who) of organizational communication networks. using this approach, novel measures for analysis of (i) closeness between actors' perceptions about such organizational networks (agreement), (ii) divergence of an actor's perceptions about organizational network from reality (misperception) are explained. using the enron email data, we show that these techniques provide sociologists with a new tool to understand organizational networks.
probabilistic enhanced mapping with the generative tabular model. visualization of the massive datasets needs new methods which are able to quickly and easily reveal their contents. the projection of the data cloud is an interesting paradigm in spite of its difficulty to be explored when data plots are too numerous. so we study a new way to show a bi-dimensional projection from a multidimensional data cloud: our generative model constructs a tabular view of the projected cloud. we are able to show the high densities areas by their non equidistributed discretization. this approach is an alternative to the self-organizing map when a projection does already exist. the resulting pixel views of a dataset are illustrated by projecting a data sample of real images: it becomes possible to observe how are laid out the class labels or the frequencies of a group of modalities without being lost because of a zoom enlarging change for instance. the conclusion gives perspectives to this original promising point of view to get a readable projection for a statistical data analysis of large data samples.
star-structured high-order heterogeneous data co-clustering based on consistent information theory. heterogeneous object co-clustering has become an important research topic in data mining. in early years of this research, people mainly worked on two types of heterogeneous data (denoted by pair-wise co-clustering); while recently more and more attention was paid to multiple types of heterogeneous data (denoted by highorder co-clustering). in this paper, we studied the highorder co-clustering of objects with star-structured interrelationship, i.e., there is a central type of objects that connects the other types of objects. actually, this case could be a very good model for many real-world applications, such as the co-clustering of web images, their low-level visual features, and the surrounding text. we used a tripartite graph to represent the interrelationships among different objects, and proposed a consistent information theory which generates an effective algorithm to obtain the co-clusters of different types of objects. experiments on a web image show that our proposed algorithm is a better choice compared with previous work on heterogeneous object co-clustering.
temporal data mining in dynamic feature spaces. many interesting real-world applications for temporal data mining are hindered by concept drift. one particular form of concept drift is characterized by changes to the underlying feature space. seemingly little has been done in this area. this paper presents fae, an incremental ensemble approach to mining data subject to such concept drift. empirical results on large data streams demonstrate promise.
data mining methods for modeling gene expression regulation and their applications. understanding gene expression regulation at both transcriptional and post-transcriptional levels is critical for elucidation of the mechanism of stress tolerance in plants and important for understanding and diagnosis of human diseases. with the advent of high throughput gene expression profiling techniques, a huge amount of gene expression data on various organisms has been collected. such a wealth of biological data has provided excellent opportunities to elucidating transcriptional regulation mechanisms using machine learning and data mining approaches.
mining for tree-query associations in a graph. new applications of data mining, such as in biology, bioinformatics, or sociology, are faced with large datasets structured as graphs. we present an efficient algorithm for mining associations between tree queries in a large graph. tree queries are powerful tree-shaped patterns featuring existential variables and data constants. our algorithm applies the theory of conjunctive database queries to make the generation of association rules efficient. we propose a practical, database-oriented implementation in sql, and show that the approach works in practice through experiments on data about food webs, protein interactions, and citation analysis.
on the use of structure and sequence-based features for protein classification and retrieval. the need to retrieve or classify proteins using structure or sequence-based similarity underlies many biomedical applications. in drug discovery, researchers search for proteins that share specific chemical properties as sources for new treatment. with folding simulations, similar intermediate structures might be indicative of a common folding pathway. here we present two normalized, stand-alone representations of proteins that enable fast and efficient object retrieval based on sequence or structure. to create our sequence-based representation, we take the profiles returned by the psi-blast alignment algorithm and create a normalized summary using a discrete wavelet transform. for our structural representation, we transform each 3d structure into a normalized 2d distance matrix and apply a 2d wavelet decomposition to generate our descriptor. we also create a hybrid representation by concatenating together the above descriptors. we evaluate the generality of our models by using them as indices for database retrieval experiments as well as feature vectors for classification. we find that our methods provide excellent performance when compared with the state-of-the-art for each task. our results show that the sequence-based representation is generally superior to the structure-based representation and that in the classification context, the hybrid strategy affords a significant improvement over sequence or structure.
a data mining approach for capacity building of stakeholders in integrated flood management. new approaches to managing flood events are increasingly of more relevance due to recent widespread floods and the presumed changes in the climate. these approaches fall under the integrated flood management (ifm) banner and focus not only on flood prevention, but on flood resilience. this paper introduces an application (floreto) for ifm that utilizes the data mining approach, in a web based three tier system, devoted to the capacity building of stakeholders as a micro-scale resilience strategy of ifm. the intelligent models, which constitute the business logic in floreto, are used to match the input parameters or design criteria, describing properties prone to flooding, to technically justifiable flood mitigation measures. datasets from the german city of kellinghusen were collected and intelligent models were built. satisfactory results have been obtained, which shows the promise of this data mining approach and opens the door for its application for ifm in other regions.
opening the black box of feature extraction: incorporating visualization into high-dimensional data mining processes. feature extraction techniques have been used to handle high-dimensional data and experimental studies often show improved classification accuracies. unfortunately very few studies provide concrete evidences on the effectiveness of these feature extraction techniques and they largely remain to be black boxes. in this study, we design and implement a visualization prototype system that allows users to look into the classification processes, explore the links among the original and extracted features in different classifiers, examine why and how an instance is correctly or incorrectly classified. we demonstrate the prototype's capabilities by combining a feature extraction method based on hierarchical feature space clustering with j48 decision tree classifiers and perform experiments on a real hyperspectral remote sensing image dataset.
belief propagation in large, highly connected graphs for 3d part-based object recognition. we describe a part-based object-recognition framework, specialized to mining complex 3d objects from detailed 3d images. objects are modeled as a collection of parts together with a pairwise potential function. an efficient inference algorithm -- based on belief propagation (bp) -- finds the optimal layout of parts, given some input image. we introduce aggbp, a message aggregation scheme for bp, in which groups of messages are approximated as a single message. for objects consisting of n parts, we reduce cpu time and memory requirements from o( n^2 ) to o(n). we apply aggbp on synthetic data as well as a real-world task identifying protein fragments in three-dimensional images. these experiments show that our improvements result in minimal loss in accuracy in significantly less time.
social capital in friendship-event networks. in this paper, we examine a particular form of social network which we call a friendship-event network. a friendship-event network captures both the friendship relationship among a set of actors, and also the organizer and participation relationships of actors in a series of events. within these networks, we formulate the notion of social capital based on the actor-organizer friendship relationship and the notion of benefit, based on event participation. we investigate appropriate definitions for the social capital of both a single actor and a collection of actors. we ground these definitions in a real-world example of academic collaboration networks, where the actors are researchers, the friendships are collaborations, the events are conferences, the organizers are program committee members and the participants are conference authors. we show that our definitions of capital and benefit capture interesting qualitative properties of event series. in addition, we show that social capital is a better publication predictor than publication history.
geometrically inspired itemset mining. in our geometric view, an itemset is a vector (itemvector) in the space of transactions. linear and potentially non-linear transformations can be applied to the itemvectors before mining patterns. aggregation functions and interestingness measures can be applied to the transformed vectors and pushed inside the mining process. we show that interesting itemset mining can be carried out by instantiating four abstract functions: a transformation (g), an algebraic aggregation operator () and measures (f and f). for frequent itemset mining (fim), g and f are identity transformations,is intersection and f is the cardinality. based on this geometric view we present a novel algorithm that uses space linear in the number of 1-itemsets to mine all interesting itemsets in a single pass over the data, with no candidate generation. it scales (roughly) linearly in running time with the number of interesting itemsets. fim experiments show that it outperforms fpgrowth on realistic datasets above a small support threshold (0.29% and 1.2% in our experiments) .
finding "who is talking to whom" in voip networks via progressive stream clustering. technologies that use the internet network to deliver voice communications have the potential to reduce costs and improve access to communications services around the world. however, these new technologies pose several challenges in terms of confidentiality of the conversations and anonymity of the conversing parties. call authentication and encryption techniques provide a way to protect confidentiality, while anonymity is typically preserved by an anonymizing service (anonymous call). this work studies the feasibility of revealing pairs of anonymous and encrypted conversing parties (caller/callee pair of streams) by exploiting the vulnerabilities inherent to voip systems. in particular, by exploiting the aperiodic inter-departure time of voip packets, we can trivialize each voip stream into a binary time-series. we first define a simple yet intuitive metric to gauge the correlation between two voip binary streams. then we propose an effective technique that progressively pairs conversing parties with high accuracy and in a limited amount of time. our metric and method are justified analytically and validated by experiments on a very large standard corpus of conversational speech. we obtain impressively high pairing accuracy that reaches 97% after 5 minutes of voice conversations.