waashk/dblp huggingface dataset:
using unsupervised link discovery methods to find interesting facts and connections in a bibliography dataset. this paper describes a submission to the open task of the 2003 kdd cup. for this task contestants were asked to devise their own questions about the hep-th bibliography dataset, and the most interesting result would be selected as the winner. instead of taking a more traditional approach such as starting with a inspection of the data, formulating questions or hypotheses interesting to us and then devising an analysis and approach to answer these questions, we tried to go a different route: can we develop a program that automatically finds interesting facts and connections in the data?to do this we developed a set of unsupervised link discovery methods that compute interestingness based on a notion of "rarity" and "abnormality". the experiments performed on the hep-th dataset show that our approaches are able to automatically uncover interesting hidden connections (e.g. significant relationships between people) and unexpected facts (e.g. citation loops) without the support of any prerequisite knowledge or training examples. the interestingness of some of our results is self-evident. for others we were able to verify them by looking for supporting evidence on the world-wide-web, which shows that our methods can find connections between entities that actually are interestingly connected in the real world in an unsupervised way.
subspace clustering for high dimensional categorical data. data clustering has been discussed extensively, but almost all known conventional clustering algorithms tend to break down in high dimensional spaces because of the inherent sparsity of the data points. existing subspace clustering algorithms for handling high-dimensional data focus on numerical dimensions. in this paper, we designed an iterative algorithm called subcad for clustering high dimensional categorical data sets, based on the minimization of an objective function for clustering. we deduced some cluster memberships changing rules using the objective function. we also designed an objective function to determine the subspace associated with each cluster. we proved various properties of this objective function that are essential for us to design a fast algorithm to find the subspace associated with each cluster. finally, we carried out some experiments to show the effectiveness of the proposed method and the algorithm.
editorial: special issue on web content mining. with the phenomenal growth of the web, there is an everincreasing volume of data and information published in numerous web pages. the research in web mining aims to develop new techniques to effectively extract and mine useful knowledge or information from these web pages [8]. due to the heterogeneity and lack of structure of web data, automated discovery of targeted or unexpected knowledge/information is a challenging task. it calls for novel methods that draw from a wide range of fields spanning data mining, machine learning, natural language processing, statistics, databases, and information retrieval. in the past few years, there was a rapid expansion of activities in the web mining field, which consists of web usage mining, web structure mining, and web content mining. web usage mining refers to the discovery of user access patterns from web usage logs. web structure mining tries to discover useful knowledge from the structure of hyperlinks. web content mining aims to extract/mine useful information or knowledge from web page contents. for this special issue, we focus on web content mining.
mining data streams under block evolution. in this paper we survey recent work on incremental data mining model maintenance and change detection under block evolution. in block evolution, a dataset is updated periodically through insertions and deletions of blocks of records at a time. we describe two techniques: (1) we describe a generic algorithm for model maintenance that takes any traditional incremental data mining model maintenance algorithm and transforms it into an algorithm that allows restrictions on a temporal subset of the database. (2) we also describe a generic framework for change detection, that quantifies the difference between two datasets in terms of the data mining models they induce.
a survey of kernels for structured data. kernel methods in general and support vector machines in particular have been successful in various learning tasks on data represented in a single table. much 'real-world' data, however, is structured - it has no natural representation in a single table. usually, to apply kernel methods to 'real-world' data, extensive pre-processing is performed to embed the data into areal vector space and thus in a single table. this survey describes several approaches of defining positive definite kernels on structured instances directly.
support vector machines classification with a very large-scale taxonomy. very large-scale classification taxonomies typically have hundreds of thousands of categories, deep hierarchies, and skewed category distribution over documents. however, it is still an open question whether the state-of-the-art technologies in automated text categorization can scale to (and perform well on) such large taxonomies. in this paper, we report the first evaluation of support vector machines (svms) in web-page classification over the full taxonomy of the yahoo! categories. our accomplishments include: 1) a data analysis on the yahoo! taxonomy; 2) the development of a scalable system for large-scale text categorization; 3) theoretical analysis and experimental evaluation of svms in hierarchical and non-hierarchical settings for classification; 4) an investigation of threshold tuning algorithms with respect to time complexity and their effect on the classification accuracy of svms. we found that, in terms of scalability, the hierarchical use of svms is efficient enough for very large-scale classification; however, in terms of effectiveness, the performance of svms over the yahoo! directory is still far from satisfactory, which indicates that more substantial investigation is needed.
mining biologically active patterns in metabolic pathways using microarray expression profiles. we present a new probabilistic framework for analyzing a metabolic pathway with microarray expression profiles. our purpose is to find biologically significant paths and patterns in a given metabolic pathway. our approach first builds a markov model using a graph structure of a known metabolic pathway, and then estimates parameters of a mixture of the markov models using microarray data, based on an em algorithm. in our experiments, we used a main pathway of glycolysis to evaluate the effectiveness of our method. we first measured the performance of our method comparing with that of another method, in a supervised learning manner, and found that our method significantly outperformed another method, which was trained by microarray data only. we further analyzed the trained models and obtained a number of new biological findings on frequent patterns (paths) and long-range correlations in a metabolic pathway.
5th international workshop on knowledge discovery in inductive databases (kdid'06): workshop report. this report presents a review of the 5th international workshop on knowledge discovery in inductive databases (kdid'06), which was organized by the authors and held in berlin, germany, on september 18, 2006, in conjunction with ecml/pkdd'06, the 17th european conference on machine learning and the 10th european conference on principles and practice of knowledge discovery in databases. the goal of the workshop was to bring together the researchers that are interested in the area of inductive databases, inductive queries, constraint-based data mining, and data mining query languages.
data mining methods for anomaly detection kdd-2005 workshop report. for many applications, data mining systems are required to detect anomalous (abnormal, unmodeled, or unexpected) observations. this has so far proven to be a difficult challenge because anomalies are usually considered to be "non-normal" observations, where "normality" is typically defined by very complex concepts. because of these and other reasons, there are no standard and principled approaches for anomaly detection, yet, and the data mining processes that have led to successful solutions include most of the times ad-hoc (algorithmic, design, and implementation) decisions that incorporate prior or commonsense knowledge about the tasks that are addressed.consequently, we considered that it would be beneficial for both researchers and practitioners interested in anomaly detection and data mining, to organize workshop that would bring together people interested in this topic. we considered that the international conference on knowledge discovery and data mining would be a good venue for such a workshop because of the diversity of interests, backgrounds, and problems that motivate people to attend the conference.this paper describes the workshop on "data mining methods for anomaly detection" - a one day event held in conjunction with kdd-2005 in chicago, on august 21, 2005.
webkdd-99: workshop on web usage analysis and user profiling. the webkdd'99 workshop on "web usage analysis and user profiling" took place at aug. 15, 1999 under the auspices of the sigkdd international conference on knowledge discovery and data mining (kdd'99). we report on the topics addressed in the workshop, the contributions and the discussions that took place in its framework.
webkdd 2002 - web mining for usage patterns & profiles. in this paper, we provide a summary of the webkdd 2002 workshop, whose theme was 'web mining for usage patterns and profiles'. this workshop was held in conjunction with the acm sigkdd international conference on knowledge discovery and data mining (kdd-2002).
phenomenal data mining: from data to phenomena. phenomenal data mining finds relations between the data and the phenomena that give rise to data rather than just relations among the data.for example, suppose supermarket cash register data does not identify cash customers. nevertheless, there really are customers, and these customers are characterized by sex, age, ethnicity, tastes, income distribution, and sensitivity to price changes. a data mining program might be able to identify which baskets of purchases are likely to have been made by the same customers. in this example, the receipts are the data, and the customers are phenomena not directly represented in the data. once the "baskets" of purchases are grouped by customer, the way is open to infer further phenomena about the customers, e.g. their sex, age, etc.this article concerns what can be inferred by programs about phenomena from data and what facts are relevant to doing this. we work mainly with the supermarket example, but the idea is general.in order to infer phenomena from data, facts about their relations must be supplied. sometimes these facts can be implicit in the programs that look for the phenomena, but more generality is achieved if the facts are represented as sentences of logic in a knowledge base used by the programs. the result of phenomenal data-mining might include an extended database with additional fields on existing relations and new relations. thus the relations describing supermarket baskets might be extended with a customer field, and new relations about customers and their properties might be introduced.
exploiting relational structure to understand publication patterns in high-energy physics. we analyze publication patterns in theoretical high-energy physics using a relational learning approach. we focus on four related areas: understanding and identifying patterns of citations, examining publication patterns at the author level, predicting whether a paper will be accepted by specific journals, and identifying research communities from the citation patterns and paper text. each of these analyses contributes to an overall understanding of theoretical high-energy physics.
a preprocessing scheme for high-cardinality categorical attributes in classification and prediction problems. categorical data fields characterized by a large number of distinct values represent a serious challenge for many classification and regression algorithms that require numerical inputs. on the other hand, these types of data fields are quite common in real-world data mining applications and often contain potentially relevant information that is difficult to represent for modeling purposes.this paper presents a simple preprocessing scheme for high-cardinality categorical data that allows this class of attributes to be used in predictive models such as neural networks, linear and logistic regression. the proposed method is based on a well-established statistical method (empirical bayes) that is straightforward to implement as an in-database procedure. furthermore, for categorical attributes with an inherent hierarchical structure, like zip codes, the preprocessing scheme can directly leverage the hierarchy by blending statistics at the various levels of aggregation.while the statistical methods discussed in this paper were first introduced in the mid 1950's, the use of these methods as a preprocessing step for complex models, like neural networks, has not been previously discussed in any literature.
discovering geographic knowledge in data rich environments: a report on a specialist meeting. on 18--20 march 1999, a specialist meeting on "discovering geographic knowledge in data-rich environments" was convened under the auspices of the varenius project of the national center for geographic information and analysis (ncgia). this, workshop brought together a diverse group of researchers and practitioners with interests in developing and applying new techniques for exploring large and diverse geographic datasets. the interaction prior to, during and after the three-day workshop resulted in the identification of research priorities and directions for continued development of "geographic knowledge discovery" (gkd) theory techniques.
mining knowledge from text using information extraction. an important approach to text mining involves the use of natural-language information extraction. information extraction (ie) distills structured data or knowledge from unstructured text by identifying references to named entities as well as stated relationships between such entities. ie systems can be used to directly extricate abstract knowledge from a text corpus, or to extract concrete data from a set of documents which can then be further analyzed with traditional data-mining techniques to discover more general patterns. we discuss methods and implemented systems for both of these approaches and summarize results on mining real text corpora of biomedical abstracts, job announcements, and product descriptions. we also discuss challenges that arise when employing current information extraction technology to discover knowledge in text.
gene ranking using bootstrapped p-values. recent research has shown that it is possible to find genes involved in the pathogenesis of a particular condition on the basis of microarray experiments. genes which are differentially expressed, for example between healthy and diseased tissues, are likely to be relevant to the disease under study. some of the properties of microarray datasets make the task of finding these genes a challenging one. this paper proposes a gene-ranking algorithm whose main novelty is the use of bootstrapped p-values. we present an analysis of the algorithm, showing how it takes account of small-sample variability in observed values of the test statistic, in a way conventional statistical tests cannot. experimental results show that our algorithm outperforms the widely-used two-sample t-test on challenging artificial data. gene ranking is then performed on two well-known microarray datasets, with encouraging results. for example, a number of genes from one of the datasets, whose differential expression was subsequently confirmed by a more reliable biochemical analysis, are found to be ranked higher by the bootstrapped algorithm than by the conventional t-test, suggesting that the proposed algorithm may be better able to exploit the limited data available to infer biologically useful information.
a baseline feature set for learning rhetorical zones using full articles in the biomedical domain. at a time when experimental throughput in the field of molecular biology is increasing, it is necessary for biologists and people working in related fields to have access to sophisticated tools to enable them to efficiently process large amounts of information in order to stay abreast of current research.rhetorical zone analysis is an application of natural language processing in which areas of text in scientific papers are classified in terms of argumentation and intellectual contribution in order to pinpoint and distinguish certain types of information. such analysis can be employed to assist in information extraction, helping to assess and integrate data generated by experiments into the scientific community's store of knowledge.we present results for several experiments in automatic zone identification on the zaisa-1 dataset, a new dataset composed of full biomedical research papers hand-annotated for rhetorical zones. we concentrate on general purpose and linguistically motivated features, and report results for a variety of sets of features. it is our intention to provide a baseline feature set for modeling, which can be extended in future work using combinations of heuristics and more sophisticated and task-specific modeling techniques.
a novel approach to determine normal variation in gene expression data. animal models for human diseases are of crucial importance for studying gene expression and regulation. in the last decade the development of mouse models for cancer, diabetes, neuro-degenerative and many other diseases has been on steady rise. microarray analysis of patterns of gene expression in mouse models of various pathological types and the study of molecular level changes as a result of interventions, holds lot of promise to the understanding of biological processes involved. the genes which show normal variance across genetically identical mice are of particular interest because they could serve as a databank for possible false positives in gene expression studies in similar kind of mice. also they could provide useful insights into the biological processes behind the differential expression patterns in otherwise similar mice. our approach systematically removes variance due to experimental noise in each of the mice and then mines for normal variance among the identical mice. this analysis carried over six tissues sampled from mice, resulted in several genes which showed variations among identical mice, thus enabling a comprehensive database of normal variations in gene expression for mouse models. a large number of these genes are known to be related to stress response, hypertension and heat shock. also principal component analysis was done to visualize similarity among the mice models and within the replicates. these studies help in the design of gene expression studies in mouse models and help in validation of the results.
webkdd 2004: web mining and web usage analysis post-workshop report. in this report, we summarize the contents and outcomes of the recent webkdd 2004 workshop that was held in conjunction with the 10th acm sigkdd international conference on knowledge discovery and data mining (kdd 2004), august 22-25, 2004, in seattle, washington. we also reflect on the trend in participation levels in the webkdd series of workshops over the last six years, and indicate new directions in web mining research as reflected in the latest workshop.
webkdd 2005: web mining and web usage analysis post-workshop report. in this report, we summarize the contents and outcomes of the recent webkdd 2005 workshop on web mining and web usage analysis that was held in conjunction with the 11th acm sigkdd international conference on knowledge discovery and data mining (kdd 2005), august 21-24, 2005, in chicago, illinois. the theme of this workshop was "taming evolving, expanding and multi-faceted web clickstreams". we also reflect on possible new directions in web mining research as reflected by the discussions and the talks during the workshop.
differential expression, class discovery and class prediction using s-plus and s+arrayanalyzer. microarrays are a powerful experimental platform, allowing simultaneous studies of gene expression for thousands of genes under different experimental conditions. however there is much biological variability induced throughout the experimental process that can obscure the biological signals of interest. as such, the need for experimental design, replication and statistical rigor are now widely recognized. statistical hypothesis testing has become the accepted differential expression analysis approach and many classification and prediction methods used in class discovery and class prediction now incorporate stochastic modeling components.this paper provides a review of statistical analysis approaches to the analysis of data from microarray experiments. this includes discussion of experimental design, data management, preprocessing, differential expression, clustering and class prediction, reporting and annotation. the review is illustrated with the analysis of an experiment with 3 experimental conditions using the affymetrix murine chip mgu 74av2; and with descriptions of available functionality in the statistical analysis software s-plus and its associated module for microarray analysis, s+arrayanalyzer.
prediction and ranking algorithms for event-based network data. event-based network data consists of sets of events over time, each of which may involve multiple entities. examples include email traffic, telephone calls, and research publications (interpreted as co-authorship events). traditional network analysis techniques, such as social network models, often aggregate the relational information from each event into a single static network. in contrast, in this paper we focus on the temporal nature of such data. in particular, we look at the problems of temporal link prediction and node ranking, and describe new methods that illustrate opportunities for data mining and machine learning techniques in this context. experimental results are discussed for a large set of co-authorship events measured over multiple years, and a large corporate email data set spanning 21 months.
biological applications of multi-relational data mining. biological databases contain a wide variety of data types, often with rich relational structure. consequently multi-relational data mining techniques frequently are applied to biological data. this paper presents several applications of multi-relational data mining to biological data, taking care to cover a broad range of multi-relational data mining techniques.
association against dissociation: some pragmatic considerations for frequent itemset generation under fixed and variable thresholds. traditionally, support is considered to be the standard measure for frequent itemset generation in association rule mining. this paper provides a new measure called togetherness where dissociation among items is also considered as a parameter in the frequent itemset generation process. results of performance analysis show that association against dissociation is a more pragmatic approach and discovers truly associated candidate itemsets. second part of the paper extends this togetherness measure to the domain of variable threshold. here, like variable minimum support, a variable minimum togetherness has been proposed where this minimum value decreases as the itemset size increases. a simple and pragmatic process has been described, which can be easily implemented. it also provides ample control facilities in the hand of the users. necessary change and extension of the existing algorithms have been made to establish the concepts. here as well, results of performance analysis justify the approach.