waashk/dblp huggingface dataset:
an adaptive nearest neighbor search for a parts acquisition eportal. one of the major hurdles in maintaining long-lived electronic systems is that electronic parts become obsolete, no longer available from the original suppliers. when this occurs, an engineer is tasked with resolving the problem by finding a replacement that is "as similar as possible" to the original part. the current approach involves a laborious manual search through several electronic portals and data books. the search is difficult because potential replacements may differ from the original and from each other by one or more parameters. worse still, the cumbersome nature of this process may cause the engineers to miss appropriate solutions amid the many thousands of parts listed in industry catalogs.in this paper, we address this problem by introducing the notion of a parametric "distance" between electronic components. we use this distance to search a large parts data set and recommend likely replacements. recommendations are based on an adaptive nearest-neighbor search through the parametric data set. for each user, we learn how to scale the axes of the feature space in which the nearest neighbors are sought. this allows the system to learn each user's judgment of the phrase "as similar as possible."
fast ordering of large categorical datasets for better visualization. an important issue in visualizing categorical data is how to order categorical values. the focus of this paper is on constructing such orderings efficiently without compromising their visual quality.
scalable robust covariance and correlation estimates for data mining. covariance and correlation estimates have important applications in data mining. in the presence of outliers, classical estimates of covariance and correlation matrices are not reliable. a small fraction of outliers, in some cases even a single outlier, can distort the classical covariance and correlation estimates making them virtually useless. that is, correlations for the vast majority of the data can be very erroneously reported; principal components transformations can be misleading; and multidimensional outlier detection via mahalanobis distances can fail to detect outliers. there is plenty of statistical literature on robust covariance and correlation matrix estimates with an emphasis on affine-equivariant estimators that possess high breakdown points and small worst case biases. all such estimators have unacceptable exponential complexity in the number of variables and quadratic complexity in the number of observations. in this paper we focus on several variants of robust covariance and correlation matrix estimates with quadratic complexity in the number of variables and linear complexity in the number of observations. these estimators are based on several forms of pairwise robust covariance and correlation estimates. the estimators studied include two fast estimators based on coordinate-wise robust transformations embedded in an overall procedure recently proposed by [14]. we show that the estimators have attractive robustness properties, and give an example that uses one of the estimators in the new insightful miner data mining product.
mining high dimensional data for classifier knowledge. we present in this paper the problem of discovering sets of attribute-value pairs in high dimensional data sets that are of interest not because of co-occurrence alone, but due to their value in serving as cores for potential classifiers of clusters. we present our algorithm in the context of a gene-expression dataset. gene expression data, in most situations, is insufficient for clustering algorithms and any statistical inference because for 6000+ genes, typically only 10s and at most 100s of data points become available. it is difficult to use statistical techniques to design a classifier for such immensely under-specified data. the observed data, though statistically, insufficient contains some information about the domain. our goal is to discover as much information about all potential classifiers as possible from the data and then summarize this knowledge. this summarization provides insights into the composition of potential classifiers. we present here algorithms and methods for mining a high dimensional data set, exemplified by a gene expression data set, for mining such information.
challenges for knowledge discovery in biology. bioinformatics is the study of information flow in biology. interest in the field has exploded in the last 10 years with the emergence of techniques for large scale experimental data collection-including genome sequencing, gene expression analysis, protein interaction detection, high-throughput structure determination and others. these techniques, in the context of a large online published literature, have created relatively large data sets (at least by biological standards) that are not possible to analyze manually. there is therefore a critical need for methods to analyze these data and reduce them to new knowledge. the principle challenges to the field include the great diversity of data types and questions that are asked of the data, and the communication difficulties that can exist between experts in biology and experts in machine learning. in this talk, i will provide an introduction to the major biological questions that are being addressed, why they are important, and how the field is trying to address them with technical approaches.
on the potential of domain literature for clustering and bayesian network learning. thanks to its increasing availability, electronic literature can now be a major source of information when developing complex statistical models where data is scarce or contains much noise. this raises the question of how to integrate information from domain literature with statistical data. because quantifying similarities or dependencies between variables is a basic building block in knowledge discovery, we consider here the following question. which vector representations of text and which statistical scores of similarity or dependency support best the use of literature in statistical models? for the text source, we assume to have annotations for the domain variables as short free-text descriptions and optionally to have a large literature repository from which we can further expand the annotations. for evaluation, we contrast the variables similarities or dependencies obtained from text using different annotation sources and vector representations with those obtained from measurement data or expert assessments. specifically, we consider two learning problems: clustering and bayesian network learning. firstly, we report performance (against an expert reference) for clustering yeast genes from textual annotations. secondly, we assess the agreement between text-based and data-based scores of variable dependencies when learning bayesian network substructures for the task of modeling the joint distribution of clinical measurements of ovarian tumors.
query-time entity resolution. the goal of entity resolution is to reconcile database references corresponding to the same real-world entities. given the abundance of publicly available databases where entities are not resolved, we motivate the problem of quickly processing queries that require resolved entities from such 'unclean' databases. we propose a two-stage collective resolution strategy for processing queries. we then show how it can be performed on-the-fly by adaptively extracting and resolving those database references that are the most helpful for resolving the query. we validate our approach on two large real-world publication databases where we show the usefulness of collective resolution and at the same time demonstrate the need for adaptive strategies for query processing. we then show how the same queries can be answered in real time using our adaptive approach while preserving the gains of collective resolution.
segmentation-based modeling for advanced targeted marketing. fingerhut business intelligence (bi) has a long and successful history of building statistical models to predict consumer behavior. the models constructed are typically segmentation-based models in which the target audience is split into subpopulations (i.e., customer segments) and individually tailored statistical models are then developed for each segment. such models are commonly employed in the direct-mail industry; however, segmentation is often performed on an ad-hoc basis without directly considering how segmentation affects the accuracy of the resulting segment models. fingerhut bi approached ibm research with the problem of how to build segmentation-based models more effectively so as to maximize predictive accuracy. the ibm advanced targeted marketing-single eventstm (ibm atm-setm) solution is the result of ibm research and fingerhut bi directing their efforts jointly towards solving this problem. this paper presents an evaluation of atm-se's modeling capabilities using data from fingerhut's catalog mailings.
computer aided detection via asymmetric cascade of sparse hyperplane classifiers. this paper describes a novel classification method for computer aided detection (cad) that identifies structures of interest from medical images. cad problems are challenging largely due to the following three characteristics. typical cad training data sets are large and extremely unbalanced between positive and negative classes. when searching for descriptive features, researchers often deploy a large set of experimental features, which consequently introduces irrelevant and redundant features. finally, a cad system has to satisfy stringent real-time requirements.this work is distinguished by three key contributions. the first is a cascade classification approach which is able to tackle all the above difficulties in a unified framework by employing an asymmetric cascade of sparse classifiers each trained to achieve high detection sensitivity and satisfactory false positive rates. the second is the incorporation of feature computational costs in a linear program formulation that allows the feature selection process to take into account different evaluation costs of various features. the third is a boosting algorithm derived from column generation optimization to effectively solve the proposed cascade linear programs.we apply the proposed approach to the problem of detecting lung nodules from helical multi-slice ct images. our approach demonstrates superior performance in comparison against support vector machines, linear discriminant analysis and cascade adaboost. especially, the resulting detection system is significantly sped up with our approach.
the "dgx" distribution for mining massive, skewed data. skewed distributions appear very often in practice. unfortunately, the traditional zipf distribution often fails to model them well. in this paper, we propose a new probability distribution, the discrete gaussian exponential (dgx), to achieve excellent fits in a wide variety of settings; our new distribution includes the zipf distribution as a special case. we present a statistically sound method for estimating the dgx parameters based on maximum likelihood estimation (mle). we applied dgx to a wide variety of real world data sets, such as sales data from a large retailer chain, us-age data from at&t, and internet clickstream data; in all cases, dgx fits these distributions very well, with almost a 99% correlation coefficient in quantile-quantile plots. our algorithm also scales very well because it requires only a single pass over the data. finally, we illustrate the power of dgx as a new tool for data mining tasks, such as outlier detection.
style mining of electronic messages for multiple authorship discrimination: first results. this paper considers the use of computational stylistics for performing authorship attribution of electronic messages, addressing categorization problems with as many as 20 different classes (authors). effective stylistic characterization of text is potentially useful for a variety of tasks, as language style contains cues regarding the authorship, purpose, and mood of the text, all of which would be useful adjuncts to information retrieval or knowledge-management tasks. we focus here on the problem of determining the author of an anonymous message, based only on the message text. several multiclass variants of the winnow algorithm were applied to a vector representation of the message texts to learn models for discriminating different authors. we present results comparing the classification accuracy of the different approaches. the results show that stylistic models can be accurately learned to determine an author's identity.
column-generation boosting methods for mixture of kernels. we devise a boosting approach to classification and regression based on column generation using a mixture of kernels. traditional kernel methods construct models based on a single positive semi-definite kernel with the type of kernel predefined and kernel parameters chosen according to cross-validation performance. our approach creates models that are mixtures of a library of kernel models, and our algorithm automatically determines kernels to be used in the final model. the 1-norm and 2-norm regularization methods are employed to restrict the ensemble of kernel models. the proposed method produces sparser solutions, and thus significantly reduces the testing time. by extending the column generation (cg) optimization which existed for linear programs with 1-norm regularization to quadratic programs with 2-norm regularization, we are able to solve many learning formulations by leveraging various algorithms for constructing single kernel models. by giving different priorities to columns to be generated, we are able to scale cg boosting to large datasets. experimental results on benchmark data are included to demonstrate its effectiveness.
systematic data selection to mine concept-drifting data streams. one major problem of existing methods to mine data streams is that it makes ad hoc choices to combine most recent data with some amount of old data to search the new hypothesis. the assumption is that the additional old data always helps produce a more accurate hypothesis than using the most recent data only. we first criticize this notion and point out that using old data blindly is not better than "gambling"; in other words, it helps increase the accuracy only if we are "lucky." we discuss and analyze the situations where old data will help and what kind of old data will help. the practical problem on choosing the right example from old data is due to the formidable cost to compare different possibilities and models. this problem will go away if we have an algorithm that is extremely efficient to compare all sensible choices with little extra cost. based on this observation, we propose a simple, efficient and accurate cross-validation decision tree ensemble method.
exploring the community structure of newsgroups. we propose to use the community structure of usenet for organizing and retrieving the information stored in newsgroups. in particular, we study the network formed by cross-posts, messages that are posted to two or more newsgroups simultaneously. we present what is, to our knowledge, by far the most detailed data that has been collected on usenet cross-postings. we analyze this network to show that it is a small-world network with significant clustering. we also present a spectral algorithm which clusters newsgroups based on the cross-post matrix. the result of our clustering provides a topical classification of newsgroups. our clustering gives many examples of significant relationships that would be missed by semantic clustering methods.
efficient anonymity-preserving data collection. the output of a data mining algorithm is only as good as its inputs, and individuals are often unwilling to provide accurate data about sensitive topics such as medical history and personal finance. individuals maybe willing to share their data, but only if they are assured that it will be used in an aggregate study and that it cannot be linked back to them. protocols for anonymity-preserving data collection provide this assurance, in the absence of trusted parties, by allowing a set of mutually distrustful respondents to anonymously contribute data to an untrusted data miner.to effectively provide anonymity, a data collection protocol must be collusion resistant, which means that even if all dishonest respondents collude with a dishonest data miner in an attempt to learn the associations between honest respondents and their responses, they will be unable to do so. to achieve collusion resistance, previously proposed protocols for anonymity-preserving data collection have quadratically many communication rounds in the number of respondents, and employ (sometimes incorrectly) complicated cryptographic techniques such as zero-knowledge proofs.we describe a new protocol for anonymity-preserving, collusion resistant data collection. our protocol has linearly many communication rounds, and achieves collusion resistance without relying on zero-knowledge proofs. this makes it especially suitable for data mining scenarios with a large number of respondents.
learning to match and cluster large high-dimensional data sets for data integration. part of the process of data integration is determining which sets of identifiers refer to the same real-world entities. in integrating databases found on the web or obtained by using information extraction methods, it is often possible to solve this problem by exploiting similarities in the textual names used for objects in different databases. in this paper we describe techniques for clustering and matching identifier names that are both scalable and adaptive, in the sense that they can be trained to obtain better performance in a particular domain. an experimental evaluation on a number of sample datasets shows that the adaptive method sometimes performs much better than either of two non-adaptive baseline systems, and is nearly always competitive with the best baseline system.
exploiting dictionaries in named entity extraction: combining semi-markov extraction processes and data integration methods. we consider the problem of improving named entity recognition (ner) systems by using external dictionaries---more specifically, the problem of extending state-of-the-art ner systems by incorporating information about the similarity of extracted entities to entities in an external dictionary. this is difficult because most high-performance named entity recognition systems operate by sequentially classifying words as to whether or not they participate in an entity name; however, the most useful similarity measures score entire candidate names. to correct this mismatch we formalize a semi-markov extraction process, which is based on sequentially classifying segments of several adjacent words, rather than single words. in addition to allowing a natural way of coupling high-performance ner methods and high-performance similarity functions, this formalism also allows the direct use of other useful entity-level features, and provides a more natural formulation of the ner problem than sequential word classification. experiments in multiple domains show that the new model can substantially improve extraction performance over previous methods for using external dictionaries in ner.
understanding captions in biomedical publications. from the standpoint of the automated extraction of scientific knowledge, an important but little-studied part of scientific publications are the figures and accompanying captions. captions are dense in information, but also contain many extra-grammatical constructs, making them awkward to process with standard information extraction methods. we propose a scheme for "understanding" captions in biomedical publications by extracting and classifying "image pointers" (references to the accompanying image). we evaluate a number of automated methods for this task, including hand-coded methods, methods based on existing learning techniques, and methods based on novel learning techniques. the best of these methods leads to a usefully accurate tool for caption-understanding, with both recall and precision in excess of 94% on the most important single class in a combined extraction/classification task.
mining optimized gain rules for numeric attributes. association rules are useful for determining correlations between attributes of a relation and have applications in the marketing, financial, and retail sectors. furthermore, optimized association rules are an effective way to focus on the most interesting characteristics involving certain attributes. optimized association rules are permitted to contain uninstantiated attributes and the problem is to determine instantiations such that either the support, confidence, or gain of the rule is maximized. in this paper, we generalize the optimized gain association rule problem by permitting rules to contain disjunctions over uninstantiated numeric attributes. our generalized association rules enable us to extract more useful information about seasonal and local patterns involving the uninstantiated attribute. for rules containing a single numeric attribute, we present an algorithm with linear complexity for computing optimized gain rules. furthermore, we propose a bucketing technique that can result in a significant reduction in input size by coalescing contiguous values without sacrificing optimality. we also present an approximation algorithm based on dynamic programming for two numeric attributes. using recent results on binary space partitioning trees, we show that the approximations are within a constant factor of the optimal optimized gain rules. our experimental results with synthetic data sets for a single numeric attribute demonstrate that our algorithm scales up linearly with the attribute's domain size as well as the number of disjunctions. in addition, we show that applying our optimized rule framework to a population survey real-life data set enables us to discover interesting underlying correlations among the attributes.
fast window correlations over uncooperative time series. data arriving in time order (a data stream) arises in fields including physics, finance, medicine, and music, to name a few. often the data comes from sensors (in physics and medicine for example) whose data rates continue to improve dramatically as sensor technology improves. further, the number of sensors is increasing, so correlating data between sensors becomes ever more critical in order to distill knowlege from the data. in many applications such as finance, recent correlations are of far more interest than long-term correlation, so correlation over sliding windows (windowed correlation) is the desired operation. fast response is desirable in many applications (e.g., to aim a telescope at an activity of interest or to perform a stock trade). these three factors -- data size, windowed correlation, and fast response -- motivate this work.previous work [10, 14] showed how to compute pearson correlation using fast fourier transforms and wavelet transforms, but such techniques don't work for time series in which the energy is spread over many frequency components, thus resembling white noise. for such "uncooperative" time series, this paper shows how to combine several simple techniques -- sketches (random projections), convolution, structured random vectors, grid structures, and combinatorial design -- to achieve high performance windowed pearson correlation over a variety of data sets.