waashk/dblp huggingface dataset:
turning clusters into patterns: rectangle-based discriminative data description. the ultimate goal of data mining is to extract knowledge from massive data. knowledge is ideally represented as human-comprehensible patterns from which end-users can gain intuitions and insights. yet not all data mining methods produce such readily understandable knowledge, e.g., most clustering algorithms output sets of points as clusters. in this paper, we perform a systematic study of cluster description that generates interpretable patterns from clusters. we introduce and analyze novel description formats leading to more expressive power, motivate and define novel description problems specifying different trade-offs between interpretability and accuracy. we also present effective heuristic algorithms together with their empirical evaluations.
object identification with constraints. object identification aims at identifying different representations of the same object based on noisy attributes such as descriptions of the same product in different online shops or references to the same paper in different publications. numerous solutions have been proposed for solving this task, almost all of them based on similarity functions of a pair of objects. although today the similarity functions are learned from a set of labeled training data, the structural information given by the labeled data is not used. by formulating a generic model for object identification we show how almost any proposed identification model can easily be extended for satisfying structural constraints. therefore we propose a model that uses structural information given as pairwise constraints to guide collective decisions about object identification in addition to a learned similarity measure. we show with empirical experiments on public and on real-life data that combining both structural information and attribute-based similarity enormously increases the overall performance for object identification tasks.
deploying approaches for pattern refinement in text mining. text mining is the technique that helps users find useful information from a large amount of digital text documents on the web or databases. instead of the keyword-based approach which is typically used in this field, the pattern-based model containing frequent sequential patterns is employed to perform the same concept of tasks. however, how to effectively use these discovered patterns is still a big challenge. in this study, we propose two approaches based on the use of pattern deploying strategies. the performance of the pattern deploying algorithms for text mining is investigated on the reuters dataset rcv1 and the results show that the effectiveness is improved by using our proposed pattern refinement approaches.
rule-based platform for web user profiling. this paper discusses a research project: rule-based web user profiling platform. in this platform, usage data are encoded as a sequence of events, each of which represents an action performed by a user on a web service at a given time. an event template is proposed to define event models for different web services. the platform is rule-based. rules define profile metrics and determine how to compute profile metrics from usage events. a prototype of the platform was implemented and was applied to generate profiles from page view events. the major contribution of the work is the rule-based approach to user profiling. it is the rules and the event template that provide the flexibility to allow the platform to be configured for different web services.
high-performance unsupervised relation extraction from large corpora. we present uries -- an unsupervised relation identification and extraction system. the system automatically identifies interesting binary relations between entities in the input corpus, and then proceeds to extract a large number of instances of these relations. the system discovers relations by clustering frequently co-occuring pairs of entities, based on the contexts in which they appear. its complex pattern-based representation of the contexts allows the clustering step to achieve very high precision, sufficient for the clusters to perform as sets of seeds for bootstrapping a high-recall relation extraction process. in a series of experiments we demonstrate the successful performance of uries and compare it to the two existing systems -- a weakly supervised high-recall web relation extraction system called sres, and an unsupervised relation identification system that uses a simpler bag-of-words representation of contexts. the experiments show that uries performs comparably to sres, but without any supervision, and that such performance is due to the power of its complex contexts representation and to its novel candidate selection method.
improving nearest neighbor classifier using tabu search and ensemble distance metrics. the nearest-neighbor (nn) classifier has long been used in pattern recognition, exploratory data analysis, and data mining problems. a vital consideration in obtaining good results with this technique is the choice of distance function, and correspondingly which features to consider when computing distances between samples. in this paper, a new ensemble technique is proposed to improve the performance of nn classifier. the proposed approach combines multiple nn classifiers, where each classifier uses a different distance function and potentially a different set of features (feature vector). these feature vectors are determined for each distance metric using simple voting scheme incorporated in tabu search (ts). the proposed ensemble classifier with different distance metrics and different feature vectors (ts-df/nn) is evaluated using various benchmark data sets from uci machine learning repository. results have indicated a significant increase in the performance when compared with various well-known classifiers. furthermore, the proposed ensemble method is also compared with ensemble classifier using different distance metrics but with same feature vector (with or without feature selection (fs)).
high quality, efficient hierarchical document clustering using closed interesting itemsets. high dimensionality remains a significant challenge for document clustering. recent approaches used frequent itemsets and closed frequent itemsets to reduce dimensionality, and to improve the efficiency of hierarchical document clustering. in this paper, we introduce the notion of "closed interesting" itemsets (i.e. closed itemsets with high interestingness). we provide heuristics such as "super item" to efficiently mine these itemsets and show that they provide significant dimensionality reduction over closed frequent itemsets. using "closed interesting" itemsets, we propose a new, sub-linearly scalable, hierarchical document clustering method that outperforms state of the art agglomerative, partitioning and frequent-itemset based methods both in terms of clustering quality and runtime performance, without requiring dataset specific parameter tuning. we evaluate twenty interestingness measures and show that when used to generate "closed interesting" itemsets, and to select parent nodes, mutual information, added value, yule's q and chi- square offer best clustering performance.
saxually explicit images: finding unusual shapes. over the past three decades, there has been a great deal of research on shape analysis, focusing mostly on shape indexing, clustering, and classification. in this work, we introduce the new problem of finding shape discords, the most unusual shapes in a collection. we motivate the problem by considering the utility of shape discords in diverse domains including zoology, anthropology, and medicine. while the brute force search algorithm has quadratic time complexity, we avoid this by using locality-sensitive hashing to estimate similarity between shapes which enables us to reorder the search more efficiently. an extensive experimental evaluation demonstrates that our approach can speed up computation by three to four orders of magnitude.
plagiarism detection in arxiv. we describe a large-scale application of methods for finding plagiarism in research document collections. the methods are applied to a collection of 284,834 documents collected by arxiv.org over a 14 year period, covering a few different research disciplines. the methodology effi- ciently detects a variety of problematic author behaviors, and heuristics are developed to reduce the number of false positives. the methods are also efficient enough to imple- ment as a real-time submission screen for a collection many times larger.
semantic kernels for text classification based on topological measures of feature similarity. in this paper we propose a new approach to the design of semantic smoothing kernels for text classification. these kernels implicitly encode a superconcept expansion in a semantic network using well-known measures of term similarity. the experimental evaluation on two different datasets indicates that our approach consistently improves performance in situations of little training data and data sparseness.
an interactive semantic video mining and retrieval platform--application in transportation surveillance video for incident detection. understanding and retrieving videos based on their semantic contents is an important research topic in multimedia data mining and has found various real-world applications. most existing video analysis techniques focus on the low level visual features of video data. however, there is a "semantic gap" between the machine-readable features and the high level human concepts i.e. human understanding of the video content. in this paper, an interactive platform for semantic video mining and retrieval is proposed using relevance feedback (rf), a popular technique in the area of content-based image retrieval (cbir). by tracking semantic objects in a video and then modeling spatio-temporal events based on object trajectories and object interactions, the proposed interactive learning algorithm in the platform is able to mine the spatio-temporal data extracted from the video. an iterative learning process is involved in the proposed platform, which is guided by the user's response to the retrieved results. although the proposed video retrieval platform is intended for general use and can be tailored to many applications, we focus on its application in traffic surveillance video database retrieval to demonstrate the design details. the effectiveness of the algorithm is demonstrated by our experiments on real-life traffic surveillance videos.
solution path for semi-supervised classification with manifold regularization. with very low extra computational cost, the entire solution path can be computed for various learning algorithms like support vector classification (svc) and support vector regression (svr). in this paper, we extend this promising approach to semi-supervised learning algorithms. in particular, we consider finding the solution path for the laplacian support vector machine (lapsvm) which is a semi-supervised classification model based on manifold regularization. one advantage of the this algorithm is that the coefficient path is piecewise linear with respect to the regularization parameter, hence its computational complexity is quadratic in the number of labeled examples.
a simple yet effective data clustering algorithm. in this paper, we use a simple concept based on k-reverse nearest neighbor digraphs, to develop a framework record for clustering and outlier detection. we developed three algorithms - (i) record algorithm (requires one parameter), (ii) agglomerative record algorithm (no parameters required) and (iii) stability-based record algorithm( no parameters required). our experimental results with published datasets, synthetic and real-life datasets show that record not only handles noisy data, but also identifies the relevant clusters. our results are as good as (if not better than) the results got from other algorithms.
coala: a novel approach for the extraction of an alternate clustering of high quality and high dissimilarity. cluster analysis has long been a fundamental task in data mining and machine learning. however, traditional clustering methods concentrate on producing a single solution, even though multiple alternative clusterings may exist. it is thus difficult for the user to validate whether the given solution is in fact appropriate, particularly for large and complex datasets. in this paper we explore the critical requirements for systematically finding a new clustering, given that an already known clustering is available and we also propose a novel algorithm, coala, to discover this new clustering. our approach is driven by two important factors; dissimilarity and quality. these are especially important for finding a new clustering which is highly informative about the underlying structure of data, but is at the same time distinctively different from the provided clustering. we undertake an experimental analysis and show that our method is able to outperform existing techniques, for both synthetic and real datasets.
optimal segmentation using tree models. sequence data are abundant in application areas such as computational biology, environmental sciences, and telecommunications. many real-life sequences have a strong segmental structure, with segments of different complexities. in this paper we study the description of sequence segments using variable length markov chains (vlmcs), also known as tree models. we discover the segment boundaries of a sequence and at the same time we compute a vlmc for each segment. we use the bayesian information criterion (bic) and a variant of the minimum description length (mdl) principle that uses the krichevsky-trofimov (kt) code length to select the number of segments of a sequence. on dna data the method selects segments that closely correspond to the annotated regions of the genes.
local correlation tracking in time series. we address the problem of capturing and tracking local correlations among time evolving time series. our approach is based on comparing the local auto-covariance matrices (via their spectral decompositions) of each series and generalizes the notion of linear cross-correlation. in this way, it is possible to concisely capture a wide variety of local patterns or trends. our method produces a general similarity score, which evolves over time, and accurately reflects the changing relationships. finally, it can also be estimated incrementally, in a streaming setting. we demonstrate its usefulness, robustness and efficiency on a wide range of real datasets.
identifying follow-correlation itemset-pairs. an association rule a\to b is useful to predict that b will likely occur when a occurs. this is a classical association rule. in real world applications, such as bioinformatics and medical research, there are many follow correlations between itemsets a and b: b likely occurs n times after a occurred m times, wrote to \le a^m , b^n \ge. we refer to this follow-correlation as p3.1 itemset-pairs because \le a^3 , b^1\ge like that in example 2 should be uninterested in association analysis. this paper designs an efficient algorithm for identifying p3.1 itemset-pairs in sequential data. we experimentally evaluate our approach, and demonstrate that the proposed approach is efficient and promising.
large scale detection of irregularities in accounting data. in recent years, there have been several large accounting frauds where a company's financial results have been intentionally misrepresented by billions of dollars. in response, regulatory bodies have mandated that auditors perform analytics on detailed financial data with the intent of discovering such misstatements. for a large auditing firm, this may mean analyzing millions of records from thousands of clients. this paper proposes techniques for automatic analysis of company general ledgers on such a large scale, identifying irregularities - which may indicate fraud or just honest errors - for additional review by auditors. these techniques have been implemented in a prototype system, called sherlock, which combines aspects of both outlier detection and classification. in developing sherlock, we faced three major challenges: developing an efficient process for obtaining data from many heterogeneous sources, training classifiers with only positive and unlabeled examples, and presenting information to auditors in an easily interpretable manner. in this paper, we describe how we addressed these challenges over the past two years and report on experiments evaluating sherlock.
applying data mining to pseudo-relevance feedback for high performance text retrieval. in this paper, we investigate the use of data mining, in particular the text classification and co-training techniques, to identify more relevant passages based on a small set of labeled passages obtained from the blind feedback of a retrieval system. the data mining results are used to expand query terms and to re-estimate some of the parameters used in a probabilistic weighting function. we evaluate the data mining based feedback method on the trec hard data set. the results show that data mining can be successfully applied to improve the text retrieval performance. we report our experimental findings in detail.
automatic single-organ segmentation in computed tomography images. in this paper, we propose a hybrid approach for automatic single-organ segmentation in computed tomography (ct) data. the approach consists of three stages: first, a probability image of the organ of interest is obtained by applying a binary classification model obtained using pixel-based texture features; second, an adaptive split-and-merge segmentation algorithm is applied on the organ probability image to remove the noise introduced by the misclassified pixels; and third, the segmented organ's boundaries from the previous stage are iteratively refined using a region growing algorithm. while we applied our approach for liver segmentation in 2-d ct images, a challenging and important task in many medical applications, the proposed approach can be applied for the segmentation of any other organ in ct images. moreover, the proposed approach can be extended to perform automatic multiple organ segmentation and to build context-sensitive reporting tools for computer-aided diagnosis applications.
probabilistic segmentation and analysis of horizontal cells. because images of neurons show interweaved processes from multiple cells, it is hard to determine which pixels belong to each cell, and consequently to analyze the images automatically. to manage these difficulties, we introduce probabilistic segmentation, in which each pixel is assigned a probability of belonging to each cell instead of being categorically assigned to one cell. we propose a randomized algorithm for probabilistic segmentation. the algorithm is based on repeated, intensity-weighted random walks on the image, and leads to improved segmentation quality. analysis and mining techniques can utilize the more nuanced and complete information that the probabilistic segmentation yields about an image. such techniques can then compute probabilistic values, which indicate the level of confidence that can be placed in them.
mixed-drove spatio-temporal co-occurence pattern mining: a summary of results. mixed-drove spatio-temporal co-occurrence patterns (mdcops) represent subsets of object-types that are located together in space and time. discovering mdcops is an important problem with many applications such as identifying tactics in battlefields, games, and predator-prey interactions. however, mining mdcops is computationally very expensive because the interest measures are computationally complex, datasets are larger due to the archival history, and the set of candidate patterns is exponential in the number of object-types. we propose a monotonic composite interest measure for discovering mdcops and a novel mdcop mining algorithm. analytical and experimental results show that the proposed algorithm is correct and complete. results also show the proposed method is computationally more efficient than naïve alternatives.
dirichlet aspect weighting: a generalized em algorithm for integrating external data fields with semantically structured queries by using gradient projection method. in this paper we address the problem of document retrieval with semantically structured queries - queries where each term has a tagged field label. we introduce dirichlet aspect weighting model which integrates terms from external databases into the query language model in a bayesian learning framework. for this model, the dirichlet prior distribution is governed by parameters which depend on the number of fields in the external databases. this model needs additional examples to be augmented to the semantically structured query. these examples are obtained using pseudo relevance feedback. we formulate a loglikelihood function for the dirichlet aspect weighting model and maximize it using a novel generalized em algorithm. comparison of the results of dirichlet aspect weighting model on trec 2005 genomics track dataset with baseline methods using pseudo relevance feedback, while incorporating terms from external databases shows an improvement.
manifold clustering of shapes. shape clustering can significantly facilitate the automatic labeling of objects present in image collections. for example, it could outline the existing groups of pathological cells in a bank of cyto-images; the groups of species on photographs collected from certain aerials; or the groups of objects observed on surveillance scenes from an office building. here we demonstrate that a nonlinear projection algorithm such as isomap can attract together shapes of similar objects, suggesting the existence of isometry between the shape space and a low dimensional nonlinear embedding. whenever there is a relatively small amount of noise in the data, the projection forms compact, convex clusters that can easily be learned by a subsequent partitioning scheme. we further propose a modification of the isomap projection based on the concept of degree-bounded minimum spanning trees. the proposed approach is demonstrated to move apart bridged clusters and to alleviate the effect of noise in the data.
an efficient reference-based approach to outlier detection in large datasets. a bottleneck to detecting distance and density based outliers is that a nearest-neighbor search is required for each of the data points, resulting in a quadratic number of pairwise distance evaluations. in this paper, we propose a new method that uses the relative degree of density with respect to a fixed set of reference points to approximate the degree of density defined in terms of nearest neighbors of a data point. the running time of our algorithm based on this approximation is o(r_n log n) where n is the size of dataset and r is the number of reference points. candidate outliers are ranked based on the outlier score assigned to each data point. theoretical analysis and empirical studies show that our method is effective, efficient, and highly scalable to very large datasets.