waashk/dblp huggingface dataset:
finding informative commonalities in concept collections. the problem of finding commonalities characterizes several knowledge management scenarios involving collection of resources. the automatic extraction of shared features in a collection of resource descriptions formalized in accordance with a logic language has been in fact widely investigated in the past. in particular, with reference to description logics concept descriptions, least common subsumers have been specifically introduced. nevertheless, such studies focused on identifying features shared by the whole collection. the paper proposes instead novel reasoning services in description logics, aimed at identifying commonalities in a significant portion of the collection, rather than in the collection as a whole. in particular, common subsumers adding informative content to the one provided by the least common subsumer are here investigated. the new services are useful in all scenarios where features are not required to be fully shared, like the one motivating our research: core competence extraction in knowledge intensive companies.
on low dimensional random projections and similarity search. random projection (rp) is a common technique for dimensionality reduction under l2 norm for which many significant space embedding results have been demonstrated. however, many similarity search applications often require very low dimension embeddings in order to reduce overhead and boost performance. inspired by the use of symmetric probability distributions in previous work, we propose a novel rp algorithm, beta random projection, and give its probabilistic analyses based on beta and gaussian approximations. we evaluate the algorithm in terms of standard similarity metrics with other rp algorithms as well as the singular value decomposition (svd). our experimental results show that brp preserves both similarity metrics well and, under various dataset types including random point sets, text (trec5) and images, provides sharper and consistent performance.
real-time new event detection for video streams. online detection of video clips that present previously unseen events in a video stream is still an open challenge to date. for this online new event detection (oned) task, existing studies mainly focus on optimizing the detection accuracy instead of the detection efficiency. as a result, it is difficult for existing systems to detect new events in real time, especially for large-scale video collections such as the video content available on the web. in this paper, we propose several scalable techniques to improve the video processing speed of a baseline oned system by orders of magnitude without sacrificing much detection accuracy. first, we use text features alone to filter out most of the non-new-event clips and to skip those expensive but unnecessary steps including image feature extraction and image similarity computation. second, we use a combination of indexing and compression methods to speed up text processing. we implemented a prototype of our optimized oned system on top of ibm's system s. the effectiveness of our techniques is evaluated on the standard trecvid 2005 benchmark, which demonstrates that our techniques can achieve a 480-fold speedup with detection accuracy degraded less than 5%.
characterization of tpc-h queries for a column-oriented database on a dual-core amd athlon processor. in this paper, we characterize the performance of the tpc-h benchmark for a popular column-oriented database called monetdb running on a dual-core amd athlon machine. specifically, we measure the performance of key microarchitectural components and analyze in detail the nature of various stalls namely cache stalls, branch misprediction stalls and resource stalls. we compare our results with published results on the characterization of tpc-h for row-oriented databases. as opposed to the previous approaches, we use thread-level monitoring of database threads to study the performance of the database in isolation from the rest of the system.
blog site search using resource selection. a blog site consists of many individual blog postings. current blog search services focus on retrieving postings but there is also a need to identify relevant blog sites. blog site search is similar to resource selection in distributed information retrieval, in that the target is to find relevant collections of documents. we introduce resource selection techniques for blog site search and evaluate their performance. further, we propose a "diversity factor" that measures the topic diversity of each blog site. our results show that the appropriate combination of the resource selection techniques and the diversity factor can achieve significant improvements in retrieval performance compared to baselines. we also report results using these techniques on the trec blog distillation task.
academic conference homepage understanding using constrained hierarchical conditional random fields. we address the problem of academic conference homepage understanding for the semantic web. this problem consists of three labeling tasks - labeling conference function pages, function blocks, and attributes. different from traditional information extraction tasks, the data in academic conference homepages has complex structural dependencies across multiple web pages. in addition, there are logical constraints in the data. in this paper, we propose a unified approach, constrained hierarchical conditional random fields, to accomplish the three labeling tasks simultaneously. in this approach, complex structural dependencies can be well described. also, the constrained viterbi algorithm in the inference process can avoid logical errors. experimental results on real world conference data have demonstrated that this approach performs better than cascaded labeling methods by 3.6% in f1-measure and that the constrained inference process can improve the accuracy by 14.3%. based on the proposed approach, we develop a prototype system of use-oriented semantic academic conference calendar. the user simply needs to specify what conferences he/she is interested in. subsequently, the system finds, extracts, and updates the semantic information from the web, and then builds a calendar automatically for the user. the semantic conference data can be used in other applications, such as finding sponsors and finding experts. the proposed approach can be used in other information extraction tasks as well.
are click-through data adequate for learning web search rankings? learning-to-rank algorithms, which can automatically adapt ranking functions in web search, require a large volume of training data. a traditional way of generating training examples is to employ human experts to judge the relevance of documents. unfortunately, it is difficult, time-consuming and costly. in this paper, we study the problem of exploiting click-through data for learning web search rankings that can be collected at much lower cost. we extract pairwise relevance preferences from a large-scale aggregated click-through dataset, compare these preferences with explicit human judgments, and use them as training examples to learn ranking functions. we find click-through data are useful and effective in learning ranking functions. a straightforward use of aggregated click-through data can outperform human judgments. we demonstrate that the strategies are only slightly affected by fraudulent clicks. we also reveal that the pairs which are very reliable, e.g., the pairs consisting of documents with large click frequency differences, are not sufficient for learning.
collaborative partitioning with maximum user satisfaction. our collaborative partitioning model posits a bicriteria objective in which we seek the best item clustering that satisfies the most users at the highest level of satisfaction. we consider two basic methods for determining user satisfaction. the first method is based on how well each user's preferences match a given partition, and the second method is based on average correlation scores taken over sufficiently large subpopulations of users. we show these problems are np-hard and develop a set of heuristic approaches for solving them. we provide lower bounds on the satisfaction level on random data, and error bounds in the planted partition model, which provide confidence levels for our heuristic methods. finally, we present experiments on several real examples that demonstrate the effectiveness of our framework.
a two-stage text mining model for information filtering. mismatch and overload are the two fundamental issues regarding the effectiveness of information filtering. both term-based and pattern (phrase) based approaches have been employed to address these issues. however, they all suffer from some limitations with regard to effectiveness. this paper proposes a novel solution that includes two stages: an initial topic filtering stage followed by a stage involving pattern taxonomy mining. the objective of the first stage is to address mismatch by quickly filtering out probable irrelevant documents. the threshold used in the first stage is motivated theoretically. the objective of the second stage is to address overload by apply pattern mining techniques to rationalize the data relevance of the reduced document set after the first stage. substantial experiments on rcv1 show that the proposed solution achieves encouraging performance.
detecting significant distinguishing sets among bi-clusters. a fundamental task of data analysis is comprehending what distinguishes clusters found within the data. we present the problem of mining distinguishing sets; which seeks to find sets of objects or attributes that induce the most incremental change between adjacent bi-clusters of a binary dataset. viewing the lattice of bi-clusters formed within a data set as a weighted directed graph, we mine the most significant distinguishing sets by growing a maximal-cost spanning tree of the lattice.
supporting sub-document updates and queries in an inverted index. inverted indexes have become the standard indexing method for supporting search queries in a variety of content-based applications. examples of such applications include enterprise document management, e-mail, web search, and social networks. one shortcoming in current inverted index designs is that they support only document-level updates, forcing a full document to be reindexed even if just part of it changes. this paper describes a new inverted index design that enables applications to break a document into semantically meaningful sub-documents or "sections". each section of a document can be updated separately, but search queries can still work seamlessly across sections. our index design is motivated by applications where there is metadata associated with each document that tends to be smaller and more frequently updated than the document's content, but at the same time, it is desireable to search the metadata and content with the same index structure. a novel self-optimizing query execution algorithm is described to efficiently join the sections of a document in the inverted index. experimental results on trec and patent data are provided, showing that sections can dramatically improve overall system throughput on a mixed workload of updates and queries.
intra-document structural frequency features for semi-supervised domain adaptation. in this work we try to bridge the gap often encountered by researchers who find themselves with few or no labeled examples from their desired target domain, yet still have access to large amounts of labeled data from other related, but distinct source domains, and seemingly no way to transfer knowledge from one to the other. experimentally, we focus on the problem of extracting protein mentions from academic publications in the field of biology, where the source domain data are abstracts labeled with protein mentions, and the target domain data are wholly unlabeled captions. we mine the large number of such full text articles freely available on the internet in order to supplement the limited amount of annotated data available. by exploiting the explicit and implicit common structure of the different subsections of these documents, including the unlabeled full text, we are able to generate robust features that are insensitive to changes in marginal and conditional distributions of classes and data across domains. we supplement these domain-insensitive features with automatically obtained high-confidence positive and negative predictions on the target domain to learn extractors that generalize well from one section of a document to another. finally, lacking labeled target testing data, we employ comparative user preference studies to evaluate the relative performance of the proposed methods with respect to existing baselines.
a novel optimization approach to efficiently process aggregate similarity queries in metric access methods. a similarity query considers an element as the query center and searches a dataset to find either the elements far up to a bounding radius or the k nearest ones from the query center. several algorithms have been developed to efficiently execute similarity queries. however, there are queries that require more than one center, which we call aggregate similarity queries. such queries appear when the user gives multiple desirable examples, and requests data elements that are similar to all of the examples, as in the case of applying relevance feedback. here we give the first algorithms that can handle aggregate similarity queries on metric access methods (mam) such as the m-tree and slim-tree. our method, which we call metric aggregate similarity search (mass) has the following properties: (a) it requires only the triangle inequality property; (b) it guarantees no false-dismissals, as we prove that it lower-bounds the aggregate distance scores; (c) it can work with any mam; (d) it can handle any number of query centers, which are either scattered all over the space or concentrated on a restricted region. experiments on both real and synthetic data show that our method scales on both the number of elements and, if the dataset is in a spatial domain, also on its dimensionality. moreover, it achieves better results than previous related methods.
answering general time sensitive queries. time is an important dimension of relevance for a large number of searches, such as over blogs and news archives. so far, research on searching over such collections has largely focused on locating topically similar documents for a query. unfortunately, topic similarity alone is not always sufficient for document ranking. in this paper, we observe that, for an important class of queries that we call time-sensitive queries, the publication time of the documents in a news archive is important and should be considered in conjunction with the topic similarity to derive the final document ranking. earlier work has focused on improving retrieval for "recency" queries that target recent documents. we propose a more general framework for handling time-sensitive queries and we automatically identify the important time intervals that are likely to be of interest for a query. then, we build scoring techniques that seamlessly integrate the temporal aspect into the overall ranking mechanism. we extensively evaluated our techniques using a variety of news article data sets, including trec data as well as real web data analyzed using the amazon mechanical turk. we examined several alternatives for detecting the important time intervals for a query over a news archive and for incorporating this information in the retrieval process. our techniques are robust and significantly improve result quality for time-sensitive queries compared to state-of-the-art retrieval techniques.
decomposition of terminology graphs for domain knowledge acquisition. we propose a graph decomposition algorithm for analyzing the structure of complex graph networks. after multi-word term extraction, we apply techniques from text mining and visual analytics in a novel way by integrating symbolic and numeric information to build clusters of domain topics. terms are clustered based on surface linguistic variations and clusters are inserted in an association network based on their intersection with documents. the graph is then decomposed based on atom graph structure into central (non-decomposable) atom and peripheral atoms. the whole process is applied to publications from the sloan digital sky survey (sdss) project in the astronomy field. the mapping obtained was evaluated by a domain expert and appeared to have captured interesting conceptual relations between different domain topics.
a consensus based approach to constrained clustering of software requirements. managing large-scale software projects involves a number of activities such as viewpoint extraction, feature detection, and requirements management, all of which require a human analyst to perform the arduous task of organizing requirements into meaningful topics and themes. automating these tasks through the use of data mining techniques such as clustering could potentially increase both the efficiency of performing the tasks and the reliability of the results. unfortunately, the unique characteristics of this domain, such as high dimensional, sparse, noisy data sets, resulting from short and ambiguous expressions of need, as well as the need for the interactive engagement of stakeholders at various stages of the process, present difficult challenges for standard clustering algorithms. in this paper, we propose a semi-supervised clustering framework, based on a combination of consensus-based and constrained clustering techniques, which can effectively handle these challenges. specifically, we provide a probabilistic analysis for informative constraint generation based on a co-association matrix, and utilize consensus clustering to combine multiple constrained partitions in order to generate high-quality, robust clusters. our approach is validated through a series of experiments on six well-studied trec data sets and on two sets of user requirements.
using sequence classification for filtering web pages. web pages often contain text that is irrelevant to their main content, such as advertisements, generic format elements, and references to other pages on the same site. when used by automatic content-processing systems, e.g., for web indexing, text classification, or information extraction, this irrelevant text often produces substantial amount of noise. this paper describes a trainable filtering system based on a feature-rich sequence classifier that removes irrelevant parts from pages, while keeping the content intact. most of the features the system uses are purely form-related: html tags and their positions, sizes of elements, etc. this keeps the system general and domain-independent. we also experiment with content words and show that while they perform very poorly alone, they can slightly improve the performance of pure-form features, without jeopardizing the domain-independence. our system achieves very high accuracy (95% and above) on several collections of web pages. we also do a series of tests with different features and different classifiers, comparing the contribution of different components to the system performance, and comparing two known sequence classifiers, robust risk minimization (rrm) and conditional random fields (crf), in a novel setting.
an effective statistical approach to blog post opinion retrieval. finding opinionated blog posts is still an open problem in information retrieval, as exemplified by the recent trec blog tracks. most of the current solutions involve the use of external resources and manual efforts in identifying subjective features. in this paper, we propose a novel and effective dictionary-based statistical approach, which automatically derives evidence for subjectivity from the blog collection itself, without requiring any manual effort. our experiments show that the proposed approach is capable of achieving remarkable and statistically significant improvements over robust baselines, including the best trec baseline run. in addition, with relatively little computational costs, our proposed approach provides an effective performance in retrieving opinionated blog posts, which is as good as a computationally expensive approach using natural language processing techniques.
proqid: partial restarts of queries in distributed databases. in a number of application areas, distributed database systems can be used to provide persistent storage of data while providing efficient access for both local and remote data. with an increasing number of sites (computers) involved in a query, the probability of failure at query time increases. recovery has previously only focused on database updates while query failures have been handled by complete restart of the query. this technique is not always applicable in the context of large queries and queries with deadlines. in this paper we present an approach for partial restart of queries that incurs minimal extra network traffic during query recovery. based on results from experiments on an implementation of the partial restart technique in a distributed database system, we demonstrate its applicability and significant reduction of query cost in the presence of failures.
fast correlation analysis on time series datasets. there has been increasing interest for efficient techniques for fast correlation analysis of time series data in different application domains. we present three algorithms for (1) bivariate correlation queries, (2) multivariate correlation queries, and (3) correlation queries based on a new correlation measure we introduce using dynamic time warping. to support these algorithms, we use a variant of the compact multi-resolution index (cmri). in addition to conventional nearest neighbor and range queries supported by cmri, the proposed algorithms compute all answers to user-defined, ad hoc and parametric correlation queries. the results of our experiments indicate a speed-up of two orders of magnitude over the brute force algorithm, and an order of magnitude improvement on average, while offering more functionalities than provided by existing techniques such as statstream and the spatial cone tree.
group-based learning: a boosting approach. this paper points out that many machine learning problems in ir should be and can be formalized in a novel way, referred to as 'group-based learning'. in group-based learning, it is assumed that training data as well as testing data consist of groups. the classifier is created and utilized across groups. furthermore, evaluation in testing and also in training are conducted at group level, with the use of evaluation measures defined on a group. this paper addresses the problem and presents a boosting algorithm to perform the new learning task. the algorithm, referred to as adaboost.group, is proved to be able to improve accuracies in terms of group-based measures during training.
a novel statistical chinese language model and its application in pinyin-to-character conversion. in this paper, we present a novel chinese language model, and study its applications, in particular in chinese pinyin-to-character conversion. in the new model, each word is associated with supporting context constructed by mining the frequent sets of nearby phrases and their distances to the word. such information was usually overlooked in previous n-gram model and its variants. we apply the model to chinese pinyin-to-character conversion and find that it offers a better solution to chinese input. the model has lower perplexity in our evaluation and higher prediction accuracy than the state-of-the-art n-gram markov model for chinese language.