waashk/dblp huggingface dataset:
trada: tree based ranking function adaptation. machine learned ranking approaches have shown successes in web search engines. with the increasing demands on developing effective ranking functions for different search domains, we have seen a big bottleneck, i.e., the problem of insufficient training data, which has significantly limited the fast development and deployment of machine learned ranking functions for different web search domains. in this paper, we propose a new approach called tree based ranking function adaptation ("tree adaptation") to address this problem. tree adaptation assumes that ranking functions are trained with regression-tree based modeling methods, such as gradient boosting trees. it takes such a ranking function from one domain and tunes its tree-based structure with a small amount of training data from the target domain. the unique features include (1) it can automatically identify the part of model that needs adjustment for the new domain, (2) it can appropriately weight training examples considering both local and global distributions. experiments are performed to show that tree adaptation can provide better-quality ranking functions for a new domain, compared to other modeling methods.
modeling multi-step relevance propagation for expert finding. an expert finding system allows a user to type a simple text query and retrieve names and contact information of individuals that possess the expertise expressed in the query. this paper proposes a novel approach to expert finding in large enterprises or intranets by modeling candidate experts (persons), web documents and various relations among them with so-called expertise graphs. as distinct from the state of-the-art approaches estimating personal expertise through one-step propagation of relevance probability from documents to the related candidates, our methods are based on the principle of multi-step relevance propagation in topic specific expertise graphs. we model the process of expert finding by probabilistic random walks of three kinds: finite, infinite and absorbing. experiments on trec enterprise track data originating from two large organizations show that our methods using multi-step relevance propagation improve over the baseline one-step propagation based method in almost all cases.
identification of gene function using prediction by partial matching (ppm) language models. in this paper, we describe the utilization of text encoding and prediction by partial matching language modeling to identify gene functions within abstracts of biomedical papers. the national center for biotechnology information has "generif" - a collection of the best possible functional representations for a subset of abstracts from pubmed. we use generif to test the efficiency of our technique. we discuss the methodology adopted to construct models necessary to enable the text mining toolkit to distinguish between gene functions and the rest of the abstract (non gene functions). we also describe the similarity based approach we deploy on the list of automatically annotated functions to generate the most likely gene function representative of the paper. the results indicate that our combined approach to identify gene functions in scientific abstracts performs very well on both precision and recall, and therefore presents exciting opportunities for use in extracting other entities embedded in scientific text.
combining concept hierarchies and statistical topic models. statistical topic models provide a general data-driven framework for automated discovery of high-level knowledge from large collections of text documents. while topic models can potentially discover a broad range of themes in a data set, the interpretability of the learned topics is not always ideal. human-defined concepts, on the other hand, tend to be semantically richer due to careful selection of words to define concepts but they tend not to cover the themes in a data set exhaustively. in this paper, we propose a probabilistic framework to combine a hierarchy of human-defined semantic concepts with statistical topic models to seek the best of both worlds. experimental results using two different sources of concept hierarchies and two collections of text documents indicate that this combination leads to systematic improvements in the quality of the associated language models as well as enabling new techniques for inferring and visualizing the semantics of a document.
online spam-blog detection through blog search. in this work, we propose a novel post-indexing spam-blog (or splog) detection method, which capitalizes on the results returned by blog search engines. more specifically, we analyze the search results of a sequence of temporally-ordered queries returned by a blog search engine, and build and maintain blog profiles for those blogs whose posts frequently appear in the top-ranked search results. with the blog profiles, 4 splog scoring functions were evaluated using real data collected from a popular blog search engine. our experiments show that the proposed method could effectively detect splogs with a high accuracy.
achieving both high precision and high recall in near-duplicate detection. to find near-duplicate documents, fingerprint-based paradigms such as broder's shingling and charikar's simhash algorithms have been recognized as effective approaches and are considered the state-of-the-art. nevertheless, we see two aspects of these approaches which may be improved. first, high score under these algorithms' similarity measurement implies high probability of similarity between documents, which is different from high similarity of the documents. but how similar two documents are is what we really need to know. second, there has to be a tradeoff between hash-code length and hash-code multiplicity in fingerprint paradigms, which makes it hard to maintain a satisfactory recall level while improving precision. in this paper our contributions are two-folded. first, we propose a framework for implementing the longest common subsequence (lcs) as a similarity measurement in reasonable computing time, which leads to both high precision and recall. second, we present an algorithm to get a trustable partition from the lcs to reduce the negative impact from templates used in web page design. a comprehensive experiment was conducted to evaluate our method in terms of its effectiveness, efficiency, and quality of result. more specifically, the method has been successfully used to partition a set of 430 million web pages into 68 million subsets of similar pages, which demonstrates its effectiveness. for quality, we compared our method with simhash and a cosine-based method through a sampling process (cosine is compared to lcs as an alternative similarity measurement). the result showed that our algorithm reached an overall precision of 0.95 while simhash was 0.71 and cosine was 0.82. at the same time our method obtains 1.86 times as much recall as simhash and 1.56 times as much recall as cosine. comparison experiment was also done for documents in the same web sites. for that, our algorithm, simhash and cosine find almost the same number of true-positives at a precision of 0.91, 0.50 and 0.63 respectively. in terms of efficiency, our algorithm takes 118 hours to process the whole archive of 430 million topic-type pages on a cluster of six linux boxes, at the same time the processing time of simhash and cosine is 94 hours and 68 hours respectively. when considering the need of word segmentation for languages such as chinese, the processing time of cosine should be multiplied and in our experiment it is 602 hours.
some rewrite optimizations of db2 xquery navigation. ibm® db2® 9 is a truly hybrid commercial database system that combines xml and relational data. it provides native support for xml storage and indexing, and query evaluation support for xquery. by building a hybrid system, the designers of db2 9 were able to use the existing sql query evaluation and optimization techniques to develop similar methods for xquery. however, sql and xquery are sufficiently different that new optimization techniques can and are being developed in the new xquery domain. this paper describes a few such techniques, all based on static rewrites of xquery expressions.
a language for manipulating clustered web documents results. we propose a novel conception language for exploring the results retrieved by several internet search services (like search engines) that cluster retrieved documents. the goal is to offer users a tool to discover relevant hidden relationships between clustered documents. the proposal is motivated by the observation that visualization paradigms, based on either the ranked list or clustered results, do not allow users to fully exploit the combined use of several search services to answer a request. when the same query is submitted to distinct search services, they may produce partially overlapped clustered results, where clusters identified by distinct labels collect some common documents. moreover, clusters with similar labels, but containing distinct documents, may be produced as well. in such a situation, it may be useful to compare, combine and rank the cluster contents, to filter out relevant documents. in the proposed language, we define several operators (inspired by relational algebra) that work on groups of clusters. new clusters (and groups) can be generated by combining (i.e., overlapping, refining and intersecting) clusters (and groups), in a set oriented fashion. furthermore, several ranking functions are also proposed, to model distinct semantics of the combination.
ranking in folksonomy systems: can context help? folksonomy systems have shown to contribute to the quality of web search ranking strategies. in this paper, we analyze and compare different graph-based ranking algorithms, namely folkrank, socialpagerank, and socialsimrank. we enhance these algorithms by exploiting the context of tag assignmets, and evaluate the results on the groupme! dataset. in groupme!, users can organize and maintain arbitrary web resources in self-defined groups. when users annotate resources in groupme!, this can be interpreted in context of a certain group. the grouping activity delivers valuable semantic information about resources and their context. we show how to use this information to improve the detection of relevant search results, and compare different strategies for ranking result lists in folksonomy systems.
to swing or not to swing: learning when (not) to advertise. web textual advertising can be interpreted as a search problem over the corpus of ads available for display in a particular context. in contrast to conventional information retrieval systems, which always return results if the corpus contains any documents lexically related to the query, in web advertising it is acceptable, and occasionally even desirable, not to show any results. when no ads are relevant to the user's interests, then showing irrelevant ads should be avoided since they annoy the user and produce no economic benefit. in this paper we pose a decision problem "whether to swing", that is, whether or not to show any of the ads for the incoming request. we propose two methods for addressing this problem, a simple thresholding approach and a machine learning approach, which collectively analyzes the set of candidate ads augmented with external knowledge. our experimental evaluation, based on over 28,000 editorial judgments, shows that we are able to predict, with high accuracy, when to "swing" for both content match and sponsored search advertising.
statistical power in retrieval experimentation. the power of a statistical test specifies the sample size required to reliably detect a given true effect. in ir evaluation, the power corresponds to the number of topics that are likely to be sufficient to detect a certain degree of superiority of one system over another. to predict the power of a test, one must estimate the variability of the population being sampled from; here, of between-system score deltas. this paper demonstrates that basing such an estimation either on previous experience or on trial experiments leaves wide margins of error. iteratively adding more topics to the test set until power is achieved is more efficient; however, we show that it leads to a bias in favour of finding both power and significance. a hybrid methodology is proposed, and the reporting requirements of the experimenter using this methodology are laid out. we also demonstrate that greater statistical power is achieved for the same relevance assessment effort by evaluating a large number of topics shallowly than a small number deeply.
extremely fast text feature extraction for classification and indexing. most research in speeding up text mining involves algorithmic improvements to induction algorithms, and yet for many large scale applications, such as classifying or indexing large document repositories, the time spent extracting word features from texts can itself greatly exceed the initial training time. this paper describes a fast method for text feature extraction that folds together unicode conversion, forced lowercasing, word boundary detection, and string hash computation. we show empirically that our integer hash features result in classifiers with equivalent statistical performance to those built using string word features, but require far less computation and less memory.
modeling lsh for performance tuning. although locality-sensitive hashing (lsh) is a promising approach to similarity search in high-dimensional spaces, it has not been considered practical partly because its search quality is sensitive to several parameters that are quite data dependent. previous research on lsh, though obtained interesting asymptotic results, provides little guidance on how these parameters should be chosen, and tuning parameters for a given dataset remains a tedious process. to address this problem, we present a statistical performance model of multi-probe lsh, a state-of-the-art variance of lsh. our model can accurately predict the average search quality and latency given a small sample dataset. apart from automatic parameter tuning with the performance model, we also use the model to devise an adaptive lsh search algorithm to determine the probing parameter dynamically for each query. the adaptive probing method addresses the problem that even though the average performance is tuned for optimal, the variance of the performance is extremely high. we experimented with three different datasets including audio, images and 3d shapes to evaluate our methods. the results show the accuracy of the proposed model: the recall errors predicted are within 5% from the real values for most cases; the adaptive search method reduces the standard deviation of recall by about 50% over the existing method.
humane data mining. data mining has made tremendous strides in the last decade. it is time to take data mining to the next level of contributions, while continuing to innovate for the current mainstream market. we postulate that a fruitful future direction could be humane data mining: applications to benefit individuals. the potential applications include personal data mining (e.g. personal health), enable people to get a grip on their world (e.g. dealing with the long tail of search), enable people to become creative (e.g. inventions arising from linking non-interacting scientific literature), enable people to make contributions to society (e.g. education collaboration networks), and data-driven science (e.g. study ecological disasters, brain disorders). rooting our future work in these (and similar) applications, will lead to new data mining abstractions, algorithms, and systems.
a new method for indexing genomes using on-disk suffix trees. we propose a new method to build persistent suffix trees for indexing the genomic data. our algorithm digest (disk-based genomic suffix tree) improves significantly over previous work in reducing the random access to the input string and performing only two passes over disk data. digest is based on the two-phase multi-way merge sort paradigm using a concise binary representation of the dna alphabet. furthermore, our method scales to larger genomic data than managed before.
estimating the number of answers with guarantees for structured queries in p2p databases. structured p2p overlays supporting standard database functionalities are a popular choice for building large-scale distributed data management systems. in such systems, estimating the number of answers for structured queries can help approximating query completeness, but is especially challenging. in this paper, we propose to use routing graphs in order to achieve this. we introduce the general approach and briefly discuss further aspects like overhead and guarantees.
query optimization in xml-based information integration. the problem of decentralized data sharing is relevant for a wide range of applications and is still a source of major theoretical and practical challenges, in spite of many years of sustained research in information integration. we focus on the challenge of efficiency of query evaluation in information-integration systems, with the objective of developing query-processing strategies that are widely applicable and easy to implement in real-life applications. in our algorithms we take into account important features of today's data-sharing applications, namely: xml as likely interface to or representation for data sources; potential for information overlap across data sources; and the need for inter-source processing (i.e., joins of data across data sources) in many applications. to the best of our knowledge, our methods are the first to account for the practical issues of information overlap across data sources and of inter-source processing. while most of our algorithms are platform- and implementation-independent, we also propose xml-specific optimization techniques that allow for system-level tuning of query processing performance. finally, using real-life datasets and our implementation of an information-integration system shell, we provide experimental results that demonstrate that our algorithms are efficient and competitive in the information-integration setting. for all the details, please see [1].
an extension of plsa for document clustering. in this paper we propose an extension of the plsa model in which an extra latent variable allows the model to co-cluster documents and terms simultaneously. we show on three datasets that our extended model produces statistically significant improvements with respect to two clustering measures over the original plsa and the multinomial mixture mm models.
yizkor books: a voice for the silent past. yizkor book collections contain firsthand commemorative accounts of events from the era surrounding the rise and fall of nazi germany, including documents from before, during, and after the holocaust. prior to our effort, information regarding the content and location of each yizkor book volume was limited. we established a centralized index and metadata repository for the yizkor book collection and developed a detailed search interface accessible worldwide.
information shared by many objects. if kolmogorov complexity [25] measures information in one object and information distance measures information shared by two objects, how do we measure information shared by many objects? this paper provides an initial pragmatic study of this fundamental data mining question. firstly, em(x1,x2,...,xn) is defined to be the minimum amount of thermodynamic energy needed to convert from any xi to any xj. with this definition several theoretical problems have been solved. second, our newly proposed theory is applied to select a comprehensive review and a specialized review from many reviews: (1) core feature words, expanded words and dependent words are extracted respectively. (2) comprehensive and specialized reviews are selected according to the information among them. this method of selecting a single review can be extended to select multiple reviews as well. finally, experiments show that this comprehensive and specialized review mining method based on our new theory can do the job efficiently.
improve the effectiveness of the opinion retrieval and opinion polarity classification. opinion retrieval is a document retrieving and ranking process. a relevant document must be relevant to the query and contain opinions toward the query. opinion polarity classification is an extension of opinion retrieval. it classifies the retrieved document as positive, negative or mixed, according to the overall polarity of the query relevant opinions in the document. this paper (1) proposes several new techniques that help improve the effectiveness of an existing opinion retrieval system; (2) presents a novel two-stage model to solve the opinion polarity classification problem. in this model, every query relevant opinionated sentence in a document retrieved by our opinion retrieval system is classified as positive or negative respectively by a svm classifier. then a second classifier determines the overall opinion polarity of the document. experimental results show that both the opinion retrieval system with the proposed opinion retrieval techniques and the polarity classification model outperformed the best reported systems respectively.
nested region algebra extended with variables for tag-annotated text search. this paper presents a framework for searching text regions with specifying annotated information in tag-annotated text by using region algebra. we extend the efficient algorithm for region algebra to handle both nested and crossed regions and introduce variables for attribute values to treat tag-annotations in which attributes indicate another tag regions. our framework have been implemented in a text search engine for medline, which is a large textbase of abstracts in medical science. experiments in tag-annotated medline abstracts demonstrate the effectiveness of specifying annotations and the efficiency of our framework.
natural language retrieval of grocery products. in this paper we describe modifications to a natural language grocery retrieval system, introduced in our earlier work. we also compare our system against an off-the-shelf retrieval tool, and show that our system is significantly better for top-ranked retrieval results.
transaction reordering with application to synchronized scans. traditional workload management methods mainly focus on the current system status while information about the interaction between queued and running transactions is largely ignored. an exception to this is the transaction reordering method, which reorders the transaction sequence submitted to the rdbms and improves the transaction throughput by considering both the current system status and information about the interaction between queued and running transactions. the existing transaction reordering method only considers the reordering opportunities provided by analyzing the lock conflict information among multiple transactions. this significantly limits the applicability of the transaction reordering method. in this paper, we extend the existing transaction reordering method into a general transaction reordering framework that can incorporate various factors as the reordering criteria. we show that by analyzing the resource utilization information of transactions, the transaction reordering method can also improve the system throughput by increasing the resource sharing opportunities among multiple transactions. we provide a concrete example on synchronized scans and demonstrate the advantages of our method through experiments with a commercial parallel rdbms.