waashk/dblp huggingface dataset:
online duplicate document detection: signature reliability in a dynamic retrieval environment. as online document collections continue to expand, both on the web and in proprietary environments, the need for duplicate detection becomes more critical. few users wish to retrieve search results consisting of sets of duplicate documents, whether identical duplicates or close matches. our goal in this work is to investigate the phenomenon and determine one or more approaches that minimize its impact on search results. recent work has focused on using some form of signature to characterize a document in order to reduce the complexity of document comparisons. a representative technique constructs a 'fingerprint' of the rarest or richest features in a document using collection statistics as criteria for feature selection. one of the challenges of this approach, however, arises from the fact that in production environments, collections of documents are always changing, with new documents, or new versions of documents, arriving frequently, and other documents periodically removed. when an enterprise proceeds to freeze a training collection in order to stabilize the underlying repository of such features and its associated collection statistics, issues of coverage and completeness arise. we show that even with very large training collections possessing extremely high feature correlations before and after updates, underlying fingerprints remain sensitive to subtle changes. we explore alternative solutions that benefit from the development of massive meta-collections made up of sizable components from multiple domains. this technique appears to offer a practical foundation for fingerprint stability. we also consider mechanisms for updating training collections while mitigating signature instability. our research is divided into three parts. we begin with a study of the distribution of duplicate types in two broad-ranging news collections consisting of approximately 50 million documents. we then examine the utility of document signatures in addressing identical or nearly identical duplicate documents and their sensitivity to collection updates. finally, we investigate a flexible method of characterizing and comparing documents in order to permit the identification of non-identical duplicates. this method has produced promising results following an extensive evaluation using a production-based test collection created by domain experts.
clustering declustered data for efficient retrieval. as databases increasingly integrate multimedia information in the form of image, video, and audio data, both the dimensionality and the amount of data that need to be processed is increasing rapidly. it becomes necessary to support the efficient retrieval of large amounts of multimedia data. declustering techniques for multi-disk architectures have been effectively used for storage in relational databases. in this paper, we first establish that besides exploiting the parallelism, a careful organization of each disk must be considered for fast searching. we introduce the notion of page allocation and data space mapping which can be used to organize and retrieve multidimensional data. we work on these notions based on three different partitioning strategies: regular grid partitioning, concentric hypercubes and hyperpyramids. we develop techniques that satisfy efficient retrieval by optimizing the number of buckets retrieved by the query, disk arm movement and i/o parallelism. we prove that concentric hypercube based mapping satisfies the optimal clustering and optimal parallelism. we develop techniques based on hyperpyramid partitioning which reduces the number of buckets retrieved by the query and has very efficient inter and intra disk organizations. we evaluate the performance of proposed techniques by comparing it with the current best approaches. the new techniques lead to very significant improvement, up to 43 times, over the existing techniques, therefore resulting in fast retrieval of multimedia data.
detecting similar documents using salient terms. we describe a system for rapidly determining document similarity among a set of documents obtained from an information retrieval (ir) system. we obtain a ranked list of the most important terms in each document using a rapid phrase recognizer system. we store these in a database and compute document similarity using a simple database query. if the number of terms found to not be contained in both documents is less than some predetermined threshold compared to the total number of terms in the document, these documents are determined to be very similar. we compare this to the shingles approach.
approximately common patterns in shared-forests. we present a proposal intended to demonstrate the applicability of tabulation techniques for detecting approximately common patterns when dealing with structures sharing some common parts. this sharing saves on the space needed to represent the structures and also on their later processing, by factorizing the filtering of substructure matching. as a consequence, preliminary experimental tests indicate a reduction of the running time.
termination analysis of active rules modular sets. this paper presents an algorithm for static termination analysis of active rules in a context of modular design. several recent works have suggested proving termination by using the concept of triggering graph. we propose here an original approach, based on these works, and that allows to guarantee the termination of a set of rules, conceived by several designers, even when none of the designers knows the set of the active rules. we introduce the notions of private event and of public event, and we refine the notion of triggering graph (by enclosing also events in graphs). we replace then the notion of cycle (which is no more relevant in a context of modular design) by the notion of maximal private path preceding a rule. by means of these tools, we show that it is possible to prove termination of active rules modular sets.
kqml as an agent communication language. this paper describes the design of and experimentation with the knowledge query and manipulation language (kqml), a new language and protocol for exchanging information and knowledge. this work is part of a larger effort, the arpa knowledge sharing effort which is aimed at developing techniques and methodology for building large-scale knowledge bases which are sharable and reusable. kqml is both a message format and a message-handling protocol to support run-time knowledge sharing among agents. kqml focuses on an extensible set of performatives, which defines the permissible &ldquo;speech acts&rdquo; agents may use and comprise a substrate on which to develop higher-level models of interagent interaction such as contract nets and negotiation. in addition, kqml provides a basic architecture for knowledge sharing through a special class of agent called communication facilitors which coordinate the interactions of other agents. the ideas which underlie the evolving design of kqml are currently being explored through experimental prototype systems which are being used to support several testbeds in such areas as concurrent engineering, intelligent design and intelligent planning and scheduling.
representing interests as a hyperlinked document collection. we describe a latent variable model for representing a user's interests as a hyperlinked document collection. by collecting hyper-text documents that a user views, creates or updates whilst at their computer, we are able to use not only the content of these documents but also the inter-connectivity of the collection to model the user's interests. the model uses probabilistic latent semantic analysis and probabilistic hypertext induced topic selection and decomposes the user's document collection into a set of factors each of which represents a user's interest. this model can be used to personalise information access tasks such as a personalised search engine or a personalised news service. our latent variable model's performance is compared with that of a more conventional vector space clustering algorithm.
semantic similarity over the gene ontology: family correlation and selecting disjunctive ancestors. many bioinformatics applications would benefit from comparing proteins based on their biological role rather than their sequence. in most biological databases, proteins are already annotated with ontology terms. previous studies identified a correlation between the sequence similarity and the semantic similarity of proteins. the semantic similarity of proteins was computed from their annotated go terms. however, proteins sharing a biological role do not necessarily have a similar sequence.this paper introduces our study of the correlation between go and family similarity. family similarity overcomes some of the limitations of sequence similarity, thus we obtained a strong correlation between go and family similarity. additionally, this paper introduces grasm, a novel method that uses all the information in the graph structure of the go, instead of considering it as a hierarchical tree. when calculating the semantic similarity of two concepts, grasm selects the disjunctive common ancestors rather than only using the most informative common ancestor. grasm produced a higher family similarity correlation than the original semantic similarity measures.
efficient ordering for xml data. with the increasing popularity of xml, there arises the need for managing and querying information in this form. several query languages, such as xquery, have been proposed which return their results in document order. however, most recent efforts focused on query optimization have disregarded order. this paper presents a simple yet elegant method to maintain document ordering for xml data. analysis of our method shows that it is indeed efficient and scalable, even for changing data.
a framework for selective query expansion. query expansion is a well-known technique that has been shown to improve <i>average</i> retrieval performance. this technique has not been used in many operational systems because of the fact that it can greatly degrade the performance of some individual queries. we show how comparison between language models of the unexpanded and expanded retrieval results can be used to predict when the expanded retrieval has strayed from the original sense of the query. in these cases, the unexpanded results are used while the expanded results are used in the remaining cases (where such straying is not detected). we evaluate this method on a wide variety of trec collections.
joint deduplication of multiple record types in relational data. record deduplication is the task of merging database records that refer to the same underlying entity. in relational data-bases, accurate deduplication for records of one type is often dependent on the decisions made for records of other types. whereas nearly all previous approaches have merged records of different types independently, this work models these inter-dependencies explicitly to collectively deduplicate records of multiple types. we construct a conditional random field model of deduplication that captures these relational dependencies, and then employ a novel relational partitioning algorithm to jointly deduplicate records. for two citation matching datasets, we show that collectively deduplicating paper and venue records results in up to a 30% error reduction in venue deduplication, and up to a 20% error reduction in paper deduplication.
the role of the database community in the national information infrastructure. the computer science community is increasingly focusing its efforts on tasks related to the national information infrastructure. hardware and software advances are being sought to make wide-area networks and technological resources in general more useful to mainstream society. with this transition comes a set of unavoidable political and social issues that have never been satisfactorily dealt with in the past. grand challenges face the computer science community on both the technical and the socio-political sides of this &ldquo;major upgrade.&rdquo; this paper discusses various aspects of the enormous coordination problem that faces all of us and considers what the database community can do to help.
an evaluation of evolved term-weighting schemes in information retrieval. this paper presents an evaluation of evolved term-weighting schemes on short, medium and long trec queries. a previously evolved global (collection-wide) term-weighting scheme is evaluated on unseen trec data and is shown to increase mean average precision over idf. a local (within-document) evolved term-weighting scheme is presented which is dependent on the best performing global scheme. the full evolved scheme (i.e. the combined local and global scheme) is compared to both the bm25 scheme and the pivoted normalisation scheme.our results show that the local evolved solution does not perform well on some collections due to its document normalisation properties and we conclude that okapi-tf can be tuned to interact effectively with the evolved global weighting scheme presented and increase mean average precision over the standard bm25 scheme.
concept-based interactive query expansion. despite the recent advances in search quality, the fast increase in the size of the web collection has introduced new challenges for web ranking algorithms. in fact, there are still many situations in which the users are presented with imprecise or very poor results. one of the key difficulties is the fact that users usually submit very short and ambiguous queries, and they do not fully specify their information needs. that is, it is necessary to improve the query formation process if better answers are to be provided. in this work we propose a novel concept-based query expansion technique, which allows disambiguating queries submitted to search engines. the concepts are extracted by analyzing and locating cycles in a special type of query relations graph. this is a directed graph built from query relations mined using association rules. the concepts related to the current query are then shown to the user who selects the one concept that he interprets is most related to his query. this concept is used to expand the original query and the expanded query is processed instead. using a web test collection, we show that our approach leads to gains in average precision figures of roughly 32%. further, if the user also provides information on the type of relation between his query and the selected concept, the gains in average precision go up to roughly 52%.
minimal document set retrieval. this paper presents a novel formulation and approach to the minimal document set retrieval problem. minimal document set retrieval (mdsr) is a promising information retrieval task in which each query topic is assumed to have different subtopics; the task is to retrieve and rank relevant document sets with maximum coverage but minimum redundancy of subtopics in each set. for this task, we propose three document set retrieval and ranking algorithms: novelty based method, cluster based method and subtopic extraction based method. in order to evaluate the system performance, we design a new evaluation framework for document set ranking which evaluates both relevance between set and query topic, and redundancy within each set. finally, we compare the performance of the three algorithms using the trec interactive track dataset. experimental results show the effectiveness of our algorithms.
connecting topics in document collections with stepping stones and pathways. in this paper, we present stepping stones and pathways (ssp), an alternative model of building and presenting answers for the cases when queries on document collections cannot be answered just by a ranked list. stepping stones can handle questions like: "what is the relation of topics x and y?" ssp addresses when the contents of a small set of related documents is needed as an answer rather than a single document, or when "query splitting" is required to satisfactorily explore a document space. query results are networks of document groups representing topics, each group relating to and connecting (by documents) to other groups in the network. thus, a network answers the user's information need. we devise new and more effective representations and techniques to visualize such answers, and to involve users as part of the answer-finding process. in order to verify the validity of our approach, and since the questions we aim to answer involve multiple topics, we performed a study involving a custom built broad collection of operating systems research papers, and evaluated the results with interested computer science students, using multiple measures.
time-based language models. we explore the relationship between time and relevance using trec ad-hoc queries. a type of query is identified that favors very recent documents. we propose a time-based language model approach to retrieval for these queries. we show how time can be incorporated into both query-likelihood models and relevance models. these models were used for experiments comparing time-based language models to heuristic techniques for incorporating document recency in the ranking. our results show that time-based models perform as well as or better than the best of the heuristic techniques.
optimizing cursor movement in holistic twig joins. holistic twig join algorithms represent the state of the art for evaluating path expressions in xml queries. using inverted indexes on xml elements, holistic twig joins move a set of index cursors in a coordinated way to quickly find structural matches. because each cursor move can trigger i/o, the performance of a holistic twig join is largely determined by how many cursor moves it makes, yet, surprisingly, existing join algorithms have not been optimized along these lines. in this paper, we describe twigoptimal, a new holistic twig join algorithm with optimal cursor movement. we sketch the proof of twigoptimal's optimality, and describe how twigoptimal can use information in the return clause of xquery to boost its performance. finally, experimental results are presented, showing twigoptimal's superiority over existing holistic twig join algorithms.
tempus fugit: a system for making semantic connections. tempus fugit ("time flies") is the first of a new generation of personal information management (pim) systems. a pim system incorporates an electronic calendar, "to-do" list and address book. the premise behind tempus fugit is that information stored in electronic calendars, to-do lists and address books can be given richer semantic interpretation and automatically processed to make its users more effective. tempus fugit also tracks the physical and virtual locations of users and uses this information to predict meeting attendance and help them as they travel during their day.
a design space approach to analysis of information retrieval adaptive filtering systems. in this paper we suggest a new approach to analysis and design of ir systems. we argue for design space exploration in constructing ir systems and in analyzing the effects of individual modules and parameters. we present results of experiments with parametric interpolation, or "homotopy", between two systems, and show, incidentally, that the best results are not achieved at the endpoints, and may lie outside the bounding hypercube defined by our choice of parameterization. three distinct classes of interpolation are introduced to deal with the complexities of the specific example.
exploiting a controlled vocabulary to improve collection selection and retrieval effectiveness. vocabulary incompatibilities arise when the terms used to index a document collection are largely unknown, or at least not well-known to the users who eventually search the collection. no matter how comprehensive or well-structured the indexing vocabulary, it is of little use if it is not used effectively in query formulation. this paper demonstrates that techniques for mapping user queries into the controlled indexing vocabulary have the potential to radically improve document retrieval performance. we also show how the use of controlled indexing vocabulary can be employed to achieve performance gains for collection selection. finally, we demonstrate the potential benefit of combining these two techniques in an interactive retrieval environment. given a user query, our evaluation approach simulates the human user's choice of terms for query augmentation given a list of controlled vocabulary terms suggested by a system. this strategy lets us evaluate interactive strategies without the need for human subjects.
applications of approximate word matching in information retrieval. as more online databases are integrated into digital libraries, the issue of quality control of the data becomes increasingly important, especially as it relates to the effective retrieval of information. the need to discover and reconcile variant forms of strings in bibliographic entries, i.e., authority work, will become more difficult. spelling variants, misspellings, and transliteration differences will all increase the difficulty of retrieving information. approximate string matching has traditionally been used to help with this problem. in this paper we introduce the notion of approximate word matching and show how it can be used to improve detection and categorization of variant forms.
a hybrid approach to ner by memm and manual rules. this paper describes a framework for defining domain specific feature functions in a user friendly form to be used in a maximum entropy markov model (memm) for the named entity recognition (ner) task. our system called merge allows defining general feature function templates, as well as linguistic rules incorporated into the classifier. the simple way of translating these rules into specific feature functions are shown. we show that merge can perform better from both purely machine learning based systems and purely-knowledge based approaches by some small expert interaction of rule-tuning.
on scalable information retrieval systems. implementing scalable information retrieval systems requires the design and development of efficient methods to ingest data from multiple sources, search and retrieve results from both english and foreign language document collections and from collections comprising of multiple data types, harness high performance computer technology, and accurately answer user questions. some recent efforts related to the development of scalable information retrieval systems are described. particular emphasis is placed on those efforts that were adopted into commercial use.
reorganizing web sites based on user access patterns. in this paper, an approach for reorganizing web sites based on user access patterns is proposed. the approach consists of three steps: preprocessing, page classification, and site reorganization. in preprocessing, pages on a web site are processed to create an internal representation of the site, and page access information of its users is extracted from its server log. in page classification, the web pages on the site are classified into two categories, index pages and content pages, based on the page access information. after the pages are classified, in site reorganization, the web site is examined to find better ways to organize and arrange the pages on the site. our experiments on a large real data set show that the approach is efficient and practical for adaptive web sites.