waashk/dblp huggingface dataset:
multi-resolution disambiguation of term occurrences. we describe a system for extracting mentions of terms such as company and product names, in a large and noisy corpus of documents, such as the world wide web. since natural language terms are highly ambiguous, a significant challenge in this task is disambiguating which occurrences of each term are truly related to the right meaning, and which are not. we describe our approach for disambiguation, and show that it achieves very high accuracy with only limited training. this serves as a necessary first step for applications that strive to do analytics on term mentions.
topic-based document segmentation with probabilistic latent semantic analysis. this paper presents a new method for topic-based document segmentation, i.e., the identification of boundaries between parts of a document that bear on different topics. the method combines the use of the probabilistic latent semantic analysis (plsa) model with the method of selecting segmentation points based on the similarity values between pairs of adjacent blocks. the use of plsa allows for a better representation of sparse information in a text block, such as a sentence or a sequence of sentences. furthermore, segmentation performance is improved by combining different instantiations of the same model, either using different random initializations or different numbers of latent classes. results on commonly available data sets are significantly better than those of other state-of-the-art systems.
efficient query evaluation using a two-level retrieval process. we present an efficient query evaluation method based on a two level approach: at the first level, our method iterates in parallel over query term postings and identifies candidate documents using an approximate evaluation taking into account only partial information on term occurrences and no query independent factors; at the second level, promising candidates are fully evaluated and their exact scores are computed. the efficiency of the evaluation process can be improved significantly using dynamic pruning techniques with very little cost in effectiveness. the amount of pruning can be controlled by the user as a function of time allocated for query evaluation. experimentally, using the trec web track data, we have determined that our algorithm significantly reduces the total number of full evaluations by more than 90%, almost without any loss in precision or recall. at the heart of our approach there is an efficient implementation of a new boolean construct called wand or weak and that might be of independent interest.
key problems in integrating structured and unstructured information. this forum will host a discussion among the panelists - and with the audience - on key trends in information technology, as well as on industry and government problems in search of technology solutions. the panel will focus on the core problems inherent in integrating structured and unstructured textual information, moving from application-centric to user-centric software, and merging knowledge management with data management.
towards speech as a knowledge resource. speech is a tantalizing mode of human communication. on the one hand, humans understand speech with ease and use speech to express complex ideas, information, and knowledge. on the other hand, automatic speech recognition with computers is still very hard, and extracting knowledge from speech is even harder. in this paper we motivate the study of speech as a knowledge resource and briefly survey a family of related applications and systems being developed at ibm research aimed towards the goal of exploiting speech as a knowledge resource.
a learning approach to processor allocation in parallel systems. given a typical parallel system and a collection of applications that are to execute on the system, a common problem is determining an effective allocation of processors among the applications. in this paper a learning approach is applied to processor allocation. the approach is to use a stochastic learning automaton (sla) as a decision tool. an sla uses values of the current state description, makes an allocation decision, evaluates its decision at some later time, modifies its decision making process, and tries to find the best allocation strategy by learning from its previous mistakes. the method is applied to the problem of allocating processors to parallel applications in a distributed system such as a cluster of workstations, and is validated through simulation. the result of this study show that a learning approach that utilizes a stochastic learning automaton is effective at making processor allocation decisions in a parallel system.
faciliating knowledge flow through the enterprise. this paper is concerned with the issues we encountered when attempting to achieve enterprise level knowledge reuse. we present 3 pilot studies where new visualization techniques were used to allow manufacturing and service operations take advantage of engineering knowledge embodied in 3d models. though all these studies showed dramatic productivity increases, only one business unit from the studies is currently working to achieve the reuse. there are a number of reasons why this is so, but the key underlying theme is a lack of enterprise level commitment to knowledge sharing and a lack of an adequate knowledge architecture for sharing knowledge across organizational boundaries. we conclude with an approach for facilitating knowledge flow across functional units.
archiving telemeetings. this paper presents a prototype system for modeling and managing the complete life-time of telemeetings/teleconferences. the system provides services for modeling telemeetings, storing telemeetings in a telemeeting database, annotating telemeetings and querying the telemeeting database.
experimental evaluation of dynamic data allocation strategies in a distributed database with changing workloads. traditionally, allocation of data in distributed database management systems has been determined by off-line analysis and optimization. this technique works well for static database access patterns, but is often inadequate for frequently changing workloads. in this paper we address how to dynamically reallocate data for partionable distributed databases with changing access patterns. rather than complicated and expensive optimization algorithms, a simple heuristic is presented and shown, via an implementation study, to improve system throughput by 3 based system. based on artificial wide area network delays, we show that dynamic reallocation can improve system throughput by a factor of two and a half for wide area networks. we also show that individual site load must be taken into consideration when reallocating data, and provide a simple policy that incorporates load in the reallocation decision.
hashing by proximity to process duplicates in spatial databases. in a spatial database, an object may extend arbitrarily in space. as a result, many spatial data structures (e.g., the quadtree, the cell tree, the r+-tree) represent an object by partitioning it into multiple, yet simple, pieces, each of which is stored separately inside the data structure. many operations on these data structures are likely to produce duplicate results because of the multiplicity of object pieces. a novel approach for duplicate processing based on proximity of spatial objects is presented. this is different from conventional duplicate elimination in database systems because, with spatial databases, different pieces of the same object can span multiple buckets of the underlying data structure. example algorithms are presented to perform duplicate processing using proximity for quadtree representation of line segments and arbitrary rectangles. the complexity of the algorithms is seen to depend on a geometric classification of different instances of the spatial objects. by using proximity and the spatial properties of the objects, the number of disk-i/o requests as well as the run-time storage during duplicate processing can be reduced.
intelligent caching: selecting, representing, and reusing data in an information server. accessing information sources to retrieve data requested by a user can be expensive, especially when dealing with distributed information sources. one way to reduce this cost is to cache the results of queries, or related classes of data. this paper presents an approach to caching and addresses the issues of which information to cache, how to describe what has been cached, and how to use the cached information to answer future queries. we consider these issues in the context of the sims information server, which is a system for retrieving information from multiple heterogeneous and distributed information sources. the design of this information server is ideal for representing and reusing cached information since each class of cached information is simply viewed as another information source that is available for answering future queries.
inferring query models by computing information flow. the language modelling approach to information retrieval can also be used to compute query models. a query model can be envisaged as an expansion of an initial query. the more prominent query models in the literature have a probabilistic basis. this paper introduces an alternative, non-probabilistic approach to query modelling whereby the strength of information flow is computed between a query q and a term w. information flow is a reflection of how strongly w is informationally contained within the query q. the information flow model is based on hyperspace analogue to language (hal) vector representations, which reflects the lexical co-occurrence information of terms. research from cognitive science has demonstrated the cognitive compatibility of hal representations with human processing. query models computed from trec queries by hal-based information flow are compared experimentally with two probabilistic query language models. experimental results are provided showing the hal-based information flow model be superior to query models computed via markov chains, and seems to be as effective as a probabilistically motivated relevance model.
generalized contextualization method for xml information retrieval. a general re-weighting method, called contextualization, for more efficient element ranking in xml retrieval is introduced. re-weighting is based on the idea of using the ancestors of an element as a context: if the element appears in a good context -- good interpreted as probability of relevance -- its weight is increased in relevance scoring; if the element appears in a bad context, its weight is decreased. the formal presentation of contextualization is given in a general xml representation and manipulation frame, which is based on utilization of structural indices. this provides a general approach independent of weighting schemas or query languages.contextualization is evaluated with the inex test collection. we tested four runs: no contextualization, parent, root and tower contextualizations. the contextualization runs were significantly better than no contextualization. the root contextualization was the best among the re-weighted runs.
itopn: incremental extraction of the n most visible objects. the visual exploration of large databases calls for a tight coupling of database and visualization systems. current visualization systems typically fetch all the data and organize it in a scene tree, which is then used to render the visible data. for immersive data explorations, where an observer navigates in a potentially huge data space and explores selected data regions this approach is inadequate. a scalable approach is to make the database system observer-aware and exchange the data that is visible and most relevant to the observer.in this paper we present itopn an incremental algorithm for extracting the most visible objects relative to the current position of the observer. we implement itopn and compare it to an improved version of the r-tree that extends lru with the caching of the top levels of the r-tree (lw-lru). our experiments show that itopn is orders of magnitude faster than lw-lru given the same amount of memory. our experiments also show that for lw-lru to perform as fast as itopn it needs three times as much memory.
indexing time vs. query time: trade-offs in dynamic information retrieval systems. we examine issues in the design of fully dynamic information retrieval systems supporting both document insertions and deletions. the two main components of such a system, index maintenance and query processing, affect each other, as high query performance is usually paid for by additional work during update operations. two aspects of the system -- incremental updates and garbage collection for delayed document deletions -- are discussed, with a focus on the respective indexing vs. query performance trade-offs. depending on the relative number of queries and update operations, different strategies lead to optimal overall performance.
extracting meaningful labels for websom text archives. self-organizing maps, being used mainly with data that are not pre-labeled, need automatic procedures for extracting keywords as labels for each of the map units. the websom methodology for building very large text archives has a very slow method for extracting such unit labels. it computes the relative frequencies of all the words of all the documents associated to each unit and then compares these to the relative frequencies of all the words of all the other units of the map. since maps may have more than 100,000 units and the archive may contain up to 7 million documents, the existing websom method is not practical. this paper describes how the meaningful labels per map unit can be deduced by analyzing the relative weight distribution of the som weight vectors and by taking advantage of some characteristics of the random projection method used in dimensionality reduction. the effectiveness of this technique is demonstrated on archives of the well studied reuters and cnn collections. comparisons with the websom method are provided.
hierarchical document categorization with support vector machines. automatically categorizing documents into pre-defined topic hierarchies or taxonomies is a crucial step in knowledge and content management. standard machine learning techniques like support vector machines and related large margin methods have been successfully applied for this task, albeit the fact that they ignore the inter-class relationships. in this paper, we propose a novel hierarchical classification method that generalizes support vector machine learning and that is based on discriminant functions that are structured in a way that mirrors the class hierarchy. our method can work with arbitrary, not necessarily singly connected taxonomies and can deal with task-specific loss functions. all parameters are learned jointly by optimizing a common objective function corresponding to a regularized upper bound on the empirical loss. we present experimental results on the wipo-alpha patent collection to show the competitiveness of our approach.
structural proximity searching for large collections of semi-structured data. the richness of the xml data format allows data to be structured in a way which precisely captures the semantics required by the author. it is the structure of the data, however, which forms the basis of all xml query languages. without at least some notion of the structure, a user cannot meaningfully query the data. this problem is compounded when one considers that heterogeneous data adhering to different schema are likely to exist in the database(s) being queried. this paper proposes a solution based on an efficient proximity index. in particular, we describe a family of encoding and compression schemes which enable us to build an index to efficiently implement the proximity search. our index is extremely small, and can reflect updates in the underlying database in modest time. experiments show that our algorithm and implementation are fast and scale well.
a study of parameter tuning for term frequency normalization. most current term frequency normalization approaches for information retrieval involve the use of parameters. the tuning of these parameters has an important impact on the overall performance of the information retrieval system. indeed, a small variation in the involved parameter(s) could lead to an important variation in the precision/recall values. most current tuning approaches are dependent on the document collections. as a consequence, the effective parameter value cannot be obtained for a given new collection without extensive training data. in this paper, we propose a novel and robust method for the tuning of term frequency normalization parameter(s), by measuring the normalization effect on the within document frequency of the query terms. as an illustration, we apply our method on amati \& van rijsbergen's so-called normalization 2. the experiments for the ad-hoc trec-6,7,8 tasks and trec-8,9,10 web tracks show that the new method is independent of the collections and able to provide reliable and good performance.
automatic query expansion based on divergence. in this paper we are mainly concerned with discussion of a formal model, based on the basic concept of divergence from information theory, for automatic query expansion. the basic principles and ideas on which our study is based are described. a theoretical framework is established, which allows the comparison and evaluation of different term scoring functions for identifying good terms for query expansion. the approaches proposed in this paper have been implemented and evaluated on collections from trec. preliminary results show that our approaches are viable and worthy of continued investigation.
query expansion using term relationships in language models for information retrieval. language modeling (lm) has been successfully applied to information retrieval (ir). however, most of the existing lm approaches only rely on term occurrences in documents, queries and document collections. in traditional unigram based models, terms (or words) are usually considered to be independent. in some recent studies, dependence models have been proposed to incorporate term relationships into lm, so that links can be created between words in the same sentence, and term relationships (e.g. synonymy) can be used to expand the document model. in this study, we further extend this family of dependence models in the following two ways: (1) term relationships are used to expand query model instead of document model, so that query expansion process can be naturally implemented; (2) we exploit more sophisticated inferential relationships extracted with information flow (if). information flow relationships are not simply pairwise term relationships as those used in previous studies, but are between a set of terms and another term. they allow for context-dependent query expansion. our experiments conducted on trec collections show that we can obtain large and significant improvements with our approach. this study shows that lm is an appropriate framework to implement effective query expansion.
combining link-based and content-based methods for web document classification. this paper studies how link information can be used to improve classification results for web collections. we evaluate four different measures of subject similarity, derived from the web link structure, and determine how accurate they are in predicting document categories. using a bayesian network model, we combine these measures with the results obtained by traditional content-based classifiers. experiments on a web directory show that best results are achieved when links from pages outside the directory are considered. link information alone is able to obtain gains of up to 46 points in f1, when compared to a traditional content-based classifier. the combination with content-based methods can further improve the results, but too much noise may be introduced, since the text of web pages is a much less reliable source of information. this work provides an important insight on which measures derived from links are more appropriate to compare web documents and how these measures can be combined with content-based algorithms to improve the effectiveness of web classification.
multi-resolution modeling of large scale scientific simulation data. to provide scientists and engineers with the ability to explore and analyze tera-scale size data-sets we are using a twofold approach. first, we model the data with the objective of creating a compressed yet manageable representation. second, with that compressed representation, we provide the ability to query the resulting approximation in order to obtain approximate yet sufficient answers; a process called ad-hoc querying. this paper is concerned with a wavelet modeling technique that seeks to capture the important physical characteristics of the target scientific data. our approach is driven by the compression, which is necessary for viable throughput, along with the end user requirements from the discovery process. our work contrasts existing research which applies wavelets to range querying, change detection, and clustering problems by working directly with the wavelet decomposition of the data. the difference in this procedure is due primarily to the nature of the data and the requirements of the scientists and engineers. our approach directly uses the wavelet coefficients of the data to compress as well as query. we describe how the wavelet decomposition is used to facilitate data compression and how queries are posed on the resulting compressed model. results of this process will be shown for several problems of interest.
discovery of decision rules in relational databases: a rough set approach. we develop an attribute-oriented rough set approach for the discovery of decision rules in relational databases. our approach combines machine learning techniques and rough set theory. we consider a learning procedure to consist of the two phases data generalization and data reduction. in the data generalization phase, utilizing knowledge about concept hierarchies and relevance of the data, an attribute-oriented induction is performed attribute by attribute. some undesirable attributes of the discovery task are removed and the primitive data in the databases are generalized to the desirable level; this process greatly decreases the number of tuples which must be examined for the discovery task and substantially reduces the computational complexity of the database learning processes. subsequently, in data reduction phase, rough set theory is applied to the generalized relation; the cause-effect relationships among the condition and decision attributes in the databases are analyzed and the non-essential or irrelevant attributes to the discovery task are eliminated without losing information of the original database system. this process further reduces the generalized relation. thus very concise and more accurate decision rules for each class in the decision attribute with little or no redundancy information, can be extracted automatically from the reduced relation during the learning process. our study shows that attribute-oriented induction combined with rough set theory provide an efficient and effective mechanism for discovering decision rules in database systems.
addressing the lack of direct translation resources for cross-language retrieval. most cross language information retrieval research concentrates on language pairs for which direct, rich, and often multiple translation resources already exist. however, for most language pairs, translation via an intermediate language is necessary. two distinct methods for dealing with the additional ambiguity introduced by the extra translation step have been proposed and individually, shown to improve retrieval effectiveness. two previous works indicated that in combination, the methods were ineffective. this paper provides strong empirical evidence that the methods can be combined to produce consistent and often significant improvements in retrieval effectiveness. the improvement is shown across a number of different intermediate languages and test collections.