waashk/dblp huggingface dataset:
evaluation of methods for relative comparison of retrieval systems based on clickthroughs. the cranfield evaluation method has some disadvantages, including its high cost in labor and inadequacy for evaluating interactive retrieval techniques. as a very promising alternative, automatic comparison of retrieval systems based on observed clicking behavior of users has recently been studied. several methods have been proposed, but there has so far been no systematic way to assess which strategy is better, making it difficult to choose a good method for real applications. in this paper, we propose a general way to evaluate these relative comparison methods with two measures: utility to users(utu) and effectiveness of differentiation(eod). we evaluate two state of the art methods by systematically simulating different retrieval scenarios. inspired by the weakness of these methods revealed through our evaluation, we further propose a novel method by considering the positions of clicked documents. experiment results show that our new method performs better than the existing methods.
learning to recommend questions based on user ratings. at community question answering services, users are usually encouraged to rate questions by votes. the questions with the most votes are then recommended and ranked on the top when users browse questions by category. as users are not obligated to rate questions, usually only a small proportion of questions eventually gets rating. thus, in this paper, we are concerned with learning to recommend questions from user ratings of a limited size. to overcome the data sparsity, we propose to utilize questions without users rating as well. further, as there exist certain noises within user ratings (the preference of some users expressed in their ratings diverges from that of the majority of users), we design a new algorithm called 'majority-based perceptron algorithm' which can avoid the influence of noisy instances by emphasizing its learning over data instances from the majority users. experimental results from a large collection of real questions confirm the effectiveness of our proposals.
beyond hyperlinks: organizing information footprints in search logs to support effective browsing. while current search engines serve known-item search such as homepage finding very well, they generally cannot support exploratory search effectively. in exploratory search, users do not know their information needs precisely and also often lack the needed knowledge to formulate effective queries, thus querying alone, as supported by the current search engines, is insufficient, and browsing into related information would be very useful. currently, browsing is mostly done by following hyperlinks embedded on web pages. in this paper, we propose to leverage search logs to allow a user to browse beyond hyperlinks with a multi-resolution topic map constructed based on search logs. specifically, we treat search logs as "footprints" left by previous users in the information space and build a multi-resolution topic map to semantically capture and organize them in multiple granularities. such a topic map can support a user to zoom in, zoom out, and navigate horizontally over the information space, and thus provide flexible and effective browsing capabilities for end users. to test the effectiveness of the proposed methods of supporting browsing, we rely on real search logs and a commercial search engine to implement our proposed methods. our experimental results show that the proposed topic map is effective to support browsing beyond hyperlinks.
what happens after an ad click?: quantifying the impact of landing pages in web advertising. unbeknownst to most users, when a query is submitted to a search engine two distinct searches are performed: the organic or algorithmic search that returns relevant web pages and related data (maps, images, etc.), and the sponsored search that returns paid advertisements. while an enormous amount of work has been invested in understanding the user interaction with organic search, surprisingly little research has been dedicated to what happens after an ad is clicked, a situation we aim to correct. to this end, we define and study the process of context transfer, that is, the user's transition from web search to the context of the landing page that follows an ad-click. we conclude that in the vast majority of cases the user is shown one of three types of pages, namely, homepage (the homepage of the advertiser), category browse (a browse-able sub-catalog related to the original query), and search transfer (the search results of the same query re-executed on the target site). we show that these three types of landing pages can be accurately distinguished using automatic text classification. finally, using such an automatic classifier, we correlate the landing page type with conversion data provided by advertisers, and show that the conversion rate (i.e., users' response rate to ads) varies considerably according to the type. we believe our findings will further the understanding of users' response to search advertising in general, and landing pages in particular, and thus help advertisers improve their web sites and help search engines select the most suitable ads.
scalable indexing of rdf graphs for efficient join processing. current approaches to rdf graph indexing suffer from weak data locality, i.e., information regarding a piece of data appears in multiple locations, spanning multiple data structures. weak data locality negatively impacts storage and query processing costs. towards stronger data locality, we propose a three-way triple tree (triplet) secondary memory indexing technique to facilitate flexible and efficient join evaluation on rdf data. the novelty of triplet is that the index is built over the atoms occurring in the data set, rather than at a coarser granularity, such as whole triples occurring in the data set; and, the atoms are indexed regardless of the roles (i.e., subjects, predicates, or objects) they play in the triples of the data set. we show through extensive empirical evaluation that triplet exhibits multiple orders of magnitude improvement over the state-of-the-art, in terms of both storage and query processing costs.
interactive relevance feedback with graded relevance and sentence extraction: simulated user experiments. research on relevance feedback (rfb) in information retrieval (ir) has given mixed results. success in rfb seems to depend on the searcher's willingness to provide feedback and ability to identify relevant documents or query keys. the paper is based on simulating many user scenarios regarding the amount and quality of rfb. in addition, we experiment with query-biased sentence extraction for query reformulation. the baselines are initial no-feedback queries and queries based on pseudo-relevance feedback. the core question is: under which conditions would rfb based on sentence extraction be successful? the answer depends on user's behavior, implementation of feedback query formulation, and the evaluation methods. a small amount of feedback from a short browsing window seems to improve the final ranking the most. longer browsing allows more feedback and better queries but also consumes the available relevant documents.
generating sql/xml query and update statements. the xml support in relational databases and the sql/xml language are still relatively new as compared to purely relational databases and traditional sql. today, most database users have a strong relational and sql background. sql/xml enables users to perform queries and updates across xml and relational data, but many struggle with writing sql/xml statements or xquery update expressions. one reason is the novelty of sql/xml and of the xquery expressions that must be included. another problem is that the tree structure of the xml data may be unknown or difficult to understand for the user. evolving xml schemas as well as hybrid xml/relational schemas make it even harder to write sql/xml statements. also, legacy applications use sql but may require access to xml data without costly code changes. motivated by these challenges, we developed a method to generate sql/xml query and update statements automatically. the input is either a gui or a regular sql statement that uses logical data item names irrespective of their actual location in relational or xml columns in the database. the output is a sql/xml statement that queries or updates relational and xml data as needed to carry out the original user statement. this relieves the user and simplifies schema evolution and integration. we have prototyped and tested the proposed method on top of db2 9.5.
classification-based resource selection. in some retrieval situations, a system must search across multiple collections. this task, referred to as federated search, occurs for example when searching a distributed index or aggregating content for web search. resource selection refers to the subtask of deciding, given a query, which collections to search. most existing resource selection methods rely on evidence found in collection content. we present an approach to resource selection that combines multiple sources of evidence to inform the selection decision. we derive evidence from three different sources: collection documents, the topic of the query, and query click-through data. we combine this evidence by treating resource selection as a multiclass machine learning problem. although machine learned approaches often require large amounts of manually generated training data, we present a method for using automatically generated training data. we make use of and compare against prior resource selection work and evaluate across three experimental testbeds.
tsa'09 workshop summary: topic-sentiment analysis. this workshop seeks to bring together researchers in both computer science and social sciences who are interested in developing and using topic-sentiment analysis methods to measure mass opinion, and to foster communications between the research community and industry practitioners as well.
clustering and exploring search results using timeline constructions. time is an important dimension of any information space and can be very useful in information retrieval and in particular clustering and exploration of search results. search result clustering is a feature integrated in some of today's search engines, allowing users to further explore search results. however, only little work has been done on exploiting temporal information embedded in documents for the presentation, clustering, and exploration of search results along well-defined timelines. in this paper, we present an add-on to traditional information retrieval applications in which we exploit various temporal information associated with documents to present and cluster documents along timelines. temporal information expressed in the form of, e.g., date and time tokens or temporal references, appear in documents as part of the textual context or metadata. using temporal entity extraction techniques, we show how temporal expressions are made explicit and used in the construction of multiple-granularity timelines. we discuss how hit-list based search results can be clustered according to temporal aspects, anchored in the constructed timelines, and how time-based document clusters can be used to explore search results that include temporal snippets. we also outline a prototypical implementation and evaluation that demonstrates the feasibility and functionality of our framework.
what makes categories difficult to classify?: a study on predicting classification performance for categories. in this paper, we try to predict which category will be less accurately classified compared with other categories in a classification task that involves multiple categories. the categories with poor predicted performance will be identified before any classifiers are trained and additional steps can be taken to address the predicted poor accuracies of these categories. inspired by the work on query performance prediction in ad-hoc retrieval, we propose to predict classification performance using two measures, namely, category size and category coherence. our experiments on 20-newsgroup and reuters-21578 datasets show that the spearman rank correlation coefficient between the predicted rank of classification performance and the expected classification accuracy is as high as 0.9.
luposdate: a semantic web database system. managing and querying semantic web are important issues for semantic web applications. therefore, we have developed a semantic web database system with logically and physically optimized sparql engines to manage and query rdf data, named luposdate. in order to present the functionalities of the luposdate system and engines, we have developed an online demonstration, which is available at http://www.ifis.uni-luebeck.de/index.php?id=luposdate-demo.
improvements that don't add up: ad-hoc retrieval results since 1998. the existence and use of standard test collections in information retrieval experimentation allows results to be compared between research groups and over time. such comparisons, however, are rarely made. most researchers only report results from their own experiments, a practice that allows lack of overall improvement to go unnoticed. in this paper, we analyze results achieved on the trec ad-hoc, web, terabyte, and robust collections as reported in sigir (1998--2008) and cikm (2004--2008). dozens of individual published experiments report effectiveness improvements, and often claim statistical significance. however, there is little evidence of improvement in ad-hoc retrieval technology over the past decade. baselines are generally weak, often being below the median original trec system. and in only a handful of experiments is the score of the best trec automatic run exceeded. given this finding, we question the value of achieving even a statistically significant result over a weak baseline. we propose that the community adopt a practice of regular longitudinal comparison to ensure measurable progress, or at least prevent the lack of it from going unnoticed. we describe an online database of retrieval runs that facilitates such a practice.
a framework for semantic link discovery over relational data. discovering links between different data items in a single data source or across different data sources is a challenging problem faced by many information systems today. in particular, the recent linking open data (lod) community project has highlighted the paramount importance of establishing semantic links among web data sources. currently, lod sources provide billions of rdf triples, but only millions of links between data sources. many of these data sources are published using tools that operate over relational data stored in a standard rdbms. in this paper, we present a framework for discovery of semantic links from relational data. our framework is based on declarative specification of linkage requirements by a user. we illustrate the use of our framework using several link discovery algorithms on a real world scenario. our framework allows data publishers to easily find and publish high-quality links to other data sources, and therefore could significantly enhance the value of the data in the next generation of web.
magiccube: choosing the best snippet for each aspect of an entity. wikis are currently used in business to provide knowledge management systems, especially for individual organizations. however, building wikis manually is a laborious and time-consuming work. to assist founding wikis, we propose a methodology in this paper to automatically select the best snippets for entities as their initial explanations. our method consists of two steps. first, we focus on extracting snippets from a given set of web pages for each entity. starting from a seed sentence, a snippet grows up by adding the most relevant neighboring sentences into itself. the sentences are chosen by the snippet growth model, which employs a distance function and an influence function to make decisions. secondly, we pick out the best snippet for each aspect of an entity. the combination of all the selected snippets serves as the primary description of the entity. we present three ever-increasing methods to handle selection process. experimental results based on a real data set show that our proposed method works effectively in producing primary descriptions for entities such as employee names.
location cache for web queries. this paper proposes a strategy to reduce the amount of hardware involved in the solution of search engine queries. it proposes using a secondary compact cache that keeps minimal information stored in the query receptionist machine to register the processors that must get involved in the solution of queries which are evicted from the standard result cache or are not admitted in it. this cache strategy produces exact answers by using very few processors.
linear inclusion for xml regular expression types. type inclusion is a fundamental operation in every type-checking compiler, but it is quite expensive for xml manipulation languages. we recently presented an inclusion checking algorithm for an expressive family of xml type languages which is polynomial, but runs in quadratic time both in the best and in the worst cases. we present here an algorithm that has a linear-time backbone, and resorts to the quadratic approach for some specific parts of the compared types. our experiments show that the new algorithm typically runs in linear time, hence can be used as a building block for a practical type-checking compiler.
answer typing for information retrieval. answer typing is commonly thought of as finding appropriate responses to given questions. we extend the notion of answer typing to information retrieval to ensure results contain plausible answers to queries. identification of a large class of applicable queries is performed using a discriminative classifier, and discriminative preference ranking methods are employed for the selection of type-appropriate terms. experimental results show that type-appropriate terms identified by the model are superior to terms most commonly associated with the query, providing strong evidence that answer typing techniques can find meaningful and appropriate terms. further experiments show that snippets containing correct answers are ranked higher by our model than by the baseline google search engine in those instances in which a query does indeed seek a short answer.
generating comparative summaries of contradictory opinions in text. this paper presents a study of a novel summarization problem called contrastive opinion summarization (cos). given two sets of positively and negatively opinionated sentences which are often the output of an existing opinion summarizer, cos aims to extract comparable sentences from each set of opinions and generate a comparative summary containing a set of contrastive sentence pairs. we formally formulate the problem as an optimization problem and propose two general methods for generating a comparative summary using the framework, both of which rely on measuring the content similarity and contrastive similarity of two sentences. we study several strategies to compute these two similarities. we also create a test data set for evaluating such a novel summarization problem. experiment results on this test set show that the proposed methods are effective for generating comparative summaries of contradictory opinions.
soire: a service-oriented ir evaluation architecture. we have developed a system that offers comprehensive analysis functionality for information retrieval experiments combined with a storage facility for persisting experiment data in a uniform fashion to facilitate repeatability and comparability of experiments. our service-oriented ir evaluation framework - soire offers a number of technological interfaces based on open networking standards and modeling languages to connect to other systems while regarding ease-of-use for researchers as the feature of utmost importance.
a proactive personalised retrieval system. we present a personalised retrieval system that captures explicit relevance feedback to build an evolving user profile with multiple aspects. the user profile is used to proactively retrieve results between search sessions to support multi-session search tasks. this approach to supporting users with their multi-session search tasks is evaluated in a between-subjects multiple time-series study with ten subjects performing two simulated work situation tasks over five sessions. system interaction data shows that subjects using the personalised retrieval system issue fewer queries and interact with fewer results than subjects using a baseline system. the interaction data also shows a trend of subjects interacting with the proactively retrieved results in the personalised retrieval system.
a general markov framework for page importance computation. we propose a general markov framework for computing page importance. under the framework, a markov skeleton process is used to model the random walk conducted by the web surfer on a given graph. page importance is then defined as the product of page reachability and page utility, which can be computed from the transition probability and the mean staying time of the pages in the markov skeleton process respectively. we show that this general framework can cover many existing algorithms as its special cases, and that the framework can help us define new algorithms to handle more complex problems. in particular, we demonstrate the use of the framework with the exploitation of a new process named mirror semi-markov process. the experimental results validate that the mirror semi-markov process model is more effective than previous models in several tasks.
effective xml content and structure retrieval with relevance ranking. xml documents can be retrieved by means of not only content-only (co) queries, but also content-and-structure (cas) queries. though promising better retrieval precision, cas queries introduce several new challenges. to address these challenges, we propose a novel approach for xml cas retrieval. the distinctive feature of the approach is that it adopts a content-oriented point of view. specifically, the approach first decomposes a cas query into several fragments, then retrieves results for each query fragment in a content-centric way, and finally scores each answer node. the approach is adaptive to versatile homogeneous and heterogeneous data environments. to assess the relevance of retrieval results to a query fragment, we present a scoring strategy that measures relevance from both content and structure perspectives. in addition, an effective approach is proposed to infer answer nodes based on the cas query and document structure. an efficient algorithm is also presented for cas retrieval. finally, we demonstrate the effectiveness of the proposed methods through comprehensive experimental studies.