waashk/dblp huggingface dataset:
real-time data pre-processing technique for efficient feature extraction in large scale datasets. due to the continuous and rampant increase in the size of domain specific data sources, there is a real and sustained need for fast processing in time-sensitive applications, such as medical record information extraction at the point of care, genetic feature extraction for personalized treatment, as well as off-line knowledge discovery such as creating evidence based medicine. since parallel multi-string matching is at the core of most data mining tasks in these applications, faster on-line matching in static and streaming data is needed to improve the overall efficiency of such knowledge discovery. to solve this data mining need not efficiently handled by traditional information extraction and retrieval techniques, we propose a block suffix shifting-based approach, which is an improvement over the state of the art multi-string matching algorithms such as aho-corasick, commentz-walter, and wu-manber. the strength of our approach is its ability to exploit the different block structures of domain specific data for off-line and online parallel matching. experiments on several real world datasets show how our approach translates into significant performance improvements.
modeling document features for expert finding. we argue that expert finding is sensitive to multiple document features in an organization, and therefore, can benefit from the incorporation of these document features. we propose a unified language model, which integrates multiple document features, namely, multiple levels of associations, pagerank, indegree, internal document structure, and url length. our experiments on two trec enterprise track collections, i.e., the w3c and csiro datasets, demonstrate that the natures of the two organizational intranets and two types of expert finding tasks, i.e., key contact finding for csiro and knowledgeable person finding for w3c, influence the effectiveness of different document features. our work provides insights into which document features work for certain types of expert finding tasks, and helps design expert finding strategies that are effective for different scenarios.
data degradation: making private data less sensitive over time. trail disclosure is the leakage of privacy sensitive data, resulting from negligence, attack or abusive scrutinization or usage of personal digital trails. to prevent trail disclosure, data degradation is proposed as an alternative to the limited retention principle. data degradation is based on the assumption that long lasting purposes can often be satisfied with a less accurate, and therefore less sensi-tive, version of the data. data will be progressively degraded such that it still serves application purposes, while decreasing accuracy and thus privacy sensitivity.
the effect of contextualization at different granularity levels in content-oriented xml retrieval. in the hierarchical xml structure, the ancestors form the context of an xml element. the process of taking element's context into account in element scoring is called contextualization. the aim of this paper is to separate different granularity levels and test the effect of contextualization on these levels.
generalized inverse document frequency. inverse document frequency (idf) is one of the most useful and widely used concepts in information retrieval. there have been various attempts to provide theoretical justifications for idf. one of the most appealing derivations follows from the robertson-sparck jones relevance weight. however, this derivation, and others related to it, typically make a number of strong assumptions that are often glossed over. in this paper, we re-examine these assumptions from a bayesian perspective, discuss possible alternatives, and derive a new, more generalized form of idf that we call generalized inverse document frequency. in addition to providing theoretical insights into idf, we also undertake a rigorous empirical evaluation that shows generalized idf outperforms classical versions of idf on a number of ad hoc retrieval tasks.
comparing citation contexts for information retrieval. in previous work, we have shown that using terms from around citations in citing papers to index the cited paper, in addition to the cited paper's own terms, can improve retrieval effectiveness. now, we investigate how to select text from around the citations in order to extract good index terms. we compare the retrieval effectiveness that results from a range of contexts around the citations, including no context, the entire citing paper, some fixed windows and several variations with linguistic motivations. we conclude with an analysis of the benefits of more complex, linguistically motivated methods for extracting citation index terms, over using a fixed window of terms. we speculate that there might be some advantage to using computational linguistic techniques for this task.
searching the wikipedia with contextual information. we propose a framework for searching the wikipedia with contextual information. our framework extends the typical keyword search, by considering queries of the type (q,p), where q is a set of terms (as in classical web search), and p is a source wikipedia document. the query terms q represent the information that the user is interested in finding, and the document p provides the context of the query. the task is to rank other documents in wikipedia with respect to their relevance to the query terms q given the context document p. by associating a context to the query terms, the search results of a search initiated in a particular page can be made more relevant. we suggest a number of features that extend the classical query-search model so that the context document p is considered. we then use ranksvm (joachims 2002) to learn weights for the individual features given suitably constructed training data. documents are ranked at query time using the inner product of the feature and the weight vectors. the experiments indicate that the proposed method considerably improves results obtained by a more traditional approach that does not take the context into account.
predicting individual disease risk based on medical history. the monumental cost of health care, especially for chronic disease treatment, is quickly becoming unmanageable. this crisis has motivated the drive towards preventative medicine, where the primary concern is recognizing disease risk and taking action at the earliest signs. however, universal testing is neither time nor cost efficient. we propose care, a collaborative assessment and recommendation engine, which relies only on a patient's medical history using icd-9-cm codes in order to predict future diseases risks. care uses collaborative filtering to predict each patient's greatest disease risks based on their own medical history and that of similar patients. we also describe an iterative version, icare, which incorporates ensemble concepts for improved performance. these novel systems require no specialized information and provide predictions for medical conditions of all kinds in a single run. we present experimental results on a medicare dataset, demonstrating that care and icare perform well at capturing future disease risks.
summarization of social activity over time: people, actions and concepts in dynamic networks. we present a framework for automatically summarizing social group activity over time. the problem is important in understanding large scale online social networks, which have diverse social interactions and exhibit temporal dynamics. in this work we construct summarization by extracting activity themes. we propose a novel unified temporal multi-graph framework for extracting activity themes over time. we use non-negative matrix factorization (nmf) approach to derive two interrelated latent spaces for users and concepts. activity themes are extracted from the derived latent spaces to construct group activity summary. experiments on real-world flickr datasets demonstrate that our technique outperforms baseline algorithms such as lsi, and is additionally able to extract temporally representative activities to construct meaningful group activity summary.
learning a two-stage svm/crf sequence classifier. learning a sequence classifier means learning to predict a sequence of output tags based on a set of input data items. for example, recognizing that a handwritten word is "cat", based on three images of handwritten letters and on general knowledge of english letter combinations, is a sequence classification task. this paper describes a new two-stage approach to learning a sequence classifier that is (i) highly accurate, (ii) scalable, and (iii) easy to use in data mining applications. the two-stage approach combines support vector machines (svms) and conditional random fields (crfs). it is (i) highly accurate because it benefits from the maximum-margin nature of svms and also from the ability of crfs to model correlations between neighboring output tags. it is (ii) scalable because the input to each svm is a small training set, and the input to the crf has a small number of features, namely the svm outputs. it is (iii) easy to use because it combines existing published software in a straightforward way. in detailed experiments on the task of recognizing handwritten words, we show that the two-stage approach is more accurate, or faster and more scalable, or both, than leading other methods for learning sequence classifiers, including max-margin markov networks (m3ns) and standard crfs.
estimating retrieval effectiveness using rank distributions. in this paper, we consider the task of estimating query effectiveness, i.e., assessment of the retrieval system performance in absence of the user relevance judgments. in our approach we model the score associated with each document in the result set as a gaussian random variable. the mean and the variance of each document score can then be used to estimate the probability that a document will be ranked above another one and thus calculate the expected rank of the document in the ranked list. we propose to measure the effectiveness of the system performance by comparing the predicted and actual ranks of the retrieved documents. in our experiments we consider two retrieval models and five document scoring methods and evaluate their impact on the proposed estimation measures. our experiments with standardized data sets that include document relevance judgments and the task of predicting the relative query effectiveness show that the expected rank metric is robust to variations in document scoring and retrieval algorithms.
improved query difficulty prediction for the web. query performance prediction aims to predict whether a query will have a high average precision given retrieval from a particular collection, or low average precision. an accurate estimator of the quality of search engine results can allow the search engine to decide to which queries to apply query expansion, for which queries to suggest alternative search terms, to adjust the sponsored results, or to return results from specialized collections. in this paper we present an evaluation of state of the art query prediction algorithms, both post-retrieval and pre-retrieval and we analyze their sensitivity towards the retrieval algorithm. we evaluate query difficulty predictors over three widely different collections and query sets and present an analysis of why prediction algorithms perform significantly worse on web data. finally we introduce improved clarity, and demonstrate that it outperforms state-of-the-art predictors on three standard collections, including two large web collections.
metadata extraction and indexing for map search in web documents. in academic scientific articles, maps are widely used to provide the related geographic information and to give readers a visual understanding of the document content. as more digital documents containing maps become accessible on the web, there is a growing demand for a web search system to provide users with tools to retrieve documents based on the information available within a document's maps. in this paper, we design methods and algorithms to extract, identify, and index maps from academic and scientific documents in digital libraries. experimental results show that our approach can accurately locate maps and significantly improve the retrieve quality for maps in digital documents.
identification of class specific discourse patterns. in this paper we address the problem of extracting important (and unimportant) discourse patterns from call center conversations. call centers provide dialog based calling-in support for customers to address their queries, requests and complaints. a call center is the direct interface between an organization and its customers and it is important to capture the voice-of-customer by gathering insights into the customer experience. we have observed that the calls received at a call center contain segments within them that follow specific patterns that are typical of the issue being addressed in the call. we present methods to extract such patterns from the calls. we show that by aggregating over a few hundred calls, specific discourse patterns begin to emerge for each class of calls. further, we show that such discourse patterns are useful for classifying calls and for identifying parts of the calls that provide insights into customer behaviour.
learning to link with wikipedia. this paper describes how to automatically cross-reference documents with wikipedia: the largest knowledge base ever known. it explains how machine learning can be used to identify significant terms within unstructured text, and enrich it with links to the appropriate wikipedia articles. the resulting link detector and disambiguator performs very well, with recall and precision of almost 75%. this performance is constant whether the system is evaluated on wikipedia articles or "real world" documents. this work has implications far beyond enriching documents with explanatory links. it can provide structured knowledge about any unstructured fragment of text. any task that is currently addressed with bags of words - indexing, clustering, retrieval, and summarization to name a few - could use the techniques described here to draw on a vast network of concepts and semantics.
identifying table boundaries in digital documents via sparse line detection. most prior work on information extraction has focused on extracting information from text in digital documents. however, often, the most important information being reported in an article is presented in tabular form in a digital document. if the data reported in tables can be extracted and stored in a database, the data can be queried and joined with other data using database management systems. in order to prepare the data source for table search, accurately detecting the table boundary plays a crucial role for the later table structure decomposition. table boundary detection and content extraction is a challenging problem because tabular formats are not standardized across all documents. in this paper, we propose a simple but effective preprocessing method to improve the table boundary detection performance by considering the sparse-line property of table rows. our method easily simplifies the table boundary detection problem into the sparse line analysis problem with much less noise. we design eight line label types and apply two machine learning techniques, conditional random field (crf) and support vector machines (svm), on the table boundary detection field. the experimental results not only compare the performances between the machine learning methods and the heuristics-based method, but also demonstrate the effectiveness of the sparse line analysis in the table boundary detection.
efficient processing of probabilistic spatio-temporal range queries over moving objects. range queries for querying the current and future positions of the moving objects have received growing interests in the research community. existing methods, however, assume that an object only moves along an anticipated path. in this paper, we study the problem of answering probabilistic range queries on moving objects based on an uncertainty model, which captures the possible movements of objects with probabilities. we conduct a performance study, which shows our proposal significantly reduces the number of object examinations and the overall cost of the query evaluation.
representative entry selection for profiling blogs. many applications on blog search and mining often meet the challenge of handling huge volume of blog data, in which one single blog could contain hundreds or even thousands of entries. we investigate novel techniques for profiling blogs by selecting a subset of representative entries for each blog. we propose two principles for guiding the entry selection task: representativeness and diversity. further, we formulate the entry selection task into a combinatorial optimization problem and propose a greedy yet effective algorithm for finding a good approximate solution by exploiting the theory of submodular functions. we suggest blog classification for judging the performance of the proposed entry selection techniques and evaluate their performance on a real blog dataset, in which encouraging results were obtained.
sorec: social recommendation using probabilistic matrix factorization. data sparsity, scalability and prediction quality have been recognized as the three most crucial challenges that every collaborative filtering algorithm or recommender system confronts. many existing approaches to recommender systems can neither handle very large datasets nor easily deal with users who have made very few ratings or even none at all. moreover, traditional recommender systems assume that all the users are independent and identically distributed; this assumption ignores the social interactions or connections among users. in view of the exponential growth of information generated by online social networks, social network analysis is becoming important for many web applications. following the intuition that a person's social network will affect personal behaviors on the web, this paper proposes a factor analysis approach based on probabilistic matrix factorization to solve the data sparsity and poor prediction accuracy problems by employing both users' social network information and rating records. the complexity analysis indicates that our approach can be applied to very large datasets since it scales linearly with the number of observations, while the experimental results shows that our method performs much better than the state-of-the-art approaches, especially in the circumstance that users have made few or no ratings.
a step towards incremental maintenance of the composed schema mapping. schema mapping plays a fundamental role in modern information systems. mapping composition is an operator that combines a chain of successive schema mappings into a single schema mapping. by pre-computing the composed schema mapping, the system can achieve significant performance benefits. however, when a change occurs on any mapping in the chain, the composed schema mapping has to be maintained correspondingly. in this paper we consider a restricted form of the problem in the xml setting and propose an incremental maintenance approach. specifically, given a chain of successive mappings, we transform intermediately them into trees that consist of atomic rules and then divide the composition into sub-compositions of the atomic rules. the dividing composition approach provides a fine-grained perspective of the composition relationships between the mappings. we depict such information through an auxiliary data structure called composition relationship graph (crg). when changes occur on any mapping in the chain, the corresponding maintenance algorithms are developed based on the dividing approach and the crg, which compute the changes on the composed mapping and then repair it into the new version, such that the computation involves only the atomic rules that are relevant with the maintenance. we evaluate our maintenance approach and report the first experiments results, which show that it is efficient.
markov logic: a unifying language for knowledge and information management. modern information and knowledge management is characterized by high degrees of complexity and uncertainty. complexity is well handled by first-order logic, and uncertainty by probabilistic graphical models. what has been sorely missing is a seamless combination of the two. markov logic provides this by attaching weights to logical formulas and treating them as templates for features of markov random fields. this talks surveys markov logic representation, inference, learning and applications. inference algorithms combine ideas from satisfiability testing, resolution, markov chain monte carlo and belief propagation. learning algorithms involve statistical weight learning and inductive logic programming. markov logic has been successfully applied to a wide range of information and knowledge management problems, including information extraction, entity resolution, ontology learning, link prediction, heterogeneous knowledge bases, and others. it is the basis of the open-source alchemy system (http://alchemy.cs.washington.edu).
peer production of structured knowledge -: an empirical study of ratings and incentive mechanisms. creating and maintaining semantic structures such as ontologies on a large scale is a labor-intensive task, which a sole individual cannot perform. established automated solutions for this task do not yet exist. peer production is a promising approach to create structured knowledge: members of an online community create and maintain semantic structures collaboratively. to motivate members to participate and to ensure the quality of the data, rating-based incentive mechanisms are promising. members mutually rate the quality of their contributions and are rewarded for good contributions and truthful ratings. until now, there has been no systematic evaluation of such rating mechanisms in the context of structured knowledge. we have developed a platform for the collaborative creation of semantic structures. to evaluate the effect of ratings and incentive mechanisms on the quality of peer-produced data, we have conducted an extensive empirical study in an online community. we show that ratings are a reliable measure of the quality of contributions by comparing user ratings with an ex post evaluation by experts. further experimental results are that incentive mechanisms increase the quality of contributions. we conclude that ratings and incentive mechanisms are promising to foster and improve the peer production of structured knowledge.
how does clickthrough data reflect retrieval quality? automatically judging the quality of retrieval functions based on observable user behavior holds promise for making retrieval evaluation faster, cheaper, and more user centered. however, the relationship between observable user behavior and retrieval quality is not yet fully understood. we present a sequence of studies investigating this relationship for an operational search engine on the arxiv.org e-print archive. we find that none of the eight absolute usage metrics we explore (e.g., number of clicks, frequency of query reformulations, abandonment) reliably reflect retrieval quality for the sample sizes we consider. however, we find that paired experiment designs adapted from sensory analysis produce accurate and reliable statements about the relative quality of two retrieval functions. in particular, we investigate two paired comparison tests that analyze clickthrough data from an interleaved presentation of ranking pairs, and we find that both give accurate and consistent results. we conclude that both paired comparison tests give substantially more accurate and sensitive evaluation results than absolute usage metrics in our domain.