waashk/dblp huggingface dataset:
spam filtering for short messages. we consider the problem of content-based spam filtering for short text messages that arise in three contexts: mobile (sms) communication, blog comments, and email summary information such as might be displayed by a low-bandwidth client. short messages often consist of only a few words, and therefore present a challenge to traditional bag-of-words based spam filters. using three corpora of short messages and message fields derived from real sms, blog, and spam messages, we evaluate feature-based and compression-model-based spam filters. we observe that bag-of-words filters can be improved substantially using different features, while compression-model filters perform quite well as-is. we conclude that content filtering for short messages is surprisingly effective.
a segment-based hidden markov model for real-setting pinyin-to-chinese conversion. hidden markov model (hmm) is frequently used for pinyin-to-chinese conversion. but it only captures the dependency with the preceding character. higher order markov models can bring higher accuracy, but are computationally unaffordable to average pc settings. we propose a segment-based hidden markov model (shmm), which has the same magnitude of complexity as first-order hmm, but generates higher decoding accuracy. shmm tells a word from a bigram connecting two words, and assigns a reasonable probability to words as a whole. it is more powerful than hmm to decode words containing over two characters. we conduct a comprehensive pinyin-to-chinese conversion evaluation on lancaster corpus. the experiment shows the perfect sentence accuracy is improved from 34.7% (hmm) to 43.3% (shmm). the one-error sentence accuracy is increased from 72.7% to 78.3%. furthermore, shmm can seamlessly integrate with pinyin typing correction, acronym pinyin input, user-defined words, and self-adaptive learning all of which are a must for a commercial pinyin-to-chinese conversion product in order to improve the efficiency of pinyin input.
efficient range-constrained similarity search on wavelet synopses over multiple streams. due to the resource limitation in the data stream environment, it has been reported that answering user queries according to the wavelet synopsis of a stream is an essential ability of a data stream management system (dsms). in this paper, motivated by the fact that a user may be interested in an arbitrary range of the data streams, we investigate two important types of range-constrained queries in time series streaming environments: the distance queries (which aim at obtaining the euclidean distance between two streams) and the knn queries (which aim at discovering k nearest neighbors to a reference stream). to achieve high efficiency in processing these two types of queries, we propose procedure red (standing for range-constrained euclidean distance) and algorithm eks (standing for enhanced knn search). compared to the existing methods in the prior research, the advantageous features of our approaches are in two folds. first, our approaches are capable of processing the queries directly from the wavelet synopses retained in the main memory without using idwt to reconstruct the data cells. this feature allows us to save the complexity in both memory and time. moreover, our approaches enable the users to query the dsms within their range of interest. unlike the conventional methods which only support the full-range query processing, this feature will enhance the flexibility at the client side. we evaluate procedure red and algorithm eks on live and synthetic datasets empirically and show that the proposed approaches are efficient in similarity search and knn discovery within arbitrary ranges in the time series streaming environments.
concept frequency distribution in biomedical text summarization. text summarization is a data reduction process. the use of text summarization enables users to reduce the amount of text that must be read while still assimilating the core information. the data reduction offered by text summarization is particularly useful in the biomedical domain, where physicians must continuously find clinical trial study information to incorporate into their patient treatment efforts. such efforts are often hampered by the high-volume of publications. our contribution is two-fold: 1) to propose the frequency of domain concepts as a method to identify important sentences within a full-text; and 2) propose a novel frequency distribution model and algorithm for identifying important sentences based on term or concept frequency distribution. an evaluation of several existing summarization systems using biomedical texts is presented in order to determine a performance baseline. for domain concept comparison, a recent high-performing frequency-based algorithm using terms is adapted to use concepts and evaluated using both terms and concepts. it is shown that the use of concepts performs closely with the use of terms for sentence selection. our proposed frequency distribution model and algorithm outperforms a state-of-the-art approach.
exploiting asymmetry in hierarchical topic extraction. topic or feature extraction is often used as an important step in document classification and text mining. topics are succinct representation of content in a document collection and hence are very effective when used as content identifiers in peer-to-peer systems and other large scale distributed content management systems. effective topic extraction is dependent on the accuracy of term clustering that often has to deal with problems like synonymy and polysemy. retrieval techniques based on spectral analysis like latent semantic indexing (lsi) are often used to effectively solve these problems. most of the spectral retrieval schemes produce term similarity measures that are symmetric and often, not an accurate characterization of term relationships. another drawback of lsi is its running time that is polynomial in the dimensions of the m x n matrix, a. this can get prohibitively large for some ir applications. in this paper, we present efficient algorithms using the technique of locality-sensitive hashing (lsh) to extract topics from a document collection based on the asymmetric relationships between terms in a collection. the relationship is characterized by the term co-occurrences and other higher-order similarity measures. our lsh based scheme can be viewed as a simple alternative to lsi. we show the efficacy of our algorithms via experiments on a set of large documents. an interesting feature of our algorithms is that it produces a natural hierarchical decomposition of the topic space instead of a flat clustering.
privacy preserving sequential pattern mining in distributed databases. research in the areas of privacy preserving techniques in databases and subsequently in privacy enhancement technologies have witnessed an explosive growth-spurt in recent years. this escalation has been fueled by the growing mistrust of individuals towards organizations collecting and disbursing their personally identifiable information (pii). digital repositories have become increasingly susceptible to intentional or unintentional abuse, resulting in organizations to be liable under the privacy legislations that are being adopted by governments the world over. these privacy concerns have necessitated new advancements in the field of distributed data mining wherein, collaborating parties may be legally bound not to reveal the private information of their customers. in this paper, we present a new algorithm pripsep (privacy preserving sequential patterns) for the mining of sequential patterns from distributed databases while preserving privacy. a salient feature of pripsep is that due to its flexibility it is more pertinent to mining operations for real world applications in terms of efficiency and functionality. under some reasonable assumptions, we prove that our architecture and protocol employed by our algorithm for multi-party computation is secure.
the real-time nature and value of homeland security information. ensuring the security of our homeland depends in large measure on two quite distinct factors: having the knowledge necessary to prevent, predict, prepare for, or respond, if necessary, to any manner of terrorist attack or a natural or manmade disaster and collaborating or sharing knowledge with a broad range of international, federal, state, local, and tribal agencies, as well as other private or public organizations. the essential problem with adequately addressing these factors, despite the many advancements made in the past decade, is twofold. first, it is not so much the mass but rather the diffuse nature and complexity of the data, information, and knowledge required for understanding terrorism and accounting for the manifold consequences of disasters that make possession of the right knowledge difficult. and, second, that diffuseness and complexity is magnified by the extreme diversity and wide distribution of the many potential homeland security collaborators. retrospective analysis, and even knowledge discovery, is less useful under these conditions than prospective, real-time synthesis of information for multiple users. also, privacy is as important as, if not more important than, security. this suggests that database designs and techniques for information retrieval, and knowledge management must take advantage of such technologies as semantic nets, visualization, and discrete mathematics to build knowledge systems capable for homeland security applications.
finding dense and isolated submarkets in a sponsored search spending graph. methods for improving sponsored search revenue are often tested or deployed within a small submarket of the larger marketplace. for many applications, the ideal submarket contains a small number of nodes, a large amount of spending within the submarket, and a small amount of spending leaving the submarket. we introduce an efficient algorithm for finding submarkets that are optimal for a user-specified tradeoff between these three quantities. we apply our algorithm to find submarkets that are both dense and isolated in a large spending graph from yahoo! sponsored search.
on gmap: and other transformations. as an alternative to the usual mean average precision, some use is currently being made of the geometric mean average precision (gmap) as a measure of average search effectiveness across topics. gmap is specifically used to emphasise the lower end of the average precision scale, in order to shed light on poor performance of search engines. this paper discusses the status of this measure and how it should be understood.
type nanotheories: a framework for term comparison. we present in this paper type nanotheories (tn), a framework for representing the knowledge necessary for performing similarity comparisons between pairs of terms of the same type. tn itself uses another methodology, namely support outcomes, which is also introduced. many ir and nlp applications use redundancy as a factor to increase confidence, and tn-based comparisons can determine redundancy better than simple string comparisons. results include a showing of a 14% increase in confidence-weighted score for an end-to-end qa system and an up to 68% improvement over baseline in an answer-key equivalencing experiment.
concept-based document readability in domain specific information retrieval. domain specific information retrieval has become in demand. not only domain experts, but also average non-expert users are interested in searching domain specific (e.g., medical and health) information from online resources. however, a typical problem to average users is that the search results are always a mixture of documents with different levels of readability. non-expert users may want to see documents with higher readability on the top of the list. consequently the search results need to be re-ranked in a descending order of readability. it is often not practical for domain experts to manually label the readability of documents for large databases. computational models of readability needs to be investigated. however, traditional readability formulas are designed for general purpose text and insufficient to deal with technical materials for domain specific information retrieval. more advanced algorithms such as textual coherence model are computationally expensive for re-ranking a large number of retrieved documents. in this paper, we propose an effective and computationally tractable concept-based model of text readability. in addition to textual genres of a document, our model also takes into account domain specific knowledge, i.e., how the domain-specific concepts contained in the document affect the document's readability. three major readability formulas are proposed and applied to health and medical information retrieval. experimental results show that our proposed readability formulas lead to remarkable improvements in terms of correlation with users' readability ratings over four traditional readability measures.
autonomously semantifying wikipedia. berners-lee's compelling vision of a semantic web is hindered by a chicken-and-egg problem, which can be best solved by a bootstrapping method - creating enough structured data to motivate the development of applications. this paper argues that autonomously "semantifying wikipedia" is the best way to solve the problem. we choose wikipedia as an initial data source, because it is comprehensive, not too large, high-quality, and contains enough manually-derived structure to bootstrap an autonomous, self-supervised process. we identify several types of structures which can be automatically enhanced in wikipedia (e.g., link structure, taxonomic data, infoboxes, etc.), and we describea prototype implementation of a self-supervised, machine learning system which realizes our vision. preliminary experiments demonstrate the high precision of our system's extracted data - in one case equaling that of humans.
matching directories and owl ontologies with aroma. this paper presents a simple and adaptable matching method dealing with web directories, catalogs and owl ontologies. by using a well-known knowledge discovery in databases model, such as the association rule paradigm, this method has the originality to be both extensional and asymmetric. it works at the terminological level (by selecting concept-relevant terms contained in documents) and permits to discover equivalence and also subsumption relations holding between entities (concepts and properties). this method relies on the implication intensity measure, a probabilistic model of deviation from independence. selection of significant rules between concepts (or properties) is lead by two criteria permitting to assess respectively the implication quality and the generativity of the rule. finally, the proposed method is evaluated on two benchmarks. the first contains two conceptual hierarchies populated with textual documents and the second one is composed of owl ontologies.
on subspace clustering with density consciousness. in this paper, a problem, called "the density divergence problem" is explored. this problem is related to the phenomenon that the densities of the clusters vary in different subspace cardinalities. we take the densities into consideration in subspace clustering and explore an algorithm to adaptively determine different density thresholds to discover clusters in different subspace cardinalities.
a system for query-specific document summarization. there has been a great amount of work on query-independent summarization of documents. however, due to the success of web search engines query-specific document summarization (query result snippets) has become an important problem, which has received little attention. we present a method to create query-specific summaries by identifying the most query-relevant fragments and combining them using the semantic associations within the document. in particular, we first add structure to the documents in the preprocessing stage and convert them to document graphs. then, the best summaries are computed by calculating the top spanning trees on the document graphs. we present and experimentally evaluate efficient algorithms that support computing summaries in interactive time. furthermore, the quality of our summarization method is compared to current approaches using a user survey.
efficient model selection for regularized linear discriminant analysis. classical linear discriminant analysis (lda) is not applicable for small sample size problems due to the singularity of the scatter matrices involved. regularized lda (rlda) provides a simple strategy to overcome the singularity problem by applying a regularization term, which is commonly estimated via cross-validation from a set of candidates. however, cross-validation may be computationally prohibitive when the candidate set is large. an efficient algorithm for rlda is presented that computes the optimal transformation of rlda for a large set of parameter candidates, with approximately the same cost as running rlda a small number of times. thus it facilitates efficient model selection for rlda.an intrinsic relationship between rlda and uncorrelated lda (ulda), which was recently proposed for dimension reduction and classification is presented. more specifically, rlda is shown to approach ulda when the regularization value tends to zero. that is, rlda without any regularization is equivalent to ulda. it can be further shown that ulda maps all data points from the same class to a common point, under a mild condition which has been shown to hold for many high-dimensional datasets. this leads to the overfitting problem in ulda, which has been observed in several applications. thetheoretical analysis presented provides further justification for the use of regularization in rlda. extensive experiments confirm the claimed theoretical estimate of efficiency. experiments also show that, for a properly chosen regularization parameter, rlda performs favorably in classification, in comparison with ulda, as well as other existing lda-based algorithms and support vector machines (svm).
a cdd-based formal model for expert finding. searching an organization's document repositories for experts is a frequently faced problem in intranet information management. this paper proposes a candidate-centered model which is referred as candidate description document (cdd)-based retrieval model. the expertise evidence about an expert candidate scattered over repositories is mined and aggregated automatically to form a profile called the candidate's cdd, which represents his knowledge. we present the model from its foundations through its logical development and argue in favor of this model for expert finding. we devise and compare the different strategies for exploring a variety of expertise evidence. the experiments on trec enterprise corpora demonstrate that the cdd-based model achieves significant and consistent improvement on performance through comparative studies with non-cdd methods.
pseudo-anchor text extraction for searching vertical objects. this paper examines the problem of utilizing pseudo-anchor text to help ranking web objects in vertical search. we adopt a machine learning based approach to extract pseudo-anchor text for a vertical object from its candidate anchor blocks. experiments in academic search domain indicate that our approach is able to dramatically improve search performance.
heuristic containment check of partial tree-pattern queries in the presence of index graphs. the wide adoption of xml has increased the interest of the database community on tree-structured data management techniques. querying capabilities are provided through tree-pattern queries. the need for querying tree-structured data sources when their structure is not fully known, and the need to integrate multiple data sources with different tree structures have driven, recently, the suggestion of query languages that relax the complete specification of a tree pattern. in this paper, we use a query language which allows partial tree-pattern queries (ptpqs). the structure in a ptpq can be flexibly specified fully, partially or not at all. to evaluate a ptpq, we exploit index graphs which generate an equivalent set of "complete" tree-pattern queries.in order to process ptpqs, we need to efficiently solve the ptpq satisfiability and containment problems. these problems become more complex in the context of ptpqs because the partial specification of the structure allows new, non-trivial, structural expressions to be derived from those explicitly specified in a ptpq. we address the problem of ptpq satisfiability and containment in the absence and in the presence of index graphs, and we provide necessary and sufficient conditions for each case. to cope with the high complexity of ptpq containment in the presence of index graphs,we study a family of heuristic approaches for ptpq containment based on structural information extracted from the index graph in advance and on-the-fly. we implement our approaches and we report on their extensive experimental evaluation and comparison.
comparing the effectiveness of hits and salsa. this paper compares the effectiveness of two well-known query-dependent link-based ranking algorithms, "hyperlink-induced topic search" (hits) and the "stochastic approach for link-structure analysis" (salsa). the two algorithms are evaluated on a very large web graph induced by 463 million crawled web pages and a set of 28,043 queries and 485,656 results labeled by human judges. we employed three different performance measures - mean average precision (map), mean reciprocal rank (mrr), and normalized discounted cumulative gain (ndcg). we found that as an isolated feature, salsa substantially outperforms hits. this is quite surprising, given that the two algorithms operate over the same neighborhood graph induced by the query result set. we also studied the combination of salsa and hits with bm25f, a state-of-the-art text-based scoring function that incorporates anchor text. we found that the combination of salsa and bm25f outperforms the combination of hits and bm25f. finally, we broke down our query set by query specificity, and found that salsa (and to a lesser extent hits) is most effective for general queries.
task-based process know-how reuse and proactive information delivery in tasknavigator. knowledge management approaches for weakly-structured, adhoc knowledge work processes need to be lightweight, i.e., they cannot rely on high upfront modeling efforts. this paper presents tasknavigator, a novel prototype to support weakly-structured processes by integrating a standard task list application with a state-of-the-art document classification system. the resulting system allows for a task-oriented view on office workers' personal knowledge spaces in order to realize a proactive and contextsensitive information support during daily, knowledge-intensive tasks. moreover, tasknavigator supports process know-how reuse by proactively suggesting similar tasks or relevant process models, based on textual similarities. finally, we report on a feasibility test and a case study that have been conducted in order to evaluate the system in the context of daily research task management and software requirements analysis.