waashk/dblp huggingface dataset:
data mining using high performance data clouds: experimental studies using sector and sphere. we describe the design and implementation of a high performance cloud that we have used to archive, analyze and mine large distributed data sets. by a cloud, we mean an infrastructure that provides resources and/or services over the internet. a storage cloud provides storage services, while a compute cloud provides compute services. we describe the design of the sector storage cloud and how it provides the storage services required by the sphere compute cloud. we also describe the programming paradigm supported by the sphere compute cloud. sector and sphere are designed for analyzing large data sets using computer clusters connected with wide area high performance networks (for example, 10+ gb/s). we describe a distributed data mining application that we have developed using sector and sphere. finally, we describe some experimental studies comparing sector/sphere to hadoop.
combinational collaborative filtering for personalized community recommendation. rapid growth in the amount of data available on social networking sites has made information retrieval increasingly challenging for users. in this paper, we propose a collaborative filtering method, combinational collaborative filtering (ccf), to perform personalized community recommendations by considering multiple types of co-occurrences in social data at the same time. this filtering method fuses semantic and user information, then applies a hybrid training strategy that combines gibbs sampling and expectation-maximization algorithm. to handle the large-scale dataset, parallel computing is used to speed up the model training. through an empirical study on the orkut dataset, we show ccf to be both effective and scalable.
unsupervised deduplication using cross-field dependencies. recent work in deduplication has shown that collective deduplication of different attribute types can improve performance. but although these techniques cluster the attributes collectively, they do not model them collectively. for example, in citations in the research literature, canonical venue strings and title strings are dependent -- because venues tend to focus on a few research areas -- but this dependence is not modeled by current unsupervised techniques. we call this dependence between fields in a record a cross-field dependence. in this paper, we present an unsupervised generative model for the deduplication problem that explicitly models cross-field dependence. our model uses a single set of latent variables to control two disparate clustering models: a dirichlet-multinomial model over titles, and a non-exchangeable string-edit model over venues. we show that modeling cross-field dependence yields a substantial improvement in performance -- a 58% reduction in error over a standard dirichlet process mixture.
anomaly pattern detection in categorical datasets. we propose a new method for detecting patterns of anomalies in categorical datasets. we assume that anomalies are generated by some underlying process which affects only a particular subset of the data. our method consists of two steps: we first use a "local anomaly detector" to identify individual records with anomalous attribute values, and then detect patterns where the number of anomalous records is higher than expected. given the set of anomalies flagged by the local anomaly detector, we search over all subsets of the data defined by any set of fixed values of a subset of the attributes, in order to detect self-similar patterns of anomalies. we wish to detect any such subset of the test data which displays a significant increase in anomalous activity as compared to the normal behavior of the system (as indicated by the training data). we perform significance testing to determine if the number of anomalies in any subset of the test data is significantly higher than expected, and propose an efficient algorithm to perform this test over all such subsets of the data. we show that this algorithm is able to accurately detect anomalous patterns in real-world hospital, container shipping and network intrusion data.
simultaneous tensor subspace selection and clustering: the equivalence of high order svd and k-means clustering. singular value decomposition (svd)/principal component analysis (pca) have played a vital role in finding patterns from many datasets. recently tensor factorization has been used for data mining and pattern recognition in high index/order data. high order svd (hosvd) is a commonly used tensor factorization method and has recently been used in numerous applications like graphs, videos, social networks, etc. in this paper we prove that hosvd does simultaneous subspace selection (data compression) and k-means clustering widely used for unsupervised learning tasks. we show how to utilize this new feature of hosvd for clustering. we demonstrate these new results using three real and large datasets, two on face images datasets and one on hand-written digits dataset. using this new hosvd clustering feature we provide a dataset quality assessment on many frequently used experimental datasets with expected noise levels.
stable feature selection via dense feature groups. many feature selection algorithms have been proposed in the past focusing on improving classification accuracy. in this work, we point out the importance of stable feature selection for knowledge discovery from high-dimensional data, and identify two causes of instability of feature selection algorithms: selection of a minimum subset without redundant features and small sample size. we propose a general framework for stable feature selection which emphasizes both good generalization and stability of feature selection results. the framework identifies dense feature groups based on kernel density estimation and treats features in each dense group as a coherent entity for feature selection. an efficient algorithm drags (dense relevant attribute group selector) is developed under this framework. we also introduce a general measure for assessing the stability of feature selection algorithms. our empirical study based on microarray data verifies that dense feature groups remain stable under random sample hold out, and the drags algorithm is effective in identifying a set of feature groups which exhibit both high classification accuracy and stability.
real memory mach. mach was designed with the assumption that the underlying hardware includes a page-based memory management unit, enabling the use of a large, sparse virtual address space. therefore the mach micro-kernel in its current state cannot be used on segmented machines such as cray vector supercomputers or real memory machines such as transputers. this paper describes the results of initial work to adapt the mach microkernel to such architectures, including the architectural changes and results obtained from a prototype system.
mike: a distributed object-oriented programming platform on top of the mach micro-kernel. this paper describes the architecture and implementation of mike - a version of the ik distributed persistent object-oriented programming platform built on top of the mach microkernel. mike's primary goal is to offer a single object-oriented programming paradigm for writing distributed applications. in mike an application programmer can use c++ almost as he would in a non-distributed system. the platform supports fine grained objects which can be invoked in a location transparent way and whose references can be exchanged freely as invocation parameters. these objects are potentially persistent. mike supports the abstraction of one-level store, persistent objects are transparently loaded on demand when first invoked and saved to disk when the application terminates. class objects are special persistent objects which are dynamically linked when needed. the platform also offers distributed garbage collection of non-persistent objects. this paper discusses how mike makes use of mach's features to offer the functionality described above and the techniques used to achieve good performance. mike is compared with the unix versions of ik to evaluate the benefits of using mach abstractions.
kernel support for recoverable-persistent virtual memory. the buffering facilities typically provided by operating systems are not powerful enough to support the performance and consistency requirements of database systems. as a result, most database systems are structured as buffer pool database (bpdb) systems, providing their own buffering facilities, with their own paging policies and recovery schemes. the emergence of operating systems with very large address spaces and flexible memory management makes virtual memory database (vmdb) systems feasible. in such systems, the database is mapped into virtual memory and the buffering facilities of the underlying virtual memory system are used. vmdb systems do not experience many of the problems faced by bpdb systems. to support the consistency and recoverability requirements of vmdb systems, we have proposed that the virtual memory system be extended to support the recoverable-persistent updates (rpu) model. this model is powerful and general enough to support a wide variety of policies for ensuring database recoverability. in this paper we discuss our approach to and progress in extending the mach 3.0 kernel to provide direct support for this rpu model.
using continuations to build a user-level threads library. we have designed and built a user-level threads library that uses continuations for transfers of control. the use of continuations reduces the amount of state that needs to be saved and restored at context switch time thereby reducing the instruction count in the critical sections. our multiprocessor contention benchmarks indicate that this reduction and the use of busy spinning, busy waiting and spin polling increases throughput by as much as 75% on a multiprocessor. in addition, flattening the locking hierarchy reduces context switch latency by 5% to 49% on both uniprocessors and multiprocessors. this paper describes the library's design and compares its overall performance characteristics to the existing implementation.
using the mach communication primitives in x11. we have modified the x11 windowing system to use the native communication facilities of the ach 3.0 microkernel. our new implementation can rely on ach's low-overhead ipc facility as a direct replacement for sockets, or it can use shared memory as a transport between x11 clients and the server. on conventional bsd unix systems. x11 communication is done through sockets. because a user-level process implements unix functionality on top of ach 3.0, a socket-based version of x11 performs substantially worse than when running on a monolithic unix kernel. using ach ipc as the transport between x11 clients and the server. x11 performance is slightly better than that of a monolithic system in which sockets are implemented inside the kernel as opposed to within a user level process. using ach's shared memory facilities as the transport, we have measured performance improvements of over 40%.
sprite on mach. sprite is a distributed operating system that supports a fast, single-image network file system and transparent process migration. over a period of 19 months we ported sprite to run as a server on top of the mach 3.0 microkernel. although the resulting server does not implement some sprite features, it can run in an existing sprite cluster, and it supports standard unix programs like vi, gcc, and make. porting sprite to mach was generally straightforward, though there were some difficulties. many of the problems were related to asynchronous interactions between the sprite server, mach, and sprite user processes. others resulted from trying to maintain native sprite's internal interfaces in the sprite server. the sprite server is 22% smaller than an equivalent sprite kernel, and it contains almost no machine-dependent code. these improvements should significantly simplify porting sprite to new hardware platforms. unfortunately, the sprite server runs the andrew benchmark at only 38% of the speed of native sprite. none of the performance problems appears insurmountable, but they could require a long time to track down and fix.
unix file access and caching in a multicomputer environment. this paper describes the unix file access and caching mechanisms in a version of the osf/1 unix operating system designed to run in a multicomputer environment. the multicomputer hard-ware platforms targeted can consist of hundreds or even thousands of individual nodes, where each node consists of one or more processors. the multicomputer version of osf/1 (called osf/1 ad) uses mach memory objects to cache data from unix files, and relies on an in-kernel distributed shared memory implementation to maintain coherency for data cached across multiple nodes. the focus of this paper is on the modifications made to standard osf/1 functionality to support distributed, efficient access to memory objects. of particular interest are the introduction of a mapped files module for synchronizing clients and maintaining file meta data, the elimination of the traditional unix buffer cache from the file data access path, and the implementation of a disk block reservation scheme to correctly support unix write() semantics. an evaluation of the technology is presented, providing insight into how it can be improved in the future, including several possible enhancements to mach. as will be seen, most of this insight would equally apply to a single-node operating system based on mach.
automatic 3d registration of lung surfaces in computed tomography scans. abstract we developed an automated system that registers chest ct scans temporally. our registration method matches corresponding anatomical landmarks to obtain initial registration parameters. the initial point-to-point registration is then generalized to an iterative surface-to-surface registration method. our ``goodness-of-fit'''' measure is evaluated at each step in the iterative scheme until the registration performance is sufficient. we applied our method to register the 3d lung surfaces of 11 pairs of chest ct scans and report promising registration performance.
towards whole brain segmentation by a hybrid model. segmenting cortical and sub-cortical structures from 3d brain images is of significant practical importance. however, various anatomical structures have similar intensity patterns in mri, and the automatic segmentation of them is a challenging task. in this paper, we present a new brain segmentation algorithm using a hybrid model. (1) a multi-class classifier, pbt.m2, is proposed for learning/computing multi-class discriminative models. the pbt.m2 handles multi-class patterns more easily than the original probabilistic boosting tree (pbt) [11], and it facilitates the process, eventually, toward whole brain segmentation. (2) we use an edge field, by learning, to constraint the region boundaries. we show the improvements due to the two new aspects both numerically and visually, and also compare the results with those by freesurfer [2]. our algorithm is general and easy to use, and the results obtained are encouraging.
nonlinear analysis of bold signal: biophysical modeling, physiological states, and functional activation. there is an increasing interest in exploiting the biophysical plausible models to investigate the physiological mechanisms that underlie observed bold response. however, most existing studies do not produce reliable model parameter estimates, are not robust due to the linearization of the nonlinear model, and do not perform statistics test to detect functional activation. to overcome these limitations, we developed a general framework for the analysis of fmri data based on nonlinear physiological models. it performs system dynamics analysis to gain meaningful insight, followed by global sensitivity analysis for model reduction which leads to better system identifiability. subsequently, a nonlinear filter is used to simultaneously estimate the state and parameter of the dynamic system, and statistics test is performed to derive activation maps based on such model. furthermore, we investigate the change of the activation maps of these hidden physiological variables with experimental paradigm through time as well.
markov dependence tree-based segmentation of deep brain structures. we propose a new framework for multi-object segmentation of deep brain structures, which have significant shape variations and relatively small sizes in medical brain images. in the images, the structure boundaries may be blurry or even missing, and the surrounding background is a clutter and full of irrelevant edges. we suggest a template-based framework, which fuses the information of edge features, region statistics and inter-structure constraints to detect and locate all the targeted brain structures such that manual initialization is unnecessary. the multi-object template is organized in the form of a hierarchical markov dependence tree. it makes the matching of multiple objects efficient. our approach needs only one example as training data and alleviates the demand of a large training set. the obtained segmentation results on real data are encouraging and the proposed method enjoys several important advantages over existing methods.
toward optimal matching for 3d reconstruction of brachytherapy seeds. x-ray c-arm fluoroscopy is a natural choice for intraoperative seed localization in prostate brachy therapy. resolving the correspondence of seeds in the projection images can be modeled as an assignment problem that is np-hard. our approach rests on the practical observation that the optimal solution has almost zero cost if the pose of the c-arm is known accurately. this allowed us to to derive an equivalent problem of reduced dimensionality that, with linear programming, can be solved efficiently in polynomial time. additionally, our method demonstrates significantly increased robustness to c-arm pose errors when compared to the prior art. because under actual clinical circumstances it is exceedingly difficult to track the c-arm, easing on this constraint has additional practical utility.
segmentation of q-ball images using statistical surface evolution. in this article, we develop a new method to segment q-ball imaging (qbi) data. we first estimate the orientation distribution function (odf) using a fast and robust spherical harmonic (sh) method. then, we use a region-based statistical surface evolution on this image of odfs to efficiently find coherent white matter fiber bundles. we show that our method is appropriate to propagate through regions of fiber crossings and we show that our results outperform state-of-the-art diffusion tensor (dt) imaging segmentation methods, inherently limited by the dt model. results obtained on synthetic data, on a biological phantom, on real datasets and on all 13 subjects of a public qbi database show that our method is reproducible, automatic and brings a strong added value to diffusion mri segmentation.
bias image correction via stationarity maximization. automated acquisitions in microscopy may come along with strong illumination artifacts due to poor physical imaging conditions. such artifacts obviously have direct consequences on the efficiency of an image analysis algorithm and on the quantitative measures. in this paper, we propose a method to correct illumination artifacts on biological images. this correction is based on orthogonal polynomial modeling, combined with stationary maximization criteria. to validate the proposed method we show that we improve particle detection algorithm.
primal/dual linear programming and statistical atlases for cartilage segmentation. in this paper we propose a novel approach for automatic segmentation of cartilage using a statistical atlas and efficient primal/dual linear programming. to this end, a novel statistical atlas construction is considered from registered training examples. segmentation is then solved through registration which aims at deforming the atlas such that the conditional posterior of the learned (atlas) density is maximized with respect to the image. such a task is reformulated using a discrete set of deformations and segmentation becomes equivalent to finding the set of local deformations which optimally match the model to the image. we evaluate our method on 56 mri data sets (28 used for the model and 28 used for evaluation) and obtain a fully automatic segmentation of patella cartilage volume with an overlap ratio of 0.84 with a sensitivity and specificity of 94.06% and 99.92%, respectively.
segmentation-driven 2d-3d registration for abdominal catheter interventions. 2d-3d registration of abdominal angiographic data is a difficult problem due to hard time constraints during the intervention, different vessel contrast in volume and image, and motion blur caused by breathing. we propose a novel method for aligning 2d digitally subtracted angiograms (dsa) to computed tomography angiography (cta) volumes, which requires no user interaction intrainterventionally. in an iterative process, we link 2d segmentation and 2d-3d registration using a probability map, which creates a common feature space where outliers in 2d and 3d are discarded consequently. unlike other approaches, we keep user interaction low while high capture range and robustness against vessel variability and deformation are maintained. tests on five patient data sets and a comparison to two recently proposed methods show the good performance of our method.
towards subject-specific models of the dynamic heart for image-guided mitral valve surgery. surgeons need a robust interventional system capable of providing reliable, real-time information regarding the position and orientation of the surgical targets and tools to compensate for the lack of direct vision and to enhance manipulation of intracardiac targets during minimally-invasive, off-pump cardiac interventions. in this paper, we describe a novel method for creating dynamic, pre-operative, subject-specific cardiac models containing the surgical targets and surrounding anatomy, and how they are used to augment the intra-operative virtual environment for guidance of valvular interventions. the accuracy of these pre-operative models was established by comparing the target registration error between the mitral valve annulus characterized in the pre-operative images and their equivalent structures manually extracted from 3d us data. on average, the mitral valve annulus was extracted with a 3.1 mm error across all cardiac phases. in addition, we also propose a method for registering the pre-operative models into the intraoperative virtual environment.
lv motion tracking from 3d echocardiography using textural and structural information. automated motion reconstruction of the left ventricle (lv) from 3d echocardiography provides insight into myocardium architecture and function. low image quality and artifacts make 3d ultrasound image processing a challenging problem. we introduce a lv tracking method, which combines textural and structural information to overcome the image quality limitations. our method automatically reconstructs the motion of the lv contour (endocardium and epicardium) from a sequence of 3d ultrasound images.
real-time tissue tracking with b-mode ultrasound using speckle and visual servoing. we present a method for real-time tracking of moving soft tissue with b-mode ultrasound (us). the method makes use of the speckle information contained in the us images to estimate the in-plane and out-of-plane motion of a fixed target relative to the ultrasound scan plane. the motion information is then used as closed-loop feedback to a robot which corrects for the target motion. the concept is demonstrated for translation motions in an experimental setup consisting of an ultrasound speckle phantom, a robot for simulating tissue motion, and a robot that performs motion stabilization from us images. this concept shows promise for us-guided procedures that require real-time motion tracking and compensation.