waashk/dblp huggingface dataset:
surviving internet catastrophes. in this paper, we propose a new approach for designing distributed systems to survive internet catastrophes called informed replication, and demonstrate this approach with the design and evaluation of a cooperative backup system called the phoenix recovery service. informed replication uses a model of correlated failures to exploit software diversity. the key observation that makes our approach both feasible and practical is that internet catastrophes result from shared vulnerabilities. by replicating a system service on hosts that do not have the same vulnerabilities, an internet pathogen that exploits a vulnerability is unlikely to cause all replicas to fail. to characterize software diversity in an internet setting, we measure the software diversity of host operating systems and network services in a large organization. we then use insights from our measurement study to develop and evaluate heuristics for computing replica sets that have a number of attractive features. our heuristics provide excellent reliability guarantees, result in low degree of replication, limit the storage burden on each host in the system, and lend themselves to a fully distributed implementation. we then present the design and prototype implementation of phoenix, and evaluate it on the planetlab testbed.
auto-pilot: a platform for system software benchmarking. when developing software, it is essential to evaluate its performance and stability, making benchmarking an essential and significant part of the software development cycle. benchmarking is also used to show that a system is useful or provide insight into how systems behave. however, benchmarking is a tedious task that few enjoy, but every programmer or systems researcher must do. developers need an easy-to-use system for collecting and analyzing benchmark results. we introduce auto-pilot, a tool for producing accurate and informative benchmark results. auto-pilot provides an infrastructure for running tests, sample test scripts, and analysis tools. auto-pilot is not just another metric or benchmark: it is a system for automating the repetitive tasks of running, measuring, and analyzing the results of arbitrary programs. auto-pilot can run a given test until results stabilize, automatically highlight outlying results, and automatically detect memory leaks. we have used autopilot for over three years on eighteen distinct projects and have found it to be an invaluable tool that saved us significant effort.
sprockets: safe extensions for distributed file systems. sprockets are a lightweight method for extending the functionality of distributed file systems. they specifically target file systems implemented at user level and small extensions that can be expressed with up to several hundred lines of code. each sprocket is akin to a procedure call that runs inside a transaction that is always rolled back on completion, even if sprocket execution succeeds. sprockets therefore make no persistent changes to file system state; instead, they communicate their result back to the core file system through a restricted format using a shared memory buffer. the file system validates the result and makes any necessary changes if the validations pass. sprockets use binary instrumentation to ensure that a sprocket can safely execute file system code without making changes to persistent state. we have implemented sprockets that perform type-specific handling within file systems such as querying application metadata, application-specific conflict resolution, and handling custom devices such as digital cameras. our evaluation shows that sprockets can be up to an order of magnitude faster to execute than extensions that utilize operating system services such as fork. we also show that sprockets allow fine-grained isolation and, thus, can catch some bugs that a fork-based implementation cannot.
a dollar from 15 cents: cross-platform management for internet services. as internet services become ubiquitous, the selection and management of diverse server platforms now affects the bottom line of almost every firm in every industry. ideally, such cross-platform management would yield high performance at low cost, but in practice, the performance consequences of such decisions are often hard to predict. in this paper, we present an approach to guide cross-platform management for real-world internet services. our approach is driven by a novel performance model that predicts application-level performance across changes in platform parameters, such as processor cache sizes, processor speeds, etc., and can be calibrated with data commonly available in today's production environments. our model is structured as a composition of several empirically observed, parsimonious sub-models. these sub-models have few free parameters and can be calibrated with lightweight passive observations on a current production platform. we demonstrate the usefulness of our cross-platform model in two management problems. first, our model provides accurate performance predictions when selecting the next generation of processors to enter a server farm. second, our model can guide platform-aware load balancing across heterogeneous server farms.
bridging the gap between software and hardware techniques for i/o virtualization. the paravirtualized i/o driver domain model, used in xen, provides severl advantages including device driver isolation in a safe execution environment, support for guest vm transparent services including live migration, and hardware independence for guests. however, these advantages currently come at the cost of high cpu overhead which can lead to low throughput for high bandwidth links such as 10 gigabit ethernet. direct i/o has been proposed as the solution to this performance problem but at the cost of removing the benefits of the driver domain model. in this paper we show how to significantly narrow the performance gap by improving the performance of the driver domain model. in particular, we reduce execution costs for conventional nics by 56% on the receive path, and we achieve close to direct i/o performance for network devices supporting multiple hardware receive queues. these results make the xen driver domain model an attractive solution for i/o virtualization for a wider range of scenarios.
cutting corners: workbench automation for server benchmarking. a common approach to benchmarking a server is to measure its behavior under load from a workload generator. often a set of such experiments is required-- perhaps with different server configurations or workload parameters--to obtain a statistically sound result for a given benchmarking objective. this paper explores a framework and policies to conduct such benchmarking activities automatically and efficiently. the workbench automation framework is designed to be independent of the underlying benchmark harness, including the server implementation, configuration tools, and workload generator. rather, we take those mechanisms as given and focus on automation policies within the framework. as a motivating example we focus on rating the peak load of an nfs file server for a given set of workload parameters, a common and costly activity in the storage server industry. experimental results show how an automated workbench controller can plan and coordinate the benchmark runs to obtain a result with a target threshold of confidence and accuracy at lower cost than scripted approaches that are commonly practiced. in more complex benchmarking scenarios, the controller can consider various factors including accuracy vs. cost tradeoffs, availability of hardware resources, deadlines, and the results of previous experiments.
a compacting real-time memory management system. we propose a real real-time memory management system called compact-fit that offers both time and space predictability. compact-fit is a compacting memory management system for allocating, deallocating, and accessing memory in real time. the system provides predictable memory fragmentation and response times that are constant or linear in the size of the request, independently of the global memory state. we present two compact-fit implementations and compare them to established memory management systems, which all fail to provide predictable memory fragmentation. the experiments confirm our theoretical complexity bounds and demonstrate competitive performance. in addition, we can control the performance versus fragmentation trade-off via our concept of partial compaction. the system can be parameterized with the needed level of compaction, improving the performance while keeping memory fragmentation predictable.
using causality to diagnose configuration bugs. we present a novel method for diagnosing configuration management errors. our proposed approach deduces the state of a buggy computer by running predicates that test system correctness and comparing the resulting execution to that generated by running the same predicates on a reference computer. our approach generates signatures that represent the execution path of a predicate by recording the causal dependencies of its execution. our results show that comparisons based on dependency sets significantly outperform comparisons based on predicate success or failure, uniquely identifying the correct bug 86-100% of the time. in the remaining cases, the dependency set method identifies the correct bug as one of two equally likely bugs.
free factories: unified infrastructure for data intensive web services. we introduce the free factory, a platform for deploying data-intensive web services using small clusters of commodity hardware and free software. independently administered virtual machines called freegols give application developers the flexibility of a general purpose web server, along with access to distributed batch processing, cache and storage services. each cluster exploits idle ram and disk space for cache, and reserves disks in each node for high bandwidth storage. the batch processing service uses a variation of the mapreduce model. virtualization allows every cpu in the cluster to participate in batch jobs. each 48-node cluster can achieve 4-8 gigabytes per second of disk i/o. our intent is to use multiple clusters to process hundreds of simultaneous requests on multi-hundred terabyte data sets. currently, our applications achieve 1 gigabyte per second of i/o with 123 disks by scheduling batch jobs on two clusters, one of which is located in a remote data center.
fast, inexpensive content-addressed storage in foundation. foundation is a preservation system for users' personal, digital artifacts. foundation preserves all of a user's data and its dependencies--fonts, programs, plugins, kernel, and configuration state--by archiving nightly snapshots of the user's entire hard disk. users can browse through these images to view old data or recover accidentally deleted files. to access data that a user's current environment can no longer interpret, foundation boots the disk image in which that data resides under an emulator, allowing the user to view and modify the data with the same programs with which the user originally accessed it. this paper describes foundation's archival storage layer, which uses content-addressed storage (cas) to retain nightly snapshots of users' disks indefinitely. current state-of-the-art cas systems, such as venti [34], require multiple high-speed disks or other expensive hardware to achieve high performance. foundation's archival storage layer, in contrast, matches the storage efficiency of venti using only a single usb hard drive. foundation archives disk snapshots at an average throughput of 21 mb/s and restores them at an average of 14 mb/s, more than an order of magnitude improvement over venti running on the same hardware. unlike venti, foundation does not rely on the assumption that sha-1 is collision-free.
power-aware remote replication for enterprise-level disaster recovery systems. electric energy consumed in data centers is rapidly growing. power-aware it, recently called 'green it', is widely recognized as a significant challenge. disk storage is a non-negligible energy consumer. rather, in light of recent data-intensive systems where a number of disk drives are incorporated, the disk storage may be what we must consider primarily. yet, all of the disk drives are not used for primary datasets, but rather larger portion of them are utilized for storing a variety of copies such as backups and snapshots. saving the energy of such storage resources that manage copies is a promising approach. the paper presents a power-aware disaster recovery system, in which the reflection of transferred updated information can be deferred through eager compaction technique. great energy saving of storage systems is expected in the remote secondary site. our experiments using a commercial database system show that 80-85% energy of the secondary-site disk storage can be saved with small penalties of possible service breakdown time.
diverse replication for single-machine byzantine-fault tolerance. new single-machine environments are emerging from abundant computation available through multiple cores and secure virtualization. in this paper, we describe the research challenges and opportunities around diversified replication as a method to increase the byzantine-fault tolerance (bft) of single-machine servers to software attacks or errors. we then discuss the design space of bft protocols enabled by these new environments.
handling flash crowds from your garage. the garage innovator creates new web applications which may rocket to popular success - or sink when the flash crowd that arrives melts the web server. in the web context, utility computing provides a path by which the innovator can, with minimal capital, prepare for overwhelming popularity. many components required for web computing have recently become available as utilities. we analyze the design space of building a load-balanced system in the context of garage innovation. we present six experiments that inform this analysis by highlighting limitations of each approach. we report our experience with three services we deployed in "garage" style, and with the flash crowds that each drew.
idle read after write - iraw. despite a low occurrence rate, silent data corruption represents a growing concern for storage systems designers. throughout the storage hierarchy, from the file system down to the disk drives, various solutions exist to avoid, detect, and correct silent data corruption. undetected errors during the completion of writes may cause silent data corruption. a portion of the write errors may be detected and corrected successfully by verifying the data written on the disk with the data in the disk cache. write verification traditionally is scheduled immediately after a write completion (read after write - raw) which is unattractive, because it degrades user performance. to reduce the performance penalty associated with raw, we propose to retain the written content in the disk cache and verify it once the disk drive becomes idle. although attractive, this approach (called iraw - idle read after write) contends for resources, i.e., cache and idle time, with user traffic and other background activities. in this paper, we present a trace-driven evaluation of iraw and show its feasibility. our analysis indicates that idleness is present in disk drives and can be utilized for write verification with minimal effect on user performance. iraw benefits significantly if some amount of cache, i.e., 1 or 2 mb, is dedicated to retain the unverified writes. if the cache is shared with the user requests then a cache retention policy that places both reads and writes upon completion at the most recently used cache segment, yields best iraw performance without effecting user reads cache hit ratio and overall user performance.
vx32: lightweight user-level sandboxing on the x86. code sandboxing is useful for many purposes, but most sandboxing techniques require kernel modifications, do not completely isolate guest code, or incur substantial performance costs. vx32 is a multipurpose user-level sandbox that enables any application to load and safely execute one or more guest plug-ins, confining each guest to a system call api controlled by the host application and to a restricted memory region within the host's address space. vx32 runs guest code efficiently on several widespread operating systems without kernel extensions or special privileges; it protects the host program from both reads and writes by its guests; and it allows the host to restrict the instruction set available to guests. the key to vx32's combination of portability, flexibility, and efficiency is its use of x86 segmentation hardware to sandbox the guest's data accesses, along with a lightweight instruction translator to sandbox guest instructions. we evaluate vx32 using microbenchmarks and whole system benchmarks, and we examine four applications based on vx32: an archival storage system, an extensible public-key infrastructure, an experimental user-level operating system running atop another host os, and a linux system call jail. the first three applications export custom apis independent of the host os to their guests, making their plug-ins binary-portable across host systems. compute-intensive workloads for the first two applications exhibit between a 30% slowdown and a 30% speedup on vx32 relative to native execution; speedups result from vx32's instruction translator improving the cache locality of guest code. the experimental user-level operating system allows the use of the guest os's applications alongside the host's native applications and runs faster than whole-system virtual machine monitors such as vmware and qemu. the linux system call jail incurs up to 80% overhead but requires no kernel modifications and is delegation-based, avoiding concurrency vulnerabilities present in other interposition mechanisms.
improving ssh-style host authentication with multi-path probing. the popularity of "trust-on-first-use" (tofu) authentication, used by ssh and https with self-signed certificates, demonstrates significant demand for host authentication that is low-cost and simple to deploy. while tofu-based applications are a clear improvement over completely insecure protocols, they can leave users vulnerable to even simple network attacks. our system, perspectives, thwarts many of these attacks by using a collection of "notary" hosts that observes a server's public key via multiple network vantage points (detecting localized attacks) and keeps a record of the server's key over time (recognizing short-lived attacks). clients can download these records on-demand and compare them against an unauthenticated key, detecting many common attacks. perspectives explores a promising part of the host authentication design space: trust-on-first-use applications gain significant attack robustness without sacrificing their ease-of-use. we also analyze the security provided by perspectives and describe our experience building and deploying a publicly available implementation.
confidns: leveraging scale and history to detect compromise. while cooperative dns resolver systems, such as co-dns, have demonstrated improved reliability and performance over standard approaches, their security has been weaker, since any corruption or misbehavior of a single resolver can easily propagate throughout the system. we address this weakness in a new system called confi-dns, which augments the cooperative lookup process with configurable policies that utilize multi-site agreement and per-site lookup histories. not only does confi-dns provide better security than cooperative approaches, but for up to 99.8% of unique lookups, confidns exceeds the security of standard dns resolvers. confidns provides these benefits while retaining the other benefits of codns, such as incremental deployability, higher reliability, and improved performance, in some cases faster than codns.
a tcp-layer name service for tcp ports. this paper presents a simple name service for tcp ports, allowing services to be reached by name instead of number. names are arbitrary byte arrays that are bound to listening ports. name resolutions take place during the tcp three-way handshake, not requiring extra message exchanges. the new tcp handshake conforms with the standard and is fully compatible with existing tcp implementations. a prototype implementation was developed in linux, paying special attention to backward compatibility with legacy systems (kernels and applications). among the many opportunities created by the name service, it allows services with unusual names, known only by small communities, to remain undetected by port scanners (though not by network sniffers).
adaptive file transfers for diverse environments. this paper presents dsync, a file transfer system that can dynamically adapt to a wide variety of environments. while many transfer systems work well in their specialized ontext, their performance comes at the cost of generality, and they perform poorly when used elsewhere. in contrast, dsync adapts to its environment by intelligently determining which of its available resources is the best to use at any given time. the resources dsync can draw from include the sender, the local disk, and network peers. while combining these resources may appear easy, in practice it is difficult because these resources may have widely different performance or contend with each other. in particular, the paper presents a novel mechanism that enables dsync to aggressively search the receiver's local disk for useful data without interfering with concurrent network transfers. our evaluation on several workloads in various network environments shows that dsync outperforms existing systems by a factor of 1.4 to 5 in one-to-one and one-to-many transfers.
a linux implementation validation of track-aligned extents and track-aligned raids. through clean-slate implementation of two storage optimizations--track-aligned extents and track-aligned raids--this paper shows the values of independent validations. the experience revealed many unanticipated disk and storage data path behaviors as potential roadblocks for wide deployment of these optimizations, and also identified implementation issues to retrofit these concepts to legacy data paths.
load shedding in network monitoring applications. monitoring and mining real-time network data streams is crucial for managing and operating data networks. the information that network operators desire to extract from the network traffic is of different size, granularity and accuracy depending on the measurement task (e.g., relevant data for capacity planning and intrusion detection are very different). to satisfy these different demands, a new class of monitoring systems is emerging to handle multiple arbitrary and continuous traffic queries. such systems must cope with the effects of overload situations due to the large volumes, high data rates and bursty nature of the network traffic. in this paper, we present the design and evaluation of a system that can shed excess load in the presence of extreme traffic conditions, while maintaining the accuracy of the traffic queries within acceptable levels. the main novelty of our approach is that it is able to operate without explicit knowledge of the traffic queries. instead, it extracts a set of features from the traffic streams to build an on-line predictionmodel of the query resource requirements. this way the monitoring system preserves a high degree of flexibility, increasing the range of applications and network scenarios where it can be used. we implemented our scheme in an existing network monitoring system and deployed it in a research isp network. our results show that the system predicts the resources required to run each traffic query with errors below 5%, and that it can efficiently handle extreme load situations, preventing uncontrolled packet losses, with minimum impact on the accuracy of the queries' results.