waashk/dblp huggingface dataset:
file system logging versus clustering: a performance comparison. the log-structured file system (lfs), introduced in 1991 [8], has received much attention for its potential order-of-magnitude improvement in file system performance. early research results [9] showed that small file performance could scale with processor speed and that cleaning costs could be kept low, allowing lfs to write at an effective bandwidth of 62 to 83% of the maximum. later work showed that the presence of synchronous disk operations could degrade performance by as much as 62% and that cleaning overhead could become prohibitive in transaction processing workloads, reducing performance by as much as 40% [10]. the same work showed that the addition of clustered reads and writes in the berkeley fast file system [6] (ffs) made it competitive with lfs in large-file handling and software development environments as approximated by the andrew benchmark [4]. these seemingly inconsistent results have caused confusion in the file system research community. this paper presents a detailed performance comparison of the 4.4bsd log-structured file system and the 4.4bsd fast file system. ignoring cleaner overhead, our results show that the order-of-magnitude improvement in performance claimed for lfs applies only to meta-data intensive activities, specifically the creation of files one-kilobyte or less and deletion of files 64 kilobytes or less. for small files, both systems provide comparable read performance, but lfs offers superior performance on writes. for large files (one megabyte and larger), the performance of the two file systems is comparable. when ffs is tuned for writing, its large-file write performance is approximately 15% better than lfs, but its read performance is 25% worse. when ffs is optimized for reading, its large-file read and write performance is comparable to lfs. both lfs and ffs can suffer performance degradation, due to cleaning and disk fragmentation respectively. we find that active ffs file systems function at approximately 85-95% of their maximum performance after two to three years. we examine lfs cleaner performance in a transaction processing environment and find that cleaner overhead reduces lfs performance by more than 33% when the disk is 50% full.
ditools: application-level support for dynamic extension and flexible composition. today, operating systems set-up process images from executable files using fixed rules. programs are restricted to run in essentially the same environment at every execution. however, we believe that this behavior is not always convenient, and that many times it is interesting to make variations to the execution environment, either to introduce new functionality or to specialize critical services, even when their source code is not available. this problem can be mitigated through application-level extensibility and flexible composition of binary modules. in this paper, we describe ditools an application-level tool that supports dynamic interposition on dynamically-linked procedure-call boundaries. this tool enables both global and per-module dynamic interposition. we also present a detailed use of ditools and various short examples of extensions.
sendmail evolution: 8.10 and beyond. sendmailtm has been the de facto mail transfer agent implementation since the dawn of the internet. today, sendmail development is still driven by a continually changing set of network requirements and user demands. lately, two new driving forces have also contributed to sendmail development. first, as more open source mail transfer agents, such as exim and postfix, become available, a new friendly competition has developed in which the authors of the various mtas share their ideas via open source and help to advance open standards as opposed to advancing their own particular implementation. second, a new "hybrid" company, sendmail, inc., has been created to offer commercial versions of the open source software while continuing to fuel open source development. this paper will briefly discuss the evolution of sendmail; the influences which drive sendmail development; and how the creation of sendmail, inc. has contributed to the open source version. the paper will also describe the new features appearing in the next "functionality release" of open source sendmail. in particular, changes in queueing and new protocol support are discussed. finally, the authors will speculate on future directions for sendmail.
how xlib is implemented (and what we're doing about it). the x window system is the de facto standard graphical environment for linux and unix hosts, and is usable on nearly any class of computer one could find today. its success is partially due to its flexible, extensible design. unfortunately, as research proceeds on cutting-edge window system functionality, the brittleness of the underlying software is a critical impediment to progress. xlib, the client-side implementation of the network protocol that underlies x, is one source of these issues. many developers working on new features in the x protocol are discovering that xlib requires changes to support these features, but xlib makes those changes difficult. for more than 15 years, new features have been added to xlib by accretion, rather than with careful design. we discuss the implementation of xlib and analyze some specific difficulties in it that cause problems in understanding and maintaining this code base. we also present our current work on migrating the x window system to a more maintainable, carefully designed architecture.
sawmill: a high-bandwidth logging file system. this paper describes the implementation of sawmill, a network file system using the raid-ii storage system. sawmill takes advantage of the direct data path in raid-ii between the disks and the network, which bypasses the file server cpu. the key ideas in the implementation of sawmill are combining logging (lfs) with raid to obtain fast small writes, using new log layout techniques to improve bandwidth, and pipelining through the controller memory to reduce latency. the file system can currently read data at 21 mb/s and write data at 15 mb/s, close to the raw disk array bandwidth, while running on a relatively slow sun-4. performance measurements show that lfs improved performance of a stream of small writes by over a order of magnitude compared to writing directly to the raid, and this improvement would be even larger with a faster cpu. sawmill demonstrates that by using a storage system with a direct data path, a file system can provide data at bandwidths much higher than the file server itself could handle. however, processor speed is still an important factor, especially when handling many small requests in parallel.
why does file system prefetching work? most file systems attempt to predict which disk blocks will be needed in the near future and prefetch them into memory; this technique can improve application throughput as much as 50%. but why? the reasons include that the disk cache comes into play, the device driver amortizes the fixed cost of an i/o operation over a larger amount of data, total disk seek time can be decreased, and that programs can overlap computation and i/o. however, intuition does not tell us the relative benefit of each of these causes, or techniques for increasing the effectiveness of prefetching. to answer these questions, we constructed an analytic performance model for file system reads. the model is based on a 4.4bsd-derived file system, and parameterized by the access patterns of the files, layout of files on disk, and the design characteristics of the file system and of the underlying disk. we then validated the model against several simple workloads; the predictions of our model were typically within 4% of measured values, and differed at most by 9% from measured values. using the model and experiments, we explain why and when prefetching works, and make proposals for how to tune file system and disk parameters to improve overall system throughput.
deceit: a flexible distributed file system. deceit, a distributed file system being developed at cornell, focuses on flexible file semantics in relation to efficiency, scalability, and reliability. deceit servers are interchangeable and collectively provide the illusion of a single, large server machine to any clients of the deceit service. non-volatile replicas of each file are stored on a subset of the file servers. the user is able to set parameters on a file to achieve different levels of availability, performance, and one-copy serializability. deceit also supports a file version control mechanism. in contrast with many recent dfs efforts, deceit can behave like a plain sun network file system server and can be used by any nfs client without modifying any client software. the current deceit prototype uses the isis distributed programming environment for all communication and process group management, an approach that reduces system complexity and increases system robustness.
ubc: an efficient unified i/o and memory caching subsystem for netbsd. this paper introduces ubc ("unified buffer cache"), a design for unifying the filesystem and virtual memory caches of file data, thereby providing increased system performance. in this paper we discuss both the traditional bsd caching interfaces and new ubc interfaces, concentrating on the design decisions that were made as the design progressed. we also discuss the designs used by other operating systems to solve the same problems that ubc solves, with emphasis on the practical implications of the differences between these designs. this project is still in progress, and once completed will be part of a future release of netbsd.
network subsystems reloaded: a high-performance, defensible network subsystem. traditionally, operating systems have used monolithic network stack implementations: implementations where the whole network stack executes in the kernel or (in microkernels) in a single, trusted, user level server. code maintenance issues, ease of debugging, need for simultaneous existence of multiple protocols, and security benefit have argued for removing the networking implementation from kernel and dividing it into multiple user level protection domains. previous attempts to do so have failed to deliver adequate performance. given the advances made in both hardware (cpu, memory, nic) and micro-kernel design over the last decade, it is now appropriate to re-evaluate how these re-factored implementations perform, and to examine the reasons for earlier failures in greater detail. building on the primitives of the eros microkernel, we have implemented two network subsystems: one a conventional, user mode, monolithic design and the other a domain-factored user level networking stack that restructures the network subsystem into several protection domains. we show that the restructuring maintains performance very close to that of the monolithic design, and that both designs compare favorably to a conventional in-kernel implementation. we discuss the issues faced in engineering the domain-factored implementation to achieve high performance, and present the quantitative evaluation of the resulting network subsystems.
a comparison of os extension technologies. the current trend in operating systems research is to allow applications to dynamically extend the kernel to improve application performance or extend functionality, but the most effective approach to extensibility remains unclear. some systems use safe languages to permit code to be downloaded directly into the kernel; other systems provide in-kernel interpreters to execute extension code; still others use software techniques to ensure the safety of kernel extensions. the key characteristics that distinguish these systems are the philosophy behind extensibility and the technology used to implement extensibility. this paper presents a taxonomy of the types of extensions that might be desirable in an extensible operating system, evaluates the performance cost of various extension technologies currently being employed, and compares the cost of adding a kernel extension to the benefit of having the extension in the kernel. our results show that compiled technologies (e.g. modula-3 and software fault isolation) are good candidates for implementing general-purpose kernel extensions, but that the overhead of interpreted languages is sufficiently high that they are inappropriate for this use.
performing replacement in modem pools. we examine a policy for managing modem pools that disconnects users only if not enough modems are available for other users to connect. managing the modem pool then becomes a replacement problem, similar to buffer cache management (e.g., in virtual memory systems). when a new connection request is received, the system needs to find a user to "replace". in this paper we examine such demand-disconnect schemes using extensive activity data from actual isps. we discuss various replacement policies and propose cirg: a novel replacement algorithm that is well suited for modem pools. in general, the choice of algorithm is significant. a naive algorithm (e.g., one that randomly replaces any user who has been inactive for a while) incurs many tens of percent more "faults" (i.e., disconnections of users who are likely to want to be active again soon) than the lru algorithm, which, in turn, incurs 10% more faults than cirg. for good replacement algorithms, the impact can be significant in terms of resource requirements. we show that the same standards of service as a system that does not disconnect idle users can be achieved with up to 13% fewer modems.
a comparison of ffs disk allocation policies. the 4.4bsd file system includes a new algorithm for allocating disk blocks to files. the goal of this algorithm is to improve file clustering, increasing the amount of sequential i/o when reading or writing files, thereby improving file system performance. in this paper we study the effectiveness of this algorithm at reducing file system fragmentation. we have created a program that artificially ages a file system by replaying a workload similar to that experienced by a real file system. we used this program to evaluate the effectiveness of the new disk allocation algorithm by replaying ten months of activity on two file systems that differed only in the disk allocation algorithms that they used. at the end of the ten month simulation, the file system using the new allocation algorithm had approximately half the fragmentation of a similarly aged file system that used the traditional disk allocation algorithm. measuring the performance difference between the two file systems by reading and writing the same set of files on the two systems showed that this decrease in fragmentation improved file write throughput by 20% and read throughput by 32%. in certain test cases, the new allocation algorithm provided a performance improvement of greater than 50%.
a usage profile and evaluation of a wide-area distributed file system. the evolution of the andrew file system (afs) into a wide-area distributed file system has encouraged collaboration and information dissemination on a much broader scale than ever before. in this paper, we examine afs as a provider of wide-area file services to over 80 organizations around the world. we discuss usage characteristics of afs derived from empirical measurements of the system, and from user responses to a questionnaire. our observations indicate that afs provides robust and efficient data access in its current configuration, thus confirming its viability as a design point for wide-area distributed file systems.
outwit: unix tool-based programming meets the windows world. the ubiquity of windows-based desktop environments has not been matched by a corresponding emergence of tools supporting the unix tool composition paradigm. outwit is a suite of tools based on the unix tool design principles allowing the processing of windows application data with sophisticated data manipulation pipelines. the outwit tools offer access to the windows clipboard, the registry, relational databases, document properties, and shell links. we demonstrate a number of applications of the outwit tools used in conjuction with existing unix commands, and discuss future directions of our work.
wux: unix tools under windows. wux is a port of unix tools to the microsoft windows environment. it is based on a library providing a unix-compatible set of system calls on top of windows. unix-derived tools run in parallel, communicating using the unix pipe abstraction. all processes are run within an application template that gives them basic windows compatibility such as input and output windows and an icon. the performance of the system is comparable to that of unix ports to the pc architecture.
flashback: a lightweight extension for rollback and deterministic replay for software debugging. software robustness has significant impact on system availability. unfortunately, finding software bugs is a very challenging task because many bugs are hard to reproduce. while debugging a program, it would be very useful to rollback a crashed program to a previous execution point and deterministically re-execute the "buggy" code region. however, most previous work on rollback and replay support was designed to survive hardware or operating system failures, and is therefore too heavyweight for the fine-grained rollback and replay needed for software debugging. this paper presents flashback, a lightweight os extension that provides fine-grained rollback and replay to help debug software. flashback uses shadow processes to efficiently roll back in-memory state of a process, and logs a process' interactions with the system to support deterministic replay. both shadow processes and logging of system calls are implemented in a lightweight fashion specifically designed for the purpose of software debugging. we have implemented a prototype of flashback in the linux operating system. our experimental results with micro-benchmarks and real applications show that flashback adds little overhead and can quickly roll back a debugged program to a previous execution point and deterministically replay from that point.
sbox: put cgi scripts in a box. sbox is a cgi wrapper script that allows web sites to safely grant cgi authoring privileges to untrusted or naive authors. the script increases security in several ways. it changes the process privileges of cgi scripts to match their owners, preventing one script from interfering with another's data files or operations. it establishes configurable ceilings on script resource usage, avoiding intentional or unintentional denial of service attacks. most importantly, sbox can also be used to run untrusted cgi scripts within a chroot()-ed directory, thereby preventing cgi scripts from accessing sensitive portions of the file system. sbox can be used and redistributed freely. the complete package is available for download at http://stein.cshl.org/www/software/sbox/
splicing unix into a genome mapping laboratory. the whitehead institute/mit center for genome research is responsible for a number of large genome mapping efforts, the scale of which create problems of data and workflow management that dictate reliance on computer support. two years ago, when we started to design the informatics support for the laboratory, we realized that the fluid and ever-changing nature of the experimental protocols precluded any effort to create a single monolithic piece of software. instead we designed a system that relied on multiple distributed data analysis and processing tools knit together by a centralized database. the obvious choice of operating systems was unix. in order to make this choice palatable to the laboratory biologists--who rightly consider it their job to do experiments rather than to interact with computers, and who have come to expect all software to be as intuitive and responsive as the apple macintoshes on their desks--we designed a system that runs automatically and essentially invisibly. whenever it is necessary for the informatics system to interact with a member of the laboratory we have carefully chosen a user interface paradigm that best balances the user's expectations against the system's capabilities. when possible we have chosen to adapt familiar software to our user interface needs rather than to write user interfaces from scratch. we've managed to hide the power of unix behind the innocuous personal computer-based front ends our users know and love, using techniques that should be applicable in other environments as well.
portals in 4.4bsd. portals were added to 4.4bsd as an experimental feature and are in the publicly available 4.4bsd-lite distribution. portals provide access to alternate file types or devices using names in the normal filesystem that a process just opens. for example, an open of /p/tcp/foo.com/smtp returns a tcp socket descriptor to the calling process that is connected to the smtp server on the specified host. by providing access through the normal filesystem, the calling process need not be aware of the special functions necessary to create a tcp socket and establish a tcp connection. this makes tcp connections, for example, available to programs such as awk, tcl, and shell scripts. this paper describes the implementation of portals in 4.4bsd as another type of filesystem and provides some examples.
mach-us: unix on generic os object servers. this paper examines the mach-us operating system, its unique architecture, and the lessons demonstrated through its implementation. mach-us is an object-oriented multi-server os which runs on the mach3.0 kernel. mach-us has a set of separate servers supplying orthogonal os services and a library which is loaded into each user process. this library uses the services to generate the semantics of the mach2.5/4.3bsd application programmers interface (api). this architecture makes mach-us a flexible research platform and a powerful tool for developing and examining various os service options. we will briefly describe mach-us, the motivations for its design choices, and its demonstrated strengths and weaknesses. we will then discuss the insights that we've acquired in the areas of multi-server architecture, os remote method invocation, object oriented technology for os implementation, api independent os services, unix api re-implementation, and smart user-space api emulation libraries.
dmfs-a data migration file system for netbsd. i have recently developed dmfs, a data migration file system, for netbsd[1]. this file system provides kernel support for the data migration system being developed by my research group at nasa/ames. the file system utilizes an underlying file store to provide the file backing, and coordinates user and system access to the files. it stores its internal metadata in a flat file, which resides on a separate file system. this paper will first describe our data migration system to provide a context for dmfs, then it will describe dmfs. i also will describe the changes to netbsd needed to make dmfs work. then i will give an overview of the file archival and restoration procedures, and describe how some typical user actions are modified by dmfs. lastly, i will present simple performance measurements which indicate that there is little performance loss due to the use of the dmfs layer.