performance analysis case study (abstract): application of experimental design & statistical data analysis techniques _extended abstract_  a common requirement of computer vendors competitive performanceanalysis departments is to measure and report on the pcrfonnancccharacteristics of another vendors system. in many cases the amountof prior knowledge concerning the competitors systcm is limited tosales brochures and non-technical publications. availability of thesyslcm for benchmarking is minimal; there is little choiceconcerning memory and 1/0 configurations; and time to complete theproject is short. a project of this nature is not, however, uniqueto computer vendors. many users of computer systems that want tobetter understand a systems performance characteristics beforedeciding on a purchase ,are also faced with similmrestrictions.  due to time and resource constraints, a common practice is tousc an existing benchmark. the benchmarks chosen arc usuallypopular ones which claim to be representative of one or morespecific production workloads. current popular benchmarks includethe spec suite, dhrystone, linpack 100 x 100 single precision, etc.while easy to use, these generally do not provide a lot of insightas to how well a system will perform on a different benchmark orworkload.  an alternative to using a representative benchmark is to use afunctional benchmark. as an example, a functional 1/0 benchmakwould be one that can execute a number of different types orvariants of 1/0, and allow the systems performance characteristicsto be measured when executing each type. in general, functionalbenchmarks can be used to collect an overwhelming amount of datathat can be used to generate estimates as to how well a system willperform on other benchmarks and workloads.  use of a functional benchmark also provides the opportunity toexploit the benefits of experimental design and statistical dataanalysis techniques. incorporating their usage in to the processcan be very beneficial. in general terms. the benefits are:    1. a reduction in the number of tests run and the quantity of datacollected.   2. a method for choosing which tests to run to help ensure thatthe results are not biased.   3. an increase in what can be determined about the systemsperformance characteristics.   4. an improvement in the quality of the results.  some examples of benefits are: by designing an experiment inwhich the primary factors affecting 1/0 are controlled (e.g. filesize, read vs. write, sequential vs. random, etc.), the main effectas well as the interactive effects of the factors can bedetermined; statistical techniques are available to provideevidence that the benchmark is in fact testing what was intended tobe testec regression analysis can be used with the reduced set ofdata collected to provide predictive capabilities not otherwiseeasily obtained.  the use of experimental design and statistical data analysistechniques are in common practice in many other fields and are notunreasonably difficult to apply. unfortunately, their applicationis not in common usage in computer performance analysis.  the paper contains a case study of the use of a functional 1/0benchmark executed and the results analyzed under such amethodology. the experiment was executed on a cdc 4680, on an ibmrs6000/540, and on a sun 4/490. for all three architectures, themain and interactive effects of file location (cached or on disk),file size, file action (read or write), record size, record access(sequential or random), and number of concurrently active 1/0streams were tested on elapsed time, throughput, and disk servicetime. an analysis of variance determined that file size was not asignificant factor for throughput and disk service time. a stepwisetype regression analysis, developed models with a parsimonious setof factors having little statistical bias. the models identifiedthe presence of two and three factor interactions. the resulting r2and residual plots, and a visual analysis were used to determinethe efficacy of the test measuring these factors across differentarchitectures.  in summary, when there are many unknowns, and resourcesincluding time are short, using a functional benchmark as part of awell designed experiment in conjunction with statistical dataanalysis, is a methodology by which the original goal ofcharacterizing a systems performance can be meet with a high levelof quality.  robb newman