Module Expr Str Import alias Import alias ImportFrom alias ImportFrom alias ImportFrom alias Assign NameStore Str Assign NameStore Str Assign NameStore Str Expr Call AttributeLoad NameLoad attr Str NameLoad ListLoad NameLoad NameLoad NameLoad Str Expr Call AttributeLoad NameLoad attr Str ListLoad Str Expr Call AttributeLoad NameLoad attr Str Num Str Assign NameStore AttributeLoad NameLoad attr Assign NameStore Str Assign NameStore Str Assign NameStore Str Assign NameStore Str Assign NameStore Num Assign NameStore ListLoad Num Num Assign NameStore ListLoad Num Num Assign NameStore Str Assign NameStore Str FunctionDef arguments args NameParam defaults body Return Call AttributeLoad NameLoad attr NameLoad NameLoad NameLoad decorator_list FunctionDef arguments args NameParam defaults body Expr Str Assign NameStore AttributeLoad NameLoad attr Assign NameStore SubscriptLoad NameLoad Index Num Expr Call AttributeLoad NameLoad attr Str NameLoad Expr Call AttributeLoad NameLoad attr Str decorator_list FunctionDef arguments args NameParam defaults body Expr Str For NameStore NameLoad body Expr Call AttributeLoad AttributeLoad NameLoad attr attr Dict Str AttributeLoad NameLoad attr decorator_list FunctionDef arguments args NameParam defaults body Expr Str Assign NameStore AttributeLoad NameLoad attr Assign NameStore BinOpMod Str TupleLoad Call AttributeLoad NameLoad attr NameLoad NameLoad If AttributeLoad NameLoad attr body AugAssignAdd NameStore BinOpMod Str AttributeLoad NameLoad attr AugAssignAdd NameStore BinOpMod Str BinOpDiv NameLoad Num Expr Call AttributeLoad NameLoad attr Str NameLoad Assign TupleStore NameStore NameStore Call AttributeLoad NameLoad attr BinOpMod Str TupleLoad AttributeLoad NameLoad attr NameLoad keyword NameLoad Assign NameStore Call AttributeLoad NameLoad attr Call AttributeLoad NameLoad attr NameLoad Call AttributeLoad NameLoad attr NameLoad Expr Call NameLoad NameLoad Return NameLoad decorator_list FunctionDef arguments args NameParam defaults body Expr Str Assign NameStore Call NameLoad BinOpDiv AttributeLoad NameLoad attr Num Num Assign NameStore BoolOpOr AttributeLoad NameLoad attr NameLoad Assign NameStore ListLoad For NameStore NameLoad body Assign NameStore BinOpMod Str TupleLoad Call AttributeLoad NameLoad attr NameLoad NameLoad If AttributeLoad NameLoad attr body AugAssignAdd NameStore BinOpMod Str AttributeLoad NameLoad attr AugAssignAdd NameStore Str Expr Call AttributeLoad NameLoad attr Str NameLoad NameLoad Assign TupleStore NameStore NameStore Call AttributeLoad NameLoad attr BinOpMod Str TupleLoad AttributeLoad NameLoad attr NameLoad keyword NameLoad Expr Call AttributeLoad NameLoad attr Call AttributeLoad NameLoad attr Call AttributeLoad NameLoad attr NameLoad Call AttributeLoad NameLoad attr NameLoad Expr Call NameLoad NameLoad Return NameLoad decorator_list FunctionDef arguments args NameParam defaults body Expr Str Assign NameStore Call NameLoad BinOpDiv AttributeLoad NameLoad attr Num Num Assign NameStore BoolOpOr AttributeLoad NameLoad attr NameLoad Assign NameStore ListLoad For NameStore NameLoad body Assign NameStore BinOpMod Str TupleLoad Call AttributeLoad NameLoad attr NameLoad NameLoad If AttributeLoad NameLoad attr body AugAssignAdd NameStore BinOpMod Str AttributeLoad NameLoad attr AugAssignAdd NameStore Str Expr Call AttributeLoad NameLoad attr Str NameLoad Assign TupleStore NameStore NameStore Call AttributeLoad NameLoad attr BinOpMod Str TupleLoad AttributeLoad NameLoad attr NameLoad keyword NameLoad Expr Call AttributeLoad NameLoad attr Call AttributeLoad NameLoad attr Call AttributeLoad NameLoad attr NameLoad Call AttributeLoad NameLoad attr NameLoad Expr Call NameLoad NameLoad Return NameLoad decorator_list Assign NameStore Dict NameLoad NameLoad NameLoad Dict NameLoad NameLoad Str NameLoad Dict NameLoad NameLoad Str NameLoad Dict NameLoad NameLoad Str NameLoad FunctionDef arguments args NameParam defaults body Expr Str Expr Call AttributeLoad NameLoad attr Str AttributeLoad NameLoad attr Assign NameStore AttributeLoad NameLoad attr Assign NameStore SubscriptLoad NameLoad Index Num Return Call SubscriptLoad SubscriptLoad NameLoad Index AttributeLoad NameLoad attr Index NameLoad NameLoad decorator_list FunctionDef arguments args NameParam defaults body Expr Str Assign NameStore AttributeLoad NameLoad attr Assign NameStore SubscriptLoad NameLoad Index Num Expr Call AttributeLoad NameLoad attr Str NameLoad Expr Call AttributeLoad NameLoad attr BinOpAdd Call AttributeLoad NameLoad attr Str decorator_list
N/A N/A Runs fio benchmarks to simulate logging, database and streaming.

Man: http://manpages.ubuntu.com/manpages/natty/man1/fio.1.html
Quick howto: http://www.bluestop.org/fio/HOWTO.txt

Simulated logging benchmark does the following things (PD only):
0) Do NOT use direct IO for any tests below, simply go through the FS.
1) Sequentially write x GB with queue depth equal to 8, where x is decided by
   the test VM's total memory. (A larger VM will write more bytes)
2) Random read of 10% of the bytes written.
3) Sequential read of all of the bytes written.

Simulated database benchmark does the following things (PD, PD-SSD, local SSD):
1) 4K Random R on a file using queue depths 1, 16 and 64 (each queue depth
   is a different benchmark).
2) 4K Random W on a file using queue depths 1, 16 and 64 (each queue depth
   is a different benchmark).
3) 4K Random 90% R/ 10% W on a file using queue depths 1, 16 and 64 (each
   queue depth is a different benchmark).
4) The size of the test file is decided by the test VM's total memory and capped
   at 1GB to ensure this test finishes within reasonable time.

Simulated streaming benchmark (PD only):
1) 1M Seq R at queue depth 1 and 16 (streaming).
2) 1M Seq W at queue depth 1 and 16 (streaming).

For AWS, where use PD, we should use EBS-GP and EBS Magnetic, for PD-SSD use
EBS-GP and PIOPS.
 N/A json N/A logging perfkitbenchmarker configs perfkitbenchmarker flags perfkitbenchmarker.linux_packages fio N/A LOGGING logging N/A DATABASE database N/A STREAMING streaming N/A N/A N/A flags DEFINE_enum workload_mode LOGGING N/A LOGGING DATABASE STREAMING Simulate a logging, database or streaming scenario. N/A N/A N/A flags DEFINE_list iodepth_list N/A A list of iodepth parameter used by fio command in simulated database and streaming scenarios only. N/A N/A N/A flags DEFINE_integer maxjobs 0 The maximum allowed number of jobs to support. N/A FLAGS N/A flags FLAGS N/A BENCHMARK_NAME block_storage_workload N/A BENCHMARK_CONFIG 
block_storage_workload:
  description: >
      Runs FIO in sequential, random, read and
      write modes to simulate various scenarios.
  vm_groups:
    default:
      vm_spec: *default_single_core
      disk_spec: *default_500_gb
 N/A DESCRIPTION description N/A METHOD method N/A DEFAULT_IODEPTH 8 N/A DEFAULT_DATABASE_SIMULATION_IODEPTH_LIST N/A 16 64 N/A DEFAULT_STREAMING_SIMULATION_IODEPTH_LIST N/A 1 16 N/A LATENCY_REGEX [=\s]+([\d\.]+)[\s,]+ N/A BANDWIDTH_REGEX (\d+)(\w+/*\w*) GetConfig N/A N/A user_config N/A N/A N/A N/A N/A configs LoadConfig BENCHMARK_CONFIG user_config BENCHMARK_NAME N/A Prepare N/A N/A benchmark_spec N/A N/A N/A Prepare the virtual machine to run FIO.

     This includes installing fio, bc. and libaio1 and insuring that the
     attached disk is large enough to support the fio benchmark.

  Args:
    benchmark_spec: The benchmark specification. Contains all data that is
        required to run the benchmark.
   N/A vms N/A benchmark_spec vms N/A vm N/A vms N/A 0 N/A N/A N/A logging info FIO prepare on %s vm N/A N/A N/A vm Install fio N/A UpdateWorkloadMetadata N/A N/A results N/A N/A N/A Update metadata fields in results with workload_mode flag value.

  Args:
    results: A list of sample.Sample objects.
   N/A result results N/A N/A N/A N/A N/A result metadata update N/A workload_mode N/A FLAGS workload_mode N/A RunSimulatedLogging N/A N/A vm N/A N/A N/A Spawn fio to simulate logging and gather the results.
  Args:
    vm: The vm that synthetic_storage_workloads_benchmark will be run upon.
  Returns:
    A list of sample.Sample objects
   N/A test_size N/A vm total_memory_kb N/A cmd N/A --filesize=10g --directory=%s --ioengine=libaio --filename=fio_test_file --invalidate=1 --randrepeat=0 --direct=0 --size=%dk --iodepth=%d  N/A N/A N/A vm GetScratchDir test_size DEFAULT_IODEPTH N/A N/A FLAGS maxjobs N/A N/A cmd N/A --max-jobs=%s  N/A FLAGS maxjobs N/A cmd N/A --name=sequential_write --overwrite=0 --rw=write --end_fsync=1 --name=random_read --size=%dk --stonewall --rw=randread --name=sequential_read --stonewall --rw=read  N/A test_size 10 N/A N/A N/A logging info FIO Results for simulated %s LOGGING N/A N/A res _ N/A N/A vm RemoteCommand N/A %s %s N/A N/A fio FIO_CMD_PREFIX cmd should_log True N/A results N/A N/A fio ParseResults N/A N/A fio FioParametersToJob cmd N/A N/A json loads res N/A N/A UpdateWorkloadMetadata results N/A results N/A RunSimulatedDatabase N/A N/A vm N/A N/A N/A Spawn fio to simulate database and gather the results.

  Args:
    vm: The vm that synthetic_storage_workloads_benchmark will be run upon.
  Returns:
    A list of sample.Sample objects
   N/A test_size N/A min N/A N/A vm total_memory_kb 10 1000000 N/A iodepth_list N/A N/A FLAGS iodepth_list DEFAULT_DATABASE_SIMULATION_IODEPTH_LIST N/A results N/A N/A depth iodepth_list N/A N/A cmd N/A --filesize=10g --directory=%s --ioengine=libaio --filename=fio_test_file --overwrite=1 --invalidate=0 --direct=1 --randrepeat=0 --iodepth=%s --size=%dk --blocksize=4k  N/A N/A N/A vm GetScratchDir depth test_size N/A N/A FLAGS maxjobs N/A N/A cmd N/A --max-jobs=%s  N/A FLAGS maxjobs N/A cmd --name=random_write --rw=randwrite --end_fsync=1 --name=random_read --stonewall --rw=randread --name=mixed_randrw --stonewall --rw=randrw --rwmixread=90 --rwmixwrite=10 --end_fsync=1  N/A N/A N/A logging info FIO Results for simulated %s, iodepth %s DATABASE depth N/A N/A res _ N/A N/A vm RemoteCommand N/A %s %s N/A N/A fio FIO_CMD_PREFIX cmd should_log True N/A N/A N/A results extend N/A N/A fio ParseResults N/A N/A fio FioParametersToJob cmd N/A N/A json loads res N/A N/A UpdateWorkloadMetadata results N/A results N/A RunSimulatedStreaming N/A N/A vm N/A N/A N/A Spawn fio to simulate streaming and gather the results.

  Args:
    vm: The vm that synthetic_storage_workloads_benchmark will be run upon.
  Returns:
    A list of sample.Sample objects
   N/A test_size N/A min N/A N/A vm total_memory_kb 10 1000000 N/A iodepth_list N/A N/A FLAGS iodepth_list DEFAULT_STREAMING_SIMULATION_IODEPTH_LIST N/A results N/A N/A depth iodepth_list N/A N/A cmd N/A --filesize=10g --directory=%s --ioengine=libaio --overwrite=0 --invalidate=1 --direct=1 --randrepeat=0 --iodepth=%s --blocksize=1m --size=%dk --filename=fio_test_file  N/A N/A N/A vm GetScratchDir depth test_size N/A N/A FLAGS maxjobs N/A N/A cmd N/A --max-jobs=%s  N/A FLAGS maxjobs N/A cmd --name=sequential_write --rw=write --end_fsync=1 --name=sequential_read --stonewall --rw=read  N/A N/A N/A logging info FIO Results for simulated %s STREAMING N/A N/A res _ N/A N/A vm RemoteCommand N/A %s %s N/A N/A fio FIO_CMD_PREFIX cmd should_log True N/A N/A N/A results extend N/A N/A fio ParseResults N/A N/A fio FioParametersToJob cmd N/A N/A json loads res N/A N/A UpdateWorkloadMetadata results N/A results N/A N/A RUN_SCENARIO_FUNCTION_DICT N/A LOGGING DATABASE STREAMING N/A DESCRIPTION METHOD simulated_logging RunSimulatedLogging N/A DESCRIPTION METHOD simulated_database RunSimulatedDatabase N/A DESCRIPTION METHOD simulated_streaming RunSimulatedStreaming Run N/A N/A benchmark_spec N/A N/A N/A Spawn fio and gather the results.

  Args:
    benchmark_spec: The benchmark specification. Contains all data that is
        required to run the benchmark.

  Returns:
    A list of sample.Sample objects.
   N/A N/A N/A logging info Simulating %s scenario. N/A FLAGS workload_mode N/A vms N/A benchmark_spec vms N/A vm N/A vms N/A 0 N/A N/A N/A N/A RUN_SCENARIO_FUNCTION_DICT N/A N/A FLAGS workload_mode N/A METHOD vm N/A Cleanup N/A N/A benchmark_spec N/A N/A N/A Uninstall packages required for fio and remove benchmark files.

  Args:
    benchmark_spec: The benchmark specification. Contains all data that is
        required to run the benchmark.
   N/A vms N/A benchmark_spec vms N/A vm N/A vms N/A 0 N/A N/A N/A logging info FIO Cleanup up on %s vm N/A N/A N/A vm RemoveFile N/A N/A N/A vm GetScratchDir /fio_test_file N/A