Module Expr Str Import alias Import alias Import alias Import alias Import alias Import alias Import alias Import alias Import alias Import alias Import alias Import alias ImportFrom alias ImportFrom alias ImportFrom alias ImportFrom alias Assign NameStore AttributeLoad NameLoad attr Assign NameStore Str Assign NameStore Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr Str Assign NameStore Call AttributeLoad NameLoad attr NameLoad Str Str Assign NameStore TupleLoad Num Num Num Num Num Num Assign NameStore Dict Str Str Str Str Str Str Str Str Str Str Str Str AttributeLoad NameLoad attr NameLoad AttributeLoad NameLoad attr AttributeLoad NameLoad attr AttributeLoad NameLoad attr AttributeLoad NameLoad attr NameLoad AttributeLoad NameLoad attr NameLoad NameLoad NameLoad NameLoad Expr Call AttributeLoad NameLoad attr Str NameLoad Str Expr Call AttributeLoad NameLoad attr Str NameLoad Str Expr Call AttributeLoad NameLoad attr Str NameLoad Str Expr Call AttributeLoad NameLoad attr Str Num Str keyword Num Expr Call AttributeLoad NameLoad attr Str ListLoad Str Expr Call AttributeLoad NameLoad attr Str ListLoad Str Expr Call AttributeLoad NameLoad attr Str ListLoad Str Expr Call AttributeLoad NameLoad attr Str ListLoad Str Str Expr Call AttributeLoad NameLoad attr Str NameLoad Str Expr Call AttributeLoad NameLoad attr Str Num Str Expr Call AttributeLoad NameLoad attr Str Num Str Expr Call AttributeLoad NameLoad attr Str Num Str Assign NameStore Num FunctionDef arguments args defaults body Expr Str Return ListComp Call NameLoad NameLoad comprehension NameStore AttributeLoad NameLoad attr decorator_list FunctionDef arguments args defaults body Expr Str If AttributeLoad NameLoad attr body Return AttributeLoad NameLoad attr Return ListComp Call AttributeLoad NameLoad attr Call AttributeLoad AttributeLoad NameLoad attr attr Str NameLoad comprehension NameStore TupleLoad Str Str decorator_list FunctionDef arguments args defaults body For NameStore Call NameLoad body If UnaryOpNot Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad body Raise Call NameLoad Call AttributeLoad Str attr NameLoad decorator_list FunctionDef arguments args NameParam defaults body Expr Str Expr Call AttributeLoad NameLoad attr Str Expr Call AttributeLoad NameLoad attr Str Expr Call AttributeLoad NameLoad attr Call AttributeLoad Str attr NameLoad NameLoad decorator_list FunctionDef arguments args NameParam defaults body Expr Str Expr Call NameLoad NameLoad decorator_list FunctionDef arguments args NameParam defaults body Expr Str Expr Call NameLoad NameLoad decorator_list FunctionDef arguments args NameParam NameParam defaults Str body Expr Str Assign NameStore Call AttributeLoad NameLoad attr NameLoad Assign NameStore Call AttributeLoad Call NameLoad NameLoad attr If UnaryOpNot Call AttributeLoad NameLoad attr Str body Raise Call NameLoad Call AttributeLoad Str attr NameLoad Assign NameStore Call AttributeLoad Call NameLoad NameLoad attr If UnaryOpNot Call AttributeLoad NameLoad attr Str body Raise Call NameLoad Call AttributeLoad Str attr NameLoad FunctionDef arguments args NameParam defaults body Return CompareIsNot Call AttributeLoad NameLoad attr Str NameLoad NameLoad decorator_list Assign NameStore Call AttributeLoad NameLoad attr NameLoad NameLoad Assign NameStore Call AttributeLoad NameLoad attr NameLoad Assign NameStore Call AttributeLoad NameLoad attr NameLoad Call AttributeLoad NameLoad attr Num Assign NameStore Call AttributeLoad NameLoad attr ListLoad TupleLoad Str NameLoad TupleLoad Str NameLoad TupleLoad Str Call AttributeLoad NameLoad attr For TupleStore NameStore NameStore NameLoad body Assign NameStore Call AttributeLoad SubscriptLoad NameLoad Slice Num Num attr Assign NameStore Dict Str NameLoad Str NameLoad ListLoad Dict For TupleStore NameStore NameStore NameStore NameLoad body Assign NameStore Call AttributeLoad NameLoad attr Assign NameStore Call AttributeLoad NameLoad attr If Call AttributeLoad NameLoad attr Str body Assign NameStore SubscriptLoad NameLoad Slice Num Assign NameStore IfExp CompareIn Str NameLoad Call NameLoad NameLoad Call NameLoad NameLoad If Call AttributeLoad NameLoad attr body If NameLoad body Expr Call AttributeLoad SubscriptLoad NameLoad Index NameLoad attr TupleLoad Call NameLoad NameLoad NameLoad orelse If CompareIn Str NameLoad body Assign NameStore Call AttributeLoad NameLoad attr Str Str AugAssignDiv NameStore Num Assign SubscriptStore SubscriptLoad NameLoad Index Str Index NameLoad NameLoad Assign SubscriptStore SubscriptLoad NameLoad Index Str Index NameLoad NameLoad Return NameLoad decorator_list FunctionDef arguments args NameParam defaults body Assign NameStore Num For NameStore NameLoad body AugAssignAdd NameStore NameLoad Expr Yield NameLoad decorator_list FunctionDef arguments args NameParam NameParam NameParam defaults body Expr Str If CompareNotEq Call NameLoad NameLoad Call NameLoad NameLoad body Raise Call NameLoad Call AttributeLoad Str attr Call NameLoad NameLoad Call NameLoad NameLoad If BoolOpOr CompareLt NameLoad Num CompareGt NameLoad Num body Raise Call NameLoad Call AttributeLoad Str attr NameLoad Assign NameStore Call NameLoad NameLoad Assign NameStore BinOpMult NameLoad Call NameLoad NameLoad Assign NameStore Call NameLoad Call NameLoad NameLoad Assign NameStore Call AttributeLoad NameLoad attr NameLoad NameLoad If CompareEq NameLoad Call NameLoad NameLoad body Return SubscriptLoad NameLoad Index Num orelse Return SubscriptLoad NameLoad Index NameLoad decorator_list FunctionDef arguments args NameParam NameParam defaults NameLoad body Expr Str Assign NameStore Call AttributeLoad NameLoad attr Assign NameStore Call NameLoad NameLoad For NameStore NameLoad body If BoolOpOr CompareLt NameLoad Num CompareGt NameLoad Num body Raise Call NameLoad Call AttributeLoad Str attr NameLoad If CompareLt SubscriptLoad Call AttributeLoad NameLoad attr NameLoad Index Num Num body Assign NameStore Call NameLoad NameLoad Assign NameStore Call AttributeLoad Str attr NameLoad Assign TupleStore NameStore NameStore Call NameLoad NameLoad Assign NameStore Call NameLoad NameLoad NameLoad BinOpMult NameLoad Num Assign SubscriptStore NameLoad Index NameLoad NameLoad Return NameLoad decorator_list FunctionDef arguments args NameParam NameParam defaults NameLoad body Expr Str FunctionDef arguments args NameParam defaults body Expr Str Assign NameStore SetComp NameLoad comprehension TupleStore NameStore NameStore Call AttributeLoad NameLoad attr CompareIs NameLoad NameLoad For NameStore Call AttributeLoad SubscriptLoad NameLoad Index Str attr body For NameStore NameLoad body Expr Call AttributeLoad SubscriptLoad NameLoad Index Str attr NameLoad NameLoad decorator_list FunctionDef arguments args NameParam NameParam defaults body Assign NameStore Call NameLoad NameLoad Assign NameStore Call NameLoad NameLoad Assign NameStore Call NameLoad BinOpBitOr Call NameLoad NameLoad Call NameLoad NameLoad Assign NameStore ListLoad For NameStore NameLoad body Expr Call AttributeLoad NameLoad attr TupleLoad NameLoad BinOpAdd Call AttributeLoad NameLoad attr NameLoad Num Call AttributeLoad NameLoad attr NameLoad Num Return NameLoad decorator_list Assign NameStore Call AttributeLoad NameLoad attr SubscriptLoad NameLoad Index Num Expr Call NameLoad NameLoad For NameStore SubscriptLoad NameLoad Slice Num body For TupleStore NameStore NameStore Call AttributeLoad SubscriptLoad NameLoad Index Str attr body If CompareNotIn NameLoad SubscriptLoad NameLoad Index Str body Expr Call AttributeLoad NameLoad attr Str NameLoad Assign SubscriptStore SubscriptLoad NameLoad Index Str Index NameLoad Call AttributeLoad NameLoad attr NameLoad Continue For TupleStore NameStore NameStore Call AttributeLoad SubscriptLoad NameLoad Index Str attr body If CompareNotIn NameLoad NameLoad body Expr Call AttributeLoad NameLoad attr Str NameLoad Continue orelse If CompareIs SubscriptLoad NameLoad Index NameLoad NameLoad body Expr Call AttributeLoad SubscriptLoad SubscriptLoad SubscriptLoad NameLoad Index Str Index NameLoad Index Str attr NameLoad NameLoad Continue orelse If CompareNotIn NameLoad SubscriptLoad SubscriptLoad SubscriptLoad NameLoad Index Str Index NameLoad Index Str body Expr Call AttributeLoad NameLoad attr Str NameLoad NameLoad Assign SubscriptStore SubscriptLoad SubscriptLoad SubscriptLoad NameLoad Index Str Index NameLoad Index Str Index NameLoad Call AttributeLoad NameLoad attr NameLoad Continue Assign NameStore SubscriptLoad NameLoad Index NameLoad Assign SubscriptStore SubscriptLoad SubscriptLoad SubscriptLoad NameLoad Index Str Index NameLoad Index Str Index NameLoad Call NameLoad SubscriptLoad SubscriptLoad SubscriptLoad SubscriptLoad NameLoad Index Str Index NameLoad Index Str Index NameLoad NameLoad If NameLoad body Assign SubscriptStore SubscriptLoad SubscriptLoad NameLoad Index Str Index NameLoad Index Str Call NameLoad SubscriptLoad SubscriptLoad SubscriptLoad NameLoad Index Str Index NameLoad Index Str SubscriptLoad NameLoad Index Str orelse Expr Call AttributeLoad SubscriptLoad SubscriptLoad NameLoad Index Str Index NameLoad attr Str NameLoad Assign SubscriptStore NameLoad Index Str Call AttributeLoad Str attr TupleLoad SubscriptLoad NameLoad Index Str SubscriptLoad NameLoad Index Str Assign SubscriptStore NameLoad Index Str Call AttributeLoad Str attr TupleLoad SubscriptLoad NameLoad Index Str SubscriptLoad NameLoad Index Str If BoolOpAnd CompareIn Str NameLoad CompareIn Str NameLoad body AugAssignAdd SubscriptStore NameLoad Index Str SubscriptLoad NameLoad Index Str Return NameLoad decorator_list FunctionDef arguments args NameParam defaults body Expr Str Assign NameStore Call AttributeLoad NameLoad attr NameLoad Assign NameStore Dict For NameStore NameLoad body If BoolOpAnd Call AttributeLoad NameLoad attr UnaryOpNot Call AttributeLoad Call AttributeLoad NameLoad attr attr Str UnaryOpNot Call AttributeLoad Call AttributeLoad NameLoad attr attr Str body Assign TupleStore NameStore NameStore Call AttributeLoad NameLoad attr Str NameLoad keyword Num Assign SubscriptStore NameLoad Index NameLoad Call AttributeLoad NameLoad attr Return NameLoad decorator_list FunctionDef arguments args NameParam NameParam defaults NameLoad kwarg body Expr Str Assign NameStore IfExp Call AttributeLoad SubscriptLoad NameLoad Index Str attr Str Str Str Assign NameStore Dict Str Str SubscriptLoad NameLoad Index Str NameLoad Expr Call AttributeLoad NameLoad attr NameLoad For TupleStore NameStore NameStore Call AttributeLoad SubscriptLoad NameLoad Index Str attr body Assign NameStore Call AttributeLoad NameLoad attr Assign SubscriptStore NameLoad Index Str NameLoad For TupleStore NameStore NameStore Call AttributeLoad SubscriptLoad NameLoad Index Str attr body If CompareIs NameLoad NameLoad body Continue Assign NameStore Str Assign NameStore Call AttributeLoad NameLoad attr Str NameLoad If NameLoad body Assign NameStore Call AttributeLoad NameLoad attr Num Assign NameStore Call AttributeLoad NameLoad attr Num Expr Yield Call AttributeLoad NameLoad attr Call AttributeLoad Str attr ListLoad NameLoad NameLoad NameLoad NameLoad NameLoad If SubscriptLoad NameLoad Index Str body Assign NameStore Call NameLoad SubscriptLoad NameLoad Index Str For TupleStore NameStore NameStore Call AttributeLoad NameLoad attr body Expr Yield Call AttributeLoad NameLoad attr Call AttributeLoad Str attr ListLoad NameLoad NameLoad Str NameLoad Str NameLoad If NameLoad body For TupleStore NameStore NameStore SubscriptLoad NameLoad Index Str body Expr Yield Call AttributeLoad NameLoad attr Call AttributeLoad Str attr NameLoad NameLoad NameLoad Str NameLoad decorator_list ClassDef bases NameLoad body Expr Str Assign NameStore TupleLoad Str Str Str Str FunctionDef arguments args NameParam NameParam NameParam defaults NameLoad kwarg body Assign AttributeStore NameLoad attr NameLoad Assign AttributeStore NameLoad attr BoolOpOr NameLoad ListLoad Assign AttributeStore NameLoad attr Call AttributeLoad NameLoad attr decorator_list FunctionDef arguments args NameParam NameParam NameParam defaults NameLoad kwarg body Assign NameStore ListLoad NameLoad NameLoad AttributeLoad NameLoad attr Assign NameStore Call AttributeLoad AttributeLoad NameLoad attr attr Expr Call AttributeLoad NameLoad attr NameLoad For NameStore AttributeLoad NameLoad attr body Assign NameStore Call AttributeLoad NameLoad attr NameLoad NameLoad If CompareIsNot NameLoad NameLoad body Expr Call AttributeLoad NameLoad attr TupleLoad Call AttributeLoad Str attr NameLoad Call NameLoad NameLoad For NameStore BinOpAdd Call NameLoad AttributeLoad NameLoad attr Call NameLoad BoolOpOr NameLoad ListLoad body Expr Call AttributeLoad NameLoad attr TupleLoad Str NameLoad For TupleStore NameStore NameStore Call AttributeLoad NameLoad attr body Expr Call AttributeLoad NameLoad attr TupleLoad Str Call AttributeLoad Str attr NameLoad NameLoad Return BinOpMod Str TupleLoad NameLoad Call AttributeLoad Str attr NameLoad decorator_list FunctionDef arguments args NameParam defaults body Expr Str If AttributeLoad SubscriptLoad NameLoad Index Str attr body Return AttributeLoad NameLoad attr Return NameLoad decorator_list NameLoad FunctionDef arguments args NameParam NameParam defaults kwarg body Expr Str Expr Call AttributeLoad NameLoad attr Str AttributeLoad NameLoad attr Expr Call AttributeLoad NameLoad attr Str AttributeLoad NameLoad attr For NameStore AttributeLoad NameLoad attr body Assign TupleStore NameStore NameStore Call AttributeLoad NameLoad attr Str Num Assign SubscriptStore NameLoad Index NameLoad NameLoad Assign NameStore Call AttributeLoad NameLoad attr Str NameLoad Assign TupleStore NameStore NameStore Call AttributeLoad NameLoad attr NameLoad Return Call NameLoad Call NameLoad NameLoad decorator_list FunctionDef arguments args NameParam NameParam NameParam defaults kwarg body Expr Str Assign NameStore ListLoad Assign NameStore Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad Expr Call AttributeLoad NameLoad attr Str AttributeLoad NameLoad attr Expr Call AttributeLoad NameLoad attr Str AttributeLoad NameLoad attr With Call NameLoad NameLoad NameStore body Assign NameStore Call NameLoad Call AttributeLoad NameLoad attr Expr Call AttributeLoad NameLoad attr NameLoad Expr Call AttributeLoad NameLoad attr keyword Str keyword BinOpMult Call NameLoad NameLoad SubscriptLoad NameLoad Index Str keyword SubscriptLoad NameLoad Index Str keyword Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad Assign NameStore Call NameLoad Call AttributeLoad NameLoad attr Str Str Assign NameStore BinOpFloorDiv Call NameLoad NameLoad Call NameLoad NameLoad Assign NameStore ListComp BinOpAdd NameLoad IfExp CompareLt NameLoad BinOpMod NameLoad Call NameLoad NameLoad Num Num comprehension NameStore Call NameLoad Call NameLoad NameLoad FunctionDef arguments args NameParam defaults body Expr Call AttributeLoad NameLoad attr NameLoad NameLoad decorator_list Expr Call AttributeLoad NameLoad attr NameLoad NameLoad Assign SubscriptStore NameLoad Index Str ListLoad NameLoad FunctionDef arguments args NameParam defaults body Assign NameStore Call NameLoad SubscriptLoad NameLoad Slice NameLoad Assign NameStore Call AttributeLoad NameLoad attr Expr Call AttributeLoad NameLoad attr keyword NameLoad keyword SubscriptLoad NameLoad Index NameLoad Expr Call AttributeLoad NameLoad attr Call AttributeLoad NameLoad attr SubscriptLoad NameLoad Index NameLoad NameLoad Expr Call AttributeLoad NameLoad attr Str NameLoad SubscriptLoad NameLoad Index NameLoad decorator_list Expr Call AttributeLoad NameLoad attr NameLoad Call NameLoad Call NameLoad NameLoad If CompareNotEq Call NameLoad NameLoad Call NameLoad NameLoad body Raise Call NameLoad Call AttributeLoad Str attr Call NameLoad NameLoad Call NameLoad NameLoad NameLoad Assign NameStore ListLoad If BoolOpAnd AttributeLoad NameLoad attr CompareGt Call NameLoad NameLoad Num body For TupleStore NameStore NameStore Call NameLoad NameLoad body Expr Call AttributeLoad NameLoad attr Call NameLoad NameLoad keyword Str keyword NameLoad keyword AttributeLoad NameLoad attr NameLoad Assign NameStore Call NameLoad NameLoad Expr Call AttributeLoad NameLoad attr Call NameLoad NameLoad keyword Str keyword AttributeLoad NameLoad attr NameLoad Return NameLoad decorator_list FunctionDef arguments args NameParam NameParam defaults kwarg body Expr Str For NameStore AttributeLoad NameLoad attr body Assign TupleStore NameStore NameStore Call AttributeLoad NameLoad attr Str Num Assign SubscriptStore NameLoad Index NameLoad NameLoad Assign NameStore Call AttributeLoad NameLoad attr Str NameLoad Assign TupleStore NameStore NameStore Call AttributeLoad NameLoad attr NameLoad Return Call NameLoad Call NameLoad NameLoad decorator_list FunctionDef arguments args NameParam NameParam defaults kwarg body Expr Str Assign NameStore Call AttributeLoad NameLoad attr Str NameLoad If CompareIsNot NameLoad NameLoad body Assign NameStore BinOpFloorDiv NameLoad Call NameLoad NameLoad Assign NameStore ListComp BinOpAdd NameLoad IfExp CompareLt NameLoad BinOpMod NameLoad Call NameLoad NameLoad Num Num comprehension NameStore Call NameLoad Call NameLoad NameLoad orelse Assign NameStore ListComp NameLoad comprehension NameStore NameLoad Assign NameStore ListLoad FunctionDef arguments args NameParam defaults body Assign NameStore SubscriptLoad NameLoad Index NameLoad Assign SubscriptStore NameLoad Index Str SubscriptLoad NameLoad Index NameLoad Expr Call AttributeLoad NameLoad attr Call AttributeLoad NameLoad attr NameLoad NameLoad Expr Call AttributeLoad NameLoad attr Str NameLoad NameLoad decorator_list Expr Call AttributeLoad NameLoad attr NameLoad Call NameLoad Call NameLoad NameLoad If CompareNotEq Call NameLoad NameLoad Call NameLoad NameLoad body Raise Call NameLoad Call AttributeLoad Str attr Call NameLoad NameLoad Call NameLoad NameLoad NameLoad Return NameLoad decorator_list FunctionDef arguments args NameParam NameParam NameParam defaults kwarg body Expr Str Assign NameStore ListLoad For TupleStore NameStore NameStore Call NameLoad NameLoad body Assign NameStore Dict Str Str AttributeLoad NameLoad attr AttributeLoad NameLoad attr If AttributeLoad NameLoad attr body Assign SubscriptStore NameLoad Index Str AttributeLoad NameLoad attr Expr Call AttributeLoad NameLoad attr NameLoad Assign NameStore Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad With Call NameLoad NameLoad NameStore body Assign NameStore Call NameLoad Call AttributeLoad NameLoad attr Expr Call AttributeLoad NameLoad attr NameLoad Expr Call AttributeLoad NameLoad attr keyword Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad keyword NameLoad keyword Str FunctionDef arguments args NameParam defaults body Expr Call AttributeLoad NameLoad attr NameLoad NameLoad decorator_list Expr Call AttributeLoad NameLoad attr NameLoad NameLoad Assign SubscriptStore NameLoad Index Str ListLoad NameLoad For NameStore Call NameLoad body Assign SubscriptStore NameLoad Index Str NameLoad Assign NameStore Call AttributeLoad NameLoad attr NameLoad NameLoad Assign NameStore Call AttributeLoad NameLoad attr Expr Call AttributeLoad NameLoad attr keyword BinOpMult Call NameLoad NameLoad NameLoad keyword NameLoad If BoolOpAnd AttributeLoad NameLoad attr CompareGt Call NameLoad NameLoad Num body For TupleStore NameStore NameStore Call NameLoad NameLoad body Expr Call AttributeLoad NameLoad attr Call NameLoad NameLoad keyword Str keyword NameLoad keyword AttributeLoad NameLoad attr NameLoad Assign NameStore Call NameLoad NameLoad Expr Call AttributeLoad NameLoad attr Call NameLoad NameLoad keyword Str keyword AttributeLoad NameLoad attr NameLoad Return NameLoad decorator_list FunctionDef arguments args NameParam NameParam NameParam NameParam NameParam defaults NameLoad NameLoad NameLoad body Expr Str Assign NameStore BoolOpOr NameLoad Call NameLoad Assert NameLoad Str Assign NameStore Call NameLoad Call AttributeLoad NameLoad attr NameLoad SubscriptLoad NameLoad Index Num BoolOpOr NameLoad Dict Assign NameStore Call NameLoad Call AttributeLoad NameLoad attr NameLoad NameLoad BoolOpOr NameLoad Dict If AttributeLoad NameLoad attr body Return BinOpAdd NameLoad NameLoad orelse Return NameLoad decorator_list decorator_list
N/A N/A Install, execute, and parse results from YCSB.

YCSB (the Yahoo! Cloud Serving Benchmark) is a common method of comparing NoSQL
database performance.
https://github.com/brianfrankcooper/YCSB

For PerfKitBenchmarker, we wrap YCSB to:

  * Pre-load a database with a fixed number of records.
  * Execute a collection of workloads under a staircase load.
  * Parse the results into PerfKitBenchmarker samples.

The 'YCSBExecutor' class handles executing YCSB on a collection of client VMs.
Generally, clients just need this class. For example, to run against
HBase 1.0:

  >>> executor = ycsb.YCSBExecutor('hbase-10')
  >>> samples = executor.LoadAndRun(loader_vms)

By default, this runs YCSB workloads A and B against the database, 32 threads
per client VM, with an initial database size of 1GB (1k records).
Each workload runs for at most 30 minutes.
 N/A bisect N/A collections N/A copy N/A csv N/A io N/A itertools N/A math N/A re N/A logging N/A operator N/A os N/A posixpath perfkitbenchmarker data perfkitbenchmarker flags perfkitbenchmarker sample perfkitbenchmarker vm_util N/A FLAGS N/A flags FLAGS N/A YCSB_TAR_URL https://github.com/brianfrankcooper/YCSB/releases/download/0.3.0/ycsb-0.3.0.tar.gz N/A YCSB_DIR N/A N/A posixpath join N/A vm_util VM_TMP_DIR ycsb N/A YCSB_EXE N/A N/A posixpath join YCSB_DIR bin ycsb N/A _DEFAULT_PERCENTILES N/A 50 75 90 95 99 99.9 N/A AGGREGATE_OPERATORS N/A Operations RunTime(ms) Return=0 Return=-1 Return=-2 Return=-3 AverageLatency(ms) Throughput(ops/sec) 95thPercentileLatency(ms) 99thPercentileLatency(ms) MinLatency(ms) MaxLatency(ms) N/A operator add max N/A operator add N/A operator add N/A operator add N/A operator add None N/A operator add None None min max N/A N/A N/A flags DEFINE_boolean ycsb_histogram False Include individual histogram results from YCSB (will increase sample count). N/A N/A N/A flags DEFINE_boolean ycsb_load_samples True Include samples from pre-populating database. N/A N/A N/A flags DEFINE_boolean ycsb_include_individual_results False Include results from each client VM, rather than just combined results. N/A N/A N/A flags DEFINE_integer ycsb_client_vms 1 Number of YCSB client VMs. lower_bound 1 N/A N/A N/A flags DEFINE_list ycsb_workload_files N/A Path to YCSB workload file to use during *run* stage only. Comma-separated list N/A N/A N/A flags DEFINE_list ycsb_load_parameters N/A Passed to YCSB during the load stage. Comma-separated list of "key=value" pairs. N/A N/A N/A flags DEFINE_list ycsb_run_parameters N/A Passed to YCSB during the load stage. Comma-separated list of "key=value" pairs. N/A N/A N/A flags DEFINE_list ycsb_threads_per_client N/A 32 Number of threads per loader during the benchmark run. Specify a list to vary the number of clients. N/A N/A N/A flags DEFINE_integer ycsb_preload_threads None Number of threads per loader during the initial data population stage. Default value depends on the target DB. N/A N/A N/A flags DEFINE_integer ycsb_record_count 1000000 Pre-load with a total dataset of records total. N/A N/A N/A flags DEFINE_integer ycsb_operation_count 1000000 Number of operations *per client VM*. N/A N/A N/A flags DEFINE_integer ycsb_timelimit 1800 Maximum amount of time to run each workload / client count combination. Set to 0 for unlimited time. N/A DEFAULT_PRELOAD_THREADS 32 _GetThreadsPerLoaderList N/A N/A N/A N/A N/A Returns the list of client counts per VM to use in staircase load. N/A N/A N/A int thread_count N/A thread_count N/A FLAGS ycsb_threads_per_client N/A _GetWorkloadFileList N/A N/A N/A N/A N/A Returns the list of workload files to run.

  Returns:
    In order of preference:
      * The argument to --ycsb_workload_files.
      * Bundled YCSB workloads A and B.
   N/A N/A FLAGS ycsb_workload_files N/A N/A N/A FLAGS ycsb_workload_files N/A N/A N/A N/A data ResourcePath N/A N/A N/A os path join ycsb workload N/A workload N/A workloada workloadb N/A CheckPrerequisites N/A N/A N/A N/A N/A workload_file N/A _GetWorkloadFileList N/A N/A N/A N/A N/A N/A os path exists workload_file N/A N/A N/A IOError N/A N/A Missing workload file: {0} format workload_file N/A _Install N/A N/A vm N/A N/A N/A Installs the YCSB package on the VM. N/A N/A N/A vm Install openjdk7 N/A N/A N/A vm Install curl N/A N/A N/A vm RemoteCommand N/A N/A mkdir -p {0} && curl -L {1} | tar -C {0} --strip-components=1 -xzf - format YCSB_DIR YCSB_TAR_URL N/A YumInstall N/A N/A vm N/A N/A N/A Installs the YCSB package on the VM. N/A N/A _Install vm N/A AptInstall N/A N/A vm N/A N/A N/A Installs the YCSB package on the VM. N/A N/A _Install vm N/A ParseResults N/A N/A ycsb_result_string data_type N/A histogram N/A N/A Parse YCSB results.

  Example input:

    YCSB Client 0.1
    Command line: -db com.yahoo.ycsb.db.HBaseClient -P /tmp/pkb/workloada
    [OVERALL], RunTime(ms), 1800413.0
    [OVERALL], Throughput(ops/sec), 2740.503428935472
    [UPDATE], Operations, 2468054
    [UPDATE], AverageLatency(us), 2218.8513395574005
    [UPDATE], MinLatency(us), 554
    [UPDATE], MaxLatency(us), 352634
    [UPDATE], 95thPercentileLatency(ms), 4
    [UPDATE], 99thPercentileLatency(ms), 7
    [UPDATE], Return=0, 2468054
    [UPDATE], 0, 398998
    [UPDATE], 1, 1015682
    [UPDATE], 2, 532078
    ...

  Args:
    ycsb_result_string: str. Text output from YCSB.
    data_type: Either 'histogram' or 'timeseries'.

  Returns:
    A dictionary with keys:
      client: containing YCSB version information.
      command_line: Command line executed.
      groups: list of operation group descriptions, each with schema:
        group: group name (e.g., update, insert, overall)
        statistics: dict mapping from statistic name to value
        histogram: list of (ms_lower_bound, count) tuples, e.g.:
          [(0, 530), (19, 1)]
        indicates that 530 ops took between 0ms and 1ms, and 1 took between
        19ms and 20ms. Empty bins are not reported.
   N/A fp N/A N/A io BytesIO ycsb_result_string N/A client_string N/A N/A N/A next fp strip N/A N/A N/A N/A client_string startswith YCSB Client 0. N/A N/A N/A IOError N/A N/A Unexpected header: {0} format client_string N/A command_line N/A N/A N/A next fp strip N/A N/A N/A N/A command_line startswith Command line: N/A N/A N/A IOError N/A N/A Unexpected second line: {0} format command_line LineFilter N/A N/A line N/A N/A N/A N/A N/A N/A re search ^\[[A-Z]+\] line None N/A N/A lines N/A N/A itertools ifilter LineFilter fp N/A r N/A N/A csv reader lines N/A by_operation N/A N/A itertools groupby r N/A N/A operator itemgetter 0 N/A result N/A N/A collections OrderedDict N/A N/A client client_string N/A command_line command_line N/A groups N/A N/A collections OrderedDict N/A N/A operation lines by_operation N/A N/A operation N/A N/A N/A operation N/A 1 -1 lower N/A op_result N/A group data_type statistics operation N/A N/A N/A N/A _ name val lines N/A N/A name N/A N/A name strip N/A val N/A N/A val strip N/A N/A N/A name startswith > N/A N/A name N/A name N/A 1 N/A val N/A N/A . val N/A float val N/A int val N/A N/A N/A name isdigit N/A N/A val N/A N/A N/A N/A N/A op_result N/A data_type append N/A N/A int name val N/A N/A N/A (us) name N/A N/A name N/A N/A name replace (us) (ms) N/A val 1000.0 N/A N/A N/A op_result N/A statistics N/A name val N/A N/A N/A result N/A groups N/A operation op_result N/A result N/A _CumulativeSum N/A N/A xs N/A N/A N/A total 0 N/A x xs N/A N/A total x N/A N/A total N/A _WeightedQuantile N/A N/A x weights p N/A N/A N/A Weighted quantile measurement for an ordered list.

  This method interpolates to the higher value when the quantile is not a direct
  member of the list. This works well for YCSB, since latencies are floored.

  Args:
    x: List of values.
    weights: List of numeric weights.
    p: float. Desired quantile in the interval [0, 1].

  Returns:
    float.

  Raises:
    ValueError: When 'x' and 'weights' are not the same length, or 'p' is not in
      the interval [0, 1].
   N/A N/A N/A len x N/A len weights N/A N/A N/A ValueError N/A N/A Lengths do not match: {0} != {1} format N/A len x N/A len weights N/A N/A N/A p 0 N/A p 1 N/A N/A N/A ValueError N/A N/A Invalid quantile: {0} format p N/A n N/A sum weights N/A target N/A n N/A float p N/A cumulative N/A list N/A _CumulativeSum weights N/A i N/A N/A bisect bisect_left cumulative target N/A N/A i N/A len x N/A N/A N/A x N/A -1 N/A N/A N/A x N/A i N/A _PercentilesFromHistogram N/A N/A ycsb_histogram percentiles N/A _DEFAULT_PERCENTILES N/A N/A Calculate percentiles for from a YCSB histogram.

  Args:
    ycsb_histogram: List of (time_ms, frequency) tuples.
    percentiles: iterable of floats, in the interval [0, 100].

  Returns:
    dict, mapping from percentile to value.
   N/A result N/A N/A collections OrderedDict N/A histogram N/A sorted ycsb_histogram N/A percentile percentiles N/A N/A N/A N/A percentile 0 N/A percentile 100 N/A N/A N/A ValueError N/A N/A Invalid percentile: {0} format percentile N/A N/A N/A N/A N/A math modf percentile N/A 0 1e-07 N/A N/A percentile N/A int percentile N/A label N/A N/A p{0} format percentile N/A N/A latencies freqs N/A zip histogram N/A time_ms N/A _WeightedQuantile latencies freqs N/A percentile 0.01 N/A N/A result N/A label time_ms N/A result N/A _CombineResults N/A N/A result_list combine_histograms N/A True N/A N/A Combine results from multiple YCSB clients.

  Reduces a list of YCSB results (the output of ParseResults)
  into a single result. Histogram bin counts, operation counts, and throughput
  are summed; RunTime is replaced by the maximum runtime of any result.

  Args:
    result_list: List of ParseResults outputs.
    combine_histograms: If true, histogram bins are summed across results. If
      not, no histogram will be returned. Defaults to True.
  Returns:
    A dictionary, as returned by ParseResults.
   DropUnaggregated N/A N/A result N/A N/A N/A Remove statistics which 'operators' specify should not be combined. N/A drop_keys N/A k N/A N/A k v N/A N/A AGGREGATE_OPERATORS iteritems N/A v None N/A group N/A N/A N/A result N/A groups itervalues N/A N/A k drop_keys N/A N/A N/A N/A N/A group N/A statistics pop k None N/A CombineHistograms N/A N/A hist1 hist2 N/A N/A N/A h1 N/A dict hist1 N/A h2 N/A dict hist2 N/A keys N/A sorted N/A N/A frozenset h1 N/A frozenset h2 N/A result N/A N/A k keys N/A N/A N/A N/A result append N/A k N/A N/A N/A h1 get k 0 N/A N/A h2 get k 0 N/A result N/A N/A result N/A N/A copy deepcopy N/A result_list N/A 0 N/A N/A DropUnaggregated result N/A indiv N/A result_list N/A 1 N/A N/A N/A group_name group N/A N/A N/A indiv N/A groups iteritems N/A N/A N/A group_name N/A result N/A groups N/A N/A N/A N/A logging warn Found result group "%s" in individual YCSB result, but not in accumulator. group_name N/A N/A N/A result N/A groups N/A group_name N/A N/A copy deepcopy group N/A N/A N/A k v N/A N/A N/A group N/A statistics iteritems N/A N/A N/A k AGGREGATE_OPERATORS N/A N/A N/A N/A logging warn No operator for "%s". Skipping aggregation. k N/A N/A N/A N/A N/A AGGREGATE_OPERATORS N/A k None N/A N/A N/A N/A N/A N/A N/A result N/A groups N/A group_name N/A statistics pop k None N/A N/A N/A N/A k N/A N/A N/A result N/A groups N/A group_name N/A statistics N/A N/A N/A N/A logging warn Found statistic "%s.%s" in individual YCSB result, but not in accumulator. group_name k N/A N/A N/A N/A N/A result N/A groups N/A group_name N/A statistics N/A k N/A N/A copy deepcopy v N/A N/A op N/A AGGREGATE_OPERATORS N/A k N/A N/A N/A N/A N/A result N/A groups N/A group_name N/A statistics N/A k N/A op N/A N/A N/A N/A result N/A groups N/A group_name N/A statistics N/A k v N/A combine_histograms N/A N/A N/A N/A N/A result N/A groups N/A group_name N/A histogram N/A CombineHistograms N/A N/A N/A result N/A groups N/A group_name N/A histogram N/A group N/A histogram N/A N/A N/A N/A N/A N/A result N/A groups N/A group_name pop histogram None N/A N/A result N/A client N/A N/A join N/A N/A result N/A client N/A indiv N/A client N/A N/A result N/A command_line N/A N/A ; join N/A N/A result N/A command_line N/A indiv N/A command_line N/A N/A N/A target result N/A target indiv N/A N/A N/A result N/A target N/A indiv N/A target N/A result N/A _ParseWorkload N/A N/A contents N/A N/A N/A Parse a YCSB workload file.

  YCSB workloads are Java .properties format.
  http://en.wikipedia.org/wiki/.properties
  This function does not support all .properties syntax, in particular escaped
  newlines.

  Args:
    contents: str. Contents of the file.

  Returns:
    dict mapping from property key to property value for each property found in
    'contents'.
   N/A fp N/A N/A io BytesIO contents N/A result N/A N/A line fp N/A N/A N/A N/A N/A line strip N/A N/A N/A N/A N/A line lstrip startswith # N/A N/A N/A N/A N/A line lstrip startswith ! N/A N/A N/A k v N/A N/A re split \s*[:=]\s* line maxsplit 1 N/A N/A result N/A k N/A N/A v strip N/A result N/A _CreateSamples N/A N/A ycsb_result include_histogram N/A True kwargs N/A N/A Create PKB samples from a YCSB result.

  Args:
    ycsb_result: dict. Result of ParseResults.
    include_histogram: bool. If True, include records for each histogram bin.
    **kwargs: Base metadata for each sample.

  Returns:
    List of sample.Sample objects.
   N/A stage N/A N/A N/A N/A ycsb_result N/A command_line endswith -load load run N/A base_metadata N/A command_line stage N/A ycsb_result N/A command_line stage N/A N/A N/A base_metadata update kwargs N/A N/A group_name group N/A N/A N/A ycsb_result N/A groups iteritems N/A N/A meta N/A N/A base_metadata copy N/A N/A meta N/A operation group_name N/A N/A statistic value N/A N/A N/A group N/A statistics iteritems N/A N/A N/A value None N/A N/A N/A unit N/A m N/A N/A re match ^(.*) *\((us|ms|ops/sec)\)$ statistic N/A m N/A N/A statistic N/A N/A m group 1 N/A unit N/A N/A m group 2 N/A N/A N/A N/A sample Sample N/A N/A join N/A group_name statistic value unit meta N/A N/A group N/A histogram N/A N/A percentiles N/A _PercentilesFromHistogram N/A group N/A histogram N/A N/A label value N/A N/A percentiles iteritems N/A N/A N/A N/A N/A sample Sample N/A N/A join N/A group_name label latency value ms meta N/A include_histogram N/A N/A N/A time_ms count N/A group N/A histogram N/A N/A N/A N/A N/A sample Sample N/A N/A {0}_latency_histogram_{1}_ms format group_name time_ms count count meta N/A YCSBExecutor N/A object N/A N/A Load data and run benchmarks using YCSB.

  See core/src/main/java/com/yahoo/ycsb/workloads/CoreWorkload.java for
  attribute descriptions.

  Attributes:
    database: str.
    parameters: dict. May contain the following, plus database-specific fields
      (e.g., columnfamily for HBase).

      threads: int.
      target: int.
      fieldcount: int.
      fieldlengthdistribution: str.
      readallfields: boolean.
      writeallfields: boolean.
      readproportion: float.
      updateproportion: float.
      scanproportion: float.
      readmodifywriteproportion: float.
      requestdistribution: str.
      maxscanlength: int. Number of records to scan.
      scanlengthdistribution: str.
      insertorder: str.
      hotspotdatafraction: float.
   N/A FLAG_ATTRIBUTES N/A cp jvm-args target threads __init__ N/A N/A self database parameter_files N/A None kwargs N/A N/A N/A self database database N/A N/A self parameter_files N/A parameter_files N/A N/A N/A self parameters N/A N/A kwargs copy N/A _BuildCommand N/A N/A self command_name parameter_files N/A None kwargs N/A N/A command N/A YCSB_EXE command_name N/A self database N/A parameters N/A N/A N/A self parameters copy N/A N/A N/A parameters update kwargs N/A flag N/A self FLAG_ATTRIBUTES N/A N/A value N/A N/A parameters pop flag None N/A N/A value None N/A N/A N/A N/A command extend N/A N/A N/A -{0} format flag N/A str value N/A param_file N/A N/A list N/A self parameter_files N/A list N/A parameter_files N/A N/A N/A N/A N/A command extend N/A -P param_file N/A N/A parameter value N/A N/A parameters iteritems N/A N/A N/A N/A command extend N/A -p N/A N/A {0}={1} format parameter value N/A N/A cd %s; %s N/A YCSB_DIR N/A N/A join command N/A _default_preload_threads N/A N/A self N/A N/A N/A The default number of threads to use for pre-populating the DB. N/A N/A N/A FLAGS N/A ycsb_preload_threads present N/A N/A N/A FLAGS ycsb_preload_threads N/A DEFAULT_PRELOAD_THREADS N/A property _Load N/A N/A self vm N/A kwargs N/A N/A Execute 'ycsb load' on 'vm'. N/A N/A N/A kwargs setdefault threads N/A self _default_preload_threads N/A N/A N/A kwargs setdefault recordcount N/A FLAGS ycsb_record_count N/A pv N/A FLAGS ycsb_load_parameters N/A N/A N/A param value N/A N/A pv split = 1 N/A N/A kwargs N/A param value N/A command N/A N/A self _BuildCommand load kwargs N/A N/A stdout _ N/A N/A vm RobustRemoteCommand command N/A N/A ParseResults N/A str stdout N/A _LoadThreaded N/A N/A self vms workload_file N/A kwargs N/A N/A Runs "Load" in parallel for each VM in VMs.

    Args:
      vms: List of virtual machine instances. client nodes.
      workload_file: YCSB Workload file to use.
      **kwargs: Additional key-value parameters to pass to YCSB.

    Returns:
      List of sample.Sample objects.
     N/A results N/A N/A remote_path N/A N/A posixpath join N/A vm_util VM_TMP_DIR N/A N/A N/A os path basename workload_file N/A N/A N/A kwargs setdefault threads N/A self _default_preload_threads N/A N/A N/A kwargs setdefault recordcount N/A FLAGS ycsb_record_count N/A N/A open workload_file fp N/A N/A workload_meta N/A _ParseWorkload N/A N/A fp read N/A N/A N/A workload_meta update kwargs N/A N/A N/A workload_meta update stage load clients N/A N/A len vms N/A kwargs N/A threads threads_per_client_vm N/A kwargs N/A threads workload_name N/A N/A N/A os path basename workload_file N/A record_count N/A int N/A N/A workload_meta get recordcount 1000 N/A n_per_client N/A N/A long record_count N/A len vms N/A loader_counts N/A N/A n_per_client N/A N/A i N/A record_count N/A len vms 1 0 N/A i N/A xrange N/A len vms PushWorkload N/A N/A vm N/A N/A N/A N/A N/A vm PushFile workload_file remote_path N/A N/A N/A N/A vm_util RunThreaded PushWorkload vms N/A N/A kwargs N/A parameter_files N/A remote_path _Load N/A N/A loader_index N/A N/A N/A start N/A sum N/A loader_counts N/A loader_index N/A kw N/A N/A kwargs copy N/A N/A N/A kw update insertstart start insertcount N/A loader_counts N/A loader_index N/A N/A N/A results append N/A N/A self _Load N/A vms N/A loader_index kw N/A N/A N/A logging info VM %d (%s) finished loader_index N/A vms N/A loader_index N/A N/A N/A N/A vm_util RunThreaded _Load N/A range N/A len vms N/A N/A N/A len results N/A len vms N/A N/A N/A IOError N/A N/A Missing results: only {0}/{1} reported
{2} format N/A len results N/A len vms results N/A samples N/A N/A N/A N/A FLAGS ycsb_include_individual_results N/A N/A len results 1 N/A N/A N/A i result N/A enumerate results N/A N/A N/A N/A samples extend N/A _CreateSamples result result_type individual result_index i include_histogram N/A FLAGS ycsb_histogram workload_meta N/A combined N/A _CombineResults results N/A N/A N/A samples extend N/A _CreateSamples combined result_type combined include_histogram N/A FLAGS ycsb_histogram workload_meta N/A samples N/A _Run N/A N/A self vm N/A kwargs N/A N/A Run a single workload from a client vm. N/A pv N/A FLAGS ycsb_run_parameters N/A N/A N/A param value N/A N/A pv split = 1 N/A N/A kwargs N/A param value N/A command N/A N/A self _BuildCommand run kwargs N/A N/A stdout _ N/A N/A vm RobustRemoteCommand command N/A N/A ParseResults N/A str stdout N/A _RunThreaded N/A N/A self vms N/A kwargs N/A N/A Run a single workload using `vms`. N/A target N/A N/A kwargs pop target None N/A N/A target None N/A N/A target_per_client N/A target N/A len vms N/A targets N/A N/A target_per_client N/A N/A i N/A target N/A len vms 1 0 N/A i N/A xrange N/A len vms N/A N/A targets N/A target N/A _ vms N/A results N/A _Run N/A N/A loader_index N/A N/A N/A vm N/A vms N/A loader_index N/A N/A kwargs N/A target N/A targets N/A loader_index N/A N/A N/A results append N/A N/A self _Run vm kwargs N/A N/A N/A logging info VM %d (%s) finished loader_index vm N/A N/A N/A N/A vm_util RunThreaded _Run N/A range N/A len vms N/A N/A N/A len results N/A len vms N/A N/A N/A IOError N/A N/A Missing results: only {0}/{1} reported
{2} format N/A len results N/A len vms results N/A results N/A RunStaircaseLoads N/A N/A self vms workloads N/A kwargs N/A N/A Run each workload in 'workloads' in succession.

    A staircase load is applied for each workload file, for each entry in
    ycsb_threads_per_client.

    Args:
      vms: List of VirtualMachine objects to generate load from.
      **kwargs: Additional parameters to pass to each run.  See constructor for
      options.

    Returns:
      List of sample.Sample objects.
     N/A all_results N/A N/A N/A workload_index workload_file N/A enumerate workloads N/A N/A parameters N/A operationcount recordcount N/A FLAGS ycsb_operation_count N/A FLAGS ycsb_record_count N/A N/A FLAGS ycsb_timelimit N/A N/A N/A parameters N/A maxexecutiontime N/A FLAGS ycsb_timelimit N/A N/A N/A parameters update kwargs N/A remote_path N/A N/A posixpath join N/A vm_util VM_TMP_DIR N/A N/A N/A os path basename workload_file N/A N/A open workload_file fp N/A N/A workload_meta N/A _ParseWorkload N/A N/A fp read N/A N/A N/A workload_meta update kwargs N/A N/A N/A workload_meta update workload_name N/A N/A N/A os path basename workload_file workload_index workload_index stage run PushWorkload N/A N/A vm N/A N/A N/A N/A N/A vm PushFile workload_file remote_path N/A N/A N/A N/A vm_util RunThreaded PushWorkload vms N/A N/A parameters N/A parameter_files N/A remote_path N/A client_count N/A _GetThreadsPerLoaderList N/A N/A N/A parameters N/A threads client_count N/A results N/A N/A self _RunThreaded vms parameters N/A client_meta N/A N/A workload_meta copy N/A N/A N/A client_meta update clients N/A N/A len vms client_count threads_per_client_vm client_count N/A N/A N/A FLAGS ycsb_include_individual_results N/A N/A len results 1 N/A N/A N/A i result N/A enumerate results N/A N/A N/A N/A all_results extend N/A _CreateSamples result result_type individual result_index i include_histogram N/A FLAGS ycsb_histogram client_meta N/A combined N/A _CombineResults results N/A N/A N/A all_results extend N/A _CreateSamples combined result_type combined include_histogram N/A FLAGS ycsb_histogram client_meta N/A all_results N/A LoadAndRun N/A N/A self vms workloads load_kwargs run_kwargs N/A None None None N/A N/A Load data using YCSB, then run each workload/client count combination.

    Loads data using the workload defined by 'workloads', then
    executes YCSB for each workload file in 'workloads', for each
    client count defined in FLAGS.ycsb_threads_per_client.

    Generally database benchmarks using YCSB should only need to call this
    method.

    Args:
      vms: List of virtual machines. VMs to use to generate load.
      workloads: List of strings. Workload files to use. If unspecified,
        _GetWorkloadFileList() is used.
      load_kwargs: dict. Additional arguments to pass to the load stage.
      run_kwargs: dict. Additional arguments to pass to the run stage.
    Returns:
      List of sample.Sample objects.
     N/A workloads N/A workloads N/A _GetWorkloadFileList N/A workloads no workloads N/A load_samples N/A list N/A N/A self _LoadThreaded vms N/A workloads N/A 0 N/A load_kwargs N/A N/A run_samples N/A list N/A N/A self RunStaircaseLoads vms workloads N/A run_kwargs N/A N/A N/A FLAGS ycsb_load_samples N/A N/A N/A load_samples run_samples N/A N/A run_samples N/A N/A