Module Expr Str Import alias Import alias Import alias Import alias Import alias Import alias Import alias Import alias Import alias ImportFrom alias ImportFrom alias identifier Import alias identifier Import alias identifier Import alias identifier Assign NameStore Str Assign NameStore Str Assign NameStore Call AttributeLoad NameLoad attr Str Str ClassDef bases body Pass decorator_list FunctionDef arguments args defaults body Expr Str TryExcept body Assign NameStore Call AttributeLoad NameLoad attr Str keyword NameLoad keyword AttributeLoad NameLoad attr handlers ExceptHandler type NameLoad name NameStore body Expr Call AttributeLoad NameLoad attr BinOpMod Str NameLoad Raise Assign NameStore ListLoad For NameStore AttributeLoad NameLoad attr body If Call AttributeLoad NameLoad attr Str body Continue If Call AttributeLoad NameLoad attr Str body Continue If Call AttributeLoad NameLoad attr Str body Continue Assign NameStore Call NameLoad Call AttributeLoad NameLoad attr Str SubscriptLoad NameLoad Slice Num If BoolOpOr CompareEq AttributeLoad NameLoad attr Str CompareEq AttributeLoad NameLoad attr Str body Continue TryExcept body Assign NameStore Call AttributeLoad NameLoad attr keyword Call NameLoad AttributeLoad NameLoad attr handlers ExceptHandler type NameLoad body Pass Assert Call AttributeLoad AttributeLoad NameLoad attr attr Str Assign NameStore Call AttributeLoad NameLoad attr keyword Call NameLoad SubscriptLoad AttributeLoad NameLoad attr Slice Num Expr Call AttributeLoad NameLoad attr NameLoad Return NameLoad decorator_list FunctionDef arguments args NameParam NameParam defaults body Assign NameStore TupleLoad Str Str Str Str Str Str Expr Call AttributeLoad NameLoad attr BinOpMod Str Call AttributeLoad Str attr ListComp BinOpMod Str TupleLoad NameLoad NameLoad comprehension TupleStore NameStore NameStore Call AttributeLoad NameLoad attr NameLoad Call AttributeLoad Call AttributeLoad NameLoad attr attr Assign NameStore Call AttributeLoad NameLoad attr Expr Call AttributeLoad NameLoad attr For NameStore NameLoad body Assign NameStore SubscriptLoad NameLoad Index NameLoad Expr Call AttributeLoad NameLoad attr BinOpMod Str NameLoad For NameStore NameLoad body Assign NameStore ListComp Call AttributeLoad NameLoad attr NameLoad comprehension NameStore NameLoad Expr Call AttributeLoad NameLoad attr BinOpMod Str Call AttributeLoad NameLoad attr NameLoad Expr Call AttributeLoad NameLoad attr Str decorator_list FunctionDef arguments args defaults body Assign NameStore Call AttributeLoad NameLoad attr keyword Str keyword SubscriptLoad Call NameLoad Index Str Expr Call AttributeLoad NameLoad attr Str Str keyword Str keyword Str keyword TupleLoad Str Str keyword Str Expr Call AttributeLoad NameLoad attr keyword Num keyword Num keyword Num keyword Num keyword Str keyword Str Assign TupleStore NameStore NameStore Call AttributeLoad NameLoad attr NameLoad keyword NameLoad Assign NameStore ListComp NameLoad comprehension NameStore Call NameLoad CompareNotEq AttributeLoad NameLoad attr Str Expr Call AttributeLoad NameLoad attr BinOpMod Str Call NameLoad NameLoad Assign NameStore Call AttributeLoad AttributeLoad NameLoad attr attr Call AttributeLoad NameLoad attr Assign NameStore AttributeLoad NameLoad attr If CompareEq AttributeLoad NameLoad attr Str body Expr Call AttributeLoad NameLoad attr BinOpMod Str Call AttributeLoad Str attr AttributeLoad NameLoad attr For NameStore NameLoad body Expr Call AttributeLoad NameLoad attr BinOpMod Str Call AttributeLoad Str attr Call NameLoad NameLoad NameLoad orelse If CompareEq NameLoad Str body Assign NameStore Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad Str If Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad body Raise Call NameLoad BinOpMod Str NameLoad With Call NameLoad NameLoad Str NameStore body If CompareEq AttributeLoad NameLoad attr Str body Assign NameStore AttributeLoad NameLoad attr Expr Call AttributeLoad NameLoad attr BinOpMod NameLoad Call NameLoad Assign NameStore Call AttributeLoad NameLoad attr NameLoad FunctionDef arguments args NameParam NameParam NameParam defaults body Expr Str Assign NameStore AttributeLoad NameLoad attr Assign NameStore BinOpMod AttributeLoad NameLoad attr Call NameLoad Expr Call AttributeLoad NameLoad attr BinOpMod Str NameLoad TryExcept body Assign NameStore Call AttributeLoad NameLoad attr NameLoad keyword NameLoad keyword AttributeLoad NameLoad attr keyword NameLoad keyword NameLoad Assign TupleStore NameStore NameStore Call AttributeLoad NameLoad attr handlers ExceptHandler type NameLoad name NameStore body Expr Call AttributeLoad NameLoad attr BinOpMod Str NameLoad Raise Expr Call AttributeLoad NameLoad attr BinOpMod Str NameLoad Assign NameStore Call AttributeLoad NameLoad attr For NameStore Call AttributeLoad NameLoad attr Str body Expr Call AttributeLoad NameLoad attr NameLoad Expr Call AttributeLoad SubscriptLoad NameLoad Index NameLoad attr NameLoad decorator_list Assign NameStore Num For NameStore Call NameLoad Num AttributeLoad NameLoad attr body Assign NameStore Call NameLoad SubscriptLoad NameLoad Slice Expr Call AttributeLoad NameLoad attr NameLoad For NameStore Call NameLoad Num Call NameLoad NameLoad AttributeLoad NameLoad attr body For NameStore GeneratorExp SubscriptLoad NameLoad Index NameLoad comprehension NameStore Call NameLoad NameLoad Call NameLoad Call NameLoad NameLoad BinOpAdd NameLoad AttributeLoad NameLoad attr body Assign NameStore ListLoad For NameStore Call NameLoad Num Call NameLoad BinOpMult AttributeLoad NameLoad attr AttributeLoad NameLoad attr body Assign NameStore Call NameLoad keyword NameLoad keyword TupleLoad NameLoad AttributeLoad NameLoad attr NameLoad Expr Call AttributeLoad NameLoad attr NameLoad Expr Call AttributeLoad NameLoad attr Expr Call AttributeLoad NameLoad attr BinOpMod Str TupleLoad NameLoad AttributeLoad NameLoad attr Call NameLoad NameLoad For NameStore NameLoad body Expr Call AttributeLoad NameLoad attr Assign NameStore BinOpMod Str NameLoad Expr Call AttributeLoad NameLoad attr BinOpMod Str TupleLoad NameLoad NameLoad AugAssignAdd NameStore Num Assign NameStore Call NameLoad NameLoad Str Expr Call NameLoad NameLoad NameLoad Expr Call AttributeLoad NameLoad attr Expr Call NameLoad AttributeLoad NameLoad attr NameLoad Expr Call AttributeLoad NameLoad attr decorator_list If CompareEq NameLoad Str body Expr Call AttributeLoad NameLoad attr Call NameLoad
N/A N/A 
benchmark_cluster.py - script for testing CPU speed in SGE cluster
=====================================================================

:Author: Andreas Heger
:Release: $Id$
:Date: |today|
:Tags: Python

Purpose
-------

Benchmark CPU speed of nodes in the cluster.

The script sends off a script to compute fibonacci numbers to
all nodes in the cluster. 

Testing is done in multiple iterations and on host-by-host basis.
Several hosts are run and the order of hosts is randomized at each
step.

More jobs than slots are send to each host.

The output is a collection of tab separated tables with benchmarking
data for each iteration separately and a combined analysis.

Usage
-----

Example::

   python benchmark_cluster.py 

Type::

   python benchmark_cluster.py --help

for command line help.

Documentation
-------------

Command line options
---------------------

 N/A sys N/A os N/A re N/A subprocess N/A optparse N/A tempfile N/A collections N/A itertools N/A random multiprocessing Process threading Thread Process N/A CGAT.Experiment E N/A CGAT.Stats Stats N/A CGAT.Logfile Logfile N/A CODE_CPU 
import sys
sys.path.append( "/ifs/devel/andreas/cgat" )
import CGAT.Experiment as E
import operator

E.Start()

def fib(n):
   if n == 0 or n == 1:
      return n
   else:
      return fib(n-1) + fib(n-2)

for i in range(%(iterations)s):
   a,b = (i, fib(i))

E.Stop()
 N/A CODE_SCRIPT 
#! /bin/bash

python %(dir)s/benchmarkscript.py
 N/A HOST N/A N/A collections namedtuple HOST host arch ncpu load memtot memuse swapto swapus Host N/A N/A N/A N/A getSGEHosts N/A N/A N/A N/A N/A get sun grid engine hosts. N/A N/A N/A x N/A N/A subprocess Popen qhost shell True stdout N/A subprocess PIPE N/A N/A N/A OSError N/A e N/A N/A N/A N/A E fail N/A Execution of %s failed cmd N/A N/A hosts N/A N/A line N/A x stdout N/A N/A N/A N/A line startswith HOSTNAME N/A N/A N/A N/A N/A line startswith global N/A N/A N/A N/A N/A line startswith - N/A N/A N/A x N/A HOST N/A N/A re split \s+ N/A line N/A -1 N/A N/A N/A N/A x ncpu - N/A N/A x arch - N/A N/A N/A N/A N/A x N/A N/A x _replace ncpu N/A int N/A x ncpu N/A N/A N/A ValueError N/A N/A N/A N/A N/A N/A x memtot endswith G N/A x N/A N/A x _replace memtot N/A float N/A N/A x memtot N/A -1 N/A N/A N/A hosts append x N/A hosts N/A writeResults N/A N/A outfile results N/A N/A N/A fields N/A wall user sys cuser csys nchunks N/A N/A N/A outfile write N/A host	%s
 N/A N/A join N/A N/A %s_%s N/A x y N/A N/A x y N/A N/A itertools product fields N/A N/A N/A N/A Stats Summary getHeaders N/A hosts N/A N/A results keys N/A N/A N/A hosts sort N/A host hosts N/A N/A result N/A results N/A host N/A N/A N/A outfile write N/A %s host N/A f fields N/A N/A d N/A N/A N/A y __getitem__ f N/A y result N/A N/A N/A outfile write N/A 	%s N/A N/A Stats Summary d N/A N/A N/A outfile write N/A main N/A N/A N/A N/A N/A parser N/A N/A optparse OptionParser version %prog version: $Id: benchmark_cluster.py 2794 2009-09-16 15:27:49Z andreas $ usage N/A N/A globals N/A __doc__ N/A N/A N/A parser add_option -s --suite dest suite type choice choices N/A CPU hosts help suite to use [default=%default]. N/A N/A N/A parser set_defaults iterations_cpu 36 iterations_sample 5 saturation 2 host_block 10 suite CPU cluster_prefix qrsh -q %(queue)s@%(host)s -cwd -now n 'python %(script)s | grep -e "job started" -e "job finished"' N/A N/A options args N/A N/A E Start parser add_cluster_options True N/A hosts N/A x N/A x N/A getSGEHosts N/A N/A x ncpu - N/A N/A N/A E info N/A found %i active hosts N/A len hosts N/A curdir N/A N/A N/A os path abspath N/A N/A os getcwd N/A outfile N/A options stdout N/A N/A N/A options suite hosts N/A N/A N/A N/A outfile write N/A %s
 N/A N/A join N/A HOST _fields N/A host hosts N/A N/A N/A N/A outfile write N/A %s
 N/A N/A join N/A map str host N/A N/A N/A suite CPU N/A N/A pythonscript N/A N/A N/A os path join curdir benchmarkscript.py N/A N/A N/A N/A os path exists pythonscript N/A N/A N/A OSError N/A file %s already exists - not overwritten pythonscript N/A N/A open pythonscript w outfile N/A N/A N/A N/A options suite CPU N/A N/A iterations N/A options iterations_cpu N/A N/A N/A outfile write N/A CODE_CPU N/A locals N/A results N/A N/A collections defaultdict list run N/A N/A script host options N/A N/A N/A run python script on node in the cluster.
             N/A queue N/A options cluster_queue N/A command N/A N/A options cluster_prefix N/A locals N/A N/A N/A E debug N/A started: %s command N/A N/A N/A child N/A N/A subprocess Popen command shell True stdout N/A subprocess PIPE stdin None close_fds True N/A N/A stdout stderr N/A N/A child communicate N/A N/A N/A OSError N/A e N/A N/A N/A N/A E warn N/A execution of %s failed cmd N/A N/A N/A N/A E debug N/A finished: %s command N/A data N/A N/A Logfile LogFileData N/A line N/A N/A stdout split N/A N/A N/A N/A data add line N/A N/A N/A N/A results N/A host append data N/A N/A idx 0 N/A n N/A range 0 N/A options iterations_sample N/A N/A myhosts N/A list N/A hosts N/A N/A N/A N/A random shuffle myhosts N/A h N/A range 0 N/A len myhosts N/A options host_block N/A N/A host N/A N/A myhosts N/A x N/A x N/A range h N/A min N/A len myhosts N/A h N/A options host_block N/A N/A processes N/A N/A cpu N/A range 0 N/A int N/A N/A host ncpu N/A options saturation N/A N/A p N/A Process target run args N/A pythonscript N/A host host options N/A N/A N/A processes append p N/A N/A N/A p start N/A N/A N/A E info N/A iteration %i: host %s: started %i threads N/A n N/A host host N/A len processes N/A p processes N/A N/A N/A N/A p join N/A filename N/A results_%i.table idx N/A N/A N/A E info N/A iteration %i: output goes to %s N/A n filename N/A idx 1 N/A outfile N/A open filename w N/A N/A writeResults outfile results N/A N/A N/A outfile close N/A N/A writeResults N/A options stdout results N/A N/A N/A E Stop N/A N/A N/A __name__ __main__ N/A N/A N/A N/A sys exit N/A main