slurm-toys 1.0.1
helper tools for the SLURM HPC workload manager used at Fred Hutch and elsewhere
Keywords: slurm

slurm-toys (aka awesome-slurm)
==============================

A collection of slurm command line tools and wrappers mostly found on github

why slurm-toys
--------------

The purpose of slurm-toys is to package useful SLURM helper tools written in Python 3 or Shell and
publish them into a single package on PyPI

currently integrated toys
-------------------------

slurm-limiter
~~~~~~~~~~~~~

HPC clusters are optimized to maximize utilization for batch jobs. FairShare helps to ensure that
all users get an appropriate amount of resources over time. However FairShare can only influence
jobs that have not started yet. If a cluster is used 100% by "large" users, "small" users become
unhappy because they may not be able to get a single node ad hoc. Currently the only solution to
this problem appears to be setting hard account limits. Unfortunatelty these limits are often set
too high when a cluster is busy and too low when it is idle. slurm-limiter addresses this problem by
dynamically adjusting the limits based on overal partition/ queue load.

If you want a responsive HPC cluster this should take no longer than 5 sec:

::

    ~$ time srun hostname 
    srun: job 61004624 queued and waiting for resources
    srun: job 61004624 has been allocated resources
    gizmof171

    real    0m1.668s
    user    0m0.044s
    sys 0m0.012s

example use in a cron job, running every 20 min:

::

    */20 * * * * root (  ml Python/3.6.4-foss-2016b-fh1; /app/bin/slurm-limiter -p campus \ 
                       --error-email=sysadmin\@institute.org --minaccountlimit=50 --maxaccountlimit=350 \ 
                       --slaaccountlimit=300 --changestep=50 --maxpercentuse=90 \
                       --minidlenodes=5 ) >>/var/tmp/slurm-limiter.log 2>&1

example output to syslog:

::

    ~$ grep slurm-limiter: /var/log/syslog
    Apr 15 09:40:03 gizmo-ctld slurm-limiter: INFO:slurm-limiter.85: Cores: running=689, pending=3299, total=1180, Usage=58 %, Limits: 350 / 370, Nodes: idle=101
    Apr 15 10:00:03 gizmo-ctld slurm-limiter: INFO:slurm-limiter.85: Cores: running=689, pending=3274, total=1180, Usage=58 %, Limits: 350 / 370, Nodes: idle=101
    Apr 15 10:20:03 gizmo-ctld slurm-limiter: INFO:slurm-limiter.85: Cores: running=680, pending=3241, total=1180, Usage=57 %, Limits: 350 / 370, Nodes: idle=102
    Apr 15 10:40:03 gizmo-ctld slurm-limiter: INFO:slurm-limiter.85: Cores: running=680, pending=3219, total=1180, Usage=57 %, Limits: 350 / 370, Nodes: idle=102

output of slurm-limiter --help

::

    ~$ slurm-limiter --help
    usage: slurm-limiter [-h] [--debug] [--error-email ERROREMAIL]
                         [--cluster CLUSTER] [--partition PARTITION]
                         [--feature FEATURE] [--qos QOS]
                         [--maxaccountlimit MAXLIMIT] [--minaccountlimit MINLIMIT]
                         [--slaaccountlimit SLALIMIT]
                         [--userlimitoffset USERLIMITOFFSET]
                         [--changestep CHANGESTEP] [--minpending MINPENDING]
                         [--maxpercentuse MAXPERCENTUSE]
                         [--minidlenodes MINIDLENODES]

    slurm-limiter checks the current util of a slurm cluster and adjusts the
    account and user limits dynamically within certain range

    optional arguments:
      -h, --help            show this help message and exit
      --debug, -d           verbose output for all commands
      --error-email ERROREMAIL, -e ERROREMAIL
                            send errors to this email address.
      --cluster CLUSTER, -M CLUSTER
                            name of the slurm cluster, (default: current cluster)
      --partition PARTITION, -p PARTITION
                            partition of the slurm cluster (default: entire
                            cluster)
      --feature FEATURE, -f FEATURE
                            filter for only this slurm feature
      --qos QOS, -q QOS     slurm QOS to use for changing account limits (default:
                            public)
      --maxaccountlimit MAXLIMIT, -x MAXLIMIT
                            maximum account limit, never go above this (default:
                            300)
      --minaccountlimit MINLIMIT, -n MINLIMIT
                            minimum account limit, never go below this (default:
                            100)
      --slaaccountlimit SLALIMIT, -t SLALIMIT
                            min SLA limit that has been committed to customers,
                            notify via email if breached (default: 150)
      --userlimitoffset USERLIMITOFFSET, -o USERLIMITOFFSET
                            offset of userlimit from account limit, set a negative
                            number for a userlimit lower than account limit
                            (default: 20)
      --changestep CHANGESTEP, -s CHANGESTEP
                            increase or decrease the limit by this # of cores
                            (default: 10)
      --minpending MINPENDING, -i MINPENDING
                            minimum number of jobs that have to be pending to take
                            action (default: 50)
      --maxpercentuse MAXPERCENTUSE, -u MAXPERCENTUSE
                            maximum allowed % usage in this cluster or partition
                            Throttle QOS down by --changestep if exceeded.
                            (default: 90)
      --minidlenodes MINIDLENODES, -w MINIDLENODES
                            critical minimum number of idle nodes. Throttle QOS
                            down to --minaccountlimit if exceeded. (default: 5)

future toys
-----------

in the future we can integrate other tools, predominantly stuff found on github

https://github.com/search?l=Python&p=1&q=slurm+&type=Repositories

https://github.com/search?l=Shell&q=slurm+&type=Repositories

new tool
~~~~~~~~