Module Expr Str Import alias Import alias ImportFrom alias ImportFrom alias ImportFrom alias alias alias Assign NameStore Str FunctionDef arguments args defaults body Expr Str Assign NameStore Call NameLoad ListLoad Str keyword NameLoad Assign NameStore Call NameLoad Call AttributeLoad AttributeLoad NameLoad attr attr Return Call NameLoad NameLoad decorator_list FunctionDef arguments args NameParam defaults body Expr Str If Call AttributeLoad AttributeLoad NameLoad attr attr Str body Return BinOpMod Str NameLoad orelse Return BinOpMod Str NameLoad decorator_list FunctionDef arguments args NameParam defaults body Expr Str Import alias TryExcept body Assign NameStore Call AttributeLoad NameLoad attr Call NameLoad NameLoad Return AttributeLoad Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr attr handlers ExceptHandler body Return Str decorator_list FunctionDef arguments args NameParam defaults body Expr Str TryExcept body Expr Call AttributeLoad NameLoad attr Str Call NameLoad NameLoad Return NameLoad handlers ExceptHandler body Return NameLoad decorator_list FunctionDef arguments args NameParam NameParam defaults body Expr Str ImportFrom alias alias Assign NameStore NameLoad If Call AttributeLoad NameLoad attr Str body Assign NameStore SubscriptLoad NameLoad Slice Num Assign NameStore Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad If UnaryOpNot NameLoad body Assign NameStore Call AttributeLoad NameLoad attr Assign NameStore Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad Str Expr Call NameLoad ListLoad NameLoad Call NameLoad NameLoad NameLoad Str Call NameLoad NameLoad keyword Call NameLoad Str Str decorator_list FunctionDef arguments args NameParam defaults NameLoad body Expr Str Assign NameStore Num Assign NameStore Call NameLoad TryExcept body If CompareGtE NameLoad Num body If CompareIs NameLoad NameLoad body Assign NameStore Call AttributeLoad NameLoad attr Expr Call NameLoad NameLoad NameLoad handlers ExceptHandler body Expr Call NameLoad NameLoad Assign NameStore Num Return NameLoad decorator_list FunctionDef arguments args defaults body Expr Str For NameStore Call NameLoad body If Call NameLoad NameLoad body Return NameLoad Return Num decorator_list FunctionDef arguments args NameParam defaults body Expr Str TryExcept body Assign NameStore Call NameLoad NameLoad Expr Call AttributeLoad NameLoad attr NameLoad BinOpAdd NameLoad Str Expr Call AttributeLoad NameLoad attr BinOpAdd NameLoad Str Return NameLoad handlers ExceptHandler body Return NameLoad decorator_list FunctionDef arguments args defaults body Assign NameStore Call NameLoad ListLoad Str Str Str keyword NameLoad Assign NameStore Call AttributeLoad Call AttributeLoad AttributeLoad NameLoad attr attr attr TryExcept body Assign NameStore Call NameLoad NameLoad Assign NameStore Num Assign NameStore ListLoad Assign NameStore ListLoad Assign NameStore ListLoad Assign NameStore ListLoad Assign NameStore SubscriptLoad Call AttributeLoad NameLoad attr Str Index Num Assign NameStore Call AttributeLoad NameLoad attr Str For NameStore NameLoad body Assign NameStore SubscriptLoad Call AttributeLoad NameLoad attr Str Index Num Assign NameStore SubscriptLoad Call AttributeLoad NameLoad attr Str Index Num Expr Call AttributeLoad NameLoad attr Call NameLoad Call AttributeLoad AttributeLoad NameLoad attr attr Assign NameStore SubscriptLoad Call AttributeLoad NameLoad attr Str Index Num Assign NameStore SubscriptLoad Call AttributeLoad NameLoad attr Str Index Num Assign NameStore SubscriptLoad Call AttributeLoad NameLoad attr Str Index Num Assign NameStore SubscriptLoad Call AttributeLoad NameLoad attr Str Index Num Assign NameStore Call NameLoad Call AttributeLoad AttributeLoad NameLoad attr attr Assign NameStore Call NameLoad Call AttributeLoad AttributeLoad NameLoad attr attr Assign NameStore Call NameLoad Call AttributeLoad AttributeLoad NameLoad attr attr Assign NameStore Call NameLoad SubscriptLoad NameLoad Slice Num Assign NameStore Call NameLoad SubscriptLoad NameLoad Slice Num Assign NameStore Call NameLoad SubscriptLoad NameLoad Slice Num Expr Call AttributeLoad NameLoad attr BinOpAdd BinOpAdd BinOpAdd BinOpMod Str NameLoad BinOpMod Str NameLoad BinOpMod Str NameLoad Str Assign NameStore SubscriptLoad Call AttributeLoad NameLoad attr Str Index Num Expr Call AttributeLoad NameLoad attr Call NameLoad Call AttributeLoad AttributeLoad NameLoad attr attr Assign NameStore SubscriptLoad Call AttributeLoad NameLoad attr Str Index Num Expr Call AttributeLoad NameLoad attr Call NameLoad Call AttributeLoad AttributeLoad NameLoad attr attr AugAssignAdd NameStore Num Return ListLoad NameLoad NameLoad NameLoad NameLoad handlers ExceptHandler body Return BinOpMult ListLoad BinOpMult ListLoad Num Call NameLoad Call NameLoad Num decorator_list If CompareEq NameLoad Str body Assign NameStore BinOpAdd Str BinOpMult Str Num Import alias Assign NameStore SubscriptLoad AttributeLoad NameLoad attr Index Num If CompareIn Str AttributeLoad NameLoad attr body If CompareGt Call NameLoad AttributeLoad NameLoad attr Num body TryExcept body Assign NameStore Call NameLoad SubscriptLoad AttributeLoad NameLoad attr Index Num Assert Call AttributeLoad AttributeLoad NameLoad attr attr BinOpMod Str NameLoad handlers ExceptHandler body Print BinOpMod Str NameLoad Print Str Print Str Expr Call AttributeLoad NameLoad attr Num orelse Assign NameStore Call AttributeLoad NameLoad attr Print Call NameLoad NameLoad orelse If CompareIn Str AttributeLoad NameLoad attr body TryExcept body Assign NameStore Call NameLoad SubscriptLoad AttributeLoad NameLoad attr Index Num handlers ExceptHandler body Print BinOpMod Str NameLoad Expr Call AttributeLoad NameLoad attr Num If Call NameLoad NameLoad body Print NameLoad orelse Print Num orelse If CompareIn Str AttributeLoad NameLoad attr body Print Call NameLoad orelse If CompareIn Str AttributeLoad NameLoad attr body TryExcept body Assign NameStore Call NameLoad SubscriptLoad AttributeLoad NameLoad attr Index Num handlers ExceptHandler body Print BinOpMod Str NameLoad Expr Call AttributeLoad NameLoad attr Num If Call NameLoad NameLoad body Print Str orelse Assign NameStore Call NameLoad NameLoad If NameLoad body Print BinOpMod Str TupleLoad NameLoad NameLoad orelse Print Str orelse If CompareIn Str AttributeLoad NameLoad attr body Assign NameStore Call NameLoad Print NameLoad Print BinOpMod Str Str Print NameLoad For NameStore Call NameLoad body Print BinOpMod Str TupleLoad NameLoad SubscriptLoad SubscriptLoad NameLoad Index Num Index NameLoad SubscriptLoad SubscriptLoad NameLoad Index Num Index NameLoad SubscriptLoad SubscriptLoad NameLoad Index Num Index NameLoad Call NameLoad NameLoad Print BinOpAdd NameLoad Str orelse Assign NameStore Call NameLoad Print NameLoad Print Str Print BinOpMod Str NameLoad Print Str Print Print BinOpMod Str NameLoad Print BinOpMod Str NameLoad Print Print BinOpMod Str NameLoad Print Str Print Str Print BinOpAdd Str NameLoad Print Str Print NameLoad For NameStore Call NameLoad body Print BinOpMod Str TupleLoad NameLoad SubscriptLoad SubscriptLoad NameLoad Index Num Index NameLoad SubscriptLoad SubscriptLoad NameLoad Index Num Index NameLoad SubscriptLoad SubscriptLoad NameLoad Index Num Index NameLoad SubscriptLoad SubscriptLoad NameLoad Index Num Index NameLoad Call NameLoad NameLoad Print BinOpAdd NameLoad Str
N/A N/A 
A simple discretionary locking system for /dev/nvidia devices.

Iain Murray, November 2009, January 2010.

-- Additions -- Charlie Tang, Jan, 2011: 
added display of GPU usages

-- Charlie Tang, July, 2011:
improved statistics displaying
 N/A os N/A os.path xml.dom Node xml.dom.minidom parseString subprocess Popen PIPE STDOUT N/A _dev_prefix /dev/nvidia board_ids N/A N/A N/A N/A N/A Returns integer board ids available on this machine. N/A p N/A Popen N/A /u/tang/bin/get_num_gpu_boards stdout PIPE N/A nBoards N/A int N/A N/A N/A p stdout read N/A N/A range nBoards N/A _lock_file N/A N/A id N/A N/A N/A lock file from integer id N/A N/A N/A N/A os path exists /dev/shm N/A N/A N/A /dev/shm/gpu_lock_%d id N/A N/A N/A /tmp/gpu_lock_%d id N/A owner_of_lock N/A N/A id N/A N/A N/A Username that has locked the device id. (Empty string if no lock). N/A pwd N/A N/A N/A statinfo N/A N/A os lstat N/A _lock_file id N/A N/A N/A N/A pwd getpwuid N/A statinfo st_uid pw_name N/A N/A N/A N/A N/A _obtain_lock N/A N/A id N/A N/A N/A Attempts to lock id, returning success as True/False. N/A N/A N/A N/A N/A os symlink /dev/null N/A _lock_file id N/A True N/A N/A N/A N/A False N/A _launch_reaper N/A N/A id pid N/A N/A N/A Start a process that will free a lock when process pid terminates subprocess Popen PIPE N/A me __file__ N/A N/A N/A me endswith .pyc N/A N/A me N/A me N/A -1 N/A myloc N/A N/A N/A os path dirname me N/A N/A myloc N/A N/A myloc N/A N/A os getcwd N/A reaper_cmd N/A N/A N/A os path join myloc run_on_me_or_pid_quit N/A N/A Popen N/A reaper_cmd N/A str pid me --free N/A str id stdout N/A open /dev/null w N/A obtain_lock_id N/A N/A pid N/A None N/A N/A 
    Finds a free id, locks it and returns integer id, or -1 if none free.

    A process is spawned that will free the lock automatically when the
    process pid (by default the current python process) terminates.
     N/A id -1 N/A id N/A obtain_lock_id_to_hog N/A N/A N/A N/A id 0 N/A N/A N/A pid None N/A N/A pid N/A N/A os getpid N/A N/A _launch_reaper id pid N/A N/A N/A N/A N/A free_lock id N/A id -1 N/A id N/A obtain_lock_id_to_hog N/A N/A N/A N/A N/A 
    Finds a free id, locks it and returns integer id, or -1 if none free.

    * Lock must be freed manually *
     N/A id N/A board_ids N/A N/A N/A _obtain_lock id N/A N/A id N/A -1 N/A free_lock N/A N/A id N/A N/A N/A Attempts to free lock id, returning success as True/False. N/A N/A N/A filename N/A _lock_file id N/A N/A N/A os rename filename N/A filename .redundant N/A N/A N/A os remove N/A filename .redundant N/A True N/A N/A N/A N/A False N/A nvidia_gpu_stats N/A N/A N/A N/A N/A p N/A Popen N/A nvidia-smi -x -a stdout PIPE N/A output N/A N/A N/A N/A N/A p stdout read lstrip N/A N/A N/A doc N/A parseString output N/A gpucounter 0 N/A templist N/A N/A memlist N/A N/A uselist N/A N/A fanlist N/A N/A doc2 N/A N/A N/A doc getElementsByTagName nvidia_smi_log N/A 0 N/A gpulist N/A N/A doc2 getElementsByTagName gpu N/A gpu gpulist N/A N/A temp N/A N/A N/A gpu getElementsByTagName temperature N/A 0 N/A temp2 N/A N/A N/A temp getElementsByTagName gpu_temp N/A 0 N/A N/A N/A templist append N/A str N/A N/A N/A temp2 firstChild toxml N/A mem N/A N/A N/A gpu getElementsByTagName memory_usage N/A 0 N/A memtot N/A N/A N/A mem getElementsByTagName total N/A 0 N/A memused N/A N/A N/A mem getElementsByTagName used N/A 0 N/A memfree N/A N/A N/A mem getElementsByTagName free N/A 0 N/A memtot_str N/A str N/A N/A N/A memtot firstChild toxml N/A memused_str N/A str N/A N/A N/A memused firstChild toxml N/A memfree_str N/A str N/A N/A N/A memfree firstChild toxml N/A memtot_float N/A float N/A memtot_str N/A -3 N/A memused_float N/A float N/A memused_str N/A -3 N/A memfree_float N/A float N/A memfree_str N/A -3 N/A N/A N/A memlist append N/A N/A N/A N/A %03.f memused_float N/A +%03.f memfree_float N/A =%03.f memtot_float Mb N/A use N/A N/A N/A gpu getElementsByTagName gpu_util N/A 0 N/A N/A N/A uselist append N/A str N/A N/A N/A use firstChild toxml N/A fan N/A N/A N/A gpu getElementsByTagName fan_speed N/A 0 N/A N/A N/A fanlist append N/A str N/A N/A N/A fan firstChild toxml N/A gpucounter 1 N/A N/A uselist memlist fanlist templist N/A N/A N/A N/A N/A N/A N/A N/A -9999 N/A len N/A board_ids 4 N/A N/A N/A __name__ __main__ N/A N/A div N/A N/A - 90 N/A sys N/A me N/A N/A sys argv N/A 0 N/A N/A --id N/A sys argv N/A N/A N/A N/A len N/A sys argv 2 N/A N/A N/A N/A pid N/A int N/A N/A sys argv N/A 2 N/A N/A N/A N/A os path exists N/A /proc/%d pid N/A N/A N/A N/A N/A Usage: %s --id [pid_to_wait_on] me N/A The optional process id must exist if specified. N/A Otherwise the id of the parent process is used. N/A N/A N/A sys exit 1 N/A N/A pid N/A N/A os getppid N/A N/A obtain_lock_id pid N/A N/A N/A --ids N/A sys argv N/A N/A N/A N/A id N/A int N/A N/A sys argv N/A 2 N/A N/A N/A N/A N/A Usage: %s --ids [specific gpu id] me N/A N/A N/A sys exit 1 N/A N/A _obtain_lock id N/A N/A id N/A N/A -1 N/A N/A N/A --id-to-hog N/A sys argv N/A N/A N/A obtain_lock_id_to_hog N/A N/A N/A --free N/A sys argv N/A N/A N/A N/A id N/A int N/A N/A sys argv N/A 2 N/A N/A N/A N/A N/A Usage: %s --free <id> me N/A N/A N/A sys exit 1 N/A N/A free_lock id N/A N/A Lock freed N/A N/A owner N/A owner_of_lock id N/A owner N/A N/A N/A Failed to free lock id=%d owned by %s N/A id owner N/A N/A Failed to free lock, but it wasn't actually set? N/A N/A N/A --noverbose N/A sys argv N/A N/A stats N/A nvidia_gpu_stats N/A div N/A N/A %s board users: abc N/A div N/A id N/A board_ids N/A N/A N/A       Board %d {Use:%s; Mem:%s; Temp:%s}: %s N/A id N/A N/A stats N/A 0 N/A id N/A N/A stats N/A 1 N/A id N/A N/A stats N/A 2 N/A id N/A owner_of_lock id N/A N/A div N/A N/A stats N/A nvidia_gpu_stats N/A div N/A   Usage instructions:
 N/A N/A   To obtain and lock an id: %s --id me N/A   The lock is automatically freed when the parent terminates N/A N/A N/A   To get an id that won't be freed: %s --id-to-hog <id> me N/A N/A   To get a specific id: %s --ids <id> me N/A N/A N/A   You *must* manually free these ids: %s --free <id>
 me N/A   More info: http://www.cs.toronto.edu/~murray/code/gpu_monitoring/ N/A   Report any problems to: tang@cs.toronto.edu N/A N/A div N/A   NVIDIA board users: N/A div N/A id N/A board_ids N/A N/A N/A   Board %d {Use:%s; Mem(used+free=total): %s; Fan:%s; Temp:%s}: %s N/A id N/A N/A stats N/A 0 N/A id N/A N/A stats N/A 1 N/A id N/A N/A stats N/A 2 N/A id N/A N/A stats N/A 3 N/A id N/A owner_of_lock id N/A N/A div