Module Expr Str ImportFrom alias Expr Call AttributeLoad NameLoad attr Import alias Import alias Import alias Import alias Import alias Import alias Import alias Import alias TryExcept body Import alias identifier handlers ExceptHandler type NameLoad body Import alias Import alias Import alias ImportFrom alias ImportFrom alias ImportFrom alias Import alias ImportFrom alias Import alias identifier ImportFrom alias ImportFrom alias alias ImportFrom alias ImportFrom alias ImportFrom alias alias alias Assign NameStore Call AttributeLoad NameLoad attr NameLoad Assign NameStore Call NameLoad NameLoad Assign NameStore Call AttributeLoad NameLoad attr Str Num Assign NameStore Call AttributeLoad NameLoad attr Str Num Assign NameStore Num Assign NameStore Dict Assign NameStore Dict Assign NameStore Dict Assign NameStore Dict Assign NameStore Num Assign NameStore ListLoad Num Assign NameStore ListLoad Num Num Num Num Assign NameStore Num Assign NameStore Num Assign NameStore Num Assign NameStore Num Assign NameStore Num Assign NameStore ListLoad Num Num Assign NameStore Call AttributeLoad AttributeLoad NameLoad attr attr Assign NameStore Call AttributeLoad AttributeLoad NameLoad attr attr Assign NameStore Num Assign NameStore Dict Assign NameStore Call AttributeLoad NameLoad attr Assign NameStore ListLoad Assign NameStore Num Assign NameStore Num Assign NameStore Dict Assign NameStore NameLoad Assign NameStore Call NameLoad Assign NameStore Call NameLoad Assign NameStore Call NameLoad Assign NameStore BinOpMult Num Num Assign NameStore Num Assign NameStore NameLoad Assign NameStore NameLoad Assign NameStore NameLoad Assign NameStore NameLoad Assign NameStore NameLoad Assign NameStore Str Assign NameStore Str Assign NameStore Str Assign NameStore Call NameLoad Assign NameStore Call NameLoad Assign NameStore NameLoad Assign NameStore NameLoad FunctionDef arguments args NameParam NameParam NameParam defaults body Assign NameStore Call AttributeLoad NameLoad attr NameLoad TryExcept body Assign NameStore Call NameLoad Call AttributeLoad NameLoad attr Str handlers ExceptHandler body Assign NameStore Num Assign NameStore Dict Str Str Str NameLoad NameLoad NameLoad If CompareIn NameLoad NameLoad body Expr Call AttributeLoad SubscriptLoad NameLoad Index NameLoad attr NameLoad orelse Assign SubscriptStore NameLoad Index NameLoad NameLoad decorator_list FunctionDef arguments args NameParam NameParam defaults body Assign NameStore Call AttributeLoad SubscriptLoad NameLoad Index NameLoad attr Str If BoolOpOr UnaryOpNot NameLoad CompareGt BinOpSub NameLoad NameLoad NameLoad body Expr Call AttributeLoad SubscriptLoad NameLoad Index NameLoad attr Dict Str NameLoad Return NameLoad Return NameLoad decorator_list FunctionDef arguments args NameParam NameParam NameParam defaults body Assign NameStore Call AttributeLoad NameLoad attr NameLoad If NameLoad body If BoolOpAnd NameLoad CompareGt Call AttributeLoad NameLoad attr Str Num NameLoad body Expr Call AttributeLoad NameLoad attr BinOpMod Str TupleLoad NameLoad SubscriptLoad NameLoad Index Str Return NameLoad orelse If BoolOpAnd NameLoad CompareGtE Call AttributeLoad NameLoad attr Str Num NameLoad body Expr Call AttributeLoad NameLoad attr BinOpMod Str TupleLoad NameLoad SubscriptLoad NameLoad Index Str Return NameLoad Return NameLoad decorator_list FunctionDef arguments args NameParam NameParam NameParam defaults body Assign NameStore Str If CompareGt Call NameLoad NameLoad NameLoad body Assign NameStore Call AttributeLoad Call AttributeLoad NameLoad attr Str Str attr Str Str Assign NameStore Call AttributeLoad NameLoad attr NameLoad Assign NameStore BinOpAdd BinOpAdd BinOpAdd Str NameLoad Str Call NameLoad NameLoad Assign NameStore Call AttributeLoad NameLoad attr Assign NameStore Call AttributeLoad AttributeLoad NameLoad attr attr Str NameLoad NameLoad Expr Call AttributeLoad NameLoad attr Str NameLoad TryFinally body TryExcept body With Call NameLoad NameLoad Str NameStore body Expr Call AttributeLoad NameLoad attr BinOpAdd BinOpAdd NameLoad Str NameLoad handlers ExceptHandler type NameLoad name NameStore body Expr Call AttributeLoad NameLoad attr BinOpMod Str TupleLoad AttributeLoad NameLoad attr NameLoad Return TupleLoad NameLoad NameLoad finalbody Expr Call AttributeLoad NameLoad attr Assign NameStore Call AttributeLoad NameLoad attr NameLoad BinOpMod Str TupleLoad NameLoad NameLoad Return TupleLoad NameLoad NameLoad orelse Assign NameStore Call AttributeLoad NameLoad attr NameLoad BinOpMod Str TupleLoad NameLoad NameLoad Return TupleLoad NameLoad NameLoad decorator_list FunctionDef arguments args NameParam defaults body TryExcept body Assign NameStore Call AttributeLoad NameLoad attr Str Return SubscriptLoad NameLoad Index BinOpAdd Call AttributeLoad NameLoad attr Str Num handlers ExceptHandler body Expr Call AttributeLoad NameLoad attr BinOpMod Str NameLoad Return NameLoad decorator_list FunctionDef arguments args NameParam NameParam NameParam defaults body Global identifier Global identifier Assign NameStore Call NameLoad NameLoad If UnaryOpNot NameLoad body Assign NameStore Num orelse TryExcept body Assign NameStore Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad handlers ExceptHandler body Assign NameStore Num If CompareEq NameLoad Str body Assign SubscriptStore NameLoad Index NameLoad Dict Str NameLoad orelse Assign SubscriptStore NameLoad Index NameLoad Dict Str NameLoad decorator_list FunctionDef arguments args NameParam NameParam NameParam NameParam NameParam NameParam defaults NameLoad NameLoad body Expr Str Expr Call AttributeLoad NameLoad attr BinOpMod Str NameLoad Global identifier Assign NameStore Call AttributeLoad NameLoad attr NameLoad NameLoad If BoolOpAnd NameLoad NameLoad CompareGtE Call AttributeLoad NameLoad attr Str Num AttributeLoad NameLoad attr body Expr Call AttributeLoad NameLoad attr BinOpMod Str TupleLoad AttributeLoad NameLoad attr SubscriptLoad NameLoad Index Str Return orelse If NameLoad body Assign NameStore AttributeLoad NameLoad attr orelse Assign NameStore NameLoad Assign NameStore Call AttributeLoad NameLoad attr Expr Call AttributeLoad NameLoad attr BinOpMod Str TupleLoad NameLoad Call NameLoad NameLoad NameLoad Expr Call AttributeLoad NameLoad attr TupleLoad NameLoad NameLoad NameLoad NameLoad NameLoad NameLoad NameLoad decorator_list FunctionDef arguments args defaults body While NameLoad body Assign TupleStore NameStore NameStore NameStore NameStore NameStore NameStore NameStore Call AttributeLoad NameLoad attr If CompareEq NameLoad Str body Expr Call NameLoad Str If Call NameLoad NameLoad NameLoad NameLoad body Continue Expr Call AttributeLoad NameLoad attr NameLoad NameLoad NameLoad NameLoad NameLoad NameLoad NameLoad NameLoad decorator_list FunctionDef arguments args NameParam NameParam NameParam NameParam NameParam NameParam NameParam defaults body Expr Call AttributeLoad NameLoad attr BinOpMod Str TupleLoad NameLoad NameLoad Expr Call AttributeLoad NameLoad attr BinOpMod Str TupleLoad NameLoad Call NameLoad NameLoad Call NameLoad Call AttributeLoad AttributeLoad NameLoad attr attr Assign SubscriptStore NameLoad Index NameLoad Call AttributeLoad AttributeLoad NameLoad attr attr If BoolOpAnd CompareIsNot NameLoad NameLoad CompareGt NameLoad Num body Assign NameStore Call AttributeLoad NameLoad attr Num NameLoad Expr Call AttributeLoad NameLoad attr BinOpMod Str NameLoad Expr Call AttributeLoad NameLoad attr NameLoad Expr Call AttributeLoad NameLoad attr BinOpMod Str TupleLoad NameLoad Call NameLoad Call AttributeLoad AttributeLoad NameLoad attr attr If Call NameLoad NameLoad NameLoad NameLoad body Return If NameLoad body If CompareEq NameLoad Str body Assign NameStore ListComp Call NameLoad NameLoad comprehension NameStore Call NameLoad NameLoad Assign NameStore Call NameLoad Call NameLoad NameLoad Expr Call AttributeLoad NameLoad attr Assign NameStore Call AttributeLoad NameLoad attr NameLoad Str Expr Call AttributeLoad NameLoad attr Str NameLoad Assign NameStore Call AttributeLoad Call NameLoad NameLoad attr Str Str Assign TupleStore NameStore NameStore Call NameLoad NameLoad NameLoad NameLoad If CompareIs NameLoad NameLoad body Return If CompareIn NameLoad NameLoad body Expr Call AttributeLoad SubscriptLoad NameLoad Index NameLoad attr Dict Str Str NameLoad NameLoad orelse Assign SubscriptStore NameLoad Index NameLoad Dict Str Str NameLoad NameLoad Expr Call AttributeLoad NameLoad attr BinOpMod Str TupleLoad NameLoad NameLoad NameLoad NameLoad If Call NameLoad NameLoad Call AttributeLoad NameLoad attr body Expr Call NameLoad NameLoad Call AttributeLoad AttributeLoad NameLoad attr attr Call AttributeLoad NameLoad attr NameLoad NameLoad keyword NameLoad keyword NameLoad keyword NameLoad keyword NameLoad orelse Expr Call AttributeLoad NameLoad attr BinOpMod Str NameLoad Expr Call AttributeLoad NameLoad attr TupleLoad NameLoad NameLoad NameLoad NameLoad NameLoad NameLoad NameLoad If BoolOpAnd NameLoad CompareGt Call NameLoad NameLoad NameLoad Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad body Expr Call AttributeLoad NameLoad attr NameLoad decorator_list FunctionDef arguments args NameParam NameParam NameParam NameParam NameParam NameParam NameParam NameParam NameParam defaults Num NameLoad NameLoad NameLoad NameLoad NameLoad body TryExcept body If NameLoad body AugAssignAdd NameStore Str AugAssignAdd NameStore NameLoad Expr Call AttributeLoad NameLoad attr NameLoad keyword NameLoad keyword NameLoad Expr Call AttributeLoad NameLoad attr BinOpMod Str NameLoad If BoolOpAnd UnaryOpNot NameLoad CompareEq NameLoad Str NameLoad NameLoad NameLoad body Expr Call NameLoad NameLoad NameLoad NameLoad If CompareIn NameLoad NameLoad body Expr Call AttributeLoad NameLoad attr NameLoad handlers ExceptHandler type AttributeLoad NameLoad attr name NameStore body Assign NameStore AttributeLoad NameLoad attr If CompareIn NameLoad NameLoad body Expr Call AttributeLoad NameLoad attr BinOpMod Str TupleLoad NameLoad NameLoad If BoolOpAnd UnaryOpNot NameLoad CompareEq NameLoad Str NameLoad NameLoad NameLoad body Expr Call NameLoad NameLoad NameLoad NameLoad orelse If BoolOpAnd CompareIn NameLoad NameLoad CompareLt NameLoad NameLoad body AugAssignAdd NameStore Num Expr Call AttributeLoad NameLoad attr BinOpMod Str NameLoad Expr Call AttributeLoad NameLoad attr NameLoad Expr Call NameLoad NameLoad NameLoad NameLoad NameLoad NameLoad NameLoad NameLoad NameLoad orelse Expr Call AttributeLoad NameLoad attr BinOpMod Str TupleLoad NameLoad NameLoad If BoolOpAnd CompareIn NameLoad NameLoad NameLoad body If CompareNotIn NameLoad NameLoad body Expr Call AttributeLoad NameLoad attr NameLoad decorator_list FunctionDef arguments args defaults body While NameLoad body TryExcept body Expr Call AttributeLoad NameLoad attr BinOpMod Str Call NameLoad NameLoad Expr Call AttributeLoad NameLoad attr Call NameLoad NameLoad Expr Call AttributeLoad NameLoad attr Call AttributeLoad NameLoad attr Num NameLoad Expr Call AttributeLoad NameLoad attr Str If CompareLt Call AttributeLoad NameLoad attr Num Num NameLoad body Expr Call NameLoad Str Expr Call NameLoad Expr Call NameLoad Call NameLoad NameLoad handlers ExceptHandler type NameLoad body Expr Call AttributeLoad NameLoad attr Str decorator_list FunctionDef arguments args defaults body For TupleStore NameStore NameStore Call AttributeLoad NameLoad attr body TryExcept body If CompareIn NameLoad NameLoad body Assign NameStore Call AttributeLoad AttributeLoad NameLoad attr attr Assign NameStore SubscriptLoad NameLoad Index NameLoad Expr Call AttributeLoad NameLoad attr BinOpMod Str TupleLoad NameLoad Call NameLoad NameLoad Assign NameStore BinOpSub NameLoad NameLoad If CompareLtE AttributeLoad NameLoad attr NameLoad body Expr Call AttributeLoad NameLoad attr BinOpMod Str TupleLoad NameLoad NameLoad Continue Expr Call AttributeLoad NameLoad attr BinOpMod Str TupleLoad NameLoad NameLoad Assign NameStore Call AttributeLoad AttributeLoad NameLoad attr attr Assign SubscriptStore NameLoad Index Str Str Expr Call NameLoad NameLoad NameLoad Call AttributeLoad NameLoad attr NameLoad NameLoad keyword NameLoad keyword NameLoad handlers ExceptHandler body Expr Call AttributeLoad NameLoad attr Str decorator_list FunctionDef arguments args NameParam defaults body If NameLoad body Expr Call AttributeLoad NameLoad attr BinOpMod Str Call NameLoad NameLoad Assign NameStore ListComp NameLoad comprehension NameStore NameLoad UnaryOpNot Call NameLoad NameLoad Delete SubscriptDel NameLoad Slice Expr Call AttributeLoad NameLoad attr NameLoad decorator_list FunctionDef arguments args defaults body While NameLoad body Expr Call AttributeLoad NameLoad attr NameLoad Expr Call AttributeLoad NameLoad attr Str Assign NameStore Call NameLoad Call AttributeLoad NameLoad attr Expr Call AttributeLoad NameLoad attr For NameStore NameLoad body Assign NameStore Call AttributeLoad NameLoad attr NameLoad Expr Call AttributeLoad NameLoad attr Str NameLoad Call NameLoad NameLoad Call NameLoad Call AttributeLoad NameLoad attr NameLoad Expr Call AttributeLoad NameLoad attr Str decorator_list FunctionDef arguments args defaults body While NameLoad body Expr Call AttributeLoad NameLoad attr NameLoad While UnaryOpNot Call AttributeLoad NameLoad attr body Assign TupleStore NameStore NameStore NameStore NameStore NameStore NameStore NameStore Call AttributeLoad NameLoad attr If BoolOpOr CompareNotIn NameLoad NameLoad CompareGtE NameLoad SubscriptLoad SubscriptLoad NameLoad Index NameLoad Index Str body Expr Call AttributeLoad NameLoad attr TupleLoad NameLoad NameLoad NameLoad NameLoad NameLoad NameLoad NameLoad decorator_list FunctionDef arguments args defaults body Expr Str Global identifier Global identifier While NameLoad body Expr Call AttributeLoad NameLoad attr NameLoad Assign NameStore Call NameLoad Assign NameStore Call NameLoad decorator_list FunctionDef arguments args NameParam defaults body TryExcept body Expr Call AttributeLoad NameLoad attr Str Expr Call AttributeLoad NameLoad attr Num Expr Call NameLoad NameLoad Global identifier Assign NameStore NameLoad handlers ExceptHandler type NameLoad body Expr Call NameLoad Str decorator_list FunctionDef arguments args NameParam defaults body Expr Str If CompareIn NameLoad NameLoad body Expr Call AttributeLoad NameLoad attr BinOpMod Str NameLoad Return Assign NameStore Call AttributeLoad NameLoad attr NameLoad If UnaryOpNot Call AttributeLoad Call NameLoad NameLoad attr NameLoad body Expr Call AttributeLoad NameLoad attr BinOpMod Str NameLoad Return Expr Call AttributeLoad NameLoad attr BinOpMod Str NameLoad Expr Call AttributeLoad Call NameLoad NameLoad attr NameLoad NameLoad Expr Call AttributeLoad NameLoad attr NameLoad decorator_list FunctionDef arguments args NameParam defaults body If NameLoad body Expr Call NameLoad Str For NameStore NameLoad body If Call AttributeLoad NameLoad attr Str body Expr Call NameLoad NameLoad orelse If CompareIn NameLoad NameLoad body Continue Assign NameStore Call AttributeLoad NameLoad attr NameLoad If UnaryOpNot Call AttributeLoad Call NameLoad NameLoad attr NameLoad body Expr Call AttributeLoad NameLoad attr BinOpMod Str NameLoad Continue Expr Call AttributeLoad NameLoad attr BinOpMod Str NameLoad Assign NameStore Call NameLoad NameLoad NameLoad Expr Call AttributeLoad NameLoad attr Call AttributeLoad NameLoad attr NameLoad NameLoad Expr Call AttributeLoad NameLoad attr NameLoad decorator_list FunctionDef arguments args NameParam defaults vararg kwarg body Assign TupleStore NameStore NameStore Call AttributeLoad Call NameLoad NameLoad attr Call AttributeLoad NameLoad attr NameLoad Assign NameStore Call AttributeLoad NameLoad attr NameLoad Assign NameStore Call AttributeLoad NameLoad attr NameLoad NameLoad TryExcept body Assign NameStore Call AttributeLoad Call AttributeLoad NameLoad attr NameLoad NameLoad keyword NameLoad attr NameLoad handlers ExceptHandler type NameLoad name NameStore body Expr Call AttributeLoad NameLoad attr BinOpMod Str TupleLoad NameLoad NameLoad Return ExceptHandler type NameLoad name NameStore body Expr Call AttributeLoad NameLoad attr BinOpMod Str TupleLoad NameLoad NameLoad Return Assign NameStore Call AttributeLoad NameLoad attr NameLoad For NameStore NameLoad body Expr Call NameLoad NameLoad decorator_list FunctionDef arguments args NameParam defaults body Assign NameStore Call AttributeLoad NameLoad attr Str Assign NameStore Call AttributeLoad NameLoad attr Str Assign NameStore Call NameLoad Call AttributeLoad NameLoad attr Str Num Assign NameStore Call AttributeLoad Call AttributeLoad NameLoad attr Str Str attr Assign NameStore CompareEq Call AttributeLoad Call AttributeLoad NameLoad attr Str Str attr Str Return Call NameLoad NameLoad NameLoad NameLoad NameLoad NameLoad NameLoad decorator_list FunctionDef arguments args NameParam NameParam NameParam NameParam NameParam NameParam defaults Num NameLoad body Expr Str If CompareIn NameLoad NameLoad body Expr Call AttributeLoad NameLoad attr BinOpMod Str NameLoad Return NameLoad If UnaryOpNot Call NameLoad NameLoad NameLoad body Expr Call AttributeLoad NameLoad attr BinOpMod Str TupleLoad NameLoad NameLoad Expr Call AttributeLoad NameLoad attr NameLoad Return NameLoad Expr Call AttributeLoad NameLoad attr BinOpMod Str TupleLoad NameLoad NameLoad Expr Call NameLoad NameLoad NameLoad NameLoad If BoolOpAnd CompareIsNot NameLoad NameLoad CompareEq Call AttributeLoad NameLoad attr Str body Assign NameStore Call NameLoad NameLoad NameLoad orelse Assign NameStore Call NameLoad NameLoad NameLoad Expr Call AttributeLoad NameLoad attr BinOpMod Str TupleLoad NameLoad NameLoad Assign SubscriptStore NameLoad Index NameLoad NameLoad Assign SubscriptStore NameLoad Index NameLoad NameLoad If CompareEq NameLoad Str body Expr Call AttributeLoad NameLoad attr Call AttributeLoad NameLoad attr NameLoad NameLoad NameLoad NameLoad NameLoad orelse Expr Call AttributeLoad NameLoad attr Call AttributeLoad NameLoad attr NameLoad NameLoad NameLoad NameLoad NameLoad Return NameLoad decorator_list FunctionDef arguments args NameParam defaults body Global identifier Global identifier Global identifier Global identifier Global identifier Global identifier Assign NameStore Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr Assign NameStore AttributeLoad NameLoad attr Assign NameStore AttributeLoad NameLoad attr Assign NameStore AttributeLoad NameLoad attr Assign NameStore AttributeLoad NameLoad attr Assign NameStore Call AttributeLoad Str attr NameLoad NameLoad NameLoad NameLoad AttributeLoad NameLoad attr decorator_list FunctionDef arguments args defaults body Global identifier Assign NameStore Call AttributeLoad NameLoad attr keyword NameLoad Expr Call AttributeLoad NameLoad attr Str Str keyword Str keyword Str keyword NameLoad keyword Str Expr Call AttributeLoad NameLoad attr Str Str keyword Str keyword Str keyword NameLoad keyword Str Expr Call AttributeLoad NameLoad attr Str Str keyword Str keyword Str keyword NameLoad keyword Str Expr Call AttributeLoad NameLoad attr Str Str keyword Str keyword Str keyword NameLoad keyword Str Expr Call AttributeLoad NameLoad attr Str Str keyword Str keyword Str keyword NameLoad keyword Str Expr Call AttributeLoad NameLoad attr Str Str keyword Str keyword Str keyword Str keyword Str Assign NameStore Call AttributeLoad NameLoad attr Expr Call NameLoad NameLoad Assign NameStore Call AttributeLoad NameLoad attr NameLoad Assign NameStore Call AttributeLoad NameLoad attr NameLoad Assign NameStore AttributeLoad NameLoad attr Expr Call AttributeLoad NameLoad attr NameLoad NameLoad Assign NameStore Call AttributeLoad NameLoad attr NameLoad Assign NameStore Call AttributeLoad NameLoad attr NameLoad Assign NameStore Call AttributeLoad NameLoad attr NameLoad Assign SubscriptStore NameLoad Index Str NameLoad Assign SubscriptStore NameLoad Index Str NameLoad Assign SubscriptStore NameLoad Index Str NameLoad Assign SubscriptStore NameLoad Index Str NameLoad Assign SubscriptStore NameLoad Index Str NameLoad Expr Call AttributeLoad NameLoad attr Str Expr Call AttributeLoad Call AttributeLoad NameLoad attr TupleLoad Str Call NameLoad AttributeLoad NameLoad attr NameLoad attr decorator_list FunctionDef arguments args defaults body Return Str decorator_list Call AttributeLoad NameLoad attr Str FunctionDef arguments args defaults body Expr Str Expr Call AttributeLoad NameLoad attr Str Expr Call AttributeLoad NameLoad attr TupleLoad Str NameLoad NameLoad NameLoad NameLoad NameLoad NameLoad Return Str decorator_list Call AttributeLoad NameLoad attr Str FunctionDef arguments args defaults body Assign NameStore DictComp NameLoad UnaryOpNot AttributeLoad SubscriptLoad NameLoad Index NameLoad attr comprehension NameStore NameLoad Return Call AttributeLoad NameLoad attr NameLoad decorator_list Call AttributeLoad NameLoad attr Str FunctionDef arguments args NameParam NameParam NameParam defaults body Assign NameStore IfExp CompareEq NameLoad Str Call AttributeLoad SubscriptLoad NameLoad Index NameLoad attr Str Num Call AttributeLoad SubscriptLoad NameLoad Index NameLoad attr Str Num If CompareGt NameLoad NameLoad body Return If CompareEq NameLoad Str body Expr Call AttributeLoad SubscriptLoad NameLoad Index NameLoad attr Dict Str NameLoad orelse Expr Call AttributeLoad SubscriptLoad NameLoad Index NameLoad attr Dict Str NameLoad decorator_list FunctionDef arguments args defaults body Expr Str Assign NameStore Call NameLoad SubscriptLoad Call AttributeLoad AttributeLoad NameLoad attr attr Str Index Num Assign NameStore Call NameLoad SubscriptLoad Call AttributeLoad AttributeLoad NameLoad attr attr Str Index Num Assign NameStore Call NameLoad SubscriptLoad Call AttributeLoad AttributeLoad NameLoad attr attr Str Index Num Assign NameStore Call NameLoad SubscriptLoad Call AttributeLoad AttributeLoad NameLoad attr attr Str Index Num If CompareEq NameLoad Str body Assign NameStore Call NameLoad SubscriptLoad Call AttributeLoad AttributeLoad NameLoad attr attr Str Index Num Assign NameStore Call NameLoad SubscriptLoad Call AttributeLoad AttributeLoad NameLoad attr attr Str Index Num TryExcept body Expr Call AttributeLoad SubscriptLoad NameLoad Index NameLoad attr Dict Str Str Str NameLoad NameLoad NameLoad handlers ExceptHandler body Expr Call AttributeLoad NameLoad attr Call AttributeLoad Str attr NameLoad orelse Assign NameStore Call NameLoad SubscriptLoad Call AttributeLoad AttributeLoad NameLoad attr attr Str Index Num TryExcept body Expr Call AttributeLoad SubscriptLoad NameLoad Index NameLoad attr Dict Str Str NameLoad NameLoad handlers ExceptHandler body Expr Call AttributeLoad NameLoad attr Call AttributeLoad Str attr NameLoad Expr Call NameLoad NameLoad NameLoad NameLoad Return Str decorator_list Call AttributeLoad NameLoad attr Str keyword ListLoad Str FunctionDef arguments args defaults body Expr Str Assign NameStore Call NameLoad SubscriptLoad Call AttributeLoad AttributeLoad NameLoad attr attr Str Index Num Assign NameStore Call NameLoad SubscriptLoad Call AttributeLoad AttributeLoad NameLoad attr attr Str Index Num Assign NameStore Call NameLoad SubscriptLoad Call AttributeLoad AttributeLoad NameLoad attr attr Str Index Num Expr Call NameLoad NameLoad NameLoad NameLoad Return Str decorator_list Call AttributeLoad NameLoad attr Str keyword ListLoad Str FunctionDef arguments args defaults body Expr Str Assign NameStore DictComp NameLoad SubscriptLoad NameLoad Index NameLoad comprehension NameStore Call AttributeLoad NameLoad attr UnaryOpNot Call AttributeLoad NameLoad attr Str Return Call AttributeLoad NameLoad attr NameLoad decorator_list Call AttributeLoad NameLoad attr Str FunctionDef arguments args NameParam defaults body Return Call AttributeLoad NameLoad attr Call AttributeLoad NameLoad attr BinOpAdd Str NameLoad decorator_list Call AttributeLoad NameLoad attr Str FunctionDef arguments args defaults body Return Call AttributeLoad NameLoad attr NameLoad decorator_list Call AttributeLoad NameLoad attr Str FunctionDef arguments args NameParam defaults body Return Call AttributeLoad NameLoad attr Call AttributeLoad NameLoad attr BinOpAdd Str NameLoad decorator_list Call AttributeLoad NameLoad attr Str FunctionDef arguments args defaults body Return Call AttributeLoad NameLoad attr Call NameLoad NameLoad decorator_list Call AttributeLoad NameLoad attr Str FunctionDef arguments args defaults body If UnaryOpNot NameLoad body Assign NameStore Dict Str Num Assign NameStore Call NameLoad NameLoad Assign AttributeStore NameLoad attr Num orelse Assign NameStore Dict Str Num Assign NameStore Call NameLoad NameLoad Assign AttributeStore NameLoad attr Num Return NameLoad decorator_list Call AttributeLoad NameLoad attr Str FunctionDef arguments args defaults body Assign NameStore NameLoad For NameStore NameLoad body If AttributeLoad SubscriptLoad NameLoad Index NameLoad attr body Assign NameStore NameLoad Break If UnaryOpNot NameLoad body Assign NameStore Dict Str Num Assign NameStore Call NameLoad NameLoad Assign AttributeStore NameLoad attr Num orelse If UnaryOpNot NameLoad body Assign NameStore Dict Str Num Assign NameStore Call NameLoad NameLoad Assign AttributeStore NameLoad attr Num orelse Assign NameStore Dict Str Num Assign NameStore Call NameLoad NameLoad Assign AttributeStore NameLoad attr Num Return NameLoad decorator_list Call AttributeLoad NameLoad attr Str FunctionDef arguments args defaults body Assign NameStore Call NameLoad SubscriptLoad Call AttributeLoad AttributeLoad NameLoad attr attr Str Index Num Assign NameStore Call NameLoad SubscriptLoad Call AttributeLoad AttributeLoad NameLoad attr attr Str Index Num If CompareEq NameLoad Str body Expr Call AttributeLoad NameLoad attr NameLoad orelse Expr Call AttributeLoad NameLoad attr NameLoad Return Str decorator_list Call AttributeLoad NameLoad attr Str keyword ListLoad Str FunctionDef arguments args defaults body Assign NameStore Call NameLoad SubscriptLoad Call AttributeLoad AttributeLoad NameLoad attr attr Str Index Num If CompareIn NameLoad NameLoad body Return Str orelse If CompareIn NameLoad NameLoad body Return Str orelse Return Str decorator_list Call AttributeLoad NameLoad attr Str keyword ListLoad Str FunctionDef arguments args defaults body Return Call AttributeLoad NameLoad attr Call NameLoad NameLoad decorator_list Call AttributeLoad NameLoad attr Str FunctionDef arguments args defaults body Return Call AttributeLoad NameLoad attr Call NameLoad NameLoad decorator_list Call AttributeLoad NameLoad attr Str FunctionDef arguments args defaults body Return Call AttributeLoad NameLoad attr Call NameLoad NameLoad decorator_list Call AttributeLoad NameLoad attr Str FunctionDef arguments args defaults body Return Call AttributeLoad NameLoad attr Call NameLoad NameLoad decorator_list Call AttributeLoad NameLoad attr Str Assign NameStore Num FunctionDef arguments args defaults body Global identifier AugAssignAdd NameStore Num Return Str decorator_list Call AttributeLoad NameLoad attr Str If CompareEq NameLoad Str body Expr Call NameLoad keyword Str Expr Call NameLoad
N/A N/A ZK_UPDATE_MONITOR (ZUM): A daemon program that monitors changes in zookeeper.

The module runs as a daemon to monitor zookeeper paths specified in a config
file, and invokes a command if a change to the path in zookeeper is observed.
It monitors zookeeper by using DataWatch in the ``kazoo_utils`` library.

There are 2 types of data zk_update_montior can watch, serverset and config.
Serveset is a group of Zookeeper Epemeral nodes whose name are endpoints of services.
Config are decider/manageddata whose values are stored in S3 and use Zookeeper as the notification
system to notify ZUM the updates.

The following is an example "MetaConfig" JSON which tells ZK_UPDATE_MONITOR to watch a serverset,
and fork zk_download_data.py when the content of a watched serverset is changed:

 [
    {
        "config_section_name": "discovery.service.prod",
        "zk_path": "/discovery/service/prod",
        "command": "zk_download_data.py -f /var/serverset/discovery.service.prod -p /discovery/service/prod -m serverset",
        "type": "serverset",
        "max_wait_in_secs": 0
    }
 ]


This is an example "MetaConfig" JSON which tells ZK_UPDATE_MONITOR to watch a configuration (manageddata)
and fork zk_download_data.py when the content of a watched mangeddata is changed. It downloads the data from S3.

 [
    {
        "config_section_name": "config.manageddata.spam.test_grey_list",
        "zk_path": "/config/manageddata/spam/test_grey_list",
        "command": "zk_download_data.py -f /var/config/config.manageddata.spam.test_grey_list --from-s3 /data/config/manageddata/spam/test_grey_list -m config -p /config/manageddata/spam/test_grey_list",
        "type": "config",
        "max_wait_in_secs": 0
    }
 ]

The ``max_wait_in_seconds`` specifies the max number of seconds the the daemon
waits before invoking the command. The actual wait time is a random number of
seconds between 0 to the ``max_wait_in_secs`` value. This random waiting is for preventing many
clients catch the same zookeeper change and run the command at the same time
that could possibly cause a service outage.

The MetaConfig can be created using in config_meta_manager.

How to launch zk_update_monitor:
See details in Kingpin README.md or comments in config_meta_manager.


How to monitor zk_update_monitor:
ZUM locally runs a flask that outsiders can query the status of the daemon. You can simply curl the local URLS
to get the information.

 gevent monkey N/A N/A N/A monkey patch_all N/A argparse N/A datetime N/A functools N/A logging N/A os N/A random N/A socket N/A string N/A N/A N/A simplejson json N/A N/A N/A ImportError N/A N/A json N/A tempfile N/A time flask Flask flask request flask jsonify N/A gevent gevent pywsgi N/A gevent_subprocess subprocess kingpin.config_utils s3config kingpin.kazoo_utils DataWatcher ServerSet kingpin.logging_utils initialize_logger kingpin.zk_update_monitor zk_util kingpin.zk_update_monitor.zk_util _kazoo_client _zk_path_exists _kill N/A log N/A N/A logging getLogger __name__ N/A app N/A Flask __name__ N/A WAKEUP_FREQUENCY_IN_SECS N/A N/A os getenv ZK_UPDATE_MONITOR_WAKEUP_FREQUENCY_IN_SECS 1800 N/A MAX_JITTER_TIME_IN_SECS N/A N/A os getenv ZK_UPDATE_MONITOR_MAX_JITTER_TIME_IN_SECS 180 N/A RESTART_PROBABILITY 0.04 N/A _PATH_TO_DATA N/A N/A _PATH_TO_COMMAND N/A N/A _SERVERSET_METADATA N/A N/A _COMMAND_TO_NOTIFICATION_TIME N/A N/A _CONCURRENT_MIN_WINDOW_IN_SECS 60 N/A _IGNORED_RET_CODES N/A 123 N/A _RETRY_RET_CODES N/A 1 9 101 102 N/A _MAX_RETRY_COUNT 1 N/A _RETRY_WAIT_TIME_IN_SECONDS 3 N/A _TOO_FEW_SERVERS_IN_ZK_CODE 3 N/A _SERVER_COUNT_DROP_BELOW_THRESHOLD 30 N/A _UNHANDLED_EXCPETION 7 N/A _SERVERSET_REJECTION_ERROR_CODES N/A 3 30 N/A _NOTIFICATION_EVENT_QUEUE N/A N/A N/A gevent queue Queue N/A _NOTIFICATION_HOLDDOWN_QUEUE N/A N/A N/A gevent queue Queue N/A _NOTIFICATION_HOLDDOWN_WIPER_SLEEP_INTERVAL_IN_SECONDS 10 N/A _PATH_TO_ALERT_DISABLED N/A N/A _HOST_NAME N/A N/A socket gethostname N/A _CONFIGS_WITH_NONEXISTENT_PATH N/A N/A _DUMP_COMMAND_PROCESSING_INFO_INTERVAL_IN_SECONDS 60 N/A _LONG_VALUE_THRESHOLD 100000 N/A _GREENLET_DICT N/A N/A _INITIALIZATION_COMPLETED False N/A _REJECTED_UPDATED_SERVERSETS N/A set N/A _SKIP_FLAPPY_CHECK_SERVERSETS N/A set N/A _NON_SKIP_FLAPPY_CHECK_SERVERSETS N/A set N/A _FLAPPY_CHECK_SERVERSETS_TTL N/A 30 60 N/A _DOWNLOAD_PROCESS_SPAWN_INTERVAL_SECONDS 1 N/A ZK_HOSTS None N/A ZK_HOSTS_FILE None N/A S3_BUCKET None N/A AWS_KEY_FILE None N/A S3_ENDPOINT None N/A METACONFIG_ZK_PATH_FORMAT /metaconfig/metaconfig/{} N/A DEPENDENCY_ZK_PATH_FORMAT /metaconfig/dependency/{} N/A METACONFIG_S3_KEY_FORMAT /data/metaconfig/metaconfig/{} N/A _WATCHED_DEPENDENCIES N/A set N/A _WATCHED_METACONFIGS N/A set N/A EXTRA_FLAGS_FOR_ZK_DOWNLOAD_DATA None N/A _CONFIGV3_INITIALIZED False update_serverset_metadata N/A N/A zk_path notification_timestamp value N/A N/A N/A hashsum N/A N/A zk_util get_md5_hash_sum value N/A N/A N/A num_endpoints N/A len N/A N/A value split , N/A N/A N/A N/A num_endpoints -1 N/A metadata N/A hashsum num_endpoints notification_timestamp hashsum num_endpoints notification_timestamp N/A N/A zk_path _SERVERSET_METADATA N/A N/A N/A N/A N/A _SERVERSET_METADATA N/A zk_path update metadata N/A N/A N/A _SERVERSET_METADATA N/A zk_path metadata N/A _check_download_interval N/A N/A zk_path current_time N/A N/A N/A last_processed_timestamp N/A N/A N/A _PATH_TO_DATA N/A zk_path get last_processed_timestamp N/A N/A N/A last_processed_timestamp N/A N/A current_time last_processed_timestamp _DOWNLOAD_PROCESS_SPAWN_INTERVAL_SECONDS N/A N/A N/A N/A N/A _PATH_TO_DATA N/A zk_path update N/A last_processed_timestamp current_time N/A True N/A False N/A _is_older_version N/A N/A zk_path version notification_timestamp N/A N/A N/A data N/A N/A _PATH_TO_DATA get zk_path N/A version N/A N/A N/A data N/A N/A N/A data get version -1 version N/A N/A N/A N/A log info N/A Queued notification event has version %d which is not newer than the current version %d, ignore N/A version N/A data N/A version N/A True N/A N/A N/A data N/A N/A N/A data get notification_timestamp 0 notification_timestamp N/A N/A N/A N/A log info N/A Queued notification event has a timestamp %f which is not newer than the current timestamp %f, ignore N/A notification_timestamp N/A data N/A notification_timestamp N/A True N/A False N/A transform_command_with_value N/A N/A command value notification_timestamp N/A N/A N/A python_download_script zk_download_data.py N/A N/A N/A len value _LONG_VALUE_THRESHOLD N/A N/A value N/A N/A N/A N/A value replace replace N/A md5digest N/A N/A zk_util get_md5_digest value N/A tmp_filename N/A N/A N/A zk_update_largefile_ md5digest _ N/A str notification_timestamp N/A tmp_dir N/A N/A tempfile gettempprefix N/A tmp_filepath N/A N/A N/A os path join / tmp_dir tmp_filename N/A N/A N/A log info This is a long value, write it to temp file %s tmp_filepath N/A N/A N/A N/A N/A N/A open tmp_filepath w f N/A N/A N/A N/A f write N/A N/A value md5digest N/A N/A N/A Exception N/A e N/A N/A N/A N/A log exception N/A %s: Failed to generate temp file %s for storing large size values N/A N/A e message tmp_filepath N/A N/A None None N/A N/A N/A N/A f close N/A transformed_command N/A N/A command replace python_download_script N/A %s -l %s N/A python_download_script tmp_filepath N/A N/A transformed_command tmp_filepath N/A N/A transformed_command N/A N/A command replace python_download_script N/A %s -v '%s' N/A python_download_script value N/A N/A transformed_command None N/A _get_local_file_path_from_command N/A N/A command N/A N/A N/A N/A N/A parts N/A N/A command split N/A N/A parts N/A N/A N/A N/A parts index -f 1 N/A N/A N/A N/A N/A N/A log error N/A Error getting local file path for %s command N/A None N/A _get_and_set_local_file_modification_time N/A N/A zk_path command watch_type N/A N/A N/A _PATH_TO_DATA N/A _SERVERSET_METADATA N/A local_file_path N/A _get_local_file_path_from_command command N/A N/A local_file_path N/A N/A modification_timestamp 0 N/A N/A N/A N/A modification_timestamp N/A N/A N/A os path getmtime local_file_path N/A N/A N/A N/A modification_timestamp 0 N/A N/A watch_type serverset N/A N/A N/A _SERVERSET_METADATA N/A zk_path N/A modification_timestamp modification_timestamp N/A N/A N/A _PATH_TO_DATA N/A zk_path N/A modification_timestamp modification_timestamp N/A _run_command N/A N/A zk_path command max_wait_in_secs watch_type value stat N/A None None N/A N/A  Invoking the shell command
     N/A N/A N/A log debug N/A Invoking command '%s' command N/A _PATH_TO_DATA N/A data N/A N/A _PATH_TO_DATA get zk_path None N/A N/A data stat N/A N/A N/A data get version -1 N/A stat version N/A N/A N/A N/A log info N/A Version %d is not newer than the current version %d, ignore N/A N/A stat version N/A data N/A version N/A N/A N/A stat N/A N/A version N/A stat version N/A N/A version None N/A notification_timestamp N/A N/A time time N/A N/A N/A log info N/A Notification is queued for running command '%s' with version %s and event timestamp %f N/A command N/A str version notification_timestamp N/A N/A N/A _NOTIFICATION_EVENT_QUEUE put N/A zk_path command value version max_wait_in_secs watch_type notification_timestamp N/A _notification_processor N/A N/A N/A N/A N/A True N/A N/A N/A zk_path command value version max_wait_in_secs watch_type notification_timestamp N/A N/A _NOTIFICATION_EVENT_QUEUE get N/A N/A zk_path kill N/A N/A N/A _kill Restart via kill api N/A N/A _is_older_version zk_path version notification_timestamp N/A N/A N/A N/A N/A gevent spawn _process_notification command value version max_wait_in_secs watch_type zk_path notification_timestamp N/A _process_notification N/A N/A command value version max_wait_in_secs watch_type zk_path notification_timestamp N/A N/A N/A N/A N/A log info N/A Processing notification event for command %s received at %f N/A command notification_timestamp N/A N/A N/A log info N/A ZkPath: %s, Version: %s, Time: %s N/A zk_path N/A str version N/A str N/A N/A N/A datetime datetime now N/A N/A _COMMAND_TO_NOTIFICATION_TIME N/A command N/A N/A N/A datetime datetime now N/A N/A N/A max_wait_in_secs None N/A max_wait_in_secs 0 N/A N/A wait_in_seconds N/A N/A random randint 0 max_wait_in_secs N/A N/A N/A log info N/A Sleeping %d seconds wait_in_seconds N/A N/A N/A gevent sleep wait_in_seconds N/A N/A N/A log debug N/A Command %s is executed at %s N/A command N/A str N/A N/A N/A datetime datetime now N/A N/A _is_older_version zk_path version notification_timestamp N/A N/A N/A value N/A N/A N/A watch_type serverset N/A N/A value N/A N/A str server N/A server N/A list value N/A value N/A list N/A set value N/A N/A N/A value sort N/A value N/A N/A string join value , N/A N/A N/A log info Trying to pass value to the download command %s command N/A value N/A N/A N/A str value replace ' \' N/A N/A command tmp_filepath N/A transform_command_with_value command value notification_timestamp N/A N/A command None N/A N/A N/A N/A zk_path _PATH_TO_DATA N/A N/A N/A N/A N/A _PATH_TO_DATA N/A zk_path update N/A version notification_timestamp version notification_timestamp N/A N/A N/A _PATH_TO_DATA N/A zk_path N/A version notification_timestamp version notification_timestamp N/A N/A N/A log debug N/A zk_path: %s value updated to value %s, version %s, timestamp %f N/A zk_path value version notification_timestamp N/A N/A _check_download_interval zk_path N/A N/A time time N/A N/A N/A _run_command_and_check_ret_code command N/A N/A N/A os environ copy N/A N/A _PATH_TO_ALERT_DISABLED get zk_path False watch_type watch_type zk_path zk_path notification_timestamp notification_timestamp value value N/A N/A N/A N/A log warn N/A Rate limit for %s zk_path N/A N/A N/A _NOTIFICATION_HOLDDOWN_QUEUE put N/A zk_path command value version max_wait_in_secs watch_type notification_timestamp N/A N/A value N/A N/A len value _LONG_VALUE_THRESHOLD N/A N/A N/A os path isfile tmp_filepath N/A N/A N/A N/A os remove tmp_filepath N/A _run_command_and_check_ret_code N/A N/A command env alert_disabled retry_count watch_type zk_path notification_timestamp value downloader_report_metadadata N/A 0 None None None None False N/A N/A N/A N/A downloader_report_metadadata N/A N/A command  --report-metadata N/A command EXTRA_FLAGS_FOR_ZK_DOWNLOAD_DATA N/A N/A N/A subprocess check_call command shell True env env N/A N/A N/A log debug N/A Command %s is executed successfully. command N/A N/A N/A downloader_report_metadadata N/A watch_type serverset zk_path notification_timestamp value N/A N/A N/A update_serverset_metadata zk_path notification_timestamp value N/A N/A zk_path _REJECTED_UPDATED_SERVERSETS N/A N/A N/A N/A _REJECTED_UPDATED_SERVERSETS remove zk_path N/A N/A N/A N/A subprocess CalledProcessError N/A e N/A N/A error_code N/A e returncode N/A N/A error_code _IGNORED_RET_CODES N/A N/A N/A N/A log debug N/A Got return code '%d' for command '%s' and ignored N/A error_code command N/A N/A N/A downloader_report_metadadata N/A watch_type serverset zk_path notification_timestamp value N/A N/A N/A update_serverset_metadata zk_path notification_timestamp value N/A N/A N/A N/A error_code _RETRY_RET_CODES N/A retry_count _MAX_RETRY_COUNT N/A N/A retry_count 1 N/A N/A N/A log info N/A Retry command after %d seconds _RETRY_WAIT_TIME_IN_SECONDS N/A N/A N/A gevent sleep _RETRY_WAIT_TIME_IN_SECONDS N/A N/A _run_command_and_check_ret_code command env alert_disabled retry_count watch_type zk_path notification_timestamp value N/A N/A N/A N/A log error N/A Got return code %d for command '%s' N/A error_code command N/A N/A N/A error_code _SERVERSET_REJECTION_ERROR_CODES zk_path N/A N/A N/A zk_path _REJECTED_UPDATED_SERVERSETS N/A N/A N/A N/A _REJECTED_UPDATED_SERVERSETS add zk_path N/A _periodic_checking N/A N/A N/A N/A N/A True N/A N/A N/A N/A N/A N/A log info N/A sleeping... for %s seconds N/A str WAKEUP_FREQUENCY_IN_SECS N/A N/A N/A gevent sleep N/A float WAKEUP_FREQUENCY_IN_SECS N/A N/A N/A gevent sleep N/A N/A random randint 0 MAX_JITTER_TIME_IN_SECS N/A N/A N/A log info Executing all associated commands... N/A N/A N/A N/A random uniform 0 1 RESTART_PROBABILITY N/A N/A N/A _kill Restart from periodic_checking N/A N/A _refresh_all_commands N/A N/A _retry_nonexistent_zk_paths N/A list _CONFIGS_WITH_NONEXISTENT_PATH N/A N/A N/A Exception N/A N/A N/A N/A log exception Periodic checking error N/A _refresh_all_commands N/A N/A N/A N/A N/A N/A zk_path command N/A N/A _PATH_TO_COMMAND iteritems N/A N/A N/A N/A N/A command _COMMAND_TO_NOTIFICATION_TIME N/A N/A now N/A N/A N/A datetime datetime now N/A previous_start N/A _COMMAND_TO_NOTIFICATION_TIME N/A command N/A N/A N/A log debug N/A Command %s was last started at %s N/A command N/A str previous_start N/A duration N/A now previous_start N/A N/A N/A duration seconds _CONCURRENT_MIN_WINDOW_IN_SECS N/A N/A N/A N/A log info N/A It is not more %d seconds after command %s was executed, ignore N/A _CONCURRENT_MIN_WINDOW_IN_SECS command N/A N/A N/A N/A log info N/A It is time to execute the command %s for path %s N/A command zk_path N/A env N/A N/A N/A os environ copy N/A N/A env N/A __ZK_UPDATE_MONITOR_REFRESH__ True N/A N/A _run_command_and_check_ret_code command env N/A N/A _PATH_TO_ALERT_DISABLED get zk_path False zk_path zk_path downloader_report_metadadata True N/A N/A N/A N/A N/A N/A log exception Refresh commands exception N/A _retry_nonexistent_zk_paths N/A N/A configs N/A N/A N/A configs N/A N/A N/A N/A log info N/A Retrying %d configs with non-existent watched zookeeper path N/A len configs N/A failed N/A x N/A x configs N/A N/A _place_watch_from_metaconfig x N/A N/A _CONFIGS_WITH_NONEXISTENT_PATH N/A N/A N/A N/A _CONFIGS_WITH_NONEXISTENT_PATH extend failed N/A _dump_command_processing_info N/A N/A N/A N/A N/A True N/A N/A N/A N/A gevent sleep _DUMP_COMMAND_PROCESSING_INFO_INTERVAL_IN_SECONDS N/A N/A N/A log info ##### COMMAND PROCESSING INFO ##### N/A paths N/A list N/A N/A _PATH_TO_COMMAND keys N/A N/A N/A paths sort N/A path paths N/A N/A command N/A N/A _PATH_TO_COMMAND get path N/A N/A N/A log info %s: %s was executed at %s path N/A str command N/A str N/A N/A _COMMAND_TO_NOTIFICATION_TIME get command N/A N/A N/A log info ############## DONE ############### N/A _holddown_queue_wiper N/A N/A N/A N/A N/A True N/A N/A N/A N/A gevent sleep _NOTIFICATION_HOLDDOWN_WIPER_SLEEP_INTERVAL_IN_SECONDS N/A N/A N/A N/A _NOTIFICATION_HOLDDOWN_QUEUE empty N/A N/A N/A zk_path command value version max_wait_in_secs watch_type notification_timestamp N/A N/A _NOTIFICATION_HOLDDOWN_QUEUE get N/A N/A N/A zk_path _PATH_TO_DATA N/A notification_timestamp N/A N/A _PATH_TO_DATA N/A zk_path N/A notification_timestamp N/A N/A N/A N/A _NOTIFICATION_EVENT_QUEUE put N/A zk_path command value version max_wait_in_secs watch_type notification_timestamp N/A _serverset_type_set_wiper N/A N/A N/A N/A N/A 
    This is a greenlet which invalidated the cache of serverset_type
    every 30 min
     N/A _SKIP_FLAPPY_CHECK_SERVERSETS N/A _NON_SKIP_FLAPPY_CHECK_SERVERSETS N/A True N/A N/A N/A N/A gevent sleep _FLAPPY_CHECK_SERVERSETS_TTL N/A _SKIP_FLAPPY_CHECK_SERVERSETS N/A set N/A _NON_SKIP_FLAPPY_CHECK_SERVERSETS N/A set N/A gevent_load_metaconfigs N/A N/A dependency N/A N/A N/A N/A N/A N/A N/A log info Sleep 3 seconds for FLASK to start before actually loading N/A N/A N/A gevent sleep 3 N/A N/A _load_metaconfigs_from_one_dependency dependency N/A _INITIALIZATION_COMPLETED N/A _INITIALIZATION_COMPLETED True N/A N/A N/A BaseException N/A N/A N/A _kill Restart because initialization error N/A _load_metaconfigs_from_one_dependency N/A N/A dependency_name N/A N/A N/A  Load the metaconfigs from one dependency and finally from S3
    It does 3 things:
    1. Set children watch on dependencies. If the root dependency contains other
     dependencies, set watchers.
    2. Set znode watch on each MetaConfig Znode
    3. Load the MetaConfig content from S3 using S3Utils.

    If dependency/metaconfig does not exist, log and skip.
     N/A N/A dependency_name _WATCHED_DEPENDENCIES N/A N/A N/A N/A log info N/A Dependency %s already watched dependency_name N/A N/A dependency_zk_path N/A N/A DEPENDENCY_ZK_PATH_FORMAT format dependency_name N/A N/A N/A N/A N/A _kazoo_client ZK_HOSTS exists dependency_zk_path N/A N/A N/A N/A log error N/A The dependency %s does not exist dependency_zk_path N/A N/A N/A N/A log info N/A Watching dependency %s dependency_name N/A N/A N/A N/A _kazoo_client ZK_HOSTS ChildrenWatch dependency_zk_path update_dependency N/A N/A N/A _WATCHED_DEPENDENCIES add dependency_name N/A update_dependency N/A N/A dependents N/A N/A N/A _INITIALIZATION_COMPLETED N/A N/A N/A _kill Watched dependency changed, restart ZUM to catch the change N/A dependent dependents N/A N/A N/A N/A dependent endswith .dep N/A N/A N/A _load_metaconfigs_from_one_dependency dependent N/A N/A N/A dependent _WATCHED_METACONFIGS N/A N/A N/A metaconfig_zk_path N/A N/A METACONFIG_ZK_PATH_FORMAT format dependent N/A N/A N/A N/A N/A _kazoo_client ZK_HOSTS exists metaconfig_zk_path N/A N/A N/A N/A log error N/A The metaconfig %s does not exist dependent N/A N/A N/A N/A log info N/A Watching Metaconfig %s dependent N/A watcher N/A DataWatcher metaconfig_zk_path ZK_HOSTS N/A N/A N/A watcher watch N/A N/A functools partial update_metaconfig dependent N/A N/A N/A _WATCHED_METACONFIGS add dependent N/A update_metaconfig N/A N/A metaconfig_name N/A args kwargs N/A N/A N/A zk_value version N/A N/A N/A _kazoo_client ZK_HOSTS get N/A N/A METACONFIG_ZK_PATH_FORMAT format metaconfig_name N/A s3_key N/A N/A METACONFIG_S3_KEY_FORMAT format metaconfig_name N/A s3_path N/A N/A zk_util construct_s3_path s3_key zk_value N/A N/A N/A metaconfig_data N/A N/A N/A N/A s3config S3Config AWS_KEY_FILE S3_BUCKET s3_endpoint S3_ENDPOINT get_config_string s3_path N/A N/A N/A ValueError N/A ve N/A N/A N/A N/A log error N/A Abort downloading from s3 key %s due to ValueError: %s N/A s3_path ve N/A N/A N/A Exception N/A e N/A N/A N/A N/A log error N/A Abort downloading from s3 key %s due to unexpected s3 exception: %s N/A s3_path e N/A N/A metaconfig_list N/A N/A json loads metaconfig_data N/A metaconfig metaconfig_list N/A N/A N/A _place_watch_from_metaconfig metaconfig N/A _place_watch_from_metaconfig N/A N/A metaconfig N/A N/A N/A zk_path N/A N/A metaconfig get zk_path N/A command N/A N/A metaconfig get command N/A max_wait_in_secs N/A int N/A N/A metaconfig get max_wait_in_secs 0 N/A watch_type N/A N/A N/A N/A metaconfig get type config lower N/A alert_disabled N/A N/A N/A N/A N/A metaconfig get alert_disabled False lower true N/A N/A _place_watch metaconfig zk_path command watch_type max_wait_in_secs alert_disabled N/A _place_watch N/A N/A config zk_path command watch_type max_wait_in_secs alert_disabled N/A 0 False N/A N/A 
    Place the watch for Config or Serverset giving the information extracted
    from metaconfig.
     N/A N/A zk_path _PATH_TO_COMMAND N/A N/A N/A N/A log warn N/A Path %s has already been added to PATH_TO_COMMAND zk_path N/A False N/A N/A N/A _zk_path_exists ZK_HOSTS zk_path N/A N/A N/A N/A log error N/A zk_path %s does not exist in zk_hosts %s, no watch is set. N/A zk_path ZK_HOSTS N/A N/A N/A _CONFIGS_WITH_NONEXISTENT_PATH append config N/A False N/A N/A N/A log info N/A Creating Zookeeper data watcher for path %s of zk_hosts %s N/A zk_path ZK_HOSTS N/A N/A _get_and_set_local_file_modification_time zk_path command watch_type N/A N/A N/A watch_type None N/A N/A N/A watch_type lower serverset N/A N/A watcher N/A ServerSet zk_path ZK_HOSTS N/A N/A watcher N/A DataWatcher zk_path ZK_HOSTS N/A N/A N/A log info N/A Associating command '%s' with watcher of path %s N/A command zk_path N/A N/A _PATH_TO_COMMAND N/A zk_path command N/A N/A _PATH_TO_ALERT_DISABLED N/A zk_path alert_disabled N/A N/A watch_type serverset N/A N/A N/A N/A watcher monitor N/A N/A functools partial _run_command zk_path command max_wait_in_secs watch_type N/A N/A N/A N/A watcher watch N/A N/A functools partial _run_command zk_path command max_wait_in_secs watch_type N/A True N/A initialize_zookeeper_aws_s3_configs N/A N/A args N/A N/A N/A ZK_HOSTS N/A ZK_HOSTS_FILE N/A AWS_KEY_FILE N/A S3_BUCKET N/A S3_ENDPOINT N/A EXTRA_FLAGS_FOR_ZK_DOWNLOAD_DATA N/A ZK_HOSTS N/A N/A zk_util parse_zk_hosts_file N/A args zk_hosts_file N/A ZK_HOSTS_FILE N/A args zk_hosts_file N/A AWS_KEY_FILE N/A args aws_key_file N/A S3_ENDPOINT N/A args s3_endpoint N/A S3_BUCKET N/A args s3_bucket N/A EXTRA_FLAGS_FOR_ZK_DOWNLOAD_DATA N/A N/A  -z {} -a {} -b {} -e {} -o {} format ZK_HOSTS_FILE AWS_KEY_FILE S3_BUCKET S3_ENDPOINT N/A args port N/A main N/A N/A N/A N/A N/A args N/A parser N/A N/A argparse ArgumentParser description __doc__ N/A N/A N/A parser add_argument -d --dependency dest dependency metavar DEPENDENCY required True help The root dependency name, which stores what config/serversets are needed to be watched by ZUM. The dependency can have either config/serverset name, or other dependencies. N/A N/A N/A parser add_argument -p --port dest port metavar PORT required True help The port Flask app will be running on. N/A N/A N/A parser add_argument -z --zk-hosts-file-path dest zk_hosts_file metavar ZKHOSTS required True help The path of file which have a list of Zookeeper endpoints (host:port) which keeps the metaconfig as well as the config/serversets N/A N/A N/A parser add_argument -a --aws-key-file dest aws_key_file metavar AWSKEY required True help The path of the file storing AWS access and secret keys N/A N/A N/A parser add_argument -b --s3-bucket dest s3_bucket metavar BUCKET required True help The S3 bucket storing metaconfigs / configs N/A N/A N/A parser add_argument -e --aws-s3-endpoint dest s3_endpoint metavar ENDPOINT default s3.amazonaws.com help The S3 endpoint storing metaconfig / configs N/A args N/A N/A parser parse_args N/A N/A initialize_zookeeper_aws_s3_configs args N/A notification_processor N/A N/A gevent spawn _notification_processor N/A dump_command_processing_info N/A N/A gevent spawn _dump_command_processing_info N/A dependency N/A args dependency N/A N/A N/A gevent spawn gevent_load_metaconfigs dependency N/A periodic_checking N/A N/A gevent spawn _periodic_checking N/A holddown_queue_wiper N/A N/A gevent spawn _holddown_queue_wiper N/A serverset_type_set_wiper N/A N/A gevent spawn _serverset_type_set_wiper N/A N/A _GREENLET_DICT N/A notification_processor notification_processor N/A N/A _GREENLET_DICT N/A dump_command_processing_info dump_command_processing_info N/A N/A _GREENLET_DICT N/A periodic_checking periodic_checking N/A N/A _GREENLET_DICT N/A holddown_queue_wiper holddown_queue_wiper N/A N/A _GREENLET_DICT N/A serverset_type_set_wiper serverset_type_set_wiper N/A N/A N/A log info starting a Flask WSGI server for stats reporting N/A N/A N/A N/A N/A pywsgi WSGIServer N/A 0.0.0.0 N/A int N/A args port app serve_forever N/A ping N/A N/A N/A N/A N/A Ping!
 N/A N/A N/A app route /admin/ping restart N/A N/A N/A N/A N/A  Kill the running of ZK update monitor and restart
     N/A N/A N/A log info Killed via flask API /admin/kill N/A N/A N/A _NOTIFICATION_EVENT_QUEUE put N/A kill None None None None None None N/A N/A N/A N/A app route /admin/kill greenlet_check N/A N/A N/A N/A N/A status_dict N/A name N/A N/A N/A _GREENLET_DICT N/A name dead N/A name _GREENLET_DICT N/A N/A N/A json dumps status_dict N/A N/A N/A app route /admin/greenlet _update_modification_timestamp_for_flask N/A N/A mode zk_path modification_timestamp N/A N/A N/A current_modification_timestamp N/A N/A mode CONFIG N/A N/A N/A _PATH_TO_DATA N/A zk_path get modification_timestamp 0 N/A N/A N/A _SERVERSET_METADATA N/A zk_path get modification_timestamp 0 N/A N/A current_modification_timestamp modification_timestamp N/A N/A N/A N/A mode SERVERSET N/A N/A N/A N/A N/A _SERVERSET_METADATA N/A zk_path update N/A modification_timestamp modification_timestamp N/A N/A N/A N/A N/A _PATH_TO_DATA N/A zk_path update N/A modification_timestamp modification_timestamp N/A report_metadata N/A N/A N/A N/A N/A  Internal endpoint for zk_download_data to report metadata
     N/A zk_path N/A str N/A N/A N/A N/A request form getlist zk_path N/A 0 N/A mode N/A str N/A N/A N/A N/A request form getlist mode N/A 0 N/A notification_timestamp N/A float N/A N/A N/A N/A request form getlist notification_timestamp N/A 0 N/A modification_timestamp N/A float N/A N/A N/A N/A request form getlist modification_timestamp N/A 0 N/A N/A mode SERVERSET N/A N/A hashsum N/A int N/A N/A N/A N/A request form getlist hashsum N/A 0 N/A num_endpoints N/A int N/A N/A N/A N/A request form getlist num_endpoints N/A 0 N/A N/A N/A N/A N/A N/A _SERVERSET_METADATA N/A zk_path update N/A hashsum num_endpoints notification_timestamp hashsum num_endpoints notification_timestamp N/A N/A N/A N/A N/A N/A log error N/A N/A Serverset {} is not loaded into dict but metadata is being reported format zk_path N/A N/A version N/A int N/A N/A N/A N/A request form getlist version N/A 0 N/A N/A N/A N/A N/A N/A _PATH_TO_DATA N/A zk_path update N/A version notification_timestamp version notification_timestamp N/A N/A N/A N/A N/A N/A log error N/A N/A Config {} is not loaded into dict but metadata is being reported format zk_path N/A N/A _update_modification_timestamp_for_flask mode zk_path modification_timestamp N/A N/A N/A N/A app route /admin/report_metadata methods N/A POST report_file_modification N/A N/A N/A N/A N/A  When zk_download_data actual write to the local file,
    it reports the time to this endpoint. This is the actual time
    when file get changed.
     N/A zk_path N/A str N/A N/A N/A N/A request form getlist zk_path N/A 0 N/A mode N/A str N/A N/A N/A N/A request form getlist mode N/A 0 N/A modification_timestamp N/A float N/A N/A N/A N/A request form getlist modification_timestamp N/A 0 N/A N/A _update_modification_timestamp_for_flask mode zk_path modification_timestamp N/A N/A N/A N/A app route /admin/report_file_modification methods N/A POST flask_get_configs N/A N/A N/A N/A N/A _PATH_TO_DATA also contains serverset paths,
    filter them out so only configs are exported here,
    to get data please use /serverset/ endpoint
     N/A filtered_config_metadata N/A filtered_zk_path N/A _PATH_TO_DATA N/A filtered_zk_path N/A filtered_zk_path N/A N/A _PATH_TO_DATA keys N/A N/A N/A filtered_zk_path startswith /discovery N/A N/A N/A json dumps filtered_config_metadata N/A N/A N/A app route /config flask_get_config N/A N/A config_path N/A N/A N/A N/A N/A json dumps N/A N/A _PATH_TO_DATA get N/A / config_path N/A N/A N/A app route /config/<path:config_path> flask_get_serversets N/A N/A N/A N/A N/A N/A N/A json dumps _SERVERSET_METADATA N/A N/A N/A app route /serverset flask_get_serverset N/A N/A serverset_path N/A N/A N/A N/A N/A json dumps N/A N/A _SERVERSET_METADATA get N/A / serverset_path N/A N/A N/A app route /serverset/<path:serverset_path> get_rejected_serversets N/A N/A N/A N/A N/A N/A N/A json dumps N/A list _REJECTED_UPDATED_SERVERSETS N/A N/A N/A app route /admin/rejected_serversets rejected_serversets_healthcheck N/A N/A N/A N/A N/A N/A _REJECTED_UPDATED_SERVERSETS N/A N/A message N/A status 200 N/A resp N/A jsonify message N/A N/A resp status_code 200 N/A N/A message N/A status 500 N/A resp N/A jsonify message N/A N/A resp status_code 500 N/A resp N/A N/A N/A app route /admin/rejected_serversets/healthcheck flask_healthcheck N/A N/A N/A N/A N/A healthy True N/A greenlet_name _GREENLET_DICT N/A N/A N/A N/A _GREENLET_DICT N/A greenlet_name dead N/A N/A healthy False N/A N/A N/A healthy N/A N/A message N/A status 500 N/A resp N/A jsonify message N/A N/A resp status_code 500 N/A N/A N/A _INITIALIZATION_COMPLETED N/A N/A message N/A status 204 N/A resp N/A jsonify message N/A N/A resp status_code 204 N/A N/A message N/A status 200 N/A resp N/A jsonify message N/A N/A resp status_code 200 N/A resp N/A N/A N/A app route /admin/healthcheck report_serverset_type N/A N/A N/A N/A N/A zk_path N/A str N/A N/A N/A N/A request form getlist zk_path N/A 0 N/A mode N/A str N/A N/A N/A N/A request form getlist mode N/A 0 N/A N/A mode 0 N/A N/A N/A N/A _SKIP_FLAPPY_CHECK_SERVERSETS add zk_path N/A N/A N/A N/A _NON_SKIP_FLAPPY_CHECK_SERVERSETS add zk_path N/A N/A N/A N/A app route /admin/report_serverset_type methods N/A POST query_serverset_type N/A N/A N/A N/A N/A zk_path N/A str N/A N/A N/A N/A request form getlist zk_path N/A 0 N/A N/A zk_path _SKIP_FLAPPY_CHECK_SERVERSETS N/A N/A 0 N/A N/A N/A zk_path _NON_SKIP_FLAPPY_CHECK_SERVERSETS N/A N/A 1 N/A N/A -1 N/A N/A N/A app route /admin/query_serverset_type methods N/A POST query_non_skip_check_flappy_serversets N/A N/A N/A N/A N/A N/A N/A json dumps N/A list _NON_SKIP_FLAPPY_CHECK_SERVERSETS N/A N/A N/A app route /admin/query_non_skip_check_flappy_serversets query_skip_check_flappy_serversets N/A N/A N/A N/A N/A N/A N/A json dumps N/A list _SKIP_FLAPPY_CHECK_SERVERSETS N/A N/A N/A app route /admin/query_skip_check_flappy_serversets query_watched_metaconfigs N/A N/A N/A N/A N/A N/A N/A json dumps N/A list _WATCHED_METACONFIGS N/A N/A N/A app route /admin/watched_metaconfigs query_watched_dependencies N/A N/A N/A N/A N/A N/A N/A json dumps N/A list _WATCHED_DEPENDENCIES N/A N/A N/A app route /admin/watched_dependencies N/A test_notification_count 0 report_unittest_notification N/A N/A N/A N/A N/A test_notification_count N/A test_notification_count 1 N/A N/A N/A N/A app route /admin/report_unittest_notification N/A N/A __name__ __main__ N/A N/A N/A initialize_logger logger_filename zk_update_monitor.log N/A N/A main