Module Expr Str ImportFrom alias Assign NameStore ListLoad Str Str Str Str Str Str Str Str Str Import alias Import alias Import alias Import alias Import alias Import alias Import alias ImportFrom alias ImportFrom alias ImportFrom alias ImportFrom alias ImportFrom alias ImportFrom alias ImportFrom alias ImportFrom alias TryExcept body Assign NameStore NameLoad Import alias If Call NameLoad NameLoad Str body Assign NameStore NameLoad If NameLoad body ImportFrom alias ImportFrom alias identifier handlers ExceptHandler type NameLoad body Pass If CompareIs NameLoad NameLoad body TryExcept body Import alias ImportFrom alias handlers ExceptHandler type NameLoad body Pass Assign NameStore Str Assign NameStore Str ClassDef bases AttributeLoad NameLoad attr body Expr Str FunctionDef arguments args NameParam NameParam defaults body Expr Str Raise Call NameLoad BinOpMod Str NameLoad decorator_list NameLoad FunctionDef arguments args NameParam NameParam defaults body Expr Str Pass decorator_list NameLoad FunctionDef arguments args NameParam NameParam defaults body Expr Str Pass decorator_list NameLoad FunctionDef arguments args NameParam NameParam defaults body Expr Str Raise Call NameLoad BinOpMod Str NameLoad decorator_list NameLoad FunctionDef arguments args NameParam defaults body Expr Str Raise Call NameLoad BinOpMod Str AttributeLoad NameLoad attr decorator_list FunctionDef arguments args NameParam NameParam NameParam NameParam NameParam defaults NameLoad body Expr Str Raise Call NameLoad BinOpMod Str NameLoad decorator_list NameLoad FunctionDef arguments args NameParam NameParam defaults body Expr Str Raise Call NameLoad BinOpMod Str AttributeLoad NameLoad attr decorator_list FunctionDef arguments args NameParam NameParam NameParam defaults body Expr Str Raise Call NameLoad BinOpMod Str AttributeLoad NameLoad attr decorator_list FunctionDef arguments args NameParam NameParam defaults body Expr Str Raise Call NameLoad BinOpMod Str NameLoad decorator_list NameLoad FunctionDef arguments args NameParam NameParam defaults body Expr Str Return NameLoad decorator_list FunctionDef arguments args NameParam NameParam defaults body Expr Str Return NameLoad decorator_list FunctionDef arguments args NameParam NameParam NameParam NameParam defaults body Expr Str Raise Call NameLoad decorator_list decorator_list Assign NameStore BinOpMult Num Num Assign NameStore BinOpMult Num Num FunctionDef arguments args NameParam NameParam NameParam defaults NameLoad NameLoad body Expr Str If CompareNotIn Str AttributeLoad NameLoad attr body Assign NameStore Str If BoolOpOr UnaryOpNot NameLoad NameLoad body Raise Call AttributeLoad NameLoad attr NameLoad Assign NameStore AttributeLoad NameLoad attr Assign NameStore Call NameLoad GeneratorExp TupleLoad Call NameLoad NameLoad NameLoad comprehension TupleStore NameStore NameStore Call AttributeLoad NameLoad attr orelse If UnaryOpNot Call NameLoad Call AttributeLoad AttributeLoad NameLoad attr attr Str NameLoad body Raise Call AttributeLoad NameLoad attr Str Assign NameStore Call AttributeLoad AttributeLoad NameLoad attr attr Str Assign NameStore Call NameLoad GeneratorExp TupleLoad Call NameLoad NameLoad NameLoad comprehension TupleStore NameStore NameStore Call AttributeLoad NameLoad attr If NameLoad body Assign NameStore BinOpSub Call NameLoad Call AttributeLoad NameLoad attr NameLoad If NameLoad body Raise Call AttributeLoad NameLoad attr BinOpMod Str Call AttributeLoad Str attr NameLoad Return NameLoad decorator_list ClassDef bases AttributeLoad NameLoad attr body Expr Str Assign NameStore Num FunctionDef arguments args NameParam NameParam NameParam NameParam defaults NameLoad NameLoad NameLoad body Expr Str Assign AttributeStore NameLoad attr NameLoad Assign AttributeStore NameLoad attr ListLoad Assign AttributeStore NameLoad attr Num Assign AttributeStore NameLoad attr NameLoad Assign AttributeStore NameLoad attr NameLoad decorator_list FunctionDef arguments args NameParam NameParam defaults body Expr Str Assign NameStore Call NameLoad NameLoad If CompareGt BinOpAdd AttributeLoad NameLoad attr NameLoad AttributeLoad NameLoad attr body Expr Call AttributeLoad NameLoad attr If BoolOpAnd UnaryOpNot AttributeLoad NameLoad attr CompareGt NameLoad NameLoad body Raise Call AttributeLoad NameLoad attr BinOpMod Str TupleLoad NameLoad NameLoad orelse Expr Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad AugAssignAdd AttributeStore NameLoad attr NameLoad If CompareGt AttributeLoad NameLoad attr AttributeLoad NameLoad attr body Expr Call AttributeLoad NameLoad attr decorator_list FunctionDef arguments args NameParam defaults body Expr Str Assign NameStore Call AttributeLoad NameLoad attr With Call AttributeLoad NameLoad attr NameLoad NameStore body For NameStore AttributeLoad NameLoad attr body Expr Call AttributeLoad NameLoad attr NameLoad Expr Call AttributeLoad NameLoad attr Assign NameStore Call AttributeLoad NameLoad attr Expr Call AttributeLoad NameLoad attr If BoolOpAnd UnaryOpNot AttributeLoad NameLoad attr CompareGt Call NameLoad NameLoad NameLoad body Raise Call AttributeLoad NameLoad attr BinOpMod Str TupleLoad NameLoad Call NameLoad NameLoad Assign NameStore Call AttributeLoad NameLoad attr Expr Call AttributeLoad NameLoad attr NameLoad If AttributeLoad NameLoad attr body Expr Call Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad Call NameLoad NameLoad AttributeLoad NameLoad attr Expr Call Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad Call NameLoad BinOpMult BinOpSub Call AttributeLoad NameLoad attr NameLoad Num AttributeLoad NameLoad attr Assign AttributeStore NameLoad attr ListLoad Assign AttributeStore NameLoad attr Num Expr Call AttributeLoad NameLoad attr decorator_list FunctionDef arguments args NameParam NameParam defaults body Raise Call NameLoad BinOpMod Str Call NameLoad NameLoad decorator_list FunctionDef arguments args NameParam defaults body Return NameLoad decorator_list FunctionDef arguments args NameParam NameParam NameParam NameParam defaults body Expr Call AttributeLoad NameLoad attr decorator_list decorator_list ClassDef bases NameLoad body Expr Str Assign NameStore BinOpMult Num Num FunctionDef arguments args NameParam NameParam NameParam NameParam NameParam defaults NameLoad NameLoad NameLoad body Expr Str Expr Call AttributeLoad Call NameLoad NameLoad NameLoad attr NameLoad NameLoad NameLoad Assign AttributeStore NameLoad attr NameLoad Assign AttributeStore NameLoad attr Num decorator_list FunctionDef arguments args NameParam NameParam defaults body Expr Str Expr Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad AugAssignAdd AttributeStore NameLoad attr Call NameLoad NameLoad decorator_list FunctionDef arguments args NameParam NameParam defaults NameLoad body Expr Str Expr Call AttributeLoad Call NameLoad NameLoad NameLoad attr If NameLoad body Assign NameStore BinOpMod AttributeLoad NameLoad attr AttributeLoad NameLoad attr If CompareGt NameLoad Num body Expr Call AttributeLoad NameLoad attr BinOpMult Str BinOpSub AttributeLoad NameLoad attr NameLoad Expr Call AttributeLoad AttributeLoad NameLoad attr attr decorator_list decorator_list ClassDef bases AttributeLoad NameLoad attr NameLoad body Expr Str Assign NameStore Str Assign NameStore Str Assign NameStore Str Assign NameStore Str Assign NameStore Str Assign NameStore Str Assign NameStore Str FunctionDef arguments args NameParam NameParam defaults body Return SubscriptLoad NameLoad Index AttributeLoad NameLoad attr decorator_list NameLoad FunctionDef arguments args NameParam NameParam defaults body Return Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr NameLoad decorator_list NameLoad FunctionDef arguments args NameParam NameParam defaults body Expr Str If CompareIn AttributeLoad NameLoad attr NameLoad body Return SubscriptLoad NameLoad Index AttributeLoad NameLoad attr Return Call AttributeLoad NameLoad attr NameLoad decorator_list NameLoad FunctionDef arguments args NameParam NameParam defaults body Expr Str If CompareIn AttributeLoad NameLoad attr NameLoad body Return Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr NameLoad Return Call AttributeLoad NameLoad attr NameLoad decorator_list NameLoad decorator_list ClassDef bases NameLoad body Expr Str Assign NameStore Str Assign NameStore Str Assign NameStore BinOpAdd NameLoad Str FunctionDef arguments args NameParam NameParam NameParam NameParam NameParam NameParam NameParam defaults NameLoad NameLoad body Expr Str Assign NameStore AttributeLoad NameLoad attr If CompareIs NameLoad NameLoad body Assign NameStore Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr AttributeLoad NameLoad attr Assign NameStore Call AttributeLoad NameLoad attr NameLoad TryExcept body If CompareIs NameLoad NameLoad body Return Call AttributeLoad NameLoad attr keyword NameLoad keyword NameLoad keyword NameLoad orelse Return Call AttributeLoad NameLoad attr keyword NameLoad keyword NameLoad keyword NameLoad keyword NameLoad keyword NameLoad handlers ExceptHandler type NameLoad name NameStore body Raise Call AttributeLoad NameLoad attr BinOpMod Str NameLoad ExceptHandler type NameLoad name NameStore body Raise Call AttributeLoad NameLoad attr BinOpMod Str TupleLoad NameLoad NameLoad decorator_list NameLoad FunctionDef arguments args NameParam NameParam NameParam NameParam defaults NameLoad NameLoad body Assign NameStore Call NameLoad NameLoad NameLoad NameLoad If BoolOpAnd CompareIsNot Call AttributeLoad AttributeLoad NameLoad attr attr AttributeLoad NameLoad attr NameLoad CompareIs Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr NameLoad body Assign SubscriptStore NameLoad Index AttributeLoad NameLoad attr SubscriptLoad AttributeLoad NameLoad attr Index AttributeLoad NameLoad attr Return NameLoad decorator_list NameLoad FunctionDef arguments args NameParam NameParam defaults body Expr Str Assign NameStore Call AttributeLoad NameLoad attr NameLoad keyword NameLoad If CompareNotIn AttributeLoad NameLoad attr NameLoad body Raise Call AttributeLoad NameLoad attr BinOpMod Str AttributeLoad NameLoad attr TryExcept body Expr Call AttributeLoad NameLoad attr SubscriptLoad NameLoad Index AttributeLoad NameLoad attr handlers ExceptHandler type NameLoad name NameStore body Raise Call AttributeLoad NameLoad attr BinOpMod Str NameLoad Expr Call AttributeLoad NameLoad attr NameLoad Str Str Num Expr Call AttributeLoad NameLoad attr NameLoad Str Str Num Num Num decorator_list NameLoad FunctionDef arguments args NameParam NameParam NameParam NameParam defaults NameLoad body Expr Str If NameLoad body Assign NameStore Call AttributeLoad NameLoad attr NameLoad Assign NameStore Call AttributeLoad NameLoad attr NameLoad orelse Assign NameStore Call AttributeLoad NameLoad attr NameLoad Assign NameStore Call AttributeLoad NameLoad attr NameLoad Assign NameStore BinOpMod Str TupleLoad NameLoad NameLoad Assign NameStore Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr NameLoad Assign NameStore Dict If CompareIn AttributeLoad NameLoad attr NameLoad body Assign SubscriptStore NameLoad Index Str Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr Return Call AttributeLoad NameLoad attr NameLoad keyword Str keyword NameLoad keyword NameLoad keyword NameLoad decorator_list NameLoad FunctionDef arguments args NameParam NameParam defaults body Return SubscriptLoad AttributeLoad NameLoad attr Index Str decorator_list NameLoad FunctionDef arguments args NameParam NameParam defaults body Assign NameStore ListLoad For NameStore Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad body If CompareEq AttributeLoad NameLoad attr AttributeLoad AttributeLoad NameLoad attr attr body Expr Call AttributeLoad NameLoad attr Call AttributeLoad NameLoad attr NameLoad Return NameLoad decorator_list NameLoad FunctionDef arguments args NameParam defaults body Expr Str Raise Call NameLoad decorator_list FunctionDef arguments args NameParam NameParam defaults body Expr Str Assign NameStore Call AttributeLoad NameLoad attr Expr Call AttributeLoad Call AttributeLoad NameLoad attr attr NameLoad Assign NameStore Call AttributeLoad NameLoad attr Expr Call Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad Call NameLoad NameLoad NameLoad Expr Call Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad Call NameLoad BinOpMult BinOpSub Call AttributeLoad NameLoad attr NameLoad Num NameLoad decorator_list FunctionDef arguments args NameParam NameParam defaults body Return NameLoad decorator_list decorator_list ClassDef bases NameLoad body Expr Str Assign NameStore Str Assign NameStore Str Assign NameStore Str Assign NameStore Str Assign NameStore Str Assign NameStore Str Assign NameStore Str FunctionDef arguments args NameParam NameParam NameParam defaults NameLoad body Expr Str Assign AttributeStore NameLoad attr NameLoad Assign AttributeStore NameLoad attr NameLoad If NameLoad body Assign AttributeStore NameLoad attr Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr NameLoad If AttributeLoad NameLoad attr body Assign AttributeStore NameLoad attr Call NameLoad SubscriptLoad Call AttributeLoad AttributeLoad NameLoad attr attr Str Num Index Num Assign AttributeStore NameLoad attr Num decorator_list FunctionDef arguments args NameParam NameParam defaults body Expr Str Assign NameStore Call AttributeLoad NameLoad attr NameLoad keyword NameLoad If CompareNotIn Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr NameLoad TupleLoad NameLoad NameLoad body Raise Call AttributeLoad NameLoad attr Str Expr Call AttributeLoad Call NameLoad NameLoad NameLoad attr NameLoad decorator_list NameLoad FunctionDef arguments args NameParam defaults body Return AttributeLoad NameLoad attr decorator_list FunctionDef arguments args NameParam NameParam NameParam NameParam NameParam defaults NameLoad body Expr Str Assign NameStore Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr keyword NameLoad Assign NameStore NameLoad If Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr NameLoad body Assign NameStore Num Assign NameStore Call AttributeLoad NameLoad attr NameLoad AttributeLoad NameLoad attr AttributeLoad NameLoad attr NameLoad NameLoad NameLoad Return Call AttributeLoad NameLoad attr NameLoad NameLoad decorator_list NameLoad FunctionDef arguments args NameParam NameParam NameParam defaults body Expr Str Assign NameStore Call AttributeLoad NameLoad attr NameLoad NameLoad Return Call NameLoad NameLoad keyword NameLoad decorator_list NameLoad FunctionDef arguments args NameParam NameParam defaults body Assign NameStore Call NameLoad Call AttributeLoad NameLoad attr SubscriptLoad NameLoad Index AttributeLoad NameLoad attr Assign NameStore Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr NameLoad Assign AttributeStore NameLoad attr NameLoad If NameLoad body Assign AttributeStore NameLoad attr SubscriptLoad NameLoad Index AttributeLoad NameLoad attr Assign AttributeStore NameLoad attr SubscriptLoad NameLoad Index AttributeLoad NameLoad attr Return NameLoad decorator_list NameLoad FunctionDef arguments args NameParam NameParam defaults body If UnaryOpNot AttributeLoad AttributeLoad NameLoad attr attr body Expr Call AttributeLoad AttributeLoad NameLoad attr attr decorator_list FunctionDef arguments args NameParam defaults body Assign NameStore Dict AttributeLoad NameLoad attr AttributeLoad NameLoad attr Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr AttributeLoad NameLoad attr If AttributeLoad NameLoad attr body Expr Call AttributeLoad NameLoad attr Dict AttributeLoad NameLoad attr AttributeLoad NameLoad attr Call AttributeLoad AttributeLoad NameLoad attr attr AttributeLoad NameLoad attr Return NameLoad decorator_list FunctionDef arguments args NameParam NameParam NameParam defaults body Expr Call AttributeLoad AttributeLoad NameLoad attr attr If AttributeLoad NameLoad attr body Expr Call AttributeLoad NameLoad attr AttributeLoad AttributeLoad NameLoad attr attr AttributeLoad AttributeLoad NameLoad attr attr keyword Dict AttributeLoad NameLoad attr Call AttributeLoad AttributeLoad NameLoad attr attr Assign NameStore AttributeLoad NameLoad attr Assign NameStore Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr keyword NameLoad Assign NameStore Call AttributeLoad NameLoad attr NameLoad AttributeLoad NameLoad attr AttributeLoad NameLoad attr AttributeLoad NameLoad attr Assign NameStore AttributeLoad AttributeLoad NameLoad attr attr Assign TupleStore NameStore NameStore Call AttributeLoad NameLoad attr Str Num Assign AttributeStore NameLoad attr Dict AttributeLoad NameLoad attr AttributeLoad NameLoad attr Str BinOpAdd NameLoad Str Call NameLoad NameLoad NameLoad orelse Assign AttributeStore NameLoad attr Dict Str AttributeLoad AttributeLoad NameLoad attr attr decorator_list FunctionDef arguments args NameParam NameParam defaults body Assign NameStore Call AttributeLoad NameLoad attr NameLoad keyword NameLoad Return Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr NameLoad decorator_list FunctionDef arguments args NameParam NameParam NameParam NameParam defaults body Assign NameStore AttributeLoad NameLoad attr If CompareNotEq AttributeLoad NameLoad attr Num body TryExcept body Assign NameStore BinOpAdd Call AttributeLoad AttributeLoad NameLoad attr attr Num If CompareGt NameLoad Call AttributeLoad AttributeLoad NameLoad attr attr body Expr Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad orelse Expr Call AttributeLoad AttributeLoad NameLoad attr attr handlers ExceptHandler type AttributeLoad NameLoad attr body Pass Expr Call AttributeLoad NameLoad attr AttributeLoad AttributeLoad NameLoad attr attr AttributeLoad AttributeLoad NameLoad attr attr keyword Dict AttributeLoad NameLoad attr AttributeLoad NameLoad attr Assign NameStore BinOpAdd AttributeLoad NameLoad attr Num Assign NameStore Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr keyword NameLoad Assign NameStore Call AttributeLoad NameLoad attr NameLoad AttributeLoad NameLoad attr AttributeLoad NameLoad attr NameLoad NameLoad NameLoad Assign NameStore Call AttributeLoad NameLoad attr NameLoad NameLoad Assign AttributeStore NameLoad attr NameLoad Return NameLoad decorator_list FunctionDef arguments args NameParam defaults body Return AttributeLoad AttributeLoad NameLoad attr attr decorator_list decorator_list Assign NameStore NameLoad ClassDef bases NameLoad body Expr Str FunctionDef arguments args NameParam defaults body Assign AttributeStore NameLoad attr NameLoad Assign AttributeStore NameLoad attr NameLoad Assign AttributeStore NameLoad attr NameLoad Assign AttributeStore NameLoad attr NameLoad Assign AttributeStore NameLoad attr NameLoad Assign AttributeStore NameLoad attr NameLoad decorator_list decorator_list ClassDef bases NameLoad body Expr Str Assign NameStore Str Assign NameStore Num Assign NameStore BinOpMult Num Num Assign NameStore Str Assign NameStore BinOpAdd NameLoad Str Assign NameStore BinOpAdd NameLoad Str FunctionDef arguments args NameParam NameParam defaults body Expr Str Assign AttributeStore NameLoad attr NameLoad Assign AttributeStore NameLoad attr NameLoad decorator_list FunctionDef arguments args NameParam defaults body If UnaryOpNot AttributeLoad AttributeLoad NameLoad attr attr body Raise Call AttributeLoad NameLoad attr Str Return AttributeLoad AttributeLoad NameLoad attr attr decorator_list FunctionDef arguments args NameParam defaults body Return AttributeLoad AttributeLoad AttributeLoad NameLoad attr attr attr decorator_list FunctionDef arguments args NameParam NameParam NameParam NameParam NameParam defaults NameLoad body Expr Str Assign NameStore Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr keyword NameLoad Assign NameStore Call AttributeLoad NameLoad attr NameLoad AttributeLoad NameLoad attr AttributeLoad NameLoad attr NameLoad NameLoad Assign NameStore Call NameLoad Assign AttributeStore NameLoad attr NameLoad Assign AttributeStore NameLoad attr Call AttributeLoad NameLoad attr NameLoad NameLoad Assign AttributeStore NameLoad attr AttributeLoad NameLoad attr Assign AttributeStore NameLoad attr NameLoad Return Call NameLoad NameLoad decorator_list NameLoad FunctionDef arguments args NameParam NameParam NameParam defaults body If UnaryOpNot NameLoad body Return Assign NameStore Call AttributeLoad NameLoad attr NameLoad TryExcept body Expr Call AttributeLoad NameLoad attr NameLoad keyword NameLoad handlers ExceptHandler type AttributeLoad NameLoad attr body Pass decorator_list FunctionDef arguments args NameParam NameParam NameParam NameParam defaults body Expr Str If AttributeLoad NameLoad attr body Return Assign NameStore Call AttributeLoad NameLoad attr NameLoad Assign NameStore Call AttributeLoad NameLoad attr NameLoad keyword NameLoad Assign NameStore Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr While NameLoad body Expr Call AttributeLoad NameLoad attr NameLoad Assign NameStore Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr Expr Call AttributeLoad NameLoad attr Expr Call AttributeLoad NameLoad attr decorator_list FunctionDef arguments args NameParam NameParam defaults body Expr Str Assign NameStore Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr Assign NameStore Call AttributeLoad NameLoad attr keyword AttributeLoad NameLoad attr keyword AttributeLoad NameLoad attr keyword Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr Return Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr NameLoad keyword NameLoad decorator_list NameLoad FunctionDef arguments args NameParam NameParam defaults body Assign NameStore AttributeLoad NameLoad attr Assign NameStore AttributeLoad NameLoad attr If AttributeLoad NameLoad attr body Expr Call AttributeLoad NameLoad attr AttributeLoad AttributeLoad NameLoad attr attr NameLoad Assign NameStore ListLoad If AttributeLoad NameLoad attr body Expr Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr AttributeLoad AttributeLoad NameLoad attr attr NameLoad Expr Call AttributeLoad NameLoad attr AttributeLoad AttributeLoad NameLoad attr attr Expr Call AttributeLoad NameLoad attr NameLoad keyword NameLoad Assign AttributeStore NameLoad attr AttributeLoad NameLoad attr Assign AttributeStore NameLoad attr Call AttributeLoad NameLoad attr NameLoad If AttributeLoad AttributeLoad NameLoad attr attr body Expr Call AttributeLoad AttributeLoad NameLoad attr attr Expr Call AttributeLoad NameLoad attr AttributeLoad AttributeLoad NameLoad attr attr NameLoad decorator_list FunctionDef arguments args NameParam NameParam defaults body Return Call NameLoad Call AttributeLoad NameLoad attr SubscriptLoad NameLoad Index AttributeLoad NameLoad attr decorator_list NameLoad FunctionDef arguments args NameParam NameParam defaults body Expr Call AttributeLoad AttributeLoad AttributeLoad NameLoad attr attr attr decorator_list FunctionDef arguments args NameParam defaults body Return Dict AttributeLoad NameLoad attr Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr decorator_list FunctionDef arguments args NameParam NameParam defaults body Expr Call AttributeLoad Call NameLoad NameLoad NameLoad attr NameLoad Assign AttributeStore NameLoad attr NameLoad decorator_list FunctionDef arguments args NameParam NameParam NameParam defaults TupleLoad body Expr Str Assign NameStore Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr Assign NameStore Call AttributeLoad NameLoad attr keyword AttributeLoad AttributeLoad NameLoad attr attr keyword AttributeLoad AttributeLoad NameLoad attr attr Assign NameStore Call AttributeLoad NameLoad attr NameLoad Assign NameStore Call AttributeLoad NameLoad attr NameLoad For NameStore Call AttributeLoad NameLoad attr BinOpMod Str TupleLoad NameLoad NameLoad keyword NameLoad body If CompareNotIn AttributeLoad NameLoad attr NameLoad body Expr Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr AttributeLoad AttributeLoad NameLoad attr attr decorator_list FunctionDef arguments args NameParam NameParam NameParam defaults body If AttributeLoad NameLoad attr body Raise Call AttributeLoad NameLoad attr Str If AttributeLoad AttributeLoad NameLoad attr attr body Expr Call AttributeLoad AttributeLoad AttributeLoad NameLoad attr attr attr Expr Call AttributeLoad AttributeLoad AttributeLoad NameLoad attr attr attr If AttributeLoad AttributeLoad NameLoad attr attr body Expr Call AttributeLoad NameLoad attr AttributeLoad AttributeLoad AttributeLoad NameLoad attr attr attr AttributeLoad AttributeLoad NameLoad attr attr If AttributeLoad AttributeLoad NameLoad attr attr body Expr Call AttributeLoad NameLoad attr AttributeLoad AttributeLoad AttributeLoad NameLoad attr attr attr AttributeLoad AttributeLoad NameLoad attr attr Expr Call AttributeLoad NameLoad attr AttributeLoad AttributeLoad NameLoad attr attr Assign AttributeStore NameLoad attr Dict Str AttributeLoad AttributeLoad AttributeLoad NameLoad attr attr attr decorator_list decorator_list ClassDef bases NameLoad body Expr Str Assign NameStore NameLoad FunctionDef arguments args NameParam NameParam defaults body Assign AttributeStore NameLoad attr NameLoad Assign AttributeStore NameLoad attr Call AttributeLoad NameLoad attr NameLoad decorator_list FunctionDef arguments args NameParam NameParam defaults body Return Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad decorator_list NameLoad FunctionDef arguments args NameParam NameParam defaults body Return Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad decorator_list NameLoad FunctionDef arguments args NameParam NameParam defaults body Return Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad decorator_list NameLoad FunctionDef arguments args NameParam NameParam defaults body Return Call NameLoad Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad decorator_list NameLoad FunctionDef arguments args NameParam defaults body Return Call AttributeLoad AttributeLoad NameLoad attr attr decorator_list FunctionDef arguments args NameParam NameParam NameParam NameParam NameParam defaults NameLoad body Return Call NameLoad Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad NameLoad NameLoad NameLoad decorator_list NameLoad FunctionDef arguments args NameParam NameParam defaults body Expr Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad decorator_list FunctionDef arguments args NameParam NameParam NameParam defaults body Return Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad NameLoad decorator_list FunctionDef arguments args NameParam NameParam defaults body Return Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad decorator_list NameLoad FunctionDef arguments args NameParam NameParam defaults body Return Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad decorator_list FunctionDef arguments args NameParam NameParam defaults body Return Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad decorator_list FunctionDef arguments args NameParam NameParam NameParam NameParam defaults body Return Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad NameLoad NameLoad decorator_list FunctionDef arguments args NameParam NameParam defaults body Return Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad decorator_list FunctionDef arguments args NameParam NameParam defaults body If UnaryOpNot AttributeLoad Call AttributeLoad AttributeLoad NameLoad attr attr attr body Expr Call AttributeLoad AttributeLoad NameLoad attr attr Return Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad decorator_list decorator_list ClassDef bases NameLoad body Assign NameStore NameLoad decorator_list Assign NameStore NameLoad ClassDef bases NameLoad body Assign NameStore NameLoad decorator_list ClassDef bases NameLoad body Expr Str FunctionDef arguments args NameParam NameParam defaults body If CompareNotEq Call NameLoad NameLoad Num body Expr Call AttributeLoad NameLoad attr Str Call NameLoad NameLoad NameLoad TryExcept body Assign NameStore Call NameLoad SubscriptLoad NameLoad Index Num Assign NameStore Call NameLoad SubscriptLoad NameLoad Index Num handlers ExceptHandler type NameLoad body Expr Call AttributeLoad NameLoad attr Str AttributeLoad AttributeLoad NameLoad attr attr NameLoad Assign NameStore Call AttributeLoad NameLoad attr Expr Call AttributeLoad NameLoad attr NameLoad Expr Call AttributeLoad NameLoad attr NameLoad Expr Call AttributeLoad NameLoad attr NameLoad Call AttributeLoad NameLoad attr decorator_list decorator_list Assign NameStore NameLoad
N/A N/A Output writers for MapReduce. __future__ with_statement N/A __all__ N/A GoogleCloudStorageConsistentOutputWriter GoogleCloudStorageConsistentRecordOutputWriter GoogleCloudStorageKeyValueOutputWriter GoogleCloudStorageOutputWriter GoogleCloudStorageRecordOutputWriter COUNTER_IO_WRITE_BYTES COUNTER_IO_WRITE_MSEC OutputWriter GCSRecordsPool N/A cStringIO N/A gc N/A logging N/A pickle N/A random N/A string N/A time mapreduce context mapreduce errors mapreduce json_util mapreduce kv_pb mapreduce model mapreduce operation mapreduce records mapreduce shard_life_cycle N/A N/A N/A cloudstorage None N/A cloudstorage N/A N/A hasattr cloudstorage _STUB N/A N/A cloudstorage None N/A cloudstorage N/A cloudstorage cloudstorage_api cloudstorage errors cloud_errors N/A N/A N/A ImportError N/A N/A N/A N/A cloudstorage None N/A N/A N/A N/A cloudstorage cloudstorage cloudstorage_api N/A N/A N/A ImportError N/A N/A N/A COUNTER_IO_WRITE_BYTES io-write-bytes N/A COUNTER_IO_WRITE_MSEC io-write-msec OutputWriter N/A N/A json_util JsonMixin N/A N/A Abstract base class for output writers.

  Output writers process all mapper handler output, which is not
  the operation.

  OutputWriter's lifecycle is the following:
    0) validate called to validate mapper specification.
    1) init_job is called to initialize any job-level state.
    2) create() is called, which should create a new instance of output
       writer for a given shard
    3) from_json()/to_json() are used to persist writer's state across
       multiple slices.
    4) write() method is called to write data.
    5) finalize() is called when shard processing is done.
    6) finalize_job() is called when job is completed.
    7) get_filenames() is called to get output file names.
   validate N/A N/A cls mapper_spec N/A N/A N/A Validates mapper specification.

    Output writer parameters are expected to be passed as "output_writer"
    subdictionary of mapper_spec.params. To be compatible with previous
    API output writer is advised to check mapper_spec.params and issue
    a warning if "output_writer" subdicationary is not present.
    _get_params helper method can be used to simplify implementation.

    Args:
      mapper_spec: an instance of model.MapperSpec to validate.
     N/A N/A NotImplementedError N/A validate() not implemented in %s cls N/A classmethod init_job N/A N/A cls mapreduce_state N/A N/A N/A Initialize job-level writer state.

    This method is only to support the deprecated feature which is shared
    output files by many shards. New output writers should not do anything
    in this method.

    Args:
      mapreduce_state: an instance of model.MapreduceState describing current
      job. MapreduceState.writer_state can be modified during initialization
      to save the information about the files shared by many shards.
     N/A N/A classmethod finalize_job N/A N/A cls mapreduce_state N/A N/A N/A Finalize job-level writer state.

    This method is only to support the deprecated feature which is shared
    output files by many shards. New output writers should not do anything
    in this method.

    This method should only be called when mapreduce_state.result_status shows
    success. After finalizing the outputs, it should save the info for shard
    shared files into mapreduce_state.writer_state so that other operations
    can find the outputs.

    Args:
      mapreduce_state: an instance of model.MapreduceState describing current
      job. MapreduceState.writer_state can be modified during finalization.
     N/A N/A classmethod from_json N/A N/A cls state N/A N/A N/A Creates an instance of the OutputWriter for the given json state.

    Args:
      state: The OutputWriter state as a dict-like object.

    Returns:
      An instance of the OutputWriter configured using the values of json.
     N/A N/A NotImplementedError N/A from_json() not implemented in %s cls N/A classmethod to_json N/A N/A self N/A N/A N/A Returns writer state to serialize in json.

    Returns:
      A json-izable version of the OutputWriter state.
     N/A N/A NotImplementedError N/A to_json() not implemented in %s N/A self __class__ N/A create N/A N/A cls mr_spec shard_number shard_attempt _writer_state N/A None N/A N/A Create new writer for a shard.

    Args:
      mr_spec: an instance of model.MapreduceSpec describing current job.
      shard_number: int shard number.
      shard_attempt: int shard attempt.
      _writer_state: deprecated. This is for old writers that share file
        across shards. For new writers, each shard must have its own
        dedicated outputs. Output state should be contained in
        the output writer instance. The serialized output writer
        instance will be saved by mapreduce across slices.
     N/A N/A NotImplementedError N/A create() not implemented in %s cls N/A classmethod write N/A N/A self data N/A N/A N/A Write data.

    Args:
      data: actual data yielded from handler. Type is writer-specific.
     N/A N/A NotImplementedError N/A write() not implemented in %s N/A self __class__ N/A finalize N/A N/A self ctx shard_state N/A N/A N/A Finalize writer shard-level state.

    This should only be called when shard_state.result_status shows success.
    After finalizing the outputs, it should save per-shard output file info
    into shard_state.writer_state so that other operations can find the
    outputs.

    Args:
      ctx: an instance of context.Context.
      shard_state: shard state. ShardState.writer_state can be modified.
     N/A N/A NotImplementedError N/A finalize() not implemented in %s N/A self __class__ N/A get_filenames N/A N/A cls mapreduce_state N/A N/A N/A Obtain output filenames from mapreduce state.

    This method should only be called when a MR is finished. Implementors of
    this method should not assume any other methods of this class have been
    called. In the case of no input data, no other method except validate
    would have been called.

    Args:
      mapreduce_state: an instance of model.MapreduceState

    Returns:
      List of filenames this mapreduce successfully wrote to. The list can be
    empty if no output file was successfully written.
     N/A N/A NotImplementedError N/A get_filenames() not implemented in %s cls N/A classmethod _supports_shard_retry N/A N/A self tstate N/A N/A N/A Whether this output writer instance supports shard retry.

    Args:
      tstate: model.TransientShardState for current shard.

    Returns:
      boolean. Whether this output writer instance supports shard retry.
     N/A False N/A _supports_slice_recovery N/A N/A self mapper_spec N/A N/A N/A Whether this output writer supports slice recovery.

    Args:
      mapper_spec: instance of model.MapperSpec.

    Returns:
      boolean. Whether this output writer instance supports slice recovery.
     N/A False N/A _recover N/A N/A self mr_spec shard_number shard_attempt N/A N/A N/A Create a new output writer instance from the old one.

    This method is called when _supports_slice_recovery returns True,
    and when there is a chance the old output writer instance is out of sync
    with its storage medium due to a retry of a slice. _recover should
    create a new instance based on the old one. When finalize is called
    on the new instance, it could combine valid outputs from all instances
    to generate the final output. How the new instance maintains references
    to previous outputs is up to implementation.

    Any exception during recovery is subject to normal slice/shard retry.
    So recovery logic must be idempotent.

    Args:
      mr_spec: an instance of model.MapreduceSpec describing current job.
      shard_number: int shard number.
      shard_attempt: int shard attempt.

    Returns:
      a new instance of output writer.
     N/A N/A NotImplementedError N/A N/A N/A _FILE_POOL_FLUSH_SIZE N/A 128 1024 N/A _FILE_POOL_MAX_SIZE N/A 1000 1024 _get_params N/A N/A mapper_spec allowed_keys allow_old N/A None True N/A N/A Obtain output writer parameters.

  Utility function for output writer implementation. Fetches parameters
  from mapreduce specification giving appropriate usage warnings.

  Args:
    mapper_spec: The MapperSpec for the job
    allowed_keys: set of all allowed keys in parameters as strings. If it is not
      None, then parameters are expected to be in a separate "output_writer"
      subdictionary of mapper_spec parameters.
    allow_old: Allow parameters to exist outside of the output_writer
      subdictionary for compatability.

  Returns:
    mapper parameters as dict

  Raises:
    BadWriterParamsError: if parameters are invalid/missing or not allowed.
   N/A N/A output_writer N/A mapper_spec params N/A N/A message Output writer's parameters should be specified in output_writer subdictionary. N/A N/A N/A allow_old allowed_keys N/A N/A N/A N/A errors BadWriterParamsError message N/A params N/A mapper_spec params N/A params N/A dict N/A N/A N/A str n v N/A N/A n v N/A N/A params iteritems N/A N/A N/A N/A isinstance N/A N/A N/A mapper_spec params get output_writer dict N/A N/A N/A N/A errors BadWriterParamsError Output writer parameters should be a dictionary N/A params N/A N/A N/A mapper_spec params get output_writer N/A params N/A dict N/A N/A N/A str n v N/A N/A n v N/A N/A params iteritems N/A allowed_keys N/A N/A params_diff N/A N/A set N/A N/A params keys allowed_keys N/A params_diff N/A N/A N/A N/A errors BadWriterParamsError N/A Invalid output_writer parameters: %s N/A N/A , join params_diff N/A params N/A _RecordsPoolBase N/A N/A context Pool N/A N/A Base class for Pool of append operations for records files. N/A _RECORD_OVERHEAD_BYTES 10 __init__ N/A N/A self flush_size_chars ctx exclusive N/A _FILE_POOL_FLUSH_SIZE None False N/A N/A Constructor.

    Any classes that subclass this will need to implement the _write() function.

    Args:
      flush_size_chars: buffer flush threshold as int.
      ctx: mapreduce context as context.Context.
      exclusive: a boolean flag indicating if the pool has an exclusive
        access to the file. If it is True, then it's possible to write
        bigger chunks of data.
     N/A N/A self _flush_size flush_size_chars N/A N/A self _buffer N/A N/A N/A self _size 0 N/A N/A self _ctx ctx N/A N/A self _exclusive exclusive N/A append N/A N/A self data N/A N/A N/A Append data to a file. N/A data_length N/A len data N/A N/A N/A N/A self _size data_length N/A self _flush_size N/A N/A N/A N/A self flush N/A N/A N/A N/A self _exclusive N/A data_length _FILE_POOL_MAX_SIZE N/A N/A N/A N/A errors Error N/A Too big input %s (%s). N/A data_length _FILE_POOL_MAX_SIZE N/A N/A N/A N/A N/A self _buffer append data N/A N/A self _size data_length N/A N/A N/A self _size N/A self _flush_size N/A N/A N/A N/A self flush N/A flush N/A N/A self N/A N/A N/A Flush pool contents. N/A buf N/A N/A cStringIO StringIO N/A N/A N/A records RecordsWriter buf w N/A N/A record N/A self _buffer N/A N/A N/A N/A w write record N/A N/A N/A w _pad_block N/A str_buf N/A N/A buf getvalue N/A N/A N/A buf close N/A N/A N/A N/A self _exclusive N/A N/A len str_buf _FILE_POOL_MAX_SIZE N/A N/A N/A N/A errors Error N/A Buffer too big. Can't write more than %s bytes in one request: risk of writes interleaving. Got: %s N/A _FILE_POOL_MAX_SIZE N/A len str_buf N/A start_time N/A N/A time time N/A N/A N/A self _write str_buf N/A N/A self _ctx N/A N/A N/A N/A N/A N/A operation counters Increment COUNTER_IO_WRITE_BYTES N/A len str_buf N/A self _ctx N/A N/A N/A N/A N/A operation counters Increment COUNTER_IO_WRITE_MSEC N/A int N/A N/A N/A N/A time time start_time 1000 N/A self _ctx N/A N/A self _buffer N/A N/A N/A self _size 0 N/A N/A N/A gc collect N/A _write N/A N/A self str_buf N/A N/A N/A N/A NotImplementedError N/A _write() not implemented in %s N/A type self N/A __enter__ N/A N/A self N/A N/A N/A self N/A __exit__ N/A N/A self atype value traceback N/A N/A N/A N/A N/A self flush N/A N/A GCSRecordsPool N/A _RecordsPoolBase N/A N/A Pool of append operations for records using GCS. N/A _GCS_BLOCK_SIZE N/A 256 1024 __init__ N/A N/A self filehandle flush_size_chars ctx exclusive N/A _FILE_POOL_FLUSH_SIZE None False N/A N/A Requires the filehandle of an open GCS file to write to. N/A N/A N/A N/A super GCSRecordsPool self __init__ flush_size_chars ctx exclusive N/A N/A self _filehandle filehandle N/A N/A self _buf_size 0 N/A _write N/A N/A self str_buf N/A N/A N/A Uses the filehandle to the file in GCS to write to it. N/A N/A N/A N/A self _filehandle write str_buf N/A N/A self _buf_size N/A len str_buf N/A flush N/A N/A self force N/A False N/A N/A Flush pool contents.

    Args:
      force: Inserts additional padding to achieve the minimum block size
        required for GCS.
     N/A N/A N/A N/A super GCSRecordsPool self flush N/A force N/A N/A extra_padding N/A N/A self _buf_size N/A self _GCS_BLOCK_SIZE N/A N/A extra_padding 0 N/A N/A N/A N/A self _write N/A N/A N/A self _GCS_BLOCK_SIZE extra_padding N/A N/A N/A N/A self _filehandle flush N/A N/A _GoogleCloudStorageBase N/A N/A shard_life_cycle _ShardLifeCycle OutputWriter N/A N/A Base abstract class for all GCS writers.

  Required configuration in the mapper_spec.output_writer dictionary.
    BUCKET_NAME_PARAM: name of the bucket to use (with no extra delimiters or
      suffixes such as directories. Directories/prefixes can be specifed as
      part of the NAMING_FORMAT_PARAM).

  Optional configuration in the mapper_spec.output_writer dictionary:
    ACL_PARAM: acl to apply to new files, else bucket default used.
    NAMING_FORMAT_PARAM: prefix format string for the new files (there is no
      required starting slash, expected formats would look like
      "directory/basename...", any starting slash will be treated as part of
      the file name) that should use the following substitutions:
        $name - the name of the job
        $id - the id assigned to the job
        $num - the shard number
      If there is more than one shard $num must be used. An arbitrary suffix may
      be applied by the writer.
    CONTENT_TYPE_PARAM: mime type to apply on the files. If not provided, Google
      Cloud Storage will apply its default.
    TMP_BUCKET_NAME_PARAM: name of the bucket used for writing tmp files by
      consistent GCS output writers. Defaults to BUCKET_NAME_PARAM if not set.
   N/A BUCKET_NAME_PARAM bucket_name N/A TMP_BUCKET_NAME_PARAM tmp_bucket_name N/A ACL_PARAM acl N/A NAMING_FORMAT_PARAM naming_format N/A CONTENT_TYPE_PARAM content_type N/A _ACCOUNT_ID_PARAM account_id N/A _TMP_ACCOUNT_ID_PARAM tmp_account_id _get_gcs_bucket N/A N/A cls writer_spec N/A N/A N/A N/A writer_spec N/A N/A cls BUCKET_NAME_PARAM N/A classmethod _get_account_id N/A N/A cls writer_spec N/A N/A N/A N/A N/A writer_spec get N/A cls _ACCOUNT_ID_PARAM None N/A classmethod _get_tmp_gcs_bucket N/A N/A cls writer_spec N/A N/A N/A Returns bucket used for writing tmp files. N/A N/A N/A cls TMP_BUCKET_NAME_PARAM writer_spec N/A N/A N/A writer_spec N/A N/A cls TMP_BUCKET_NAME_PARAM N/A N/A N/A cls _get_gcs_bucket writer_spec N/A classmethod _get_tmp_account_id N/A N/A cls writer_spec N/A N/A N/A Returns the account id to use with tmp bucket. N/A N/A N/A cls TMP_BUCKET_NAME_PARAM writer_spec N/A N/A N/A N/A writer_spec get N/A cls _TMP_ACCOUNT_ID_PARAM None N/A N/A N/A cls _get_account_id writer_spec N/A classmethod N/A _GoogleCloudStorageOutputWriterBase N/A _GoogleCloudStorageBase N/A N/A Base class for GCS writers directly interacting with GCS.

  Base class for both _GoogleCloudStorageOutputWriter and
  GoogleCloudStorageConsistentOutputWriter.

  This class is expected to be subclassed with a writer that applies formatting
  to user-level records.

  Subclasses need to define to_json, from_json, create, finalize and
  _get_write_buffer methods.

  See _GoogleCloudStorageBase for config options.
   N/A _DEFAULT_NAMING_FORMAT $name/$id/output-$num N/A _MR_TMP gae_mr_tmp N/A _TMP_FILE_NAMING_FORMAT N/A _MR_TMP /$name/$id/attempt-$attempt/output-$num/seg-$seg _generate_filename N/A N/A cls writer_spec name job_id num attempt seg_index N/A None None N/A N/A Generates a filename for a particular output.

    Args:
      writer_spec: specification dictionary for the output writer.
      name: name of the job.
      job_id: the ID number assigned to the job.
      num: shard number.
      attempt: the shard attempt number.
      seg_index: index of the seg. None means the final output.

    Returns:
      a string containing the filename.

    Raises:
      BadWriterParamsError: if the template contains any errors such as invalid
        syntax or contains unknown substitution placeholders.
     N/A naming_format N/A cls _TMP_FILE_NAMING_FORMAT N/A N/A seg_index None N/A N/A naming_format N/A N/A writer_spec get N/A cls NAMING_FORMAT_PARAM N/A cls _DEFAULT_NAMING_FORMAT N/A template N/A N/A string Template naming_format N/A N/A N/A N/A seg_index None N/A N/A N/A N/A template substitute name name id job_id num num N/A N/A N/A N/A template substitute name name id job_id num num attempt attempt seg seg_index N/A N/A N/A ValueError N/A error N/A N/A N/A N/A errors BadWriterParamsError N/A Naming template is bad, %s error N/A N/A KeyError N/A error N/A N/A N/A N/A errors BadWriterParamsError N/A Naming template '%s' has extra mappings, %s N/A naming_format error N/A classmethod get_params N/A N/A cls mapper_spec allowed_keys allow_old N/A None True N/A N/A params N/A _get_params mapper_spec allowed_keys allow_old N/A N/A N/A N/A N/A N/A mapper_spec params get N/A cls BUCKET_NAME_PARAM None N/A N/A N/A params get N/A cls BUCKET_NAME_PARAM None N/A N/A N/A params N/A N/A cls BUCKET_NAME_PARAM N/A N/A mapper_spec params N/A N/A cls BUCKET_NAME_PARAM N/A params N/A classmethod validate N/A N/A cls mapper_spec N/A N/A N/A Validate mapper specification.

    Args:
      mapper_spec: an instance of model.MapperSpec.

    Raises:
      BadWriterParamsError: if the specification is invalid for any reason such
        as missing the bucket name or providing an invalid bucket name.
     N/A writer_spec N/A N/A cls get_params mapper_spec allow_old False N/A N/A N/A cls BUCKET_NAME_PARAM writer_spec N/A N/A N/A N/A errors BadWriterParamsError N/A %s is required for Google Cloud Storage N/A cls BUCKET_NAME_PARAM N/A N/A N/A N/A N/A cloudstorage validate_bucket_name N/A writer_spec N/A N/A cls BUCKET_NAME_PARAM N/A N/A N/A ValueError N/A error N/A N/A N/A N/A errors BadWriterParamsError N/A Bad bucket name, %s error N/A N/A N/A cls _generate_filename writer_spec name id 0 N/A N/A N/A cls _generate_filename writer_spec name id 0 1 0 N/A classmethod _open_file N/A N/A cls writer_spec filename_suffix use_tmp_bucket N/A False N/A N/A Opens a new gcs file for writing. N/A use_tmp_bucket N/A N/A bucket N/A N/A cls _get_tmp_gcs_bucket writer_spec N/A account_id N/A N/A cls _get_tmp_account_id writer_spec N/A N/A bucket N/A N/A cls _get_gcs_bucket writer_spec N/A account_id N/A N/A cls _get_account_id writer_spec N/A filename N/A /%s/%s N/A bucket filename_suffix N/A content_type N/A N/A writer_spec get N/A cls CONTENT_TYPE_PARAM None N/A options N/A N/A N/A N/A cls ACL_PARAM writer_spec N/A N/A N/A options N/A x-goog-acl N/A N/A writer_spec get N/A cls ACL_PARAM N/A N/A N/A cloudstorage open filename mode w content_type content_type options options _account_id account_id N/A classmethod _get_filename N/A N/A cls shard_state N/A N/A N/A N/A N/A shard_state writer_state N/A filename N/A classmethod get_filenames N/A N/A cls mapreduce_state N/A N/A N/A filenames N/A N/A shard N/A N/A N/A model ShardState find_all_by_mapreduce_state mapreduce_state N/A N/A N/A N/A shard result_status N/A N/A model ShardState RESULT_SUCCESS N/A N/A N/A N/A filenames append N/A N/A cls _get_filename shard N/A filenames N/A classmethod _get_write_buffer N/A N/A self N/A N/A N/A Returns a buffer to be used by the write() method. N/A N/A NotImplementedError N/A write N/A N/A self data N/A N/A N/A Write data to the GoogleCloudStorage file.

    Args:
      data: string containing the data to be written.
     N/A start_time N/A N/A time time N/A N/A N/A N/A N/A self _get_write_buffer write data N/A ctx N/A N/A context get N/A N/A N/A N/A N/A operation counters Increment COUNTER_IO_WRITE_BYTES N/A len data ctx N/A N/A N/A N/A N/A operation counters Increment COUNTER_IO_WRITE_MSEC N/A int N/A N/A N/A N/A time time start_time 1000 ctx N/A _supports_shard_retry N/A N/A self tstate N/A N/A N/A True N/A N/A _GoogleCloudStorageOutputWriter N/A _GoogleCloudStorageOutputWriterBase N/A N/A Naive version of GoogleCloudStorageWriter.

  This version is known to create inconsistent outputs if the input changes
  during slice retries. Consider using GoogleCloudStorageConsistentOutputWriter
  instead.

  Optional configuration in the mapper_spec.output_writer dictionary:
    _NO_DUPLICATE: if True, slice recovery logic will be used to ensure
      output files has no duplicates. Every shard should have only one final
      output in user specified location. But it may produce many smaller
      files (named "seg") due to slice recovery. These segs live in a
      tmp directory and should be combined and renamed to the final location.
      In current impl, they are not combined.
   N/A _SEG_PREFIX seg_prefix N/A _LAST_SEG_INDEX last_seg_index N/A _JSON_GCS_BUFFER buffer N/A _JSON_SEG_INDEX seg_index N/A _JSON_NO_DUP no_dup N/A _VALID_LENGTH x-goog-meta-gae-mr-valid-length N/A _NO_DUPLICATE no_duplicate __init__ N/A N/A self streaming_buffer writer_spec N/A None N/A N/A Initialize a GoogleCloudStorageOutputWriter instance.

    Args:
      streaming_buffer: an instance of writable buffer from cloudstorage_api.

      writer_spec: the specification for the writer.
     N/A N/A self _streaming_buffer streaming_buffer N/A N/A self _no_dup False N/A writer_spec N/A N/A N/A self _no_dup N/A N/A writer_spec get N/A self _NO_DUPLICATE False N/A N/A self _no_dup N/A N/A N/A self _seg_index N/A int N/A N/A N/A N/A streaming_buffer name rsplit - 1 N/A 1 N/A N/A self _seg_valid_length 0 N/A validate N/A N/A cls mapper_spec N/A N/A N/A Inherit docs. N/A writer_spec N/A N/A cls get_params mapper_spec allow_old False N/A N/A N/A N/A writer_spec get N/A cls _NO_DUPLICATE False N/A True False N/A N/A N/A N/A errors BadWriterParamsError No duplicate must a boolean. N/A N/A N/A N/A super _GoogleCloudStorageOutputWriter cls validate mapper_spec N/A classmethod _get_write_buffer N/A N/A self N/A N/A N/A N/A self _streaming_buffer N/A create N/A N/A cls mr_spec shard_number shard_attempt _writer_state N/A None N/A N/A Inherit docs. N/A writer_spec N/A N/A cls get_params N/A mr_spec mapper allow_old False N/A seg_index None N/A N/A N/A writer_spec get N/A cls _NO_DUPLICATE False N/A N/A seg_index 0 N/A key N/A N/A cls _generate_filename writer_spec N/A mr_spec name N/A mr_spec mapreduce_id shard_number shard_attempt seg_index N/A N/A N/A cls _create writer_spec key N/A classmethod _create N/A N/A cls writer_spec filename_suffix N/A N/A N/A Helper method that actually creates the file in cloud storage. N/A writer N/A N/A cls _open_file writer_spec filename_suffix N/A N/A cls writer writer_spec writer_spec N/A classmethod from_json N/A N/A cls state N/A N/A N/A writer N/A cls N/A N/A pickle loads N/A state N/A N/A cls _JSON_GCS_BUFFER N/A no_dup N/A N/A state get N/A cls _JSON_NO_DUP False N/A N/A writer _no_dup no_dup N/A no_dup N/A N/A N/A writer _seg_valid_length N/A state N/A N/A cls _VALID_LENGTH N/A N/A writer _seg_index N/A state N/A N/A cls _JSON_SEG_INDEX N/A writer N/A classmethod end_slice N/A N/A self slice_ctx N/A N/A N/A N/A N/A N/A self _streaming_buffer closed N/A N/A N/A N/A N/A self _streaming_buffer flush N/A to_json N/A N/A self N/A N/A N/A result N/A N/A self _JSON_GCS_BUFFER N/A self _JSON_NO_DUP N/A N/A pickle dumps N/A self _streaming_buffer N/A self _no_dup N/A N/A self _no_dup N/A N/A N/A N/A result update N/A N/A self _VALID_LENGTH N/A self _JSON_SEG_INDEX N/A N/A N/A self _streaming_buffer tell N/A self _seg_index N/A result N/A finalize N/A N/A self ctx shard_state N/A N/A N/A N/A N/A N/A self _streaming_buffer close N/A N/A self _no_dup N/A N/A N/A N/A cloudstorage_api copy2 N/A N/A self _streaming_buffer name N/A N/A self _streaming_buffer name metadata N/A N/A self _VALID_LENGTH N/A N/A N/A self _streaming_buffer tell N/A mr_spec N/A ctx mapreduce_spec N/A writer_spec N/A N/A self get_params N/A mr_spec mapper allow_old False N/A filename N/A N/A self _generate_filename writer_spec N/A mr_spec name N/A mr_spec mapreduce_id N/A shard_state shard_number N/A seg_filename N/A N/A self _streaming_buffer name N/A N/A prefix last_index N/A N/A seg_filename rsplit - 1 N/A N/A shard_state writer_state N/A N/A self _SEG_PREFIX N/A self _LAST_SEG_INDEX filename N/A prefix - N/A int last_index filename N/A N/A N/A shard_state writer_state N/A filename N/A N/A self _streaming_buffer name N/A _supports_slice_recovery N/A N/A self mapper_spec N/A N/A N/A writer_spec N/A N/A self get_params mapper_spec allow_old False N/A N/A N/A writer_spec get N/A self _NO_DUPLICATE False N/A _recover N/A N/A self mr_spec shard_number shard_attempt N/A N/A N/A next_seg_index N/A self _seg_index N/A N/A N/A self _seg_valid_length 0 N/A N/A N/A N/A gcs_next_offset N/A N/A N/A N/A self _streaming_buffer _get_offset_from_gcs 1 N/A N/A gcs_next_offset N/A N/A N/A self _streaming_buffer tell N/A N/A N/A N/A N/A self _streaming_buffer _force_close gcs_next_offset N/A N/A N/A N/A N/A self _streaming_buffer close N/A N/A N/A N/A cloudstorage FileClosedError N/A N/A N/A N/A N/A cloudstorage_api copy2 N/A N/A self _streaming_buffer name N/A N/A self _streaming_buffer name metadata N/A N/A self _VALID_LENGTH N/A self _seg_valid_length N/A next_seg_index N/A N/A self _seg_index 1 N/A writer_spec N/A N/A self get_params N/A mr_spec mapper allow_old False N/A key N/A N/A self _generate_filename writer_spec N/A mr_spec name N/A mr_spec mapreduce_id shard_number shard_attempt next_seg_index N/A new_writer N/A N/A self _create writer_spec key N/A N/A new_writer _seg_index next_seg_index N/A new_writer N/A _get_filename_for_test N/A N/A self N/A N/A N/A N/A N/A self _streaming_buffer name N/A N/A N/A GoogleCloudStorageOutputWriter _GoogleCloudStorageOutputWriter _ConsistentStatus N/A object N/A N/A Object used to pass status to the next slice. __init__ N/A N/A self N/A N/A N/A N/A self writer_spec None N/A N/A self mapreduce_id None N/A N/A self shard None N/A N/A self mainfile None N/A N/A self tmpfile None N/A N/A self tmpfile_1ago None N/A N/A GoogleCloudStorageConsistentOutputWriter N/A _GoogleCloudStorageOutputWriterBase N/A N/A Output writer to Google Cloud Storage using the cloudstorage library.

  This version ensures that the output written to GCS is consistent.
   N/A _JSON_STATUS status N/A _RAND_BITS 128 N/A _REWRITE_BLOCK_SIZE N/A 1024 256 N/A _REWRITE_MR_TMP gae_mr_tmp N/A _TMPFILE_PATTERN N/A _REWRITE_MR_TMP /$id-tmp-$shard-$random N/A _TMPFILE_PREFIX N/A _REWRITE_MR_TMP /$id-tmp-$shard- __init__ N/A N/A self status N/A N/A N/A Initialize a GoogleCloudStorageConsistentOutputWriter instance.

    Args:
      status: an instance of _ConsistentStatus with initialized tmpfile
              and mainfile.
     N/A N/A self status status N/A N/A self _data_written_to_slice False N/A _get_write_buffer N/A N/A self N/A N/A N/A N/A N/A N/A self status tmpfile N/A N/A N/A N/A errors FailJobError write buffer called but empty, begin_slice missing? N/A N/A N/A self status tmpfile N/A _get_filename_for_test N/A N/A self N/A N/A N/A N/A N/A N/A self status mainfile name N/A create N/A N/A cls mr_spec shard_number shard_attempt _writer_state N/A None N/A N/A Inherit docs. N/A writer_spec N/A N/A cls get_params N/A mr_spec mapper allow_old False N/A key N/A N/A cls _generate_filename writer_spec N/A mr_spec name N/A mr_spec mapreduce_id shard_number shard_attempt N/A status N/A _ConsistentStatus N/A N/A status writer_spec writer_spec N/A N/A status mainfile N/A N/A cls _open_file writer_spec key N/A N/A status mapreduce_id N/A mr_spec mapreduce_id N/A N/A status shard shard_number N/A N/A cls status N/A classmethod _remove_tmpfile N/A N/A self filename writer_spec N/A N/A N/A N/A filename N/A N/A N/A account_id N/A N/A self _get_tmp_account_id writer_spec N/A N/A N/A N/A N/A cloudstorage_api delete filename _account_id account_id N/A N/A N/A N/A cloud_errors NotFoundError N/A N/A N/A _rewrite_tmpfile N/A N/A self mainfile tmpfile writer_spec N/A N/A N/A Copies contents of tmpfile (name) to mainfile (buffer). N/A N/A mainfile closed N/A N/A N/A account_id N/A N/A self _get_tmp_account_id writer_spec N/A f N/A N/A cloudstorage_api open tmpfile _account_id account_id N/A data N/A N/A f read N/A self _REWRITE_BLOCK_SIZE N/A data N/A N/A N/A N/A mainfile write data N/A data N/A N/A f read N/A self _REWRITE_BLOCK_SIZE N/A N/A N/A f close N/A N/A N/A mainfile flush N/A _create_tmpfile N/A N/A cls status N/A N/A N/A Creates a new random-named tmpfile. N/A tmpl N/A N/A string Template N/A cls _TMPFILE_PATTERN N/A filename N/A N/A tmpl substitute id N/A status mapreduce_id shard N/A status shard random N/A N/A random getrandbits N/A cls _RAND_BITS N/A N/A N/A cls _open_file N/A status writer_spec filename use_tmp_bucket True N/A classmethod begin_slice N/A N/A self slice_ctx N/A N/A N/A status N/A self status N/A writer_spec N/A status writer_spec N/A N/A status tmpfile_1ago N/A N/A N/A N/A self _remove_tmpfile N/A N/A status tmpfile_1ago name writer_spec N/A files_to_keep N/A N/A N/A status tmpfile N/A N/A N/A N/A self _rewrite_tmpfile N/A status mainfile N/A N/A status tmpfile name writer_spec N/A N/A N/A files_to_keep append N/A N/A status tmpfile name N/A N/A N/A self _try_to_clean_garbage writer_spec exclude_list files_to_keep N/A N/A status tmpfile_1ago N/A status tmpfile N/A N/A status tmpfile N/A N/A self _create_tmpfile status N/A N/A N/A status mainfile closed N/A N/A N/A N/A N/A status tmpfile close N/A N/A N/A self _remove_tmpfile N/A N/A status tmpfile name writer_spec N/A from_json N/A N/A cls state N/A N/A N/A N/A cls N/A N/A pickle loads N/A state N/A N/A cls _JSON_STATUS N/A classmethod end_slice N/A N/A self slice_ctx N/A N/A N/A N/A N/A N/A N/A self status tmpfile close N/A to_json N/A N/A self N/A N/A N/A N/A N/A self _JSON_STATUS N/A N/A pickle dumps N/A self status N/A write N/A N/A self data N/A N/A N/A N/A N/A N/A super GoogleCloudStorageConsistentOutputWriter self write data N/A N/A self _data_written_to_slice True N/A _try_to_clean_garbage N/A N/A self writer_spec exclude_list N/A N/A N/A N/A Tries to remove any files created by this shard that aren't needed.

    Args:
      writer_spec: writer_spec for the MR.
      exclude_list: A list of filenames (strings) that should not be
        removed.
     N/A tmpl N/A N/A string Template N/A self _TMPFILE_PREFIX N/A prefix N/A N/A tmpl substitute id N/A N/A self status mapreduce_id shard N/A N/A self status shard N/A bucket N/A N/A self _get_tmp_gcs_bucket writer_spec N/A account_id N/A N/A self _get_tmp_account_id writer_spec N/A f N/A N/A cloudstorage listbucket N/A /%s/%s N/A bucket prefix _account_id account_id N/A N/A N/A N/A f filename exclude_list N/A N/A N/A N/A self _remove_tmpfile N/A f filename N/A N/A self status writer_spec N/A finalize N/A N/A self ctx shard_state N/A N/A N/A N/A self _data_written_to_slice N/A N/A N/A N/A errors FailJobError finalize() called after data was written N/A N/A N/A self status tmpfile N/A N/A N/A N/A N/A N/A self status tmpfile close N/A N/A N/A N/A N/A self status mainfile close N/A N/A N/A self status tmpfile_1ago N/A N/A N/A N/A self _remove_tmpfile N/A N/A N/A self status tmpfile_1ago name N/A N/A self status writer_spec N/A N/A N/A self status tmpfile N/A N/A N/A N/A self _remove_tmpfile N/A N/A N/A self status tmpfile name N/A N/A self status writer_spec N/A N/A N/A self _try_to_clean_garbage N/A N/A self status writer_spec N/A N/A shard_state writer_state N/A filename N/A N/A N/A self status mainfile name N/A N/A _GoogleCloudStorageRecordOutputWriterBase N/A _GoogleCloudStorageBase N/A N/A Wraps a GCS writer with a records.RecordsWriter.

  This class wraps a WRITER_CLS (and its instance) and delegates most calls
  to it. write() calls are done using records.RecordsWriter.

  WRITER_CLS has to be set to a subclass of _GoogleCloudStorageOutputWriterBase.

  For list of supported parameters see _GoogleCloudStorageBase.
   N/A WRITER_CLS None __init__ N/A N/A self writer N/A N/A N/A N/A self _writer writer N/A N/A self _record_writer N/A N/A records RecordsWriter writer N/A validate N/A N/A cls mapper_spec N/A N/A N/A N/A N/A N/A cls WRITER_CLS validate mapper_spec N/A classmethod init_job N/A N/A cls mapreduce_state N/A N/A N/A N/A N/A N/A cls WRITER_CLS init_job mapreduce_state N/A classmethod finalize_job N/A N/A cls mapreduce_state N/A N/A N/A N/A N/A N/A cls WRITER_CLS finalize_job mapreduce_state N/A classmethod from_json N/A N/A cls state N/A N/A N/A N/A cls N/A N/A N/A cls WRITER_CLS from_json state N/A classmethod to_json N/A N/A self N/A N/A N/A N/A N/A N/A self _writer to_json N/A create N/A N/A cls mr_spec shard_number shard_attempt _writer_state N/A None N/A N/A N/A cls N/A N/A N/A cls WRITER_CLS create mr_spec shard_number shard_attempt _writer_state N/A classmethod write N/A N/A self data N/A N/A N/A N/A N/A N/A self _record_writer write data N/A finalize N/A N/A self ctx shard_state N/A N/A N/A N/A N/A N/A self _writer finalize ctx shard_state N/A get_filenames N/A N/A cls mapreduce_state N/A N/A N/A N/A N/A N/A cls WRITER_CLS get_filenames mapreduce_state N/A classmethod _supports_shard_retry N/A N/A self tstate N/A N/A N/A N/A N/A N/A self _writer _supports_shard_retry tstate N/A _supports_slice_recovery N/A N/A self mapper_spec N/A N/A N/A N/A N/A N/A self _writer _supports_slice_recovery mapper_spec N/A _recover N/A N/A self mr_spec shard_number shard_attempt N/A N/A N/A N/A N/A N/A self _writer _recover mr_spec shard_number shard_attempt N/A begin_slice N/A N/A self slice_ctx N/A N/A N/A N/A N/A N/A self _writer begin_slice slice_ctx N/A end_slice N/A N/A self slice_ctx N/A N/A N/A N/A N/A N/A N/A N/A self _writer _get_write_buffer closed N/A N/A N/A N/A N/A self _record_writer _pad_block N/A N/A N/A N/A self _writer end_slice slice_ctx N/A N/A _GoogleCloudStorageRecordOutputWriter N/A _GoogleCloudStorageRecordOutputWriterBase N/A N/A WRITER_CLS _GoogleCloudStorageOutputWriter N/A N/A GoogleCloudStorageRecordOutputWriter _GoogleCloudStorageRecordOutputWriter GoogleCloudStorageConsistentRecordOutputWriter N/A _GoogleCloudStorageRecordOutputWriterBase N/A N/A WRITER_CLS GoogleCloudStorageConsistentOutputWriter N/A _GoogleCloudStorageKeyValueOutputWriter N/A _GoogleCloudStorageRecordOutputWriter N/A N/A Write key/values to Google Cloud Storage files in LevelDB format. write N/A N/A self data N/A N/A N/A N/A N/A len data 2 N/A N/A N/A N/A logging error Got bad tuple of length %d (2-tuple expected): %s N/A len data data N/A N/A N/A key N/A str N/A data N/A 0 N/A value N/A str N/A data N/A 1 N/A N/A N/A TypeError N/A N/A N/A N/A logging error Expecting a tuple, but got %s: %s N/A N/A data __class__ __name__ data N/A proto N/A N/A kv_pb KeyValue N/A N/A N/A proto set_key key N/A N/A N/A proto set_value value N/A N/A N/A GoogleCloudStorageRecordOutputWriter write self N/A N/A proto Encode N/A N/A N/A GoogleCloudStorageKeyValueOutputWriter _GoogleCloudStorageKeyValueOutputWriter