Module Expr Str ImportFrom alias Import alias Import alias Import alias Import alias Import alias Import alias ImportFrom alias TryExcept body ImportFrom alias handlers ExceptHandler type NameLoad body ImportFrom alias Import alias TryExcept body ImportFrom alias handlers ExceptHandler type NameLoad body ImportFrom alias ImportFrom alias ImportFrom alias ImportFrom alias ImportFrom alias ImportFrom alias ImportFrom alias alias alias alias alias alias ImportFrom alias Assign NameStore Call AttributeLoad NameLoad attr Str Assign NameStore Str Assign NameStore Str ClassDef bases NameLoad body Pass decorator_list ClassDef bases NameLoad body Pass decorator_list ClassDef bases NameLoad body Expr Str FunctionDef arguments args NameParam NameParam NameParam defaults NameLoad NameLoad kwarg body Import alias Import alias ImportFrom alias Assign NameStore Call AttributeLoad NameLoad attr Expr Call AttributeLoad NameLoad attr NameLoad Assign NameStore Call AttributeLoad NameLoad attr Str Assign NameStore Call AttributeLoad NameLoad attr Str Assign NameStore NameLoad If BoolOpAnd NameLoad NameLoad body ImportFrom alias Assign NameStore Call AttributeLoad NameLoad attr Assign NameStore Call AttributeLoad NameLoad attr NameLoad NameLoad Assign NameStore AttributeLoad AttributeLoad NameLoad attr attr Assign NameStore AttributeLoad AttributeLoad NameLoad attr attr Assign NameStore AttributeLoad AttributeLoad NameLoad attr attr orelse If UnaryOpNot NameLoad body Assign NameStore Call AttributeLoad NameLoad attr Str If UnaryOpNot NameLoad body Assign NameStore Call AttributeLoad NameLoad attr Str For NameStore ListLoad Str Str Str Str body If CompareIn NameLoad NameLoad body Expr Call AttributeLoad NameLoad attr NameLoad Assign AttributeStore NameLoad attr Call AttributeLoad AttributeLoad AttributeLoad NameLoad attr attr attr NameLoad NameLoad keyword NameLoad NameLoad Assign AttributeStore NameLoad attr NameLoad decorator_list FunctionDef arguments args NameParam NameParam defaults body Expr Str Assign TupleStore NameStore NameStore Call AttributeLoad NameLoad attr NameLoad Assign NameStore Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad keyword NameLoad If Call AttributeLoad NameLoad attr NameLoad body Return NameLoad Assign NameStore Call AttributeLoad NameLoad attr NameLoad If NameLoad body Return NameLoad If Call AttributeLoad NameLoad attr NameLoad body Return NameLoad Expr Call AttributeLoad NameLoad attr Str NameLoad Return NameLoad decorator_list FunctionDef arguments args NameParam NameParam NameParam defaults NameLoad body Expr Str If UnaryOpNot Call AttributeLoad NameLoad attr NameLoad body Expr Call AttributeLoad NameLoad attr Str NameLoad Return NameLoad Assign TupleStore NameStore NameStore Call AttributeLoad NameLoad attr NameLoad If Call AttributeLoad NameLoad attr NameLoad body Raise Call NameLoad BinOpMod Str NameLoad Assign NameStore Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad keyword NameLoad Assign NameStore Call AttributeLoad NameLoad attr NameLoad If NameLoad body Expr Call AttributeLoad NameLoad attr NameLoad Expr Call AttributeLoad NameLoad attr Str NameLoad NameLoad Return NameLoad If BoolOpAnd Call AttributeLoad NameLoad attr NameLoad UnaryOpNot NameLoad body Raise Call NameLoad BinOpMod Str NameLoad Assign NameStore ListComp NameLoad comprehension NameStore Call AttributeLoad NameLoad attr Call AttributeLoad NameLoad attr NameLoad Assign NameStore Call AttributeLoad NameLoad attr BinOpAdd NameLoad NameLoad If NameLoad body Expr Call AttributeLoad NameLoad attr NameLoad If CompareGt Call NameLoad NameLoad Num body For NameStore NameLoad body Expr Call AttributeLoad NameLoad attr Str NameLoad NameLoad Expr Call AttributeLoad NameLoad attr NameLoad Return NameLoad Return NameLoad decorator_list FunctionDef arguments args NameParam NameParam defaults body Assign TupleStore NameStore NameStore Call AttributeLoad NameLoad attr NameLoad Assign NameStore Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad keyword NameLoad Return Call AttributeLoad NameLoad attr NameLoad decorator_list FunctionDef arguments args NameParam NameParam NameParam defaults kwarg body Expr Str Assign TupleStore NameStore NameStore Call AttributeLoad NameLoad attr NameLoad Assign NameStore Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad keyword NameLoad Assign NameStore Call AttributeLoad NameLoad attr NameLoad Assign AttributeStore NameLoad attr NameLoad Expr Call AttributeLoad NameLoad attr NameLoad NameLoad decorator_list FunctionDef arguments args NameParam NameParam NameParam defaults kwarg body Expr Str Assign TupleStore NameStore NameStore Call AttributeLoad NameLoad attr NameLoad Assign NameStore Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad keyword NameLoad Assign NameStore Call AttributeLoad NameLoad attr NameLoad Assign AttributeStore NameLoad attr NameLoad Expr Call AttributeLoad NameLoad attr NameLoad NameLoad decorator_list FunctionDef arguments args NameParam NameParam NameParam NameParam defaults Num kwarg body Expr Str Assign NameStore AttributeLoad Call AttributeLoad NameLoad attr NameLoad attr If CompareLtE NameLoad NameLoad body Return Call AttributeLoad NameLoad attr NameLoad NameLoad NameLoad Assign TupleStore NameStore NameStore Call AttributeLoad NameLoad attr NameLoad Assign NameStore Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad keyword NameLoad Assign NameStore BinOpFloorDiv BinOpSub BinOpAdd NameLoad NameLoad Num NameLoad Assign NameStore NameLoad TryExcept body Assign NameStore Call AttributeLoad NameLoad attr NameLoad NameLoad For NameStore Call NameLoad NameLoad body Assign NameStore BinOpMult NameLoad NameLoad Assign NameStore Call NameLoad NameLoad BinOpSub NameLoad NameLoad With Call NameLoad NameLoad Str NameStore body Assign NameStore BinOpAdd NameLoad Num Expr Call AttributeLoad NameLoad attr Str NameLoad NameLoad NameLoad Expr Call AttributeLoad NameLoad attr NameLoad Expr Call AttributeLoad NameLoad attr NameLoad keyword NameLoad keyword NameLoad Expr Call AttributeLoad NameLoad attr handlers ExceptHandler type NameLoad body If NameLoad body Expr Call AttributeLoad NameLoad attr Str NameLoad Expr Call AttributeLoad NameLoad attr Raise decorator_list FunctionDef arguments args NameParam NameParam NameParam defaults body Expr Str Assign TupleStore NameStore NameStore Call AttributeLoad NameLoad attr NameLoad Assign NameStore Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad keyword NameLoad Assign NameStore Call AttributeLoad NameLoad attr NameLoad Assign AttributeStore NameLoad attr NameLoad Expr Call AttributeLoad NameLoad attr NameLoad decorator_list FunctionDef arguments args NameParam NameParam defaults body Expr Str Assign TupleStore NameStore NameStore Call AttributeLoad NameLoad attr NameLoad Assign NameStore Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad keyword NameLoad Assign NameStore Call AttributeLoad NameLoad attr NameLoad Assign AttributeStore NameLoad attr NameLoad Assign NameStore Call AttributeLoad NameLoad attr Return NameLoad decorator_list FunctionDef arguments args NameParam NameParam NameParam NameParam NameParam NameParam NameParam defaults Num NameLoad NameLoad Num kwarg body Expr Str Assign NameStore Call AttributeLoad AttributeLoad NameLoad attr attr Assign TupleStore NameStore NameStore Call AttributeLoad NameLoad attr NameLoad Assign TupleStore NameStore NameStore Call AttributeLoad NameLoad attr NameLoad Assign NameStore IfExp CompareLt NameLoad Num Num NameLoad Assign NameStore Num Assign NameStore Call NameLoad keyword NameLoad If Call AttributeLoad NameLoad attr NameLoad body Assign NameStore ListLoad Assign NameStore Call NameLoad keyword NameLoad Assign NameStore Call AttributeLoad NameLoad attr NameLoad Assign NameStore Call AttributeLoad NameLoad attr NameLoad For NameStore Call AttributeLoad NameLoad attr NameLoad keyword NameLoad keyword NameLoad body If BoolOpAnd CompareNotEq NameLoad Str CompareNotEq NameLoad Str body AugAssignAdd NameStore Num Assign NameStore Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr keyword TupleLoad NameLoad NameLoad BinOpAdd NameLoad NameLoad NameLoad BinOpAdd NameLoad NameLoad NameLoad keyword NameLoad Expr Call AttributeLoad NameLoad attr NameLoad Expr Call AttributeLoad NameLoad attr Expr Call AttributeLoad NameLoad attr Expr Call AttributeLoad NameLoad attr Expr Call AttributeLoad NameLoad attr For NameStore NameLoad body Expr Call AttributeLoad NameLoad attr Assign NameStore Call AttributeLoad AttributeLoad NameLoad attr attr Assign NameStore BinOpSub NameLoad NameLoad Expr Call AttributeLoad NameLoad attr BinOpMod Str TupleLoad Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad NameLoad orelse Expr Call AttributeLoad NameLoad attr NameLoad NameLoad NameLoad NameLoad NameLoad NameLoad NameLoad Expr Call AttributeLoad NameLoad attr Expr Call AttributeLoad NameLoad attr decorator_list FunctionDef arguments args NameParam NameParam NameParam NameParam NameParam NameParam NameParam defaults kwarg body Expr Str Assign NameStore Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad keyword NameLoad Assign NameStore Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad keyword NameLoad Assign NameStore AttributeLoad Call AttributeLoad NameLoad attr NameLoad attr If BoolOpOr CompareEq NameLoad Num CompareLtE NameLoad NameLoad body Assign NameStore Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr keyword TupleLoad NameLoad NameLoad NameLoad keyword NameLoad Return Call AttributeLoad NameLoad attr Assign NameStore NameLoad TryExcept body Assign NameStore Call AttributeLoad NameLoad attr NameLoad NameLoad Assign NameStore Num Assign NameStore ListLoad Assign NameStore BinOpFloorDiv BinOpSub BinOpAdd NameLoad NameLoad Num NameLoad For NameStore Call NameLoad NameLoad body Assign NameStore NameLoad AugAssignAdd NameStore NameLoad Assign NameStore Call NameLoad BinOpSub NameLoad Num BinOpSub NameLoad Num Assign NameStore BinOpAdd NameLoad Num Expr Call AttributeLoad NameLoad attr Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr keyword TupleLoad NameLoad NameLoad NameLoad NameLoad NameLoad Expr Call AttributeLoad NameLoad attr Str NameLoad NameLoad NameLoad NameLoad Expr Call AttributeLoad NameLoad attr Str NameLoad NameLoad For NameStore NameLoad body Expr Call AttributeLoad NameLoad attr Expr Call AttributeLoad NameLoad attr Return AttributeLoad NameLoad attr handlers ExceptHandler body Expr Call AttributeLoad NameLoad attr Str NameLoad NameLoad NameLoad NameLoad If NameLoad body Expr Call AttributeLoad NameLoad attr Raise decorator_list FunctionDef arguments args NameParam defaults vararg kwarg body Expr Str Expr Call AttributeLoad NameLoad attr NameLoad NameLoad decorator_list FunctionDef arguments args NameParam NameParam NameParam defaults kwarg body Expr Str Expr Call AttributeLoad NameLoad attr NameLoad NameLoad NameLoad Expr Call AttributeLoad NameLoad attr NameLoad decorator_list FunctionDef arguments args NameParam NameParam NameParam NameParam defaults NameLoad NameLoad body Expr Str Assign TupleStore NameStore NameStore Call AttributeLoad NameLoad attr NameLoad Assign NameStore Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad keyword NameLoad Assign NameStore Call AttributeLoad NameLoad attr NameLoad Assign NameStore Call NameLoad NameLoad For NameStore Call AttributeLoad NameLoad attr keyword NameLoad body Assign NameStore Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr Str If BoolOpOr BoolOpAnd UnaryOpNot NameLoad UnaryOpNot NameLoad BoolOpAnd NameLoad UnaryOpNot NameLoad CompareLt NameLoad NameLoad BoolOpAnd UnaryOpNot NameLoad NameLoad CompareLt NameLoad NameLoad BoolOpAnd NameLoad NameLoad CompareLtLt NameLoad NameLoad NameLoad body Expr Yield BinOpAdd Call AttributeLoad NameLoad attr NameLoad SubscriptLoad AttributeLoad NameLoad attr Slice NameLoad decorator_list FunctionDef arguments args NameParam NameParam NameParam NameParam defaults NameLoad NameLoad body Assign NameStore Call NameLoad Call AttributeLoad NameLoad attr NameLoad For NameStore Call AttributeLoad NameLoad attr NameLoad keyword NameLoad keyword NameLoad body Expr Yield SubscriptLoad NameLoad Slice NameLoad decorator_list FunctionDef arguments args NameParam NameParam defaults body Expr Str Assign TupleStore NameStore NameStore Call AttributeLoad NameLoad attr NameLoad Assign NameStore Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad keyword NameLoad If Call AttributeLoad NameLoad attr NameLoad body Return NameLoad For NameStore TupleLoad NameLoad NameLoad body Assign NameStore Call AttributeLoad NameLoad attr BinOpAdd NameLoad NameLoad If NameLoad body Return NameLoad Assign NameStore Call AttributeLoad NameLoad attr NameLoad Assign NameStore Call NameLoad Call AttributeLoad NameLoad attr Call AttributeLoad NameLoad attr keyword NameLoad Num If NameLoad body Return NameLoad Return NameLoad decorator_list Assign NameStore NameLoad FunctionDef arguments args NameParam NameParam NameParam NameParam defaults NameLoad NameLoad body If BoolOpAnd NameLoad Call AttributeLoad NameLoad attr NameLoad body Raise Call NameLoad Assign TupleStore NameStore NameStore Call AttributeLoad NameLoad attr NameLoad If Call AttributeLoad NameLoad attr NameLoad body Return Assign NameStore Call AttributeLoad NameLoad attr NameLoad If BoolOpAnd UnaryOpNot NameLoad UnaryOpNot Call AttributeLoad NameLoad attr Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad body Raise Call NameLoad Return Call AttributeLoad NameLoad attr Str Call AttributeLoad NameLoad attr NameLoad decorator_list FunctionDef arguments args NameParam NameParam defaults NameLoad body TryExcept body Assign NameStore Call NameLoad Call AttributeLoad Call AttributeLoad NameLoad attr attr Str handlers ExceptHandler type NameLoad body Return Dict For TupleStore NameStore NameStore Call AttributeLoad NameLoad attr NameLoad body TryExcept body Assign SubscriptStore NameLoad Index NameLoad Call NameLoad NameLoad handlers ExceptHandler type NameLoad body Pass If NameLoad body Return Call AttributeLoad NameLoad attr NameLoad Return NameLoad decorator_list FunctionDef arguments args NameParam NameParam defaults body Assign TupleStore NameStore NameStore NameStore NameStore NameStore Call NameLoad NameLoad Assign NameStore SubscriptLoad NameLoad Slice Num Return TupleLoad NameLoad NameLoad decorator_list FunctionDef arguments args NameParam NameParam defaults body Return BoolOpOr CompareEq Call NameLoad NameLoad Num CompareEq NameLoad Str decorator_list FunctionDef arguments args NameParam NameParam defaults body Return IfExp BoolOpOr CompareEq SubscriptLoad NameLoad Slice Num Str CompareEq NameLoad Str NameLoad BinOpAdd NameLoad Str decorator_list decorator_list ClassDef bases NameLoad body Expr Str FunctionDef arguments args NameParam NameParam NameParam defaults kwarg body Assign AttributeStore NameLoad attr NameLoad Expr Call AttributeLoad Call NameLoad NameLoad NameLoad attr NameLoad Assign AttributeStore NameLoad attr NameLoad decorator_list FunctionDef arguments args NameParam defaults body Expr Call AttributeLoad AttributeLoad NameLoad attr attr AttributeLoad NameLoad attr AttributeLoad NameLoad attr AttributeLoad NameLoad attr decorator_list decorator_list ClassDef bases NameLoad body FunctionDef arguments args NameParam NameParam defaults body Assign AttributeStore NameLoad attr NameLoad Assign AttributeStore NameLoad attr ListLoad Assign AttributeStore NameLoad attr NameLoad Assign AttributeStore NameLoad attr NameLoad decorator_list FunctionDef arguments args NameParam NameParam defaults Num body Assign NameStore Call AttributeLoad AttributeLoad NameLoad attr attr keyword NameLoad If CompareEq NameLoad Str body Assign AttributeStore NameLoad attr NameLoad If AttributeLoad NameLoad attr body Return Str Return NameLoad decorator_list FunctionDef arguments args NameParam defaults body Expr Call AttributeLoad AttributeLoad NameLoad attr attr Assign AttributeStore NameLoad attr NameLoad decorator_list FunctionDef arguments args NameParam defaults body Expr Call AttributeLoad NameLoad attr decorator_list FunctionDef arguments args NameParam NameParam NameParam NameParam defaults body Expr Call AttributeLoad NameLoad attr decorator_list FunctionDef arguments args NameParam defaults body Return NameLoad decorator_list FunctionDef arguments args NameParam NameParam defaults body Expr Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad decorator_list FunctionDef arguments args NameParam defaults body Assign NameStore Call AttributeLoad Str attr AttributeLoad NameLoad attr Assign AttributeStore NameLoad attr ListLoad Return NameLoad decorator_list FunctionDef arguments args NameParam defaults body Return NameLoad decorator_list FunctionDef arguments args NameParam defaults body Return NameLoad decorator_list FunctionDef arguments args NameParam defaults body Return NameLoad decorator_list FunctionDef arguments args NameParam defaults body Assign NameStore Call AttributeLoad AttributeLoad NameLoad attr attr Assign NameStore NameLoad While NameLoad body TryExcept body Assign NameStore Call NameLoad NameLoad For NameStore Call AttributeLoad NameLoad attr NameLoad body If UnaryOpNot Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr body Expr Call AttributeLoad NameLoad attr NameLoad orelse If AttributeLoad NameLoad attr body Expr Call AttributeLoad NameLoad attr NameLoad Expr Yield Call AttributeLoad NameLoad attr orelse Expr Yield NameLoad handlers ExceptHandler type NameLoad body Assign NameStore Call AttributeLoad NameLoad attr If NameLoad body Expr Yield NameLoad Assign NameStore NameLoad Expr Call AttributeLoad NameLoad attr decorator_list decorator_list ClassDef bases NameLoad body Expr Str Assign NameStore NameLoad FunctionDef arguments args NameParam NameParam NameParam NameParam defaults NameLoad NameLoad kwarg body Expr Call AttributeLoad Call NameLoad NameLoad NameLoad attr NameLoad If CompareIs NameLoad NameLoad body Assign NameStore Call NameLoad Assign AttributeStore NameLoad attr NameLoad Assign AttributeStore NameLoad attr NameLoad Assign AttributeStore NameLoad attr BoolOpOr NameLoad Call NameLoad Assign AttributeStore NameLoad attr NameLoad decorator_list FunctionDef arguments args NameParam NameParam defaults Str body Expr Str If CompareNotIn NameLoad TupleLoad Str Str body Raise Call NameLoad BinOpMod Str NameLoad If CompareEq NameLoad Str body Assign NameStore Call AttributeLoad AttributeLoad NameLoad attr attr AttributeLoad NameLoad attr If UnaryOpNot NameLoad body Raise Call NameLoad BinOpMod Str AttributeLoad NameLoad attr Assign NameStore Call NameLoad NameLoad Return Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad orelse Return Call AttributeLoad AttributeLoad NameLoad attr attr Call NameLoad AttributeLoad NameLoad attr AttributeLoad NameLoad attr AttributeLoad NameLoad attr decorator_list decorator_list ClassDef bases NameLoad body Expr Str Assign NameStore NameLoad FunctionDef arguments args NameParam NameParam NameParam NameParam NameParam defaults NameLoad NameLoad Str body Expr Str If CompareIs NameLoad NameLoad body Assign NameStore Call NameLoad If CompareNotEq SubscriptLoad NameLoad Index Num Str body Raise Call NameLoad Str Expr Call AttributeLoad Call NameLoad NameLoad NameLoad attr NameLoad NameLoad NameLoad Assign AttributeStore NameLoad attr NameLoad decorator_list FunctionDef arguments args NameParam defaults body Assign NameStore BinOpAdd AttributeLoad NameLoad attr AttributeLoad NameLoad attr Return Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad decorator_list decorator_list ClassDef bases NameLoad body Expr Str FunctionDef arguments args NameParam defaults vararg kwarg body Expr Call AttributeLoad NameLoad attr Str Expr Call AttributeLoad Call NameLoad NameLoad NameLoad attr NameLoad NameLoad decorator_list decorator_list ClassDef bases NameLoad body Expr Str Assign NameStore Call NameLoad FunctionDef arguments args NameParam defaults body Return Call NameLoad AttributeLoad NameLoad attr decorator_list decorator_list ClassDef bases NameLoad body Expr Str Assign NameStore Call NameLoad FunctionDef arguments args NameParam defaults body Return Call NameLoad AttributeLoad NameLoad attr decorator_list decorator_list ClassDef bases NameLoad body Expr Str Assign NameStore Call NameLoad Assign NameStore Call NameLoad keyword NameLoad FunctionDef arguments args NameParam defaults body Return Call NameLoad AttributeLoad NameLoad attr keyword AttributeLoad NameLoad attr decorator_list decorator_list
N/A N/A 
Implementation of Simple Storage Service support.
:py:class:`S3Target` is a subclass of the Target class to support S3 file
system operations. The `boto` library is required to use S3 targets.
 __future__ division N/A datetime N/A itertools N/A logging N/A os N/A os.path N/A time multiprocessing.pool ThreadPool N/A N/A urlparse urlsplit N/A N/A N/A ImportError N/A urllib.parse urlsplit N/A warnings N/A N/A ConfigParser NoSectionError N/A N/A N/A ImportError N/A configparser NoSectionError luigi six luigi.six.moves range luigi configuration luigi.format get_default_format luigi.parameter Parameter luigi.target FileAlreadyExists FileSystem FileSystemException FileSystemTarget AtomicLocalFile MissingParentDirectory luigi.task ExternalTask N/A logger N/A N/A logging getLogger luigi-interface N/A S3_DIRECTORY_MARKER_SUFFIX_0 _$folder$ N/A S3_DIRECTORY_MARKER_SUFFIX_1 / InvalidDeleteException N/A FileSystemException N/A N/A N/A FileNotFoundException N/A FileSystemException N/A N/A N/A S3Client N/A FileSystem N/A N/A 
    boto-powered S3 client.
     __init__ N/A N/A self aws_access_key_id aws_secret_access_key N/A None None kwargs N/A N/A boto N/A boto.s3.connection boto.s3.key Key N/A options N/A N/A self _get_s3_config N/A N/A N/A options update kwargs N/A role_arn N/A N/A options get aws_role_arn N/A role_session_name N/A N/A options get aws_role_session_name N/A aws_session_token None N/A N/A role_arn role_session_name N/A boto sts N/A sts_client N/A N/A sts STSConnection N/A assumed_role N/A N/A sts_client assume_role role_arn role_session_name N/A aws_secret_access_key N/A N/A assumed_role credentials secret_key N/A aws_access_key_id N/A N/A assumed_role credentials access_key N/A aws_session_token N/A N/A assumed_role credentials session_token N/A N/A N/A aws_access_key_id N/A N/A aws_access_key_id N/A N/A options get aws_access_key_id N/A N/A aws_secret_access_key N/A N/A aws_secret_access_key N/A N/A options get aws_secret_access_key N/A key N/A aws_access_key_id aws_secret_access_key aws_role_session_name aws_role_arn N/A N/A N/A key options N/A N/A N/A N/A options pop key N/A N/A self s3 N/A N/A N/A N/A boto s3 connection S3Connection aws_access_key_id aws_secret_access_key security_token aws_session_token options N/A N/A self Key Key N/A exists N/A N/A self path N/A N/A N/A 
        Does provided path exist on S3?
         N/A N/A bucket key N/A N/A self _path_to_bucket_and_key path N/A s3_bucket N/A N/A N/A self s3 get_bucket bucket validate True N/A N/A N/A self _is_root key N/A N/A True N/A s3_key N/A N/A s3_bucket get_key key N/A s3_key N/A N/A True N/A N/A N/A self isdir path N/A N/A True N/A N/A N/A logger debug Path %s does not exist path N/A False N/A remove N/A N/A self path recursive N/A True N/A N/A 
        Remove a file or directory from S3.
         N/A N/A N/A N/A self exists path N/A N/A N/A N/A logger debug Could not delete %s; path does not exist path N/A False N/A N/A bucket key N/A N/A self _path_to_bucket_and_key path N/A N/A N/A self _is_root key N/A N/A N/A InvalidDeleteException N/A Cannot delete root of bucket at path %s path N/A s3_bucket N/A N/A N/A self s3 get_bucket bucket validate True N/A s3_key N/A N/A s3_bucket get_key key N/A s3_key N/A N/A N/A N/A s3_bucket delete_key s3_key N/A N/A N/A logger debug Deleting %s from bucket %s key bucket N/A True N/A N/A N/A N/A self isdir path N/A recursive N/A N/A N/A InvalidDeleteException N/A Path %s is a directory. Must use recursive delete path N/A delete_key_list N/A k N/A k N/A N/A s3_bucket list N/A N/A self _add_path_delimiter key N/A s3_dir_with_suffix_key N/A N/A s3_bucket get_key N/A key S3_DIRECTORY_MARKER_SUFFIX_0 N/A s3_dir_with_suffix_key N/A N/A N/A N/A delete_key_list append s3_dir_with_suffix_key N/A N/A N/A len delete_key_list 0 N/A N/A k delete_key_list N/A N/A N/A N/A logger debug Deleting %s from bucket %s k bucket N/A N/A N/A s3_bucket delete_keys delete_key_list N/A True N/A False N/A get_key N/A N/A self path N/A N/A N/A N/A bucket key N/A N/A self _path_to_bucket_and_key path N/A s3_bucket N/A N/A N/A self s3 get_bucket bucket validate True N/A N/A N/A s3_bucket get_key key N/A put N/A N/A self local_path destination_s3_path N/A kwargs N/A N/A 
        Put an object stored locally to an S3 path.

        :param kwargs: Keyword arguments are passed to the boto function `set_contents_from_filename`
         N/A N/A bucket key N/A N/A self _path_to_bucket_and_key destination_s3_path N/A s3_bucket N/A N/A N/A self s3 get_bucket bucket validate True N/A s3_key N/A N/A self Key s3_bucket N/A N/A s3_key key key N/A N/A N/A s3_key set_contents_from_filename local_path kwargs N/A put_string N/A N/A self content destination_s3_path N/A kwargs N/A N/A 
        Put a string to an S3 path.

        :param kwargs: Keyword arguments are passed to the boto function `set_contents_from_string`
         N/A N/A bucket key N/A N/A self _path_to_bucket_and_key destination_s3_path N/A s3_bucket N/A N/A N/A self s3 get_bucket bucket validate True N/A s3_key N/A N/A self Key s3_bucket N/A N/A s3_key key key N/A N/A N/A s3_key set_contents_from_string content kwargs N/A put_multipart N/A N/A self local_path destination_s3_path part_size N/A 67108864 kwargs N/A N/A 
        Put an object stored locally to an S3 path
        using S3 multi-part upload (for files > 5GB).

        :param local_path: Path to source local file
        :param destination_s3_path: URL for target S3 location
        :param part_size: Part size in bytes. Default: 67108864 (64MB), must be >= 5MB and <= 5 GB.
        :param kwargs: Keyword arguments are passed to the boto function `initiate_multipart_upload`
         N/A source_size N/A N/A N/A os stat local_path st_size N/A N/A source_size part_size N/A N/A N/A N/A self put local_path destination_s3_path kwargs N/A N/A bucket key N/A N/A self _path_to_bucket_and_key destination_s3_path N/A s3_bucket N/A N/A N/A self s3 get_bucket bucket validate True N/A num_parts N/A N/A N/A source_size part_size 1 part_size N/A mp None N/A N/A N/A mp N/A N/A s3_bucket initiate_multipart_upload key kwargs N/A i N/A range num_parts N/A N/A offset N/A part_size i N/A bytes N/A min part_size N/A source_size offset N/A N/A open local_path rb fp N/A N/A part_num N/A i 1 N/A N/A N/A logger info Uploading part %s/%s to %s part_num num_parts destination_s3_path N/A N/A N/A fp seek offset N/A N/A N/A mp upload_part_from_file fp part_num part_num size bytes N/A N/A N/A mp complete_upload N/A N/A N/A BaseException N/A N/A mp N/A N/A N/A N/A logger info Canceling multipart s3 upload for %s destination_s3_path N/A N/A N/A mp cancel_upload N/A N/A get N/A N/A self s3_path destination_local_path N/A N/A N/A 
        Get an object stored in S3 and write it to a local path.
         N/A N/A bucket key N/A N/A self _path_to_bucket_and_key s3_path N/A s3_bucket N/A N/A N/A self s3 get_bucket bucket validate True N/A s3_key N/A N/A self Key s3_bucket N/A N/A s3_key key key N/A N/A N/A s3_key get_contents_to_filename destination_local_path N/A get_as_string N/A N/A self s3_path N/A N/A N/A 
        Get the contents of an object stored in S3 as a string.
         N/A N/A bucket key N/A N/A self _path_to_bucket_and_key s3_path N/A s3_bucket N/A N/A N/A self s3 get_bucket bucket validate True N/A s3_key N/A N/A self Key s3_bucket N/A N/A s3_key key key N/A contents N/A N/A s3_key get_contents_as_string N/A contents N/A copy N/A N/A self source_path destination_path threads start_time end_time part_size N/A 100 None None 67108864 kwargs N/A N/A 
        Copy object(s) from one S3 location to another. Works for individual keys or entire directories.

        When files are larger than `part_size`, multipart uploading will be used.

        :param source_path: The `s3://` path of the directory or key to copy from
        :param destination_path: The `s3://` path of the directory or key to copy to
        :param threads: Optional argument to define the number of threads to use when copying (min: 3 threads)
        :param start_time: Optional argument to copy files with modified dates after start_time
        :param end_time: Optional argument to copy files with modified dates before end_time
        :param part_size: Part size in bytes. Default: 67108864 (64MB), must be >= 5MB and <= 5 GB.
        :param kwargs: Keyword arguments are passed to the boto function `copy_key`
         N/A start N/A N/A N/A datetime datetime now N/A N/A src_bucket src_key N/A N/A self _path_to_bucket_and_key source_path N/A N/A dst_bucket dst_key N/A N/A self _path_to_bucket_and_key destination_path N/A threads N/A N/A threads 3 3 threads N/A total_keys 0 N/A copy_pool N/A ThreadPool processes threads N/A N/A N/A self isdir source_path N/A N/A copy_jobs N/A N/A management_pool N/A ThreadPool processes threads N/A src_prefix N/A N/A self _add_path_delimiter src_key N/A dst_prefix N/A N/A self _add_path_delimiter dst_key N/A key N/A N/A self list source_path start_time start_time end_time end_time N/A N/A N/A N/A key N/A key / N/A N/A total_keys 1 N/A job N/A N/A management_pool apply_async N/A self __copy_multipart args N/A copy_pool src_bucket N/A src_prefix key dst_bucket N/A dst_prefix key part_size kwds kwargs N/A N/A N/A copy_jobs append job N/A N/A N/A management_pool close N/A N/A N/A management_pool join N/A N/A N/A copy_pool close N/A N/A N/A copy_pool join N/A result copy_jobs N/A N/A N/A N/A result get N/A end N/A N/A N/A datetime datetime now N/A duration N/A end start N/A N/A N/A logger info N/A %s : Complete : %s total keys copied in %s N/A N/A N/A N/A datetime datetime now total_keys duration N/A N/A N/A N/A self __copy_multipart copy_pool src_bucket src_key dst_bucket dst_key part_size kwargs N/A N/A N/A copy_pool close N/A N/A N/A copy_pool join N/A __copy_multipart N/A N/A self pool src_bucket src_key dst_bucket dst_key part_size N/A kwargs N/A N/A 
        Copy a single S3 object to another S3 object, falling back to multipart copy where necessary

        NOTE: This is a private method and should only be called from within the `luigi.s3.copy` method

        :param pool: The threadpool to put the s3 copy processes onto
        :param src_bucket: source bucket name
        :param src_key: source key name
        :param dst_bucket: destination bucket name
        :param dst_key: destination key name
        :param key_size: size of the key to copy in bytes
        :param part_size: Part size in bytes. Must be >= 5MB and <= 5 GB.
        :param kwargs: Keyword arguments are passed to the boto function `initiate_multipart_upload`
         N/A source_bucket N/A N/A N/A self s3 get_bucket src_bucket validate True N/A dest_bucket N/A N/A N/A self s3 get_bucket dst_bucket validate True N/A key_size N/A N/A N/A source_bucket lookup src_key size N/A N/A N/A key_size 0 N/A key_size part_size N/A N/A result N/A N/A pool apply_async N/A dest_bucket copy_key args N/A dst_key src_bucket src_key kwds kwargs N/A N/A N/A result get N/A mp None N/A N/A N/A mp N/A N/A dest_bucket initiate_multipart_upload dst_key kwargs N/A cur_pos 0 N/A results N/A N/A num_parts N/A N/A N/A key_size part_size 1 part_size N/A i N/A range num_parts N/A N/A part_start cur_pos N/A cur_pos part_size N/A part_end N/A min N/A cur_pos 1 N/A key_size 1 N/A part_num N/A i 1 N/A N/A N/A results append N/A N/A pool apply_async N/A mp copy_part_from_key args N/A src_bucket src_key part_num part_start part_end N/A N/A N/A logger info Requesting copy of %s/%s to %s/%s part_num num_parts dst_bucket dst_key N/A N/A N/A logger info Waiting for multipart copy of %s/%s to finish dst_bucket dst_key N/A result results N/A N/A N/A N/A result get N/A N/A N/A mp complete_upload N/A N/A mp key_name N/A N/A N/A N/A N/A N/A logger info Error during multipart s3 copy for %s/%s to %s/%s... src_bucket src_key dst_bucket dst_key N/A mp N/A N/A N/A N/A mp cancel_upload N/A N/A rename N/A N/A self N/A args kwargs N/A N/A 
        Alias for ``move()``
         N/A N/A N/A self move args kwargs N/A move N/A N/A self source_path destination_path N/A kwargs N/A N/A 
        Rename/move an object from one S3 location to another.

        :param kwargs: Keyword arguments are passed to the boto function `copy_key`
         N/A N/A N/A self copy source_path destination_path kwargs N/A N/A N/A self remove source_path N/A listdir N/A N/A self path start_time end_time N/A None None N/A N/A 
        Get an iterable with S3 folder contents.
        Iterable contains paths relative to queried path.

        :param start_time: Optional argument to copy files with modified dates after start_time
        :param end_time: Optional argument to copy files with modified dates before end_time
         N/A N/A bucket key N/A N/A self _path_to_bucket_and_key path N/A s3_bucket N/A N/A N/A self s3 get_bucket bucket validate True N/A key_path N/A N/A self _add_path_delimiter key N/A key_path_len N/A len key_path N/A item N/A N/A s3_bucket list prefix key_path N/A N/A last_modified_date N/A N/A time strptime N/A item last_modified %Y-%m-%dT%H:%M:%S.%fZ N/A N/A N/A N/A start_time N/A end_time N/A start_time N/A end_time N/A start_time last_modified_date N/A N/A start_time end_time N/A last_modified_date end_time N/A start_time end_time N/A start_time last_modified_date end_time N/A N/A N/A N/A N/A N/A self _add_path_delimiter path N/A N/A item key N/A key_path_len N/A list N/A N/A self path start_time end_time N/A None None N/A N/A key_path_len N/A len N/A N/A self _add_path_delimiter path N/A item N/A N/A self listdir path start_time start_time end_time end_time N/A N/A N/A N/A item N/A key_path_len N/A isdir N/A N/A self path N/A N/A N/A 
        Is the parameter S3 path a directory?
         N/A N/A bucket key N/A N/A self _path_to_bucket_and_key path N/A s3_bucket N/A N/A N/A self s3 get_bucket bucket validate True N/A N/A N/A self _is_root key N/A N/A True N/A suffix N/A S3_DIRECTORY_MARKER_SUFFIX_0 S3_DIRECTORY_MARKER_SUFFIX_1 N/A N/A s3_dir_with_suffix_key N/A N/A s3_bucket get_key N/A key suffix N/A s3_dir_with_suffix_key N/A N/A True N/A key_path N/A N/A self _add_path_delimiter key N/A s3_bucket_list_result N/A list N/A N/A itertools islice N/A N/A s3_bucket list prefix key_path 1 N/A s3_bucket_list_result N/A N/A True N/A False N/A N/A is_dir isdir mkdir N/A N/A self path parents raise_if_exists N/A True False N/A N/A N/A raise_if_exists N/A N/A self isdir path N/A N/A N/A FileAlreadyExists N/A N/A _ key N/A N/A self _path_to_bucket_and_key path N/A N/A N/A self _is_root key N/A N/A N/A key N/A N/A self _add_path_delimiter key N/A N/A N/A parents N/A N/A N/A self isdir N/A N/A N/A os path dirname key N/A N/A N/A MissingParentDirectory N/A N/A N/A self put_string N/A N/A self _add_path_delimiter path N/A _get_s3_config N/A N/A self key N/A None N/A N/A N/A N/A config N/A dict N/A N/A N/A N/A configuration get_config items s3 N/A N/A N/A NoSectionError N/A N/A N/A N/A N/A k v N/A N/A six iteritems config N/A N/A N/A N/A N/A config N/A k N/A int v N/A N/A N/A ValueError N/A N/A N/A key N/A N/A N/A N/A config get key N/A config N/A _path_to_bucket_and_key N/A N/A self path N/A N/A N/A N/A scheme netloc path query fragment N/A urlsplit path N/A path_without_initial_slash N/A path N/A 1 N/A N/A netloc path_without_initial_slash N/A _is_root N/A N/A self key N/A N/A N/A N/A N/A N/A len key 0 N/A key / N/A _add_path_delimiter N/A N/A self key N/A N/A N/A N/A N/A N/A N/A key N/A -1 / N/A key key N/A key / N/A N/A AtomicS3File N/A AtomicLocalFile N/A N/A 
    An S3 file that writes to a temp file and put to S3 on close.

    :param kwargs: Keyword arguments are passed to the boto function `initiate_multipart_upload`
     __init__ N/A N/A self path s3_client N/A kwargs N/A N/A N/A self s3_client s3_client N/A N/A N/A N/A super AtomicS3File self __init__ path N/A N/A self s3_options kwargs N/A move_to_final_destination N/A N/A self N/A N/A N/A N/A N/A N/A self s3_client put_multipart N/A self tmp_path N/A self path N/A self s3_options N/A N/A ReadableS3File N/A object N/A __init__ N/A N/A self s3_key N/A N/A N/A N/A self s3_key s3_key N/A N/A self buffer N/A N/A N/A self closed False N/A N/A self finished False N/A read N/A N/A self size N/A 0 N/A N/A f N/A N/A N/A self s3_key read size size N/A N/A f N/A N/A N/A self finished True N/A N/A self finished N/A N/A N/A f N/A close N/A N/A self N/A N/A N/A N/A N/A N/A self s3_key close N/A N/A self closed True N/A __del__ N/A N/A self N/A N/A N/A N/A N/A self close N/A __exit__ N/A N/A self exc_type exc traceback N/A N/A N/A N/A N/A self close N/A __enter__ N/A N/A self N/A N/A N/A self N/A _add_to_buffer N/A N/A self line N/A N/A N/A N/A N/A N/A self buffer append line N/A _flush_buffer N/A N/A self N/A N/A N/A output N/A N/A join N/A self buffer N/A N/A self buffer N/A N/A output N/A readable N/A N/A self N/A N/A N/A True N/A writable N/A N/A self N/A N/A N/A False N/A seekable N/A N/A self N/A N/A N/A False N/A __iter__ N/A N/A self N/A N/A N/A key_iter N/A N/A N/A self s3_key __iter__ N/A has_next True N/A has_next N/A N/A N/A N/A chunk N/A next key_iter N/A line N/A N/A chunk splitlines True N/A N/A N/A N/A N/A line endswith N/A os linesep N/A N/A N/A N/A self _add_to_buffer line N/A N/A N/A self buffer N/A N/A N/A N/A self _add_to_buffer line N/A N/A N/A N/A self _flush_buffer N/A N/A N/A line N/A N/A N/A StopIteration N/A N/A output N/A N/A self _flush_buffer N/A output N/A N/A N/A output N/A has_next False N/A N/A N/A self close N/A N/A S3Target N/A FileSystemTarget N/A N/A 

    :param kwargs: Keyword arguments are passed to the boto function `initiate_multipart_upload`
     N/A fs None __init__ N/A N/A self path format client N/A None None kwargs N/A N/A N/A N/A N/A super S3Target self __init__ path N/A N/A format None N/A N/A format N/A get_default_format N/A N/A self path path N/A N/A self format format N/A N/A self fs N/A client N/A S3Client N/A N/A self s3_options kwargs N/A open N/A N/A self mode N/A r N/A N/A N/A N/A mode N/A r w N/A N/A N/A ValueError N/A Unsupported open mode '%s' mode N/A N/A mode r N/A N/A s3_key N/A N/A N/A self fs get_key N/A self path N/A N/A s3_key N/A N/A N/A FileNotFoundException N/A Could not find file at %s N/A self path N/A fileobj N/A ReadableS3File s3_key N/A N/A N/A N/A self format pipe_reader fileobj N/A N/A N/A N/A N/A self format pipe_writer N/A AtomicS3File N/A self path N/A self fs N/A self s3_options N/A N/A S3FlagTarget N/A S3Target N/A N/A 
    Defines a target directory with a flag-file (defaults to `_SUCCESS`) used
    to signify job success.

    This checks for two things:

    * the path exists (just like the S3Target)
    * the _SUCCESS file exists within the directory.

    Because Hadoop outputs into a directory and not a single file,
    the path is assumed to be a directory.

    This is meant to be a handy alternative to AtomicS3File.

    The AtomicFile approach can be burdensome for S3 since there are no directories, per se.

    If we have 1,000,000 output files, then we have to rename 1,000,000 objects.
     N/A fs None __init__ N/A N/A self path format client flag N/A None None _SUCCESS N/A N/A 
        Initializes a S3FlagTarget.

        :param path: the directory where the files are stored.
        :type path: str
        :param client:
        :type client:
        :param flag:
        :type flag: str
         N/A N/A format None N/A N/A format N/A get_default_format N/A N/A N/A path N/A -1 / N/A N/A N/A ValueError S3FlagTarget requires the path to be to a directory.  It must end with a slash ( / ). N/A N/A N/A N/A super S3FlagTarget self __init__ path format client N/A N/A self flag flag N/A exists N/A N/A self N/A N/A N/A hadoopSemaphore N/A N/A self path N/A self flag N/A N/A N/A N/A self fs exists hadoopSemaphore N/A N/A S3EmrTarget N/A S3FlagTarget N/A N/A 
    Deprecated. Use :py:class:`S3FlagTarget`
     __init__ N/A N/A self N/A args kwargs N/A N/A N/A N/A warnings warn S3EmrTarget is deprecated. Please use S3FlagTarget N/A N/A N/A N/A super S3EmrTarget self __init__ args kwargs N/A N/A S3PathTask N/A ExternalTask N/A N/A 
    A external task that to require existence of a path in S3.
     N/A path N/A Parameter output N/A N/A self N/A N/A N/A N/A S3Target N/A self path N/A N/A S3EmrTask N/A ExternalTask N/A N/A 
    An external task that requires the existence of EMR output in S3.
     N/A path N/A Parameter output N/A N/A self N/A N/A N/A N/A S3EmrTarget N/A self path N/A N/A S3FlagTask N/A ExternalTask N/A N/A 
    An external task that requires the existence of EMR output in S3.
     N/A path N/A Parameter N/A flag N/A Parameter default None output N/A N/A self N/A N/A N/A N/A S3FlagTarget N/A self path flag N/A self flag N/A N/A