Module Expr Str Assign NameStore Str Import alias ImportFrom alias Import alias Import alias Assign NameStore Call AttributeLoad NameLoad attr Str ImportFrom alias ImportFrom alias TryExcept body Import alias ImportFrom alias ImportFrom alias handlers ExceptHandler type NameLoad body Assign NameStore NameStore NameStore NameLoad ExceptHandler type NameLoad name NameStore body Expr Call AttributeLoad NameLoad attr BinOpMod Str Call NameLoad NameLoad Assign NameStore NameStore NameStore NameLoad Assign NameStore Str Assign NameStore Str FunctionDef arguments args NameParam defaults body ImportFrom alias Assign NameStore Call NameLoad NameLoad Str NameLoad If UnaryOpNot AttributeLoad NameLoad attr body Expr Call AttributeLoad NameLoad attr Assign NameStore Call NameLoad Return Call AttributeLoad NameLoad attr SubscriptLoad NameLoad Index Str SubscriptLoad NameLoad Index Str decorator_list ClassDef bases NameLoad body Expr Str FunctionDef arguments args NameParam NameParam defaults body Assign AttributeStore NameLoad attr Dict Assign AttributeStore NameLoad attr NameLoad decorator_list FunctionDef arguments args NameParam defaults body Return AttributeLoad NameLoad attr decorator_list NameLoad FunctionDef arguments args NameParam NameParam NameParam NameParam defaults Str NameLoad body Assign SubscriptStore AttributeLoad NameLoad attr Index NameLoad NameStore BinOpAdd Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad Num Num Assign NameStore BinOpMod Str NameLoad Assign NameStore Call NameLoad AttributeLoad NameLoad attr Assign AttributeStore NameLoad attr NameLoad Assign TupleStore NameStore NameStore Call NameLoad NameLoad Assign NameStore NameLoad Assign NameStore BoolOpOr SubscriptLoad Call AttributeLoad NameLoad attr NameLoad Index Num Str Assign NameStore Dict Str NameLoad If CompareIs NameLoad NameLoad body Assign NameStore NameLoad If CompareEq NameLoad Str body AugAssignAdd NameStore BinOpMod Str NameLoad orelse AugAssignAdd NameStore NameLoad Expr Call AttributeLoad NameLoad attr NameLoad keyword NameLoad Return NameLoad decorator_list FunctionDef arguments args NameParam NameParam defaults body Assign SubscriptStore AttributeLoad NameLoad attr Index NameLoad Num decorator_list decorator_list FunctionDef arguments args NameParam NameParam defaults Str body Expr Str If CompareEq NameLoad Str body Return Call AttributeLoad Str attr keyword NameLoad orelse If CompareEq NameLoad Str body Return Call AttributeLoad Str attr keyword NameLoad orelse Raise Call NameLoad NameLoad decorator_list FunctionDef arguments args NameParam defaults body Expr Str Expr Call AttributeLoad NameLoad attr Call AttributeLoad NameLoad attr Str Expr Call AttributeLoad NameLoad attr Expr Call AttributeLoad NameLoad attr BinOpMod Str AttributeLoad NameLoad attr decorator_list FunctionDef arguments args NameParam defaults body Expr Call AttributeLoad NameLoad attr BinOpMod Str AttributeLoad NameLoad attr decorator_list FunctionDef arguments args NameParam defaults body Assign NameStore Call NameLoad NameLoad TryFinally body TryExcept body Assign NameStore Call AttributeLoad NameLoad attr NameLoad Expr Call AttributeLoad NameLoad attr BinOpMod Str AttributeLoad NameLoad attr Expr Call NameLoad NameLoad handlers ExceptHandler body Pass finalbody Pass Return Call AttributeLoad NameLoad attr NameLoad decorator_list FunctionDef arguments args NameParam NameParam defaults Str NameLoad body Assign NameStore Call NameLoad NameLoad Expr Call AttributeLoad NameLoad attr Str Expr Call NameLoad NameLoad Assign NameStore Call NameLoad NameLoad Expr Call NameLoad Str Expr Call NameLoad Str If NameLoad body Expr Call AttributeLoad NameLoad attr NameLoad Expr Call NameLoad Str Expr Call NameLoad Str For NameStore Call NameLoad Num body Expr Call NameLoad Str Expr Call NameLoad Str Expr Call AttributeLoad NameLoad attr Call NameLoad Str Expr Call AttributeLoad NameLoad attr Call NameLoad Str Expr Call NameLoad Str Expr Call NameLoad Str Assign NameStore Str Expr Call AttributeLoad NameLoad attr Call NameLoad NameLoad Expr Call AttributeLoad NameLoad attr NameLoad Expr Call NameLoad NameLoad Expr Call AttributeLoad NameLoad attr BinOpMod Str NameLoad Return NameLoad decorator_list FunctionDef arguments args defaults body Return Call NameLoad Str keyword NameLoad decorator_list FunctionDef arguments args defaults body Return Call NameLoad Str keyword NameLoad decorator_list FunctionDef arguments args defaults body Assign NameStore Str Assign NameStore Call NameLoad NameLoad Expr Call AttributeLoad NameLoad attr NameLoad Expr Call AttributeLoad NameLoad attr Str Expr Call NameLoad NameLoad Assign NameStore Call NameLoad NameLoad Expr Call NameLoad Str keyword Str Expr Call NameLoad Str Expr Call NameLoad Str keyword Str decorator_list ImportFrom alias alias ImportFrom alias alias alias alias alias alias FunctionDef arguments args NameParam NameParam NameParam NameParam NameParam defaults NameLoad NameLoad NameLoad NameLoad body Expr Str Assign NameStore Call NameLoad NameLoad Assign TupleStore NameStore NameStore TupleLoad NameLoad Call AttributeLoad AttributeLoad NameLoad attr attr Str If Call AttributeLoad AttributeLoad NameLoad attr attr Str body If UnaryOpNot CompareIn AttributeLoad NameLoad attr TupleLoad Str Str body Raise Call NameLoad BinOpMod Str AttributeLoad NameLoad attr Assign NameStore SubscriptLoad Call AttributeLoad AttributeLoad NameLoad attr attr Str Num Index Num orelse If CompareEq AttributeLoad NameLoad attr Str body If UnaryOpNot CompareIn AttributeLoad NameLoad attr TupleLoad Str Str body Raise Call NameLoad BinOpMod Str AttributeLoad NameLoad attr Assign TupleStore NameStore NameStore Call AttributeLoad NameLoad attr Str Num orelse If CompareEq AttributeLoad NameLoad attr Str body Assign NameStore AttributeLoad NameLoad attr Raise NameLoad Assign NameStore NameLoad Assign NameStore ListLoad If NameLoad body If CompareIs NameLoad NameLoad body ImportFrom alias Assign NameStore Call AttributeLoad NameLoad attr Assign NameStore BinOpMod Str NameLoad Assign NameStore Call AttributeLoad NameLoad attr NameLoad Assign NameStore Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad AttributeLoad NameLoad attr orelse Assign NameStore Call AttributeLoad NameLoad attr NameLoad Assign NameStore NameLoad TryExcept body Assign NameStore Call AttributeLoad NameLoad attr handlers ExceptHandler type NameLoad name NameStore body Assign NameStore Str If NameLoad body Assign NameStore Call AttributeLoad NameLoad attr NameLoad Assign NameStore SubscriptLoad ListComp NameLoad comprehension NameStore Call NameLoad NameLoad keyword Lambda arguments args NameParam defaults TupleLoad AttributeLoad NameLoad attr AttributeLoad NameLoad attr BoolOpAnd Call NameLoad NameLoad NameLoad CompareEq AttributeLoad NameLoad attr NameLoad Slice Num Assert NameLoad Assert AttributeLoad SubscriptLoad NameLoad Index Num attr For NameStore NameLoad body Assign NameStore AttributeLoad NameLoad attr Assign NameStore BinOpAdd IfExp AttributeLoad NameLoad attr BinOpAdd AttributeLoad NameLoad attr Str Str BinOpMod Str NameLoad Assign NameStore Call NameLoad Call AttributeLoad NameLoad attr keyword NameLoad Expr Call AttributeLoad NameLoad attr NameLoad If NameLoad body Assign NameStore Call NameLoad Call NameLoad NameLoad Assign NameStore NameLoad If UnaryOpNot NameLoad body Break If BoolOpAnd NameLoad UnaryOpNot NameLoad body Raise Call NameLoad BinOpMod Str NameLoad If UnaryOpNot NameLoad body Assign NameStore ListLoad Call NameLoad NameLoad If NameLoad body Return NameLoad orelse Return SubscriptLoad NameLoad Index Num decorator_list If CompareEq NameLoad Str body Import alias Expr Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr If BoolOpAnd CompareGt Call NameLoad AttributeLoad NameLoad attr Num CompareEq SubscriptLoad AttributeLoad NameLoad attr Index Num Str body If CompareLt Call NameLoad AttributeLoad NameLoad attr Num body Raise Call NameLoad Str Assign NameStore SubscriptLoad AttributeLoad NameLoad attr Index Num If CompareEq Call AttributeLoad NameLoad attr Str body For NameStore Call AttributeLoad Call NameLoad attr body If Call AttributeLoad NameLoad attr Str body Expr Call SubscriptLoad Call NameLoad Index NameLoad orelse Expr Call SubscriptLoad Call NameLoad Index BinOpMod Str NameLoad orelse Print Str
N/A N/A Variety of helpers to deal with AWS S3

Use as a script to generate test buckets via e.g.

    python -m datalad.support.s3 generate test1_dirs
 N/A __docformat__ restructuredtext N/A mimetypes os.path splitext N/A logging N/A datalad.log N/A lgr N/A N/A logging getLogger datalad.s3 keyring_ keyring dochelpers exc_str N/A N/A N/A boto boto.s3.key Key boto.exception S3ResponseError N/A N/A N/A ImportError N/A N/A boto Key S3ResponseError None N/A N/A Exception N/A e N/A N/A N/A N/A lgr warning N/A boto module failed to import although available: %s N/A exc_str e N/A boto Key S3ResponseError None N/A S3_ADMIN_CREDENTIAL datalad-datalad-admin-s3 N/A S3_TEST_CREDENTIAL datalad-datalad-test-s3 _get_bucket_connection N/A N/A credential N/A N/A downloaders.providers Credential N/A credential N/A Credential credential aws-s3 None N/A N/A N/A credential is_known N/A N/A N/A N/A credential enter_new N/A creds N/A credential N/A N/A N/A boto connect_s3 N/A creds N/A key_id N/A creds N/A secret_id N/A VersionedFilesPool N/A object N/A N/A Just a helper which would help to create versioned files in the bucket __init__ N/A N/A self bucket N/A N/A N/A N/A self _versions N/A N/A N/A self _bucket bucket N/A bucket N/A N/A self N/A N/A N/A N/A self _bucket N/A property __call__ N/A N/A self filename prefix load N/A None N/A N/A N/A N/A self _versions N/A filename version N/A N/A N/A N/A self _versions get filename 0 1 N/A version_str N/A version%d version N/A k N/A Key N/A self _bucket N/A N/A k key filename N/A N/A base ext N/A splitext filename N/A content_type None N/A mtype N/A N/A N/A N/A mimetypes guess_type filename N/A 0 application/octet-stream N/A headers N/A Content-Type mtype N/A N/A load None N/A N/A load prefix N/A N/A ext html N/A N/A load N/A <html><body>%s</body></html> version_str N/A N/A load version_str N/A N/A N/A k set_contents_from_string load headers headers N/A k N/A reset_version N/A N/A self filename N/A N/A N/A N/A N/A self _versions N/A filename 0 N/A N/A get_key_url N/A N/A e schema N/A http N/A N/A Generate an s3:// or http:// url given a key
     N/A N/A schema http N/A N/A N/A N/A http://{e.bucket.name}.s3.amazonaws.com/{e.name}?versionId={e.version_id} format e e N/A N/A N/A schema s3 N/A N/A N/A N/A s3://{e.bucket.name}/{e.name}?versionId={e.version_id} format e e N/A N/A N/A ValueError schema N/A prune_and_delete_bucket N/A N/A bucket N/A N/A N/A Deletes all the content and then deletes bucket

    Should be used with care -- no confirmation requested
     N/A N/A N/A bucket delete_keys N/A N/A bucket list_versions N/A N/A N/A bucket delete N/A N/A N/A lgr info N/A Bucket %s was removed N/A bucket name N/A set_bucket_public_access_policy N/A N/A bucket N/A N/A N/A N/A N/A bucket set_policy N/A {
      "Version":"2012-10-17",
      "Statement":[{
          "Sid":"AddPerm",
          "Effect":"Allow",
          "Principal": "*",
          "Action":["s3:GetObject", "s3:GetObjectVersion", "s3:GetObjectTorrent", "s3:GetObjectVersionTorrent"],
          "Resource":["arn:aws:s3:::%s/*"]
        }
      ]
    } N/A bucket name N/A gen_test_bucket N/A N/A bucket_name N/A N/A N/A conn N/A _get_bucket_connection S3_ADMIN_CREDENTIAL N/A N/A N/A N/A N/A bucket N/A N/A conn get_bucket bucket_name N/A N/A N/A lgr info N/A Deleting existing bucket %s N/A bucket name N/A N/A prune_and_delete_bucket bucket N/A N/A N/A N/A N/A N/A N/A N/A N/A conn create_bucket bucket_name N/A _gen_bucket_test0 N/A N/A bucket_name versioned N/A datalad-test0 True N/A N/A bucket N/A gen_test_bucket bucket_name N/A N/A N/A bucket configure_website index.html N/A N/A set_bucket_public_access_policy bucket N/A files N/A VersionedFilesPool bucket N/A N/A files 1version-nonversioned1.txt N/A N/A files 2versions-nonversioned1.txt N/A versioned N/A N/A N/A N/A bucket configure_versioning True N/A N/A files 2versions-nonversioned1.txt N/A N/A files 2versions-nonversioned1.txt_sameprefix N/A v N/A range 3 N/A N/A N/A files 3versions-allversioned.txt N/A N/A files 3versions-allversioned.txt_sameprefix N/A N/A N/A bucket delete_key N/A files 1version-removed.txt N/A N/A N/A bucket delete_key N/A files 2versions-removed-recreated.txt N/A N/A files 2versions-removed-recreated.txt N/A N/A files 2versions-removed-recreated.txt_sameprefix N/A f 1version-removed-recreated.txt N/A N/A N/A bucket delete_key N/A files f N/A N/A N/A files reset_version f N/A N/A files f N/A N/A N/A lgr info N/A Bucket %s was generated and populated bucket_name N/A bucket N/A gen_bucket_test0_versioned N/A N/A N/A N/A N/A N/A _gen_bucket_test0 datalad-test0-versioned versioned True N/A gen_bucket_test0_nonversioned N/A N/A N/A N/A N/A N/A _gen_bucket_test0 datalad-test0-nonversioned versioned False N/A gen_bucket_test1_dirs N/A N/A N/A N/A N/A bucket_name datalad-test1-dirs-versioned N/A bucket N/A gen_test_bucket bucket_name N/A N/A N/A bucket configure_versioning True N/A N/A N/A bucket configure_website index.html N/A N/A set_bucket_public_access_policy bucket N/A files N/A VersionedFilesPool bucket N/A N/A files d1 load N/A N/A files d1/file1.txt N/A N/A files d1 load smth N/A six.moves.urllib.request urlopen Request six.moves.urllib.parse urljoin urlparse urlsplit urlunsplit urlunparse urlencode get_versioned_url N/A N/A url guarantee_versioned return_all verify s3conn N/A False False False None N/A N/A Given a url return a versioned URL

    Originally targeting AWS S3 buckets with versioning enabled

    Parameters
    ----------
    url : string
    guarantee_versioned : bool, optional
      Would fail if buckets is determined to have no versioning enabled.
      It will not fail if we fail to determine if bucket is versioned or
      not
    return_all: bool, optional
      If True, would return a list with URLs for all the versions of this
      file, sorted chronologically with latest first (when possible, e.g.
      for S3).  Remove markers get ignored
    verify: bool, optional
      Verify that URL is accessible. As discovered some versioned keys might
      be denied access to

    Returns
    -------
    string or list of string
     N/A url_rec N/A urlparse url N/A N/A s3_bucket fpath N/A None N/A N/A N/A url_rec path lstrip / N/A N/A N/A N/A url_rec netloc endswith .s3.amazonaws.com N/A N/A N/A N/A N/A url_rec scheme N/A http https N/A N/A N/A ValueError N/A Do not know how to handle %s scheme N/A url_rec scheme N/A s3_bucket N/A N/A N/A N/A url_rec netloc split . 1 N/A 0 N/A N/A N/A N/A url_rec netloc s3.amazonaws.com N/A N/A N/A N/A N/A url_rec scheme N/A http https N/A N/A N/A ValueError N/A Do not know how to handle %s scheme N/A url_rec scheme N/A N/A s3_bucket fpath N/A N/A fpath split / 1 N/A N/A N/A N/A url_rec scheme s3 N/A N/A s3_bucket N/A url_rec netloc N/A NotImplementedError N/A was_versioned False N/A all_versions N/A N/A s3_bucket N/A N/A N/A s3conn None N/A downloaders.providers Providers N/A providers N/A N/A Providers from_config_files N/A s3url N/A s3://%s/ s3_bucket N/A s3provider N/A N/A providers get_provider s3url N/A bucket N/A N/A N/A s3provider authenticator authenticate s3_bucket N/A s3provider credential N/A N/A bucket N/A N/A s3conn get_bucket s3_bucket N/A supports_versioning True N/A N/A N/A supports_versioning N/A N/A bucket get_versioning_status N/A N/A N/A S3ResponseError N/A e N/A N/A supports_versioning maybe N/A supports_versioning N/A N/A all_keys N/A N/A bucket list_versions fpath N/A all_keys N/A N/A x N/A x N/A sorted all_keys key N/A N/A N/A x N/A N/A N/A x last_modified N/A x is_latest N/A N/A isinstance x Key N/A N/A x name fpath N/A -1 N/A all_keys N/A N/A N/A all_keys N/A 0 is_latest N/A key all_keys N/A N/A version_id N/A key version_id N/A query N/A N/A N/A url_rec query N/A N/A url_rec query & N/A versionId=%s version_id N/A url_versioned N/A urlunparse N/A N/A url_rec _replace query query N/A N/A N/A all_versions append url_versioned N/A verify N/A N/A _ N/A urlopen N/A Request url N/A was_versioned True N/A N/A return_all N/A N/A N/A N/A guarantee_versioned N/A was_versioned N/A N/A N/A RuntimeError N/A Could not version %s url N/A N/A all_versions N/A N/A all_versions N/A N/A urlunparse url_rec N/A return_all N/A N/A all_versions N/A N/A N/A all_versions N/A 0 N/A N/A N/A __name__ __main__ N/A N/A sys N/A N/A N/A lgr setLevel N/A logging INFO N/A N/A N/A N/A len N/A sys argv 1 N/A N/A N/A sys argv N/A 1 generate N/A N/A N/A N/A len N/A sys argv 3 N/A N/A N/A ValueError Say 'all' to regenerate all, or give a generators name N/A name N/A N/A sys argv N/A 2 N/A N/A N/A N/A name lower all N/A N/A f N/A N/A N/A locals keys N/A N/A N/A N/A f startswith gen_bucket_ N/A N/A N/A N/A N/A locals N/A f N/A N/A N/A N/A N/A locals N/A N/A gen_bucket_%s name N/A N/A nothing todo